Package: hypertag
HyperTagHyperTag helps humans intuitively express how they think about their files using tags and machine learning. Represent how you think using tags. Find what you look for using semantic search for your text documents (yes, even PDF's) and images. Instead of introducing proprietary file formats like other existing file organization tools, HyperTag just smoothly layers on top of your existing files without any fuss.Objective Function: Minimize time between a thought and access to all relevant files.Accompanying blog post:https://blog.neotree.uber.space/posts/hypertag-file-organization-made-for-humansTable of ContentsHyperTagInstallCommunityOverviewCLI FunctionsImport existing directory recursivelyAdd file/s or URL/s manuallyTag file/s  (with values)Untag file/sTag a tagMerge tagsQuery using Set TheoryIndex supported image and text filesSemantic search for text filesSemantic search for image filesStart HyperTag DaemonPrint all tags of file/sPrint all metatags of tag/sPrint all tagsPrint all filesVisualize HyperTag GraphGenerate HyperTagFSAdd directory to auto import listSet HyperTagFS directory pathArchitectureDevelopmentInspirationInstallAvailable onPyPI$ pip install hypertag(supports both CPU only & CUDA accelerated execution!)CommunityJoin the HyperTagmatrix chat roomto stay up to date on the latest developments or to ask for help.OverviewHyperTag offers a slick CLI but more importantly it creates a directory calledHyperTagFSwhich is a file system based representation of your files and tags using symbolic links and directories.Directory Import: Import your existing directory hierarchies using$ hypertag import path/to/directory. HyperTag converts it automatically into a tag hierarchy using metatagging.Semantic Text & Image Search  (Experimental): Search forimages(jpg, png) andtext documents(yes, even PDF's) content with a simple text query. Text search is powered by the awesomeSentence Transformerslibrary. Text to image search is powered by OpenAI'sCLIP model. Currently only English queries are supported.HyperTag Daemon  (Experimental): MonitorsHyperTagFSand directories added to the auto import list for user changes (see section "Start HyperTag Daemon" below). Also spawns the DaemonService which speeds up semantic search significantly (warning: daemon process is a RAM hog with ~2GB usage).Fuzzy Matching Queries: HyperTag uses fuzzy matching to minimize friction in the unlikely case of a typo.File Type Groups: HyperTag automatically creates folders containing common files (e.g. Images: jpg, png, etc., Documents: txt, pdf, etc., Source Code: py, js, etc.), which can be found inHyperTagFS.HyperTag Graph: Quickly get an overview of your HyperTag Graph! HyperTag visualizes the metatag graph on every change and saves it atHyperTagFS/hypertag-graph.pdf.CLI FunctionsImport existing directory recursivelyImport files with tags inferred from the existing directory hierarchy.$ hypertag import path/to/directoryAdd file/s or URL/s manually$ hypertag add path/to/file https://github.com/SeanPedersen/HyperTagTag file/s  (with values)Manually tag files. Shortcut:$ hypertag t$ hypertag tag humans/*.txt with human "Homo Sapiens"Add a value to a file's tag:$ hypertag tag sean.txt with name="Sean Pedersen"Untag file/sManually remove tag/s from file/s.$ hypertag untag humans/*.txt with human "Homo Sapiens"Tag a tagMetatag tag/s to create tag hierarchies. Shortcut:$ hypertag tt$ hypertag metatag human with animalMerge tagsMerge all associations (files & tags) of tag A into tag B.$ hypertag merge human into "Homo Sapiens"Query using Set TheoryPrint file names of the resulting set matching the query. Queries are composed of tags (with values) and operands. Tags are fuzzy matched for convenience. Nesting is currently not supported, queries are evaluated from left to right.Shortcut:$ hypertag qQuery with a value using a wildcard:$ hypertag query name="Sean*"Print paths:$ hypertag query human --pathPrint fuzzy matched tag:$ hypertag query man --verboseDisable fuzzy matching:$ hypertag query human --fuzzy=0Default operand is AND (intersection):$ hypertag query human name="Sean*"is equivalent to$ hypertag query human and name="Sean*"OR (union):$ hypertag query human or "Homo Sapiens"MINUS (difference):$ hypertag query human minus "Homo Sapiens"Index supported image and text filesOnly indexed files can be searched.$ hypertag indexTo parse even unparseable PDF's, install tesseract:# pacman -S tesseract tesseract-data-engIndex only image files:$ hypertag index --imageIndex only text files:$ hypertag index --textSemantic search for text filesA custom search algorithm combining semantic with token matching search.
Print text file names sorted by matching score.
Performance benefits greatly from running the HyperTag daemon.Shortcut:$ hypertag s$ hypertag search "your important text query" --path --score --top_k=10Semantic search for image filesPrint image file names sorted by matching score.
Performance benefits greatly from running the HyperTag daemon.Shortcut:$ hypertag siText to image:$ hypertag search_image "your image content description" --path --score --top_k=10Image to image:$ hypertag search_image "path/to/image.jpg" --path --score --top_k=10Start HyperTag DaemonStart daemon process with triple functionality:WatchesHyperTagFSdirectory for user changesMaps file (symlink) and directory deletions into tag / metatag removal/sOn directory creation: Interprets name as set theory tag query and automatically populates it with resultsOn directory creation inSearch ImagesorSearch Texts: Interprets name as semantic search query (add top_k=42 to limit result size) and automatically populates it with resultsWatches directories on the auto import list for user changes:Maps file changes (moves & renames) to DBOn file creation: Adds new file/s with inferred tag/s and auto-indexes it (if supported file format).Spawns DaemonService to load and expose models used for semantic search, speeding it up significantly$ hypertag daemonPrint all tags of file/s$ hypertag tags filename1 filename2Print all metatags of tag/s$ hypertag metatags tag1 tag2Print all tags$ hypertag showPrint all filesPrint names:$ hypertag show filesPrint paths:$ hypertag show files --pathVisualize HyperTag GraphVisualize the metatag graph hierarchy (saved at HyperTagFS root).$ hypertag graphSpecifylayout algorithm(default: fruchterman_reingold):$ hypertag graph --layout=kamada_kawaiGenerate HyperTagFSGenerate file system based representation of your files and tags using symbolic links and directories.$ hypertag mountAdd directory to auto import listDirectories added to the auto import list will be monitored by the daemon for new files or changes.$ hypertag add_auto_import_dir path/to/directorySet HyperTagFS directory pathDefault is the user's home directory.$ hypertag set_hypertagfs_dir path/to/directoryArchitecturePython and it's vibrant open-source community power HyperTagMany other awesome open-source projects make HyperTag possible (listed inpyproject.toml)SQLite3 serves as the meta data storage engine (located at~/.config/hypertag/hypertag.db)Added URLs are saved in~/.config/hypertag/web_pagesfor websites, others in~/.config/hypertag/downloadsSymbolic links are used to create the HyperTagFS directory structureSemantic Search: boosted usinghnswlibText to text search is powered by the awesomeDistilBERTText to image & image to image search is powered by OpenAI's impressiveCLIP modelDevelopmentFind prioritized issues here:TODO ListPick an issue and comment how you plan to tackle it before starting out, to make sure no dev time is wasted.Clone repo:$ git clone https://github.com/SeanPedersen/HyperTag.git$ cd HyperTag/InstallPoetryInstall dependencies:$ poetry installActivate virtual environment:$ poetry shellRun all tests:$ pytest -vRun formatter:$ black hypertag/Run linter:$ flake8Run type checking:$ mypy **/*.pyRun security checking:$ bandit --exclude tests/ -r .Codacy:DashboardRun HyperTag:$ python -m hypertagInspirationWhat is the point of HyperTag's existence?HyperTag offers many unique features such as the import, semantic search, graphing and fuzzy matching functions that make it very convenient to use. All while HyperTag's code base staying relatively tiny at <2000 LOC compared to similar projects likeTMSU(>10,000 LOC in Go) andSuperTag(>25,000 LOC in Rust), making it easy to hack on.