Package: intellifs
Content-Aware File System.InstallationpipinstallintellifsNote: intellifs only indexes plain text files, HTML, XML and PDF soures by default.Add support for all document typespipinstall"unstructured[all-docs]"Referunstructured installation documentationfor more control over document types.CLI UsageDisplay help sectionifsUsage: ifs COMMAND

Content-Aware File System.

╭─ Commands ─────────────────────────────────────────────╮
│ embedder  Default embedder.                            │
│ index     Index a file or directory.                   │
│ search    Perform semantic search in a directory.      │
│ version   Display application version.                 │
╰────────────────────────────────────────────────────────╯
╭─ Parameters ───────────────────────────────────────────╮
│ help,-h  Display this message and exit.                │
╰────────────────────────────────────────────────────────╯indexcommandifsindexhelpUsage: ifs index [ARGS]

Index a file or directory.

╭─ Arguments ──────────────────────────────────────────╮
│ *  PATH  Path to file or directory. [required]       │
╰──────────────────────────────────────────────────────╯Indexing a file.ifsindex./Cyber.pdfIndexing a directory.ifsindex./test_docssearchcommandifssearchhelpUsage: ifs search [ARGS] [OPTIONS]

Perform semantic search in a directory.

╭─ Arguments ───────────────────────────────────────────────────────────╮
│ DIR  Start search directory path. [default: /home/synacktra]          │
╰───────────────────────────────────────────────────────────────────────╯
╭─ Parameters ──────────────────────────────────────────────────────────╮
│ *  --query        -q  Search query string. [required]                 │
│    --max-results  -k  Maximum result count. [default: 5]              │
│    --threshold    -t  Minimum filtering threshold value.              │
│    --return       -r  Component to return. [choices: path,context]    │
╰───────────────────────────────────────────────────────────────────────╯Search in current directoryifssearch--query"How does intellifs work?"Search in specific directoryifssearchpath/to/directory--query"How does intellifs work?"Get specific amount of resultsifssearch-q"How does intellifs work?"-k8Control threshold value for better resultsifssearch-q"How does intellifs work?"-t0.5Get specific component of results[default: path mapped contexts JsON]ifssearch-q"How does intellifs work?"-rpathembeddercommandifsembedderhelpUsage: ifs embedder [OPTIONS]

Default embedder.

╭─ Parameters ────────────────────────────────────╮
│ --select  -s  Select from available embedders.  │
╰─────────────────────────────────────────────────╯Display default embedderifsembedder{
  "model": "BAAI/bge-small-en-v1.5",
  "dim": 384,
  "description": "Fast and Default English model",
  "size_in_GB": 0.13
}Select from available embeddersUseshttps://github.com/synacktraa/minifzffor selection.ifsembedder--selectshellcommandStarts an interactive shell.https://github.com/synacktraa/synacktraa/assets/91981716/b746ccf8-e27b-4abd-99cf-528677fb0ef8Library UsageInitializeFileSystemfromintellifsimportFileSystemifs=FileSystem()By default it uses default embedder. You can specify a differentEmbedderinstance too.fromintellifs.embedderimportEmbedderifs=FileSystem(embedder=Embedder(model="<model-name>",dim=<model-dimension>))UseEmbedder.available_modelsto list supported models.indexmethodIndexing a filefromintellifs.indexablesimportFileifs.index(File(__file__))Indexing a directoryfromintellifs.indexablesimportDirectoryifs.index(Directory('path/to/directory'))is_indexedmethodVerify If aFileorDirectoryhas been indexed.file=File(__file__)ifs.is_indexed(file)ifs.is_indexed(file.directory)searchmethodSearch in current directoryifs.search(query="How does intellifs work?")Search in specific directoryifs.search(directory=Directory('path/to/directory'),query="How does intellifs work?")Get specific amount of resultsifs.search(query="How does intellifs work?",max_results=8)Control threshold value for better resultsifs.search(query="How does intellifs work?",score_threshold=0.5)How It Works?TheFileSystemis a sophisticated file system management tool designed for organizing and searching through files and directories based on their content. It utilizes embeddings to represent file contents, allowing for semantic search capabilities. Here's a breakdown of its core components and functionalities:Core ComponentsMetadataandIndexMetadata: A structured representation that includes file contexts (chunks of text extracted from files), the directory path, filepath, and the last modified timestamp.Index: Consists of embeddings (vector representations of file contents) and associated metadata.FileSystemClassTheFileSystemclass is the heart of the system, integrating various components to facilitate file indexing, searching, and management.InitializationUpon initialization, theFileSystemprepares the environment for indexing and searching files and directories with the following steps:Embedder Setup: An embedder is initialized to generate vector embeddings from file content. If a custom embedder is not provided, the system defaults to a pre-configured option suitable for general-purpose text embedding.Local Storage Initialization: The system sets up a local storage mechanism to cache the embeddings and metadata. This involves:Determining the storage path based on the embedder's name, ensuring a unique cache directory for different embedders.Creating a mapping file (map.json) within the cache directory to maintain a record of collection names associated with base paths.Base Path Handling: The FileSystem intelligently handles base paths to accommodate the file system structure of different operating systems.Windows Systems: On Windows, base paths are recognized as drive letters (e.g.,C:,D:). This allows the system to manage files and directories across different drives distinctly.POSIX Systems: For POSIX-compliant systems (like Linux and macOS), base paths are identified as root directories (e.g.,/var,/home). This approach facilitates indexing and searching files in a structured manner consistent with UNIX-like directory hierarchies.Collection Management: Utilizes a local persistent vector database, managed through theqdrant_client, to store and retrieve embeddings and metadata.Indexing Files and DirectoriesFile Indexing: Generates an index for a single file by extracting text, partitioning it into manageable chunks, and converting these chunks into embeddings. Metadata is also generated to include the file's contextual information and modification timestamp.Directory Indexing: Recursively indexes all files within a directory. It checks for modifications to ensure the index is current, adds new files, and removes entries for deleted files.SearchingAllows for semantic search within specified directories or globally across all indexed files. Searches are performed using query embeddings to find the most relevant files based on their content embeddings.WorkflowGenerate Index: When a file or directory is indexed, the system extracts text, generates embeddings, and stores this information along with metadata in a dedicated collection.Search: Input a text query to search across indexed files and directories. The system converts the query into an embedding and retrieves the most relevant files based on cosine similarity.Management: The system supports adding, updating, and deleting files or directories in the index to keep the database current with the filesystem.