Package: hnsqlite
hnsqlitehnsqliteis a text-centric integration of SQLite andHnswlibto provide a persistent collection of embeddings (strings, vectors, and metadata) and search time filtering based on the metadata.ClassesCollectionTheCollectionclass represents a combination of a SQLite database and an HNSWLIB index. The purpose of this class is to provide a persistent collection of embeddings (strings, vectors, and metadata) and search time filtering based on the metadata.EmbeddingEmbeddingis a class that represents an embedding sent to or received from the Collection API.Attributes:vector: A list of float values representing the user-supplied vector element. The vector can be sent as a numpy array and will be converted to a list of floats.text: The text that was input to the model to generate this embedding.doc_id: An optional document_id associated with the embedding.metadata: An optional dictionary of metadata associated with the text.created_at: The epoch timestamp when the embedding was created (in seconds).SearchResponseSearchResponseis a class derived from theEmbeddingclass, specifically designed for returning search results. ASearchResponseobject consists of an embedding along with its distance to the query vector.Attributes:vector: A list of float values representing the user-supplied vector element. The vector can be sent as a numpy array and will be converted to a list of floats.text: The text that was input to the model to generate this embedding.doc_id: An optional document_id associated with the embedding.metadata: An optional dictionary of metadata associated with the text.created_at: The epoch timestamp when the embedding was created (in seconds).distance: A float value representing the cosine similarity distance between the search result and the query vector.  Lower distances represent closer matches.Collection MethodsCollection():- Initializes a new Collection as a SQLite database file and associated HNSWLIB index. If the specified collection name is found in the database, the collection will be initialized from database. Otherwise, a new collection of the specified name will be created in the database.add_items: Adds new items to the collection as lists of individual components.  An alternative interface to add_embeddings().add_embedding: Adds a single Embedding object to the collection.  A convenience alternative to add_embeddings().add_embeddings: Adds a list of Embedding objects to the collection.  An alternative interface to add_items().get_embeddings: return a list of embeddings from an offsetget_embeddings_doc_ids: return a list of embeddings associated with specified doc_idssearch: Queries the HNSW index for the nearest neighbors of the given vector. Supply a k parameter (defaults to 12) and an optional filter dictionary.delete: Deletes items from the collection based on a filter, a specific list of document_ids, or everything.Database classesThe following classes are the internal SqlModel data classes used to persist the embeddings and configuration in sqlite. They are not directly accessed by the user, but will be created as tables in the sqlite database:ThedbHnswIndexConfigclass represents the configuration associated with an HNSWLIB index as stored in the database.ThedbCollectionConfigclass represents the configuration associated with a collection of strings and embeddings as persisted in the database.ThedbEmbeddingclass represents an embedding as stored in the database.UsageTo usehnsqlite, you can create a new collection, add items to it, and perform search operations. Here's an example:fromhnsqliteimportCollectionimportnumpyasnp# Create a new collectioncollection=Collection(collection_name="example",dim=128)# Add items to the collectionvectors=[np.random.rand(128)for_inrange(10)]texts=[f"Text{i}"foriinrange(10)]collection.add_items(vectors,texts)# Get the number of items in the collectionitem_count=collection.count()print(f"Number of items in the collection:{item_count}")# Search for the nearest neighbors of a query vectorquery_vector=np.random.rand(128)results=collection.search(query_vector,k=5)# Print the search resultsforresultinresults:print(f"Item:{result}, Distance:{result.distance}")FilteringThe filtering function is designed to support metadata filtering similar to MongoDB.  It utilizes the hnswlib filtering function to accept or reject nearest neighbot candidates based on the embedding metadata matching a search time filtering criteria.Supported MetadataThe embeddingmetadatais a dictionary that stores metadata associated with items in the collection. The keys represent the field names of the metadata, and the supported values are strings, numbers, booleans or lists of strings.Example of a metadata dictionary:{"author":"John Doe","rating":4.5,"tags":["python","database","search"]}Filtering OperationsThe  search function supports a filter similar toMongoDB.The following operations are supported:$eq: Checks if a metadata value is equal to the specified value.$ne: Checks if a metadata value is not equal to the specified value.$gt: Checks if a metadata value is greater than the specified value.$gte: Checks if a metadata value is greater than or equal to the specified value.$lt: Checks if a metadata value is less than the specified value.$lte: Checks if a metadata value is less than or equal to the specified value.$in: Checks if a metadata value is in the specified list of values.$nin: Checks if a metadata value is not in the specified list of values.$and: Combines multiple filter conditions using an AND logical operator.$or: Combines multiple filter conditions using an OR logical operator.Usagefilter_dict={"rating":{"$gte":4},"tags":{"$in":["python","search"]},"$or":[{"author":{"$eq":"John Doe"}},{"author":{"$eq":"Jane Smith"}}]}metadata_dict={"author":"John Doe","rating":4.5,"tags":["python","database","search"]}result=filter_item(filter_dict,metadata_dict)Theresultwill beTrueif themetadata_dictsatisfies the conditions defined in thefilter_dict. In this example, the metadata has a rating greater than or equal to 4 and at least one tag from the specified list, the author is either "John Doe" or "Jane Smith", so the result will beTrue.`This will create a new collection with 10 random embeddings, get the number of items in the collection, search for the 5 nearest neighbors of a random query vector.