Package: django-vectordb
Django VectorDBAdding extremely fast, low-latency, and scalable vector similarity search to django applications.Full documentation for the project is available athttps://pkavumba.github.io/django-vectordb/.Django Vector Databaseis a powerful and flexible toolkit for adding vector similarity search capabilities to your Django applications. It is built on top of the lighteningly fast nearest neighbor search library: hnswlib.Some reasons you might want to use Django Vector DB:Low latency, because you don't need to call an external API.Scalable to a billion vectors with millisecond search results.Fast and accurate search.Native Django integration.Metadata filtering with the full power of the django queryset queries, e.g.vectordb.filter(metadata__user_id=1).search("some text").only("text")Automatic syncing between your models and the vector index, simply register the provided signals and you can continue about your day. Vectordb will sync the vector database whenever you create, update or delete an instance.Out of the box support for incremental updates, allowing you to add or update data without rebuilding the entire index.Extensive documentation and support for easy implementation and troubleshooting.RequirementsDjango VectorDB requires the following:Python(3.6, 3.7, 3.8, 3.9, 3.10, 3.11)Django(2.2, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2)HNSWLib(0.7.0)numpyWehighly recommendand only officially support the latest patch release of
each Python and Django series.The following packages are optional:Sentence-Transformers- Add support for converting text into vector embeddings used for similarity searchDjango Rest Framework- Add API endpoint for VectorDB.django-filters- Add metadata filtering support on the API endpoint.InstallationInstall usingpip, it is recommended that you install the optional packages with:# This will install the optional dependencies above.pipinstall"django-vectordb[standard]"If you dont want to install the optional packages you can run:pipinstalldjango-vectordbAdd'django-vectordb'to yourINSTALLED_APPSsetting.INSTALLED_APPS=[...'vectordb',]Run the migrations to create thevectordbtable$./manage.pymigrateIf you're intending to use the API, you'll probably also want to add vectordb.urls. Add the following to your rooturls.pyfile.urlpatterns=[...path('api/',include('vectordb.urls'))]Note: that the URL path can be whatever you want.This will expose endpoints for all CRUD actions (/api/vectordb/) and searching (/api/vectordb/search/).ExampleLets beging with a simple example for blog postsfromdjango.dbimportmodelsfromdjango.contrib.authimportget_user_modelUser=get_user_model()classPost(models.Model):title=models.CharField(max_length=100)description=models.TextField()user=models.ForeignKey(User,on_delete=models.CASCADE)def__str__(self):returnself.title1. ImportingTo begin working with VectorDB, you'll first need to import it into your project. There are two ways to do this, depending on whether you'd like to use the simple proxy to the vector models manager,Vector.objects, or the Vector model directly.Option 1: Import the simple proxy [Recommended]fromvectordbimportvectordbOption 2: Import the Vector model directlyfromvectordbimportget_vectordb_modelVectorModel=get_vectordb_model()With either of these imports, you'll have access to all the Django manager functions available on the object. Note that you can run the commands detailed below usingvectordborVectorModel.objects, whichever you've chosen to import. For the rest of this guide we will usevectordb.Now that you've imported VectorDB, it's time to dive in and explore its powerful features!Populating the Vector DatabaseFirst, let's make a few updates to the model to allow VectorDB to handle most tasks for us: addget_vectordb_textandget_vectordb_metadatamethods.fromdjango.dbimportmodelsfromdjango.contrib.authimportget_user_modelUser=get_user_model()classPost(models.Model):title=models.CharField(max_length=100)description=models.TextField()user=models.ForeignKey(User,on_delete=models.CASCADE)def__str__(self):returnself.titledefget_vectordb_text(self):# Use title and description for vector searchreturnf"{self.title}--{self.description}"defget_vectordb_metadata(self):# Enable filtering by any of these metadatareturn{"title":self.title,"description":self.description,"user_id":self.user.id,"model":"post"}In an existing project, you can run thevectordb_syncmanagement command to add all items to the database../manage.pyvectordb_sync<app_name><model_name>For this example:./manage.pyvectordb_syncblogPostManually adding items to the vector databaseVectorDB provides two utility methods for adding items to the database:vectordb.add_instanceorvectordb.add_text. Note that for adding the instance, you need to provide theget_vectordb_textand an optionalget_vectordb_metadatamethods.1. Adding Model Instancespost1=Post.objects.create(title="post1",description="post1 description",user=user1)# provide valid user# add to vector databasevectordb.add_instance(post1)2. Adding Text to the ModelTo add text to the database, you can usevectordb.add_text():vectordb.add_text(text="Hello text",id=3,metadata={"user_id":1})Thetextandidare required. Additionally, theidmust be unique, or an error will occur.metadatacan beNoneor any valid JSON.Automatically Syncing Your Model to the vector databaseTo enable auto sync, register the model to vectordb sync handlers inapps.py. The sync handlers are signals defined invectordb/sync_signals.py.fromdjango.appsimportAppConfigclassBlogConfig(AppConfig):default_auto_field="django.db.models.BigAutoField"name="blog"defready(self):from.modelsimportPostfromvectordb.shortcutsimportautosync_model_to_vectordbautosync_model_to_vectordb(Post)This will automatically sync the vectors when you create and delete instances.Note that signals are not called in bulk create, so you will need to sync manually when using those methods.Ensure that your models implement theget_vectordb_text()and/orget_vectordb_metadata()methods.SearchingTo search, simply callvectordb.search():vectordb.search("Some text",k=10)# k is the maximum number of results you want.Note: search method returns a query whose results are order from best match. Each item will have the following fields:id,content_object,object_id,content_type,text,embedding, annotatedscore, and a propertyvectorthat returns thenp.ndarrayrepresentation of the item. Because search gives us aQuerySetwe can choose the fiels we want to see like sos:vectordb.search("Some text",k=10).only('text','content_object')Ifkis not provided, the default value is 10.Metadata Filtering with Django Vector DatabaseDjango vector database provides a powerful way to filter on metadata, using the intuitive Django QuerySet methods.You can filter ontextormetadatawith the full power of Django QuerySet filtering. You can combine as many filters as needed. And since Django vector database is built on top of Django QuerySet, you can chain the filters with the search method. You can also filter on nested metadata fields.# scope the search to user with an id 1vectordb.filter(metadata__user_id=1).search("Some text",k=10)# example two with more filtersvectordb.filter(text__icontains="Apple",metadata__title__icontains="IPhone",metadata__description__icontains="2023").search("Apple new phone",k=10)If our metadata was nested like follows:{"text":"Sample text","metadata":{"date":{"year":2021,"month":7,"day":20,"time":{"hh":14,"mm":30,"ss":45}}}}We can filter on the nested fields like so:vectordb.filter(metadata__date__year=2021,metadata__date__time__hh=14).search("Sample text",k=10)We can also use model instances instead of text:post1=Post.objects.get(id=1)# Limit the search scope to a user with an id of 1results=vectordb.filter(metadata__user_id=1).search(post1,k=10)# Scope the results to text which contains France, belonging to user with id 1 and created in 2023vectordb.filter(text__icontains="Apple",metadata__title__icontains="IPhone",metadata__description__icontains="2023").search(post1,k=10)Refer to theDjango documentationon querying theJSONFieldfor more information on filtering.SettingsYou can provide your settings in thesettings.pyfile of your project. The following settings are available:# settings.pyDJANGO_VECTOR_DB={"DEFAULT_EMBEDDING_CLASS":"vectordb.embedding_functions.SentenceTransformerEncoder","DEFAULT_EMBEDDING_MODEL":"all-MiniLM-L6-v2","DEFAULT_EMBEDDING_SPACE":"l2",# Can be "cosine" or "l2""DEFAULT_EMBEDDING_DIMENSION":384,# Default is 384 for "all-MiniLM-L6-v2""DEFAULT_MAX_N_RESULTS":10,# Number of results to return from search maximum is default is 10"DEFAULT_MIN_SCORE":0.0,# Minimum score to return from search default is 0.0"DEFAULT_MAX_BRUTEFORCE_N":10_000,# Maximum number of items to search using brute force default is 10_000. If the number of items is greater than this number, the search will be done using the HNSW index.}QuickstartCan't wait to get started? Thequickstart guideis the fastest way to get up and running, and building APIs with REST framework.DevelopmentClone the repository$gitclonehttps://github.com/pkavumba/django-vectordb.gitInstall the app in editable mode with all dev dependencies:pipinstall-e.[dev]If you're using the Zsh shell, please execute the following command (pay attention to the quotes):pipinstall-e".[dev]"This command will install the app and its dev dependencies specified in thesetup.cfgfile. The-eflag installs the package in editable mode, which means that any changes you make to the app's source code will be reflected immediately without needing to reinstall the package. The[dev]part tellspipto install the dependencies listed under the "dev" section in theoptions.extras_requireof thesetup.cfgfile.Run testspytestOrtox