Package: django-native-search
django-native-searchdjango-native-search implements basic full-text search engine for Django models.The engine itself uses Django ORM to manage its index, so no additional backend is needed for searching to work. Just create a model for index, runmakemigrationsandmigrateand you are ready to feed it with data and search.InstallationInstall the package from PyPi:pip install django-native-searchThe package will be installed with all its dependencies includingdjango-expression-index.SetupSetting up the search in basic configuration is quite simple.1. Register the appAdddjango_native_searchtoINSTALLED_APPSin your settings:INSTALLED_APPS=[...'django_native_search.apps.DjangoNativeSearch',...]2. Define your Index ModelCreate a new app or in existing app, in yourmodels.py, define an index model. In this example we are creating a simple index forbooks.Bookmodel:fromdjango_native_search.modelsimportIndexEntryclassBookIndexEntry(IndexEntry):object=models.OneToOneField('books.Book',on_delete=models.CASCADE)search_template='search_index/book.txt'Theobjectfield defines a relation to a model which is being indexed.
The engine usessearch_templateto render the text withobjectvariable in template context.
By default the rendered text is tokenized with byre.searchall(r'[^\s"]+', text).
You can change this behavior by overridingtokenizeclass method in your index model.
All extracted tokens are stored in the index of respective indexed model instance.importreclassBookIndexEntry(IndexEntry):...@classmethoddeftokenize(self,text):returnre.findall(r"[^\W_]+(['_]?[^\W_]+)*",text)Index for multiple modelsIt is also possible to create index for multiple models by using model inheritance. Create a single concrete descendant model ofIndexEntrywith multiple descendants for each indexed model.You can add some common fields to this model to be used for filtering the entries, but do not addobjectsfield. Then create descendants of your IndexEntry model. Each of the derived classes should haveobjectfield which points to a model to be indexed and asearch_template.I would advise to put some additional fields to your root index model, to be able to filter entries of any kind or display the results without additional query for descendant models. You can fill the fields with data by overriding yoursavemethod in your index model.It should also be possible to useGenericForeignKeyto define theobjectfield, but I haven't tried it.Multiple indexesEach direct descendant ofIndexEntryis a separate index, so you can have multiple independent
indexes in your site.3. Prepare the databaseRun the well known commands:manage.py makemigrations
manage.py migrateThe index was tested withsqliteandPostreSQL.UsageUsually you use your index to do full-text seach within your data. Just remember to fill it with data first.Feeding the index with dataThe only thing you need to do is to create yourIndexEntrydescendant model instance and save it.frombook_index.modelsimportBookIndexEntryfrombooks.modelsimportBookforbookinBook.objects.all():BookIndexEntry(object=book).save()There is a convenient shortcut for indexing querysets:frombook_index.modelsimportBookIndexEntryBookIndexEntry.objects.rebuild()You can overrideget_index_querysetmethod in your class to doselect_relatedorfilteror anything you need, before passing the queryset for indexing.You can call therebuildmethod on your index model root class manager, to rebuild all descendant
index models.Probably you would like to create you own management command to run the indexing, but actually
you would not use it...Runtime index updatesThe indexing should be fast enough to be executed in runtime on every save of the indexed model.
Just connect a handler topost_savesignal:fromdjango.db.models.signalsimportpost_saveclassBookIndexEntry(IndexEntry):...@classmethoddefupdate_index(cls,instance,**kwargs):cls.objects.refresh([instance])post_save.connect(BookIndexEntry.update_index,sender=Book)Now your index will be always up-to-date.SearchingYou can search the index by calling the manager'ssearchmethod. The query is tokenized using
the sametokenizemethod as when indexing. All tokens must be found in a document to consider it
matched:qs=BookIndexEntry.objects.search('Monty Python')This will return aQuerySetofBookIndexEntrywhich contain both "Monty" and "Python" case
sensitively. If you want your search to be case-insensitive, then provide the query in lowercase:qs=BookIndexEntry.objects.search('circus')You can filter the search results, just as any otherQuerySet:qs=BookIndexEntry.objects.search('circus').filter(object__release_date__year__gt=1970)By default search returns matches only for whole words. If there is a single keyword in a query,
the engine does a substring search, so search results may contain documents with words matching
the keyword or containing it.For example searching for "yth" may return documents containing "python", "pythonic", "myth",
"demythologization".Substring search works fine insqlite. InPostgreSQLthere is a problem with using the db index,
so the searching might be too slow.Putting multiple words inside quotes forces searching for colocation of these words.qs=BookIndexEntry.objects.search('"Monty Python\'s Flying Circus"')This will return aQuerySetofBookIndexEntrywhich contain word "Monty" followed by "Python's",
followed by "Flying", followed by "Circus".Search formThere isSearchFormMixinavailable to easily to create your search view:fromdjango.views.generic.baseimportTemplateViewfrombook_index.modelsimportBookIndexEntryfromdjango_native_search.formsimportSearchFormMixin,searchform_factoryclassSearchView(SearchFormMixin,TemplateView):template_name="books_index/search.html"form_class=searchform_factory(BookIndexEntry)Thesearchform_factoryfunction will use all fields withdb_index = TrueinBookIndexEntryto createMultipleChoiceFieldin your form. The fields can be used to filter the results.
Each filtering field in your form will contain all possible values of the field in the database.Search templateThe templated referred bytemplate_nameis rendered withformcontaining the form instance andresultscontaining the queryset of search results if form is valid.{%blockcontent%}<h2>Search</h2><form method="get" action="."><table>{{form}}<tr><td>&nbsp;</td><td><input type="submit" value="Search" class="btn"/></td></tr></table></form>{%ifform.is_valid%}<br/><h3>Found{{results.count}}results</h3><ul>{%forresultinresults%}<li class = "search-result"><ul><li class="result-link"><a href="{{result.object.get_absolute_url}}">{{result.object.title}}</a></li><li class="result-excerpt'>{{result.excerpt}}</li></ul></li>{%endfor%}</ul>{%endif%}{%endblock%}Theexcerptmember of index entry instance returns a fragment of the indexed document with
occurrences of search keywords hihghted with<em>.SettingsThere are serveral settings to tweak the search engine.SEARCH_MIN_SUBSTR_LENGTHDefault :2Minimum number of characters in keyword to run substring search.SEARCH_MAX_SUBTSTR_COUNT_IN_QUERYDefault :300Maximum number of indexed words containing the substring to run substring search.SEARCH_MAX_EXCERPT_FRAGMENTSDefault :5Maximum number of fragments containing keywords to be returned in excerpt.SEARCH_EXCERPT_FRAGMENT_START_OFFSETDefault :-2Offset of excerpt fragment start.SEARCH_EXCERPT_FRAGMENT_END_OFFSETDefault :5Offset of excerpt fragment end.SEARCH_MAX_RANKING_KEYWORDS_COUNTDefault :3Maximum number of keywords to be used for ranking the results. If the query contains more keywords,
only the first ones will be used to calculate the ranking of results.Search APITo be described...Look into the code to check what you can do with it.PerformanceDespite the naive design, the index performs surpsisingly well, even with quite large datasets.
It can search through 100k documents containing 10M words in a fraction of a second.