Package: bungiesearch
BungiesearchTable of contentsBungiesearchPurposeFeaturesFeature examplesInstallationInstall the packageIn DjangoIn your shellStart populating the indexIn ElasticsearchQuick start exampleProcedureExampleDocumentationModelIndexSearchAliasSettingsTestingPurposeBungiesearch is a Django wrapper forelasticsearch-dsl-py.
It inherits from elasticsearch-dsl-py’sSearchclass, so all the
fabulous features developed by the elasticsearch-dsl-py team are also
available in Bungiesearch. In addition, just likeSearch,
Bungiesearch is a lazy searching class (and iterable), meaning you can
call functions in a row, or do something like the following.lazy=Article.objects.search.query('match',_all='Description')printlen(lazy)# Prints the number of hits by only fetching the number of items.foriteminlazy[5:10]:printitemFeaturesCore Python friendlyIteration ([x for x in lazy_search])Get items (lazy_search[10])Number of hits vialen(len(lazy_search))Index managementCreating and deleting an index.Creating, updating and deleting doctypes and their mappings.Update index doctypes.Django Model MappingVery easy mapping (no lies).Automatic model mapping (and supports undefined models by
returning aResultinstance ofelasticsearch-dsl-py).Efficient database fetching:One fetch for all items of a given model.Fetches only desired fields.Django ManagerEasy model integration:MyModel.search.query("match",_all="somethingto search").Search aliases (search shortcuts with as many parameters as
wanted):Tweet.object.bungie_title_search("bungie")orArticle.object.bungie_title_search("bungie"), wherebungie_title_searchis uniquely defined.Django signalsConnect to post save and pre delete signals for the elasticsearch
index to correctly reflect the database (almost) at all times.RequirementsDjango >= 1.7Python 2.7 (no Python 3 support yet)Feature examplesSee section “Full example” at the bottom of page to see the code needed
to perform these following examples. ### Query a word (or list thereof)
on a managed model.Article.objects.search.query('match',_all='Description')Use a search alias on a model’s manager.Article.objects.bsearch_title_search('title')Use a search alias on a bungiesearch instance.Article.objects.search.bsearch_title_search('title').bsearch_titlefilter('filterthis title')Iterate over search results# Will print the Django model instance.forresultinArticle.objects.search.query('match',_all='Description'):printresultFetch a single itemArticle.objects.search.query('match',_all='Description')[0]Get the number of returned itemsprintlen(Article.objects.search.query('match',_all='Description'))Deferred model instantiation# Will print the Django model instance's primary key. Will only fetch the `pk` field from the database.forresultinArticle.objects.search.query('match',_all='Description').only('pk'):printresult.pkElasticsearch limited field fetching# Will print the Django model instance. However, elasticsearch's response only has the `_id` field.forresultinArticle.objects.search.query('match',_all='Description').fields('_id'):printresultGet a specific number of items with an offset.This is actually elasticseach-dsl-py functionality, but it’s
demonstrated here because we can iterate over the results via
Bungiesearch.foriteminArticle.objects.bsearch_title_search('title').only('pk').fields('_id')[5:7]:printitemLazy objectslazy=Article.objects.bsearch_title_search('title')printlen(lazy)foriteminlazy.filter('range',effective_date={'lte':'2014-09-22'}):printitemInstallationUnless noted otherwise, each step is required.Install the packageThe easiest way is to install the package from PyPi:pip install bungiesearchNote:Check your version of Django after installing bungiesearch. It
was reported to me directly that installing bungiesearch may upgrade
your version of Django, although I haven’t been able to confirm that
myself. Bungiesearch depends on Django 1.7 and above.In DjangoUpdating your Django modelsNote:this part is only needed if you want to be able to use search
aliases, which allow you to define shortcuts to complex queries,
available directly from your Django models. I think it’s extremely
practical.Open yourmodels.pyfile.Add the bungiesearch manager import:from bungiesearch.managers import BungiesearchManagerFind the model, or models, you wish to index on Elasticsearch and set
them to be managed by Bungiesearch by adding the objects field to
them, as such:objects = BungiesearchManager(). You should now
have a Django modelsimilar to
this.Creating bungiesearch search indexesThe search indexes define how bungiesearch should serialize each of the
model’s objects. It effectively defines how your object is serialized
and how the ES index should be structured. These are referred to asModelIndexes.A good practice here is to have all the bungiesearch stuff in its own
package. For example, for the section of the Sparrho platform that uses
Django, we have a package calledsearchwhere we define the search
indexes, and a subpackage calledaliaseswhich has the many aliases
we use (more on that latter).Create a subclass ofModelIndex, which you can import from frombungiesearch.indices import ModelIndex, in a new module
preferably.In this class, define a class calledMeta: it will hold meta
information of this search index for bungiesearch’s internal working.Import the Django model you want to index (from your models file)
and, in the Meta class, define a field calledmodel, which must
be set to the model you want indexed.By default, bungiesearch will index every field of your model. This
may not always be desired, so you can define which fields must be
excluded in thisMetaclass, via the exclude field.There are plenty of options, so definitely have a read through the
documentation forModelIndex.Here’san
exampleof a
search index. There can be many such definitions in a file.Django settingsThis is the final required step. Here’s thefull
documentationof
this step.Open your settings file and add aBUNGIESEARCHvariable, which
must be a dictionary.DefineURLSas a list of URLs (which can contain only one) of
your ES servers.Define theINDICESkey as a dictionary where the key is the name
of the index on ES that you want, and the value is the full Python
path to the module which has all the ModelIndex classes for to be
indexed on that index name.SetALIASESto an empty dictionary (until you define any search
aliases).You can keep other values as their defaults.In your shellCreate the ES indexesFrom your shell, in the Django environment, run the following:python manage.py search_index--createStart populating the indexRun the following which will take each of the objects in your model,
serialize them, and add them to the elasticsearch index.python manage.py search_index--updateNote:With additional parameters, you can limit the number of
documents to be indexed, as well as set conditions on whether they
should be indexed based on updated time for example.In ElasticsearchYou can now open your elasticsearch dashboard, such as Elastic HQ, and
see that your index is created with the appropriate mapping and has
items that are indexed.Quick start exampleThis example is from thetestfolder. It may be partially out-dated,
so please refer to thetestfolder for the latest version.ProcedureIn your models.py file (or your managers.py), import bungiesearch and
use it as a model manager.Define one or more ModelIndex subclasses which define the mapping
between your Django model and elasticsearch.(Optional) Define SearchAlias subclasses which make it trivial to
call complex elasticsearch-dsl-py functions.Add a BUNGIESEARCH variable in your Django settings, which must
contain the elasticsearch URL(s), the modules for the indices, the
modules for the search aliases and the signal definitions.ExampleHere’s the code which is applicable to the previous examples. ### Django
Modelfromdjango.dbimportmodelsfrombungiesearch.managersimportBungiesearchManagerclassArticle(models.Model):title=models.TextField(db_index=True)authors=models.TextField(blank=True)description=models.TextField(blank=True)link=models.URLField(max_length=510,unique=True,db_index=True)published=models.DateTimeField(null=True)created=models.DateTimeField(auto_now_add=True)updated=models.DateTimeField(null=True)tweet_count=models.IntegerField()raw=models.BinaryField(null=True)source_hash=models.BigIntegerField(null=True)missing_data=models.CharField(blank=True,max_length=255)positive_feedback=models.PositiveIntegerField(null=True,blank=True,default=0)negative_feedback=models.PositiveIntegerField(null=True,blank=True,default=0)popularity_index=models.IntegerField(default=0)objects=BungiesearchManager()classMeta:app_label='core'ModelIndexThe following ModelIndex will generate a mapping containing all fields
fromArticle, minus those defined inArticleIndex.Meta.exclude.
When the mapping is generated, each field will the most appropriateelasticsearch core
type,
with default attributes (as defined in bungiesearch.fields).These default attributes can be overwritten withArticleIndex.Meta.hotfixes: each dictionary key must be field
defined either in the model or in the ModelIndex subclass
(ArticleIndexin this case).fromcore.modelsimportArticlefrombungiesearch.fieldsimportDateField,StringFieldfrombungiesearch.indicesimportModelIndexclassArticleIndex(ModelIndex):effectived_date=DateField(eval_as='obj.created if obj.created and obj.published > obj.created else obj.published')meta_data=StringField(eval_as='" ".join([fld for fld in [obj.link, str(obj.tweet_count), obj.raw] if fld])')classMeta:model=Articleexclude=('raw','missing_data','negative_feedback','positive_feedback','popularity_index','source_hash')hotfixes={'updated':{'null_value':'2013-07-01'},'title':{'boost':1.75},'description':{'boost':1.35},'full_text':{'boost':1.125}}SearchAliasDefines a search alias for one or more models (in this case only forcore.models.Article).fromcore.modelsimportArticlefrombungiesearch.aliasesimportSearchAliasclassSearchTitle(SearchAlias):defalias_for(self,title):returnself.search_instance.query('match',title=title)classMeta:models=(Article,)alias_name='title_search'# This is optional. If none is provided, the name will be the class name in lower case.classInvalidAlias(SearchAlias):defalias_for_does_not_exist(self,title):returntitleclassMeta:models=(Article,)Django settingsBUNGIESEARCH={'URLS':[os.getenv('ELASTIC_SEARCH_URL')],'INDICES':{'bungiesearch_demo':'core.search_indices'},'ALIASES':{'bsearch':'myproject.search_aliases'},'SIGNALS':{'BUFFER_SIZE':1}# uses BungieSignalProcessor}DocumentationModelIndexAModelIndexdefines mapping and object extraction for indexing of a
given Django model.Any Django model to be managed by bungiesearch must have a defined
ModelIndex subclass. This subclass must contain a subclass calledMetawhich must have amodelattribute (sets the model which it
represents).Class attributesAs detailed below, the doc type mapping will contain fields from the
model it related to. However, one may often need to index fields which
correspond to either a concatenation of fields of the model or some
logical operation.Bungiesearch makes this very easy: simply define a class attribute as
whichever core type, and set to theeval_asconstructor parameter to
a one line Python statement. The object is referenced asobj(notselfnorobject, justobj).ExampleThis is a partial example as the Meta subclass is not defined, yet
mandatory (cf. below).frombungiesearch.fieldsimportDateField,StringFieldfrombungiesearch.indicesimportModelIndexclassArticleIndex(ModelIndex):effective_date=DateField(eval_as='obj.created if obj.created and obj.published > obj.created else obj.published')meta_data=StringField(eval_as='" ".join([fld for fld in [obj.link, str(obj.tweet_count), obj.raw] if fld])')Here, botheffective_dateandmeta_datawill be part of the doc
type mapping, but won’t be reversed mapped since those fields do not
exist in the model.This can also be used to index foreign keys:some_field_name=StringField(eval_as='",".join([item for item in obj.some_foreign_relation.values_list("some_field", flat=True)]) if obj.some_foreign_relation else ""')Class methodsmatches_indexing_conditionOverride this function to specify whether an item should be indexed or
not. This is useful when defining multiple indices (and ModelIndex
classes) for a given model. This method’s signature and super class code
is as follows, and allows indexing of all items.defmatches_indexing_condition(self,item):returnTrueFor example, if a given elasticsearch index should contain only item
whose title starts with"Awesome", then this method can be
overridden as follows.defmatches_indexing_condition(self,item):returnitem.title.startswith("Awesome")Meta subclass attributesNote: in the following, any variable defined a being alistcould also be atuple. ##### modelRequired:defines the Django
model for which this ModelIndex is applicable.fieldsOptional:list of fields (or columns) which must be fetched when
serializing the object for elasticsearch, or when reverse mapping the
object from elasticsearch back to a Django Model instance. By default,
all fields will be fetched. Setting thiswillrestrict which fields
can be fetched and may lead to errors when serializing the object. It is
recommended to use theexcludeattribute instead (cf. below).excludeOptional:list of fields (or columns) which must not be fetched when
serializing or deserializing the object.hotfixesOptional:a dictionary whose keys are index fields and whose values
are dictionaries which definecore type
attributes.
By default, there aren’t any special settings, apart for String fields,
where theanalyzeris set to`snowball<http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-snowball-analyzer.html>`__
({'analyzer': 'snowball'}).additional_fieldsOptional:additional fields to fetch for mapping, may it be foreval_asfields or when returning the object from the database.id_fieldOptional:the model field to use as a unique ID for elasticsearch’s
metadata_id. Defaults toid(also called`pk<https://docs.djangoproject.com/en/dev/topics/db/models/#automatic-primary-key-fields>`__).updated_fieldOptional:set the model’s field which can be filtered on dates in
order to find when objects have been updated. Note, this ismandatoryto use--startand/or--endwhen updating index (withsearch_index--update).optimize_queriesOptional:set to True to make efficient queries when automatically
mapping to database objects. This willalwaysrestrict fetching to the
fields set infieldsand inadditional_fields.Note:You can
also perform an optimal database query with.only('__model'), which
will use the same fields asoptimize_queries, or.only('__fields'), which will use the fields provided in the.fields()call.indexing_queryOptional:set to a QuerySet instance to specify the query used when
the search_index command is ran to index. Thisdoes notaffect how
each piece of content is indexed.defaultEnables support for a given model to be indexed on several elasticsearch
indices. Set toFalseon all but the default index.Note: if all
managed models are set withdefault=Falsethen Bungiesearch will
fail to find and index that model.ExampleIndexes all objects ofArticle, as long as theirupdateddatetime is less than21 October 2015
04:29.fromcore.modelsimportArticlefrombungiesearch.indicesimportModelIndexfromdatetimeimportdatetimeclassArticleIndex(ModelIndex):defmatches_indexing_condition(self,item):returnitem.updated<datetime.datetime(2015,10,21,4,29)classMeta:model=Articleid_field='id'# That's actually the default value, so it's not really needed.exclude=('raw','missing_data','negative_feedback','positive_feedback','popularity_index','source_hash')hotfixes={'updated':{'null_value':'2013-07-01'},'title':{'boost':1.75},'description':{'boost':1.35},'full_text':{'boost':1.125}}optimized_queries=Trueindexing_query=Article.objects.defer(*exclude).select_related().all().prefetch_related('tags')SearchAliasASearchAliasdefine search shortcuts (somewhat similar toDjango
managers).
Often times, a given search will be used in multiple parts of the code.
SearchAliases allow you define those queries, filters, or any
bungiesearch/elasticsearch-dsl-py calls as an alias.A search alias is either applicable to alist(ortuple) of
managed models, or to any bungiesearch instance. It’s very simple, so
here’s an example which is detailed right below.ExampleThe most simple implementation of a SearchAlias is as follows. This
search alias can be called viaArticle.objects.bungie_title(orArticle.objects.search.bungie_title), supposing that the namespace
is set toNonein the settings (cf. below).Definitionfrombungiesearch.aliasesimportSearchAliasclassTitle(SearchAlias):defalias_for(self,title):returnself.search_instance.query('match',title=title)UsageArticle.objects.bungie_title('title')Method overwriteAny implementation needs to inherit frombungiesearch.aliases.SearchAliasand overwritealias_for. You
can set as many or as little parameters as you want for that function
(since bungiesearch only return the pointer to that function without
actually calling it).Since each managed model has its own doc type,self.search_instanceis a bungiesearch instance set to search the specific doctype.Meta subclass attributesAlthough not mandatory, theMetasubclass enabled custom naming and
model restrictions for a search alias.modelsOptional:list(ortuple) of Django models which are allowed
to use this search alias. If a model which is not allowed to use this
SearchAlias tries it, aValueErrorwill be raised.alias_nameOptional:A string corresponding the suffix name of this search alias.
Defaults to the lower case class name.WARNING: As explained in the “Settings” section below, all search
aliases in a given module share the prefix (or namespace). This is to
prevent aliases from accidently overwriting Django manager function
(e.g.updateorget). In other words, if you define thealias_nametotest, then it must be called asmodel_obj.objects.$prefix$_testwhere$prefix$is the prefix
defined in the settings. This prefix is also applicable to search
aliases which are available via bungiesearch instances directly. Hence,
one can define in one module search utilities (e.g.regexandrange) and define model specific aliases (e.g.title) in another
module, and use both in conjunction as such:Article.objects.search.bungie_title('searchtitle').utils_range(field='created',gte='2014-05-20',as_query=True).
These aliases can be concatenated ad vitam aeternam.Sophisticated exampleThis example shows that we can have some fun with search aliases. In
this case, we define a Range alias which is applicable to any field on
any model.classRange(SearchAlias):defalias_for(self,field,gte=None,lte=None,boost=None,as_query=False):body={field:{}}ifgte:body[field]['gte']=gteiflte:body[field]['lte']=lteifboost:ifnotas_query:logging.warning('Boost is not applicable to search alias Range when not used as a query.')else:body[field]['boost']=boostifas_query:returnself.search_instance.query({'range':body})returnself.search_instance.filter({'range':body})We can use it as suchArticle.objects.bungie_range(field='created',gte='2014-05-20',as_query=True).SettingsAdd ‘bungiesearch’ to INSTALLED_APPS.You must defineBUNGIESEARCHin your Django settings in order for
bungiesearch to know elasticsearch URL(s) and which index name contains
mappings for each ModelIndex.BUNGIESEARCH={'URLS':['localhost'],# No leading http:// or the elasticsearch client will complain.'INDICES':{'main_index':'myproject.myapp.myindices'}# Must be a module path.'ALIASES':{'bsearch':'myproject.search_aliases'},'SIGNALS':{'BUFFER_SIZE':1},'TIMEOUT':5}URLSRequired:must be a list of URLs which host elasticsearch instance(s).
This is directly sent to elasticsearch-dsl-py, so any issue with
multiple URLs should be refered to them.INDICESRequired:must be a dictionary where each key is the name of an
elasticsearch index and each value is a path to a Python module
containing classes which inherit frombungiesearch.indices.ModelIndex(cf. below).ALIASESOptional:a dictionary whose key is the alias namespace and whose
value is the Python module containing classes which inherit frombungiesearch.aliases.SearchAlias. If the namespace isNone, then
the alias will be namedbungie. If the namespace is an empty string,
there will be no alias namespace. The provided namespace will be
appended by an underscore. In the example above, each search alias
defined inmyproject.search_aliaseswill be referenced as$ModelObj$.objects.bsearch_$alias$, where$ModelObj$is a Django
model and$alias$is the name of the search alias.The purpose is to not accidently overwrite Django’s default manager
functions with search aliases.SIGNALSOptional:if it exists, it must be a dictionary (even empty), and will
connect to thepost saveandpre deletemodel functions ofallmodels usingbungiesearch.managers.BungiesearchManageras a manager.
One may also define a signal processor class for more custom
functionality by placing the string value of the module path under a key
calledSIGNAL_CLASSin the dictionary value ofSIGNALSand
definingsetupandteardownmethods, which takemodelas the
only parameter. These methods connect and disconnect the signal
processing class to django signals (signals are connected to each model
which uses a BungiesearchManager).IfSIGNALSis not defined in the settings,noneof the models
managed by BungiesearchManager will automatically update the index when
a new item is created or deleted.BUFFER_SIZEOptional:an integer representing the number of items to buffer before
making a bulk index update, defaults to100.WARNING: if your application is shut down before the buffer is
emptied, then any buffered instancewill notbe indexed on
elasticsearch. Hence, a possibly better implementation is wrappingpost_save_connectorandpre_delete_connectorfrombungiesearch.signalsin a celery task. It is not implemented as such
here in order to not requirecelery.TIMEOUTOptional:Elasticsearch connection timeout in seconds. Defaults to5.TestingThe easiest way to run the tests is to install all dev dependencies using./setup.shthen run./test.shAll Bungiesearch tests are intests/core/test_bungiesearch.py. You
can run the tests by creating a Python virtual environment, installing
the requirements fromrequirements.txt, installing the package
(pip install .) and runningpython tests/manage.py test. Make
sure to updatetests/settings.pyto use your own elasticsearch URLs,
or update the ELASTIC_SEARCH_URL environment variable.