Package: django-es
Django ES=========.. contents:: Table of contents:depth: 2Purpose=======Django ES is a Django wrapper for`elasticsearch-dsl-py <https://github.com/elasticsearch/elasticsearch-dsl-py>`__.Originally it's a fork from `bungiesearch <https://github.com/ChristopherRabotin/bungiesearch>`__ soyou'll find a lot of things in common.The big change is it uses register admin as a philosophy instead of django manager.So a lot of code has been removed and there is a lot of changes.There are no alias, no management commands.This contribution use elasticsearch 5.x and its restrictions (unique field name related to one unique mapping definition).CRUD operations are mostly done by elasticsearch-dsl library for more control and maintainability.Features========-  Django Model Mapping-  Very easy mapping (no lies).-  Automatic model mapping (and supports undefined models byreturning a ``Result`` instance of ``elasticsearch-dsl-py``).-  Django Admin register like-  Register your model as you do with Django Admin contributionin a separated file.-  Django signals-  Connect to pre_save, post save and pre delete signals for the elasticsearchindex to correctly reflect the database (almost) at all times.-  Requirements-  Django >= 1.8-  Python 2.7 (**no Python 3 support yet**)Installation============Install the package-------------------The easiest way is to install the package from github:``pip install git+ssh://git@github.com/GuillaumeCisco/django-es.git``**Note:** Check your version of Django after installing django-es. Itwas reported to me directly that installing django-es may upgradeyour version of Django, although I haven't been able to confirm thatmyself. django-es depends on Django 1.8 and above.In Django---------Updating your Django models~~~~~~~~~~~~~~~~~~~~~~~~~~~Create a ``djangoes.py`` python file (or package) and register your models.More description, in examples following.Creating Django ES search indexes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~The search indexes define how Django ES should serialize each of themodel's objects. It effectively defines how your object is serializedand how the Elasticsearch index should be structured.Now you should run a first time your server for allowing autodiscovermodule to generate mapping and communicate with your ElasticSearchserver.In Elasticsearch----------------You can now open your elasticsearch dashboard, such as Elastic HQ, andsee that your index is created with the appropriate mapping and hasitems that are indexed.Quick start example===================Procedure---------Add 'django_es' to `INSTALLED_APPS`.You can define in your own code an `ES_CLIENT` parameter for connecting to your Elasticsearch instance,By default `ES_CLIENT` is `Elasticsearch()`Example-------### Django Model.. code:: pythonfrom django.db import modelsfrom django.core.urlresolvers import reversefrom autoslug import AutoSlugFieldfrom wall.models import Wallfrom category.models import Categoryclass MyModel(models.Model):name = models.CharField(max_length=128, null=True, blank=True)created = models.DateTimeField(auto_now_add=True)wall = models.ForeignKey(Wall, related_name='mymodels', null=True, blank=True)slug = AutoSlugField(populate_from='populate_slug', unique=True)last_modified = models.DateTimeField(auto_now_add=True)is_finalized = models.BooleanField(default=False)is_recorded = models.BooleanField(default=False)desc = models.CharField(max_length=4096, null=True, blank=True)diff_date = models.DateTimeField()duration = models.DurationField(null=True, blank=True)category = models.ForeignKey(Category)def __str__(self):return self.namedef get_absolute_url(self):return reverse('video', kwargs={'slug': self.slug})# use this technique because name if from parent classdef populate_slug(self):return self.name or 'mymodel'class Meta(Media.Meta):app_label = 'media'ModelIndex~~~~~~~~~~The following ModelIndex will generate a mapping containing all fieldsfrom ``MyModel``, minus those defined in ``MyModelModelIndex.Meta.exclude``.When the mapping is generated, each field will the most appropriate`elasticsearch coretype <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html>`__,with default attributes (as defined in django_es.fields).These default attributes can be overwritten with``MyModelModelIndex.Meta.hotfixes``: each dictionary key must be fielddefined either in the model or in the ModelIndex subclass(``MyModelModelIndex`` in this case)... code:: pythonfrom django_es import mappingfrom django_es.fields import String, Date, Integerfrom django_es.indices import ModelIndexfrom media.models import MyModelfrom elasticsearch_dsl.analysis import Analyzerfrom utils.fields import Completionclass MyModelModelIndex(ModelIndex):description = String(analyzer='snowball', _model_attr='desc')created_date = Date(_model_attr='created')category = Integer(_eval_as='obj.category.id')img = String()author = String()suggest = Completion(analyzer=Analyzer('simple'),search_analyzer=Analyzer('simple'),preserve_position_increments=False,preserve_separators=False,payloads=True,context={'type': {'type': 'category','path': '_type'}})class Meta:index = 'django_es'  # optional but recommended, default is `django_es`, ever use `populate_index` methodexclude = ('last_modified', 'is_finalized', 'is_recorded', 'diff_date', 'duration',)def prepare_img(self, obj):# How we want to store this field in elasticsearchfrom media.serializers.liveVideo import MyModelSerializerreturn MyModelSerializer._img(obj, '48x48')  # getting related image passing resdef prepare_author(self, obj):return obj.wall.profile.get_full_name()def prepare_suggest(self, obj):# How we want to store this field in elasticsearchreturn {'input': [obj.name, obj.desc, obj.wall.profile.get_full_name()],'output': obj.name + ' - ' + obj.wall.profile.get_full_name(),'payload': {'slug': obj.slug,'img': self.prepare_img(obj),'category': obj.category.id}}mapping.register(MyModel, MyModelModelIndex)The last line is important, it allows Django ES to create the mapping related to this modeland to put in on the elasticsearch server.This `djangoes.py` file use a Completion Field not related to the model fieldderived from elasticsearch-dsl.fields.You can create your own fields if there are not already provided by elasticsearch-dslor this contribution... code:: pythonfrom elasticsearch_dsl import Fieldclass Completion(Field):_param_defs = {'fields': {'type': 'field', 'hash': True},'analyzer': {'type': 'analyzer'},'search_analyzer': {'type': 'analyzer'},'max_input_length': {'type': 'integer'}}name = 'completion'def _empty(self):return ''Now, for your mapping and index to be generated, you need to launch your server a first time.Your mappings can be updated following these`elasticsearch mappings rules <https://www.elastic.co/blog/changing-mapping-with-zero-downtime>`__,Creating/Updating, Deleting documents~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~By default, your documents are created on ``post_save`` signal of the model.But with an API oriented website or with django forms, you can directly useelasticsearch-dsl methods or simply use functions defined in ``utils``:``update_index`` and ``delete_index``Example:.. code:: python# for updating/deleting one or more instances simultaneouslyupdate_index([instance, ...], sender, bulk_size=1)  # chose your action : index or delete, default is index# for deletingdelete_index_item(instance, sender)The ``update_index`` functions use the ``bulk``/``bulk_index`` method of elasticsearch for performingseveral actions in a row.You can create your own utils methods.Querying your elasticsearch documents~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~You can query your documents using elasticsearch-dsl methods. It's the easier way.Example:.. code:: pythonfrom elasticsearch import Elasticsearchfrom elasticsearch_dsl import Q as _Qfrom elasticsearch_dsl import Searchfrom elasticsearch_dsl.query import MultiMatchsearchstr = 'Some terms to research'client = Elasticsearch()s = Search().using(client)fields = ["name^2.0", "description^1.5", "author^1.0"]s.query(MultiMatch(query=searchstr, type='best_fields', fields=fields, tie_breaker=0.3))#or#s.query(_Q('query_string', query=' AND '.join([x + '~2' for x in searchstr.split(' ')]), use_dis_max=True, fields=fields, tie_breaker=0.3))s.aggs.bucket('list', 'filter', term={'_type': 'mymodel'}) \.metric('obj','top_hits',**{'_source': ['name', 'slug', 'img'],'from': (int(page) - 1) * 20,'size': 20})s = s[:0]  # getting only aggregations resultsresponse = s.execute()count = response.aggregations.list.obj.hits.totalres = [x._source.to_dict() for x in response.aggregations.list.obj.hits.hits]You also can use your ``suggest`` field defined previously:.. code:: pythonfrom elasticsearch import Elasticsearchfrom elasticsearch_dsl import Searchsearchstr = 'Some terms to research'client = Elasticsearch()s = Search().using(client)\.suggest('lives', searchstr, completion={'field': 'suggest', 'fuzzy': True, 'size': 5, 'context': {'type': 'mymodel'}})s = s[:0]  # getting only suggestions resultsresponse = s.execute()def format_result(options):results = []for x in options:d = x['payload'].to_dict()d.update(name=x.text)results.append(d)return resultslives = format_result(response.suggest.lives[0]['options'])Django settings~~~~~~~~~~~~~~~You can define a ``DJANGO_ES`` dict in your settings for overriding the way signalsare dealt with models associated with Django_ES instances.You can inspect the code and find in the signals packages inspiration for your business logic,or use the classic ``BaseDjangoESSignalProcessor`` which will use a buffer of 100 objects beforecreating/updating/deleting deleting elasticsearch doctype objects... code:: pythonDJANGO_ES = {'SIGNAL_CLASS': 'BaseDjangoESSignalProcessor'  # default}Documentation=============ModelIndex----------A ``ModelIndex`` defines mapping and object extraction for indexing of agiven Django model. It is possible to create directly a mapping withouta model too, just pass a doctype.Any Django model to be managed by Django ES must have a definedModelIndex subclass. This subclass must contain a subclass called``Meta``.Class attributes~~~~~~~~~~~~~~~~As detailed below, the doc type mapping will contain fields from themodel it related to. However, one may often need to index fields whichcorrespond to either a concatenation of fields of the model or somelogical operation.Django ES makes this very easy: simply define a class attribute aswhichever core type, and set to the ``eval_as`` constructor parameter toa one line Python statement. The object is referenced as ``obj`` (not``self`` nor ``object``, just ``obj``).You can also use ``prepare_%s`` functions with name of the field for more complexserialization.Example^^^^^^^This is a partial example as the Meta subclass is not defined, yetmandatory (cf. below)... code:: pythonfrom django_es.fields import Date, String, Integerfrom django_es.indices import ModelIndexclass MyModelModelIndex(ModelIndex):description = String(analyzer='snowball', _model_attr='desc')created_date = Date(_model_attr='created')category = Integer(_eval_as='obj.category.id')img = String()def prepare_img(self, obj):# How we want to store this field in elasticsearchfrom media.serializers.liveVideo import MyModelSerializerreturn MyModelSerializer._img(obj, '48x48')Here, ``img`` will be part of the doctype mapping, but won't be reversed mapped since those fields do notexist in the model.``description`` and ``created_date`` use the ``_model_attr`` link for redefining fields name.``category`` will be evaluated as an integer related to the Category foreignkey.This can also be used to index foreign keys:.. code:: pythonsome_field_name = String(_eval_as='",".join([item for item in obj.some_foreign_relation.values_list("some_field", flat=True)]) if obj.some_foreign_relation else ""')# ordef prepare_some_field_name(self, obj):if obj.some_foreign_relation:return ','.join([item for item in obj.some_foreign_relation.values_list("some_field", flat=True)])return ''Class methods~~~~~~~~~~~~~populate\_index^^^^^^^^^^^^^^^Override this method if you want to deal with dynamic index generation.Example with dynamic date:It will create a new index every day... code:: pythondef populate_index(self):return 'my_index_name-%(now)s' % {'now': now().strftime("%Y.%m.%d")}matches\_indexing\_condition^^^^^^^^^^^^^^^^^^^^^^^^^^^^Override this function to specify whether an item should be indexed ornot. This is useful when defining multiple indices (and ModelIndexclasses) for a given model. This method's signature and super class codeis as follows, and allows indexing of all items... code:: pythondef matches_indexing_condition(self, item):return TrueFor example, if a given elasticsearch index should contain only itemwhose title starts with ``"Awesome"``, then this method can beoverridden as follows... code:: pythondef matches_indexing_condition(self, item):return item.title.startswith("Awesome")Meta subclass attributes~~~~~~~~~~~~~~~~~~~~~~~~**Note**: in the following, any variable defined as being a ``list``could also be a ``tuple``.fields^^^^^^*Optional:* list of fields (or columns) which must be fetched whenserializing the object for elasticsearch, or when reverse mapping theobject from elasticsearch back to a Django Model instance. By default,all fields will be fetched. Setting this *will* restrict which fieldscan be fetched and may lead to errors when serializing the object. It isrecommended to use the ``exclude`` attribute instead (cf. below).exclude^^^^^^^*Optional:* list of fields (or columns) which must not be fetched whenserializing or deserializing the object.hotfixes^^^^^^^^*Optional:* a dictionary whose keys are index fields and whose valuesare dictionaries which define `core typeattributes <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html>`__.By default, there aren't any special settings, apart for String fields,where the`analyzer <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-analyzers.html>`__is set to```snowball`` <http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-snowball-analyzer.html>`__(``{'analyzer': 'snowball'}``).additional\_fields^^^^^^^^^^^^^^^^^^*Optional:* additional fields to fetch for mapping, may it be for``eval_as``/``prepare_%s`` fields or when returning the object from the database.id\_field^^^^^^^^^*Optional:* the model field to use as a unique ID for elasticsearch'smetadata ``_id``. Defaults to ``id`` (also called```pk`` <https://docs.djangoproject.com/en/dev/topics/db/models/#automatic-primary-key-fields>`__).Settings--------Add 'django_es' to INSTALLED_APPS.SIGNAL_CLASS~~~~~~~~~~~~*Optional:* if it exists, it must be a dictionary (even empty), and willconnect to the ``pre_save``, ``post save``, ``pre delete`` model functions of *all*models registered.One may also define a signal processor class for more customfunctionality by placing the string value of the module path under a keycalled ``SIGNAL_CLASS`` defining ``setup`` and ``teardown`` methods,which take ``model`` as the only parameter. These methods connect and disconnect the signalprocessing class to django signals (signals are connected to each modelregistered).Example with a customized ``SIGNAL_CLASS``In the settings:.. code:: pythonDJANGO_ES = {'SIGNAL_CLASS': '.signals.DjangoESSignalProcessor'}In a separated file:.. code:: pythonfrom django.db.models import signalsclass DjangoESSignalProcessor(object):@staticmethoddef post_save_connector(sender, instance, created, **kwargs):from django_es.utils import update_index# Only create index if createdif created:update_index([instance], sender, bulk_size=1)@staticmethoddef pre_delete_connector(sender, instance, **kwargs):from django_es.utils import delete_index_itemdelete_index_item(instance, sender)def setup(self, model):signals.post_save.connect(self.post_save_connector, sender=model)signals.pre_delete.connect(self.pre_delete_connector, sender=model)def teardown(self, model):signals.pre_delete.disconnect(self.pre_delete_connector, sender=model)signals.post_save.disconnect(self.post_save_connector, sender=model)BUFFER\_SIZE^^^^^^^^^^^^*Optional:* an integer representing the number of items to buffer beforemaking a bulk index update, defaults to ``100``.**WARNING**: if your application is shut down before the buffer isemptied, then any buffered instance *will not* be indexed onelasticsearch. Hence, a possibly better implementation is wrapping``post_save_connector`` and ``pre_delete_connector`` from``django_es.signals`` in a celery task. It is not implemented as suchhere in order to not require ``celery``.