Package: django-es-drf
Django ES DRFWork in progressA simple integration layer between Django, Elasticsearch and Django rest frameworkDjango ES DRFModel and ES Document exampleDRF exampleReturned fields from listingSearch, facetsSearch via "q" GET parametersSimple searchLuqum searchFacets and faceted filteringDjango Document and mappingCustom declaration for fieldsExcluding fieldsCustom mapping between serializer fields and ES fieldsDisabling the mappingRelationsSerializerObjects and nestedModel and ES Document exampleTo declare a document, create a model and register it with a subclass of DjangoDocument. You do not need to declare any
fields on the document - they are generated automatically.fromdjango.dbimportmodelsfromdjango_es_drfimportregistry,DjangoDocumentclassSchool(models.Model):name=models.CharField(max_length=100)address=models.TextField()@registry.register(School)classSchoolDocument(DjangoDocument):classIndex:name="schools"If you want to split the file into models and documents, be free to do so but ensure that the file including documents
is loaded at the startup - in app's ready function, imported at the bottom of the model etc.Later on, useSchoolDocument(which is a subclass of elasticsearch-dsl document):s=SchoolDocument(name='Blah',address='somewhere')s.save()forsinSchoolDocument.search().filter('term',name='Blah'):print(s)s.name=s.name+'-test's.save()# will save django and index to ESYou can work with django objects if you prefer - they will get persisted into Elastic whenever they are saved:s=School.objects.create(name='blah',address='somewhere')# it was persisted, see:doc=SchoolDocument.get(s.id)print(doc.name)You can switch between django and document representations:django_school=School.objects.get(pk=1)document_school=SchoolDocument.from_django(django_school)django_again=document_school.to_django()# they are not the same instances thoughassertdjango_schoolisnotdjango_againDRF exampleESViewSet is inherited fromModelViewSet, so feel free to use any of its features. You have to remember thatget_objectwill give you DSL Document, not an instance of model.In the viewset, specify a document, not a model/queryset.fromdjango_es_drfimportESViewSetclassSchoolAPI(ESViewSet):document=SchoolDocumentIf you need to pre-filter the initial queryset, override theget_queryset, call the parent and add your own initial
filter:fromdjango_es_drfimportESViewSetclassSchoolAPI(ESViewSet):document=SchoolDocumentdefget_queryset(self):qs=super().get_queryset()returnqs.filter('term',cooperating=True)Returned fields from listingADynamicSourceFilteris added automatically to filters and can be used to filter the properties returned in listings.You can specify the props returned on the viewset:classSchoolAPI(ESViewSet):document=SchoolDocumentsource=['id','name']or alternatively, use_includeor_excludeGET parameters containing a list of fields separated by commas
(ifsourceis set, they are active only on the fields from the source, otherwise from all fields)The filter is only used for listing and listing-like operations (that is those where object id is not present in the
URL). If you want to restrict the filter only on some calls, add ais_dynamic_source_filtering_enabled(self, request)method on the viewset class.Example:GET /schools/?_exclude=addressSearch, facetsDRF filter backends (that isfilter_backendsproperty on viewset) are used for both aggregations and search.Search via "q" GET parametersImplicitlyfilter_backendscontainQueryFilterBackendthat parses and interpretsqandparserGET parameters:theparserparameter contains name of the query interpreter that will be used. The interpreters are registered on '
ESViewSet.query_interpreters' dictionary which defaults
to{"simple": simple_query_interpreter, "luqum": luqum_query_interpreter}. If the parameter is not passed,
ESViewSet.default_query_interpreter is used. If invalid value is passed, empty result list is returned.qparameter is passed to the interpreter which applies it to a querysetSimple searchThe simple interpreter performsmulti_matchoperation on all fields.Luqum searchThis search interpreter takes the content of the query and interprets it as a luqum query string. This library enables
use ofnestedmapping in query and supports boolean operators.
Seehttps://luqum.readthedocs.io/en/latest/about.htmlfor details on the language/capabilities.Facets and faceted filteringTo enable aggregations and filtering by bucket values, addaggsto the viewset class. Example:fromdjango_es_drfimportESViewSet,BucketAggclassSchoolAPI(ESViewSet):document=SchoolDocumentaggs=[BucketAgg("name"),BucketAgg("address")]You can use the following aggregations:BucketAgg(name, field=None, type='terms')- a terms bucket filtering. If field is not set, it is the same asname.
Dot notation in thefieldis supported.NestedAgg(name, path, filter)- generates a nested aggregationTranslatedBucketAgg(name, type="terms", field=None, map=None)- translates the bucket keys through a map to get
human labelsThe aggregations are chainable. ChainingBucketAggdoes not mean nesting - the aggregations are on the same level.Bigger Example:aggs=[TranslatedBucketAgg('available',label='Available',map={True:'yes',False:'no'}),NestedAgg("tagsWithYear").bucket("tagsWithYear.tag",label="Tag").bucket("tagsWithYear.year",label="Year"),]Any options not interpreted by ES (in this case 'label') are copied into the API output (and might be used, for example,
in UI renderer).Sample returned value (aggs =aggs = [BucketAgg("name"), BucketAgg("address")]):{"aggs":[{"code":"name","__missing__":0,"__count__":2,"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"test 1","doc_count":1},{"key":"test 2","doc_count":1}]},{"code":"address","__missing__":0,"__count__":2,"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"blah","doc_count":2}]}],...}Filtering by aggregation bucketsTo filter by aggregation buckets, add?f=<bucket_code>:<value>:<bucket_code>:<value>...to the url. URLEncode '%'
and ':' if there is ':' in the unlikely case these chars are in the value. If you need a different syntax, define
aparse_facet_query(self, request)method on you viewset that returns either None or dictionary with facet code as a
key and a list of values to match as the value.Example:GET /schools/?f=name:test 2Response:{
  "count": 1,
  // ...
  "hits": [
    {
      "id": 2,
      "name": "test 2",
      "address": "blah"
    }
  ],
  "aggs": [
    // ...
  ]
}Filtering without aggregationsA bucket aggregation can be used just to filter the queryset, not to generate any aggregations. To do so, specifydisplay=Falseoption to the aggregation. The?fwill still be available, but no aggregations will be generated.aggs=[BucketAgg('available',display=False),NestedAgg("tagsWithYear").bucket("tagsWithYear.tag",label="Tag").bucket("tagsWithYear.year",label="Year"),]Django Document and mappingCustom declaration for fieldsAny fields that are already present on the document will be keep as they are. In the example above, to set thatnameis atextand notkeyword(the default), just declare:importelasticsearch_dslase@registry.register(School)classSchoolDocument(DjangoDocument):name=e.Text()classIndex:name="schools"Excluding fieldsTo exclude a field from the automatic generation, add it toexcluded. If "nested/object" mapping will be generated,
you can use a dot-separated path.@registry.register(School,excluded=['address'])classSchoolDocument(DjangoDocument):classIndex:name="schools"Custom mapping between serializer fields and ES fieldsAdd your own mapping - the key is the DRF field type, value is a function that takes the field and context and returns
ES field:
The context is an instance ofRegistrationContext.importelasticsearch_dslaseimportrest_framework.serializersass@registry.register(School,mapping={s.CharField:lambdafld,ctx,**kwargs:e.Keyword()})classSchoolDocument(DjangoDocument):classIndex:name="schools"Disabling the mappingAddgenerate=Falseto decorator's parameters:importelasticsearch_dslase@registry.register(School,generate=False)classSchoolDocument(DjangoDocument):# you need to provide your own mapping hereclassIndex:name="schools"RelationsThe framework has a rudimentary support for read-only relations. If there is an M2M or ForeignKey on model and
serializer uses target's serializer to embed the representation of the relation target, the document will contain an
object mapping for the target's data. It can be changed to Nested via mapping parameter in Document's registration
decorator.The framework does not provide any support for propagating changes on related models. If you need this, write your own
change listeners or use a more complete library, such as django-elasticsearch-dsl-drf.A simple example with a foreign key:classCity(models.Model):name=models.CharField(max_length=100)classSchool(models.Model):name=models.CharField(max_length=100)city=models.ForeignKey(City,related_name="+",on_delete=models.CASCADE)@registry.register(School,serializer_meta={"depth":1})classSchoolDocument(DjangoDocument):classIndex:name="schools"Note:thedepthinserializer_metaargument - it instructs the generated model serializer to create a serializer
for city as well. This is roughly equivalent to the following code:fromrest_frameworkimportserializersclassCitySerializer(serializers.ModelSerializer):classMeta:model=Cityexclude=()classSchoolSerializer(serializers.ModelSerializer):city=CitySerializer()classMeta:model=Schoolexclude=()@registry.register(School,serializer=SchoolSerializer)classSchoolDocument(DjangoDocument):classIndex:name="schools"If you want to change the type to nested:importelasticsearch_dslase@registry.register(School,serializer_meta={"depth":1},mapping={'city':e.Nested})classSchoolDocument(DjangoDocument):classIndex:name="schools"Note:There is no support for creating nested objects. If you need this support, you'll have to write your
owncreate/updatemethod on the serializer.SerializerOptionally you might want to provide your own serializer, to transform/generate additional fields into the document.The serializer is just a plain DRF serializer that converts django fields to document's fields, there is nothing fancy
about it.Note:If you useSerializerMethodField, be sure to add the correct field into the mapping:fromrest_frameworkimportserializersclassSchoolSerializer(serializers.ModelSerializer):name_with_address=serializers.SerializerMethodField()defget_name_with_address(self,instance):returnf"{instance.name},{instance.address}"classMeta:model=Schoolexclude=()@registry.register(School,serializer=SchoolSerializer,mapping={'name_with_address':e.Text})classSchoolDocument(DjangoDocument):classIndex:name="schools"