Package: elasticmagic-qf-attrs
Elasticmagic query filters for attributesLibrary to store, filter and build facets for custom attributesThe problemEach attribute pair can be stored in the index as a nested document.
We can use following mapping for that:attrs:type:nestedproperties:attr_id:type:integer# usually only one of the next fields should be populatedvalue_int:type:integervalue_bool:type:booleanvalue_float:type:floatThis makes it possible to filter documents by an attribute id and its value
(for example we want to find all the documents withattr_id = 1234andvalue = 5678):query:bool:filter:-nested:path:attrsquery:bool:must:-term:attrs.attr_id:1234-term:attrs.value_int:5678It is also possible to build a facets for all attributes at once:aggs:attrs_nested:nested:path:attrsaggs:attrs:terms:field:attrs.attr_idaggs:values:field:attrs.value_intor for a single attribute:aggs:attrs_nested:nested:path:attrsaggs:attr_1234:filter:term:attrs.attr_id:1234aggs:values:field:attrs.value_intBut nested documents have some drawbacks. Every nested document is stored
in the index as different document. For instance, next document will be stored
as 5 lucene documents:name:"I'madocumentwithnestedattributes"attrs:-attr_id:1value_int:42-attr_id:2value_int:43-attr_id:3value_bool:true-attr_id:4value_float:99.9Nested queries are slow by itself:In particular, joins should be avoided. nested can make queries several times
slower and parent-child relations can make queries hundreds of times slower.https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-search-speed.html#_document_modelingBut what is worse regular queries are also slower when there are nested documents
in the index. It is because of all the fields of main documents becomes sparse.
This in turn degrades performance of all filters and accesses to doc_values.The solutionThe idea is to encode pair of an attribute id and a corresponding value into
a single value. If our attribute ids are 32-bit integers and all value types
also fit into 32 bits we can store them as a single 64-bit value.So our mapping can be:attrs:type:objectproperties:int:type:longbool:type:longfloat:type:longDocument with encoded attributes:name:"I'madocumentwithpackedattributes"attrs:# (1 << 32) | 42-int:0x1_0000002a# (2 << 32) | 43-int:0x2_0000002b# (3 << 1) | 1-bool:0x7# (4 << 32) | {integer representation of 99.9}# (4 << 32) | struct.unpack('=I', struct.pack('=f', 99.9))[0]-float:0x4_42c7cccdNow with a bit of bit magic we can emulate nested queries.Filtering by attribute id1234with value5678:query:bool:filter:-term:attrs.int:0x4d2_0000162eBuilding facet for all attribute values:aggs:attrs_int:terms:field:attrs.int# specify big enough aggregation size# so all flat attrite values should fitsize:10000One more step that client should do is to decode and group values by
attribute id.How to use it in pythonfromelasticsearchimportElasticsearchfromelasticmagicimportCluster,Document,Fieldfromelasticmagic.typesimportList,Longfromelasticmagic.ext.queryfilterimportQueryFilterfromelasticmagic_qf_attrsimportAttrBoolFacetFilterfromelasticmagic_qf_attrsimportAttrIntFacetFilterfromelasticmagic_qf_attrsimportAttrRangeFacetFilterfromelasticmagic_qf_attrs.utilimportmerge_attr_value_boolfromelasticmagic_qf_attrs.utilimportmerge_attr_value_floatfromelasticmagic_qf_attrs.utilimportmerge_attr_value_int# Specify documentclassAttrsDocument(Document):__doc_type__='attrs'ints=Field(List(Long))bools=Field(List(Long))floats=Field(List(Long))# Create an indexindex_name='test-attrs'client=Elasticsearch()client.indices.create(index=index_name)cluster=Cluster(client)index=cluster.get_index(index_name)index.put_mapping(AttrsDocument)# Index example documentindex.add([AttrsDocument(ints=[merge_attr_value_int(1,42),merge_attr_value_int(2,43),],bools=[merge_attr_value_bool(3,True)],floats=[merge_attr_value_float(4,99.9)],),],refresh=True)# Define a query filterclassAttrsQueryFilter(QueryFilter):ints=AttrIntFacetFilter(AttrsDocument.ints,alias='a')bools=AttrBoolFacetFilter(AttrsDocument.bools,alias='a')ranges=AttrRangeFacetFilter(AttrsDocument.floats,alias='a')# Now we can build facetsqf=AttrsQueryFilter()sq=index.search_query()sq=qf.apply(sq,{})res=sq.get_result()assertres.total==1qf_res=qf.process_result(res)# And finally lets print resultsforattr_id,facetinqf_res.ints.facets.items():print(f'>{attr_id}:')forfacet_valueinfacet.all_values:print(f'{facet_value.value}: ({facet_value.count_text})')forattr_id,facetinqf_res.bools.facets.items():print(f'>{attr_id}:')forfacet_valueinfacet.all_values:print(f'{facet_value.value}: ({facet_value.count_text})')forattr_id,facetinqf_res.ranges.facets.items():print(f'>{attr_id}: ({facet.count})')# Also we can filter documents:qf=AttrsQueryFilter()sq=index.search_query()sq=qf.apply(sq,{'a1':'42','a3':'true','a4__lte':'100',})res=sq.get_result()assertres.total==1qf=AttrsQueryFilter()sq=index.search_query()sq=qf.apply(sq,{'a4__gte':'100',})res=sq.get_result()assertres.total==0This script should print:> 1:
  42: (1)
> 2:
  43: (1)
> 3:
  True: (1)
> 4: (1)