Package: elastipy
elastipyA python wrapper to make elasticsearch queries and aggregations more fun.Lean more atreadthedocs.io.Actually i'm just learning this stuff and have the following requests:some generic convenient data access to nested bucketed aggregations and suchthe IDE/auto-completion should help a bit/lot with all the elasticsearch parameterscontentsinstallationrequirementsquickrefaggregationsmetricsqueryexportingtestingdevelopmentinstallationpipinstallelastipyrequirementsOne thing is, of course, toinstall elasticsearch.elastipyitself requireselasticsearch-pydoc building is listed indocs/requirements.txtand mainly
consists of sphinx with the readthedocs theme.generating the interface and running the tests and notebooks is listed inrequirements.txtand contains pyyaml and coverage as well as the
usual stack of jupyter, scipy, matplotlib, ..configurationBy default anelasticsearchhost is expected atlocalhost:9200. There are currently two ways
to specify a different connection.fromelasticsearchimportElasticsearchfromelastipyimportSearch# Use an explicit Elasticsearch client (or compatible class)client=Elasticsearch(hosts=[{"host":"localhost","port":9200}],http_auth=("user","pwd"))# create a Search using the specified clients=Search(index="bla",client=client)# can also be done laters=s.client(client)Check the ElasticsearchAPI referencefor all the parameters.We can also set a default client at the program start:fromelastipyimportconnectionsconnections.set("default",client)# .. or as parameters, they get converted to an Elasticsearch clientconnections.set("default",{"hosts":[{"host":"localhost","port":9200}]})# get a clientconnections.get("default")<Elasticsearch([{'host': 'localhost', 'port': 9200}])>Different connections can be specified with thealiasname:connections.set("special",{"hosts":[{"host":"special","port":1234}]})s=Search(client="special")s.get_client()<Elasticsearch([{'host': 'special', 'port': 1234}])>aggregationsMore details can be found in thetutorial.# get a search objects=Search(index="world")# create an Aggregation class connected to the Searchagg=s.agg_date_histogram(calendar_interval="1w")# (for date-specific aggregations we can leave out the 'field' parameter#  it falls back to Search.timestamp_field which is "timestamp" by default)# submit the whole requests.execute()# access the responselist(agg.keys())['1999-12-27T00:00:00.000Z',
 '2000-01-03T00:00:00.000Z',
 '2000-01-10T00:00:00.000Z',
 '2000-01-17T00:00:00.000Z']list(agg.values())[21, 77, 60, 42]Without ametricthese numbers are the document counts.Above example as a one-liner:Search(index="world").agg_date_histogram(calendar_interval="1w").execute().to_dict(){'1999-12-27T00:00:00.000Z': 21,
 '2000-01-03T00:00:00.000Z': 77,
 '2000-01-10T00:00:00.000Z': 60,
 '2000-01-17T00:00:00.000Z': 42}nested aggregations and metricss=Search(index="world")# the first parameter is the name of the aggregation#   (if omitted it will be "a0", "a1", aso..)agg=s\.agg_terms("occasion",field="occasion")\.agg_rare_terms("rare-excuses",field="excuse",max_doc_count=2)\.metric_avg("avg-length",field="conversation_length")\.metric_max("max-length",field="conversation_length")\.execute()Therare_termsaggregation is nested into thetermsaggregation and
the metrics are siblings nested insiderare_terms.keys(),values(),items()andto_dict()all operate on the current aggregation.
For bucket aggregations they typically show thedoc_countvalue.'agg.to_dict(){('dinner', 'my mouth is too dry'): 1,
 ('dinner', "i can't reach the spoon"): 2}Therows(),dict_rows()anddump.table()methods operate on the whole aggregation branch:list(agg.dict_rows())[{'occasion': 'dinner',
  'occasion.doc_count': 200,
  'rare-excuses': 'my mouth is too dry',
  'rare-excuses.doc_count': 1,
  'avg-length': 163.0,
  'max-length': 163.0},
 {'occasion': 'dinner',
  'occasion.doc_count': 200,
  'rare-excuses': "i can't reach the spoon",
  'rare-excuses.doc_count': 2,
  'avg-length': 109.5,
  'max-length': 133.0}]agg.dump.table(colors=False)occasion │ occasion.doc_count │ rare-excuses            │ rare-excuses.doc_count │ avg-length   │ max-length  
─────────┼────────────────────┼─────────────────────────┼────────────────────────┼──────────────┼─────────────
dinner   │ 200                │ my mouth is too dry     │ 1 ██████████▌          │ 163.0 ██████ │ 163.0 ██████
dinner   │ 200                │ i can't reach the spoon │ 2 ████████████████████ │ 109.5 ████   │ 133.0 ████▉queriesfromelastipyimportquerys=Search(index="prog-world")# chaining means ANDs=s\.term(field="category",value="programming")\.term("usage","widely-used")# also can use operatorss=s&(query.Term("topic","yet-another-api")|query.Term("topic","yet-another-operator-overload"))# .query() replaces the current querys=s.query(query.MatchAll())languages_per_country=s.agg_terms(field="country").agg_terms(field="language").execute()languages_per_country.to_dict(){('IT', 'PHP'): 28,
 ('IT', 'Python'): 24,
 ('IT', 'C++'): 21,
 ('ES', 'C++'): 29,
 ('ES', 'Python'): 22,
 ('ES', 'PHP'): 18,
 ('US', 'PHP'): 23,
 ('US', 'Python'): 20,
 ('US', 'C++'): 15}exportingThere is a small helper to export stuff to elasticsearch.fromelastipyimportExporterclassMyExporter(Exporter):INDEX_NAME="my-index"# mapping can be defined here# it will be sent to elasticsearch before the first document is exportedMAPPINGS={"properties":{"some_field":{"type":"text"},}}count,errors=MyExporter().export_list(a_lot_of_objects)print(f"expored{count}objects, errors:{errors}")expored 1000 objects, errors: []It uses bulk requests and is very fast, supports document transformation and
control over id and sub-index of documents.importdatetimeclassMyExporter(Exporter):INDEX_NAME="my-index-*"MAPPINGS={"properties":{"some_field":{"type":"text"},"group":{"type":"keyword"},"id":{"type":"keyword"},"timestamp":{"type":"date"},}}# if each document has a unique id value we can use it# as the elasticsearch id as well. That way we do not# create documents twice when exporting them again.# Their data just gets updated.defget_document_id(self,es_data):returnes_data["id"]# we can bucket documents into separate indicesdefget_document_index(self,es_data):returnself.index_name().replace("*",es_data["group"])# here we can adjust or add some data before it gets exported.# it's also possible to split the data into several documents#   by yielding or returning a listdeftransform_document(self,data):data=data.copy()data["timestamp"]=datetime.datetime.now()returndataMyExporter().export_list(a_lot_of_objects)(1000, [])If we are tired enough we can call:MyExporter().delete_index()TrueThis will actually delete all sub-indices because there's this wildcard*in theINDEX_NAME.More examples can be foundhere.testingTo run the tests call:pythontest.pyTo include testing against a live elasticsearch:pythontest.py--liveTo changelocalhost:9200to something different
pass any arguments as json:pythontest.py--live--elasticsearch'{"hosts": [{"host": "127.0.0.5", "port": 1200}], "http_auth": ["user", "password"]}'The live tests will create new indices and immediately destroy them afterwards.
They are prefixed withelastipy---unittest-To check the coverage of the tests add-cor-mflags.-mwill add the missing line numbers to the summary.developmentThe methods forqueriesandaggregationsas well as thequery
classesare auto-generated fromyaml files.
They include all parameters, default values and documentation.Add a missing query or aggregationCreate a yaml file with the name of it in one of the sub-directories
indefinition/queryordefinition/aggregation.The sub-directories inquery/are just for tidiness and
follow the nesting in the sidebar of the officialdocumentation.
The three directories belowaggregation/actually define the
aggregation typebucket,metricorpipeline.Create the python code via# in project rootpythongenerate_interfaces.pyThis will update the files:elastipy/query/generated_classes.pyelastipy/query/generated_interface.pyelastipy/aggregation/generated_interface.pyThe sphinx documentation will collect the respective documentation
from these files.Update the example and tutorial notebooksDo some changes or add a new notebook (and keep mainrequirements.txtup to date).Execute:pythonrun_doc_notebooks.py--executeThis will convert the notebooks to.rstfiles into thedocs/directory.Thedocs/quickref.ipynbnotebook will even be rendered
as markdown into this README.The-e/--executeflag is required for proper doc building. For debugging
purposes it can be omitted in which case the current notebook state is
rendered.Runcddocs/
pipinstall-rrequirements.txt
makeclean&&makehtmland inspect the results indocs/_build/html/index.html.Before committing changes runpipinstallpre-commit
pre-commitinstallThis will install a pre-commit hook from.pre-commit-config.yamlthat clears the output of all notebooks.
Since the interesting ones are already rendered to the document pages, i just
think this is more tidy and releases one from cleaning up the execution state
of notebooks by hand before committing.Generally, i'm stuck withrestructuredtextfor the docstrings although
besides the:param:syntax i find it simply repellent.
It still has the most supported toolchain it seems.history / changesv0.2.0 (2021/02)clean up test interface and createtest.pyaddtravis-ciandread-the-docs.move tutorial code to jupyter notebooks and unify quickref in README and in docsmove all aggregation print helpers toAggregation.dumpclean upAggregation.itemsand all those data accessorsaddAggregation.to_matrixadd interface for allSearchparameters. This might change though..apply rst doc standards to the auto-generated documentation from the yamlsput pandas plotting interface intoAggregation.plotv0.1.0 (2021/01)complete rework and renaming of interfaceauto-generated methods and classesclean-up of Exporternice table printv0.0.1 (2020/12)Initial crappy release