Package: elasticsearch-partition
Elasticsearch PartitionA Python library is written on Cython for creating Elasticsearch indexes by
date range.For time oriented data, such as logs, a common strategy is to partition data
into indexes that hold data for a certain time range. For example, the indexlogstash-2018.01.01holds data for events that happened on2018-01-01, i.e.
a time range of a day. You can of course choose bigger or smaller time ranges
as well(year,monthordayfrequencies), depending on your needs. Using
index templates, you can easily manage settings and mappings for any index
created with a name starting with e.g.logstash-*.InstallationInstall the elasticsearch partition package with pip:pip install elasticsearch-partitionHow to UseBasic usageHow to import and use partition module withsinceanduntildates:importdatetimefromelasticsearch_partitionimportpartitionpartition('logs-*',datetime.date(2016,11,29),datetime.date(2018,2,4))# ['logs-2016-11-29', 'logs-2016-11-30', 'logs-2016-12-*', 'logs-2017-*',# 'logs-2018-01-*', 'logs-2018-02-01', 'logs-2018-02-02', 'logs-2018-02-03',# 'logs-2018-02-04']When you are usingpartitiononly withsincedate,untilwill be replaced
on a current date.partition('logs-*',since=datetime.date(2018,7,10))# ['logs-2018-07-10', 'logs-2018-07-11', 'logs-2018-07-12', 'logs-2018-07-13',# 'logs-2018-07-14', 'logs-2018-07-15', 'logs-2018-07-16', 'logs-2018-07-17']Or when you are usingpartitiononly withuntilall dates fromuntilto
current date will be excluded.partition('logs-*',until=datetime.date(2018,7,10))# ['-logs-2018-07-10', '-logs-2018-07-11', '-logs-2018-07-12',# '-logs-2018-07-13', '-logs-2018-07-14', '-logs-2018-07-15',# '-logs-2018-07-16', '-logs-2018-07-17', 'logs-*']Note: Ifuntilmore then current date you will get an error.How to customize partitioningIf you want to change somepartitionbahavior you can do it ease withRangePartitionandformattersmodule, also you can use your custom datenowfunctions.fromelasticsearch_partitionimportRangePartitionfromelasticsearch_partition.partitioningimportMONTHfromelasticsearch_partition.formattersimportLittleEndianDateFormatter# frequency - Index partitioning frequency# formatter - Formatter instance# escape - Special character which will be replaced on a date# now_func - Get now date functionmy_partition=RangePartition(frequency=MONTH,formatter=LittleEndianDateFormatter(sep='.'),escape='@',now_func=custom_date_now,)my_partition('logs-@',datetime.date(2016,11,29),datetime.date(2018,2,4))# ['logs-11.2016', 'logs-12.2016', 'logs-*.2017', 'logs-01.2018', 'logs-02.2018']How to create custom date formatterAll date formatters must be inherited from abstractDateFormatterclass and
implementfmt_year,fmt_monthandfmt_daymethods. Some method accept
additional keyword parameterwildcardwhich used for creating formatted date
with specified wildcard character. For example2018-04will be replced on2018-04-*,2018on2018-*etc.classMyDateFormatter(DateFormatter):deffmt_year(self,year,wildcard):# Should be implementeddeffmt_month(self,year,month,wildcard):# Should be implementeddeffmt_day(self,year,month,day):# Should be implementedpartition=RangePartition(formatter=MyDateFormatter())How to use withelasticsearch-pyThis is useful for all Elasticsearch APIs that refer to an index parameter
support execution across multiple indices.fromelasticsearchimportElasticsearches=Elasticsearch()indexes=partition('logs-*',datetime.date(2016,11,29),datetime.date(2018,2,4))res=es.search(index=indexes,body={"query":{"match_all":{}}})How to use withelasticsearch-dsl-pyThis is useful for all Elasticsearch APIs that refer to an index parameter
support execution across multiple indices and similar for simple Search and
Persistance DSL.fromelasticsearchimportElasticsearchfromelasticsearch_dslimportSearchclient=Elasticsearch()indexes=partition('logs-*',datetime.date(2016,11,29),datetime.date(2018,2,4))search=Search(using=client,index=indexes)\.filter("term",category="search")\.query("match",title="python")\.exclude("match",description="beta")response=search.execute()ChangesA full changelog is maintained in theCAHNGELOGfile.Contributingelasticsearch-partitionis an open source project and contributions are
welcome! Check out theIssuespage to see if your idea for a contribution has already been mentioned, and feel
free to raise an issue or submit a pull request.