Package: pyesbulk
py-es-bulkA simple wrapper around the Pythonelasticsearchclientput_template(),streaming_bulk(), andparallel_bulk()helper APIs with robust error handling.This library is designed to work across various versions of the
elasticsearch Python module and of the Elasticsearch server, by
dynamically identifying the module used to create theElasticsearchobject.These names are available for import:put_templatePush a document template to the server using a specified
Elasticsearch object. This module will determine whether
a template document of the same name and version already
exists, and PUT the new template if not.Args:es: An instance of theElasticsearchclass.name: The name of the template.mapping_name: The name of the mapping used in the template.body: The payload body of the template.Returns: A tuple (start_time, end_time, retry_count, error_keys)streaming_bulkPush multiple source documents to Elasticsearch indices,
using proper error handling and retry logic.Args:'es': An instance of theElasticsearchclass.actions: An iterable of Elasticsearch action records (passed directly to Elasticsearch).errorsfp: A file pointer where HTTP 400 errors are logged.logger: ALoggerobject where messages can be logged.Returns: A tuple (start_time, end_time, successfully_indexed, duplicate, failed, retry_count).parallel_bulkPush multiple source documents to Elasticsearch indices
in parallel across multiple threads, using proper error
handling and retry logic.Args:es: An instance of theElasticsearchclass.actions: An iterable of Elasticsearch action records
(passed directly to Elasticsearch)errorsfp: A file pointer where HTTP 400 errors are logged.logger: ALoggerobject where messages can be logged.chunk_size=10000000: Number of docs sent in one chunk to Elasticsearch.max_chunk_bytes=104857600: The maximum size of a request.thread_count=8: The size of the thread pool to use.queue_size=4: The size of the task queue between the controller and processing threads.Returns: A tuple (start_time, end_time, successfully_indexed, duplicate, failed, retry_count)TemplateExceptionThis exception is raised by put_template when a
template document does not contain the required version
metadata ({"_meta": {"version": <integer>}}); or, when
multiple template documents are included in a single call
to put_template, if the versions of those documents are
not all identical.Unit testing supportThepyesbulkpackage attempts to dynamically determine the
Python module used to produce theElasticsearchobject that's passed in topyesbulkmethods. This is necessary in order to properly resolve exception classes for the error handling and retry logic.However, unit tests often work with mocked objects which
won't have "real" Python package structure, and the dynamic
module recognition algorithm may fail. When this happens,pyesbulkwill attempt to importelasticsearch. If that's not correct (e.g., if you're usingelasticsearch1orelasticsearch5), you can override the automatic search
by including aforce_elastic_search_moduleproperty on
your mockedElasticsearchobject.For example,class MockElasticsearch:
    def __init__(self):
        self.force_elastic_search_module = "elasticsearch5"ores = MockElasticSearch()
    es.force_elastic_search_module = "elasticsearch1"See alsohttps://pypi.org/project/pyesbulk/.