Package: elasticsearch-loader
Main featuresBatch upload CSV (actually any *SV) files to ElasticsearchBatch upload JSON files / JSON lines to ElasticsearchBatch upload parquet files to ElasticsearchPre defining custom mappingsDelete index before uploadIndex documents with _id from the document itselfLoad data directly from urlSSL and basic authUnicode Support ✌️PluginsIn order to install plugin, simply runpip installplugin-name-esl-redis- Read continuously
from a redis list(s) and index to elasticsearch -esl-s3- Plugin for listing and
indexing files from S3Test matrixpython / es5.6.166.8.07.1.12.7VVV3.7VVVInstallationpip installelasticsearch-loaderIn order to add parquet support run
``pip install elasticsearch-loader[parquet]``Usage(venv)/tmp $ elasticsearch_loader --help
Usage: elasticsearch_loader [OPTIONS] COMMAND [ARGS]...

Options:
  -c, --config-file TEXT          Load default configuration file from esl.yml
  --bulk-size INTEGER             How many docs to collect before writing to
                                  Elasticsearch (default 500)
  --es-host TEXT                  Elasticsearch cluster entry point. (default
                                  http://localhost:9200)
  --verify-certs                  Make sure we verify SSL certificates
                                  (default false)
  --use-ssl                       Turn on SSL (default false)
  --ca-certs TEXT                 Provide a path to CA certs on disk
  --http-auth TEXT                Provide username and password for basic auth
                                  in the format of username:password
  --index TEXT                    Destination index name  [required]
  --delete                        Delete index before import? (default false)
  --update                        Merge and update existing doc instead of
                                  overwrite
  --progress                      Enable progress bar - NOTICE: in order to
                                  show progress the entire input should be
                                  collected and can consume more memory than
                                  without progress bar
  --type TEXT                     Docs type. TYPES WILL BE DEPRECATED IN APIS
                                  IN ELASTICSEARCH 7, AND COMPLETELY REMOVED
                                  IN 8.  [required]
  --id-field TEXT                 Specify field name that be used as document
                                  id
  --as-child                      Insert _parent, _routing field, the value is
                                  same as _id. Note: must specify --id-field
                                  explicitly
  --with-retry                    Retry if ES bulk insertion failed
  --index-settings-file FILENAME  Specify path to json file containing index
                                  mapping and settings, creates index if
                                  missing
  --timeout FLOAT                 Specify request timeout in seconds for
                                  Elasticsearch client
  --encoding TEXT                 Specify content encoding for input files
  --keys TEXT                     Comma separated keys to pick from each
                                  document
  -h, --help                      Show this message and exit.

Commands:
  csv
  json     FILES with the format of [{"a": "1"}, {"b": "2"}]
  parquet
  redis
  s3ExamplesLoad 2 CSV to elasticsearchelasticsearch_loader--indexincidents--typeincident csv file1.csv file2.csvLoad JSONs to elasticsearchelasticsearch_loader--indexincidents--typeincident json *.jsonLoad all git commits into elasticsearchgit log--pretty=format:'{"sha":"%H","author_name":"%aN","author_email":"%aE","date":"%ad","message":"%f"}'| elasticsearch_loader--typegit--indexgit json--json-lines-Load parquet to elasticsearchelasticsearch_loader--indexincidents--typeincident parquet file1.parquetLoad CSV from github repo (actually any http/https is ok)elasticsearch_loader--indexdata--typeavg_height--id-fieldcountry jsonhttps://raw.githubusercontent.com/samayo/country-data/master/src/country-avg-male-height.jsonLoad data from stdingenerate_data | elasticsearch_loader--indexdata--typeincident csv -Read id from incident_id fieldelasticsearch_loader--id-fieldincident_id--indexincidents--typeincident csv file1.csv file2.csvLoad custom mappingselasticsearch_loader--index-settings-filesamples/mappings.json--indexincidents--typeincident csv file1.csv file2.csvTests and sample dataEnd to end and regression tests are located under test directory and can
run by running./test.pyInput formats can be found under samplesStargazers over time