Package: ist-pulse-data-extractor
pulse-data-extractorAneasy-button to take data out of Pulse. Downloads Pulse documents from
Elasticsearch, and saves them to.jsonl,.json,.pickle, or.csvformat.InstallationTo install in an existing environment, run this commandpip install ist-pulse-data-extractorTo define as a project requirement, add the following line torequirements.txt:ist-pulse-data-extractorUsagedownloadPerforms a multi-process sliced query for documents in a Pulse Elasticsearch index.
Saves the result to format specified by filename extension. Optional flattening of
documents is available (use with caution).from pulse.downloader import downloadRequired Parametersindex: Elasticsearch indexquery: Elasticsearch queryfilepath: Output filepath. File extension should match the desired output
format. Supported formats include:.jsonl: Fastest to download, suited for large datasets. Lowest
memory overhead in downstream processes..json: Standard, faster to load than .jsonl, but not suitable
for datasets that must be loaded into memory at once.pkl: Fastest to load if using result in Python script.csv: When consuming data with Excel or Pandas. Fields are
automatically flattened. Recommended to create a
separate post-processing script if some fields contain
data that can't be flattened automatically.es_hosts: Not required if usingES_URLenvironment variable.
A list of Elasticsearch hosts. Each item should be a fully-qualified URL
with authentication if applicable. This overrides values that may exist in
configuration. Example:https://elastic:password@node1.host.com:9200,https://elastic:password@node2.host.com:9200Optionssample_size: Maximum number of results to return (default=20000).fields: A list of fields to return from Elasticsearch. Limiting the
amount of fields reduces download time.flatten_doc: Flatten documents. Useful when working with data frames,
but has nuances. Use with caution.delimiter: Delimiter to use when flattening fieldsinclude_meta_attribs: Only applicable when flattening. When false,
all meta.*.attribs fields are discarded.no_flatten: A list of fields that should not be flattenedquery_slice_size: Maximum number of documents per slice (worker)query_concurrency: Maximum number of queries to run concurrentlyauto_mkdir: Automatically create output directory if it doesn't existExamplefrompulse.downloaderimportdownloaddownload(filepath="data/rohingya.jsonl",sample_size=10000,query={"query":{"bool":{"filter":[{"match_phrase":{"norm.body":"Rohingya"}}]}}},index='pulse-*',es_hosts=["https://user:password@dag1.istresearch.com:9200","https://user:password@dag2.istresearch.com:9200",])build_queryBuilds an Elasticsearch queryfrom pulse.downloader import build_queryOptions:start_date: Date range start (eg.2020-06-14or2020-06-14T12:00**:00.000Z)end_date: Date range endproject_id: Project IDcampaign_id: Campaign IDwhere_exists: A list or tuple containing fields that should exist in
each documentwhere_not_exists: A list or tuple containing fields that should not
exist in each documentinclude_match: A mapping of fields to match queries. Returns documents
that match a provided text, number, date or boolean value. The provided text
is analyzed before matching. The match query is the standard query for
performing a full-text search, including options for fuzzy matching.exclude_match: A mapping of fields to match queries. Filters documents
that match a provided text, number, date or boolean value.include_terms: A mapping of fields to term queries. Returns documents
that contain an exact term in a provided field.exclude_terms: A mapping of fields to term queries. Filters documents
that contain an exact term in a provided field.include_phrase: A mapping of fields to match_phrase queries.The
match_phrase query analyzes the text and creates a phrase query out of
the analyzed text.exclude_phrase: A mapping of fields to match_phrase queries. Excludes
matching documents.doc_type: Pulse document typetimestamp_field: Timestamp field to use for start_date and end_datequery_string: A prepared query stringExamplefrompulse.downloaderimportbuild_query,downloadquery=build_query(include_phrase={"norm.body":"Rohingya"},)download(filepath="data/rohingya.jsonl",sample_size=10000,query=query,index='pulse-*',es_hosts=["https://user:password@dag1.istresearch.com:9200","https://user:password@dag2.istresearch.com:9200",])DevelopmentTo deploy a new version, follow the instructions indeploy.sh. Requires access
to deployment credentials in Lastpass.