Package: bartbroere-eland
AboutEland is a Python Elasticsearch client for exploring and  analyzing data in Elasticsearch with a familiar
Pandas-compatible API.Where possible the package uses existing Python APIs and data structures to make it easy to switch between numpy,
pandas, or scikit-learn to their Elasticsearch powered equivalents. In general, the data resides in Elasticsearch and
not in memory, which allows Eland to access large datasets stored in Elasticsearch.Eland also provides tools to upload trained machine learning models from common libraries likescikit-learn,XGBoost,  andLightGBMinto Elasticsearch.Getting StartedEland can be installed fromPyPIwith Pip:$python-mpipinstallelandIf using Eland to upload NLP models to Elasticsearch install the PyTorch extras:$python-mpipinstalleland[pytorch]Eland can also be installed fromConda Forgewith Conda:$condainstall-cconda-forgeelandCompatibilitySupports Python 3.8, 3.9, 3.10 and Pandas 1.5Supports Elasticsearch clusters that are 7.11+, recommended 8.3 or later for all features to work.
If you are using the NLP with PyTorch feature make sure your Eland minor version matches the minor
version of your Elasticsearch cluster. For all other features it is sufficient for the major versions
to match.You need to use PyTorch1.13.1or earlier to import an NLP model.
Runpip install torch==1.13.1to install the aproppriate version of PyTorch.PrerequisitesUsers installing Eland on Debian-based distributions may need to install prerequisite packages for the transitive
dependencies of Eland:$sudoapt-getinstall-y\build-essentialpkg-configcmake\python3-devlibzip-devlibjpeg-devNote that other distributions such as CentOS, RedHat, Arch, etc. may require using a different package manager and
specifying different package names.DockerIf you want to use Eland without installing it just to run the available scripts, use the Docker
image.
It can be used interactively:$dockerrun-it--rm--networkhostdocker.elastic.co/eland/elandRunning installed scripts is also possible without an interactive shell, e.g.:$dockerrun-it--rm--networkhost\docker.elastic.co/eland/eland\eland_import_hub_model\--urlhttp://host.docker.internal:9200/\--hub-model-idelastic/distilbert-base-cased-finetuned-conll03-english\--task-typenerConnecting to ElasticsearchEland uses theElasticsearch low level clientto connect to Elasticsearch.
This client supports a range ofconnection options and authentication options.You can pass either an instance ofelasticsearch.Elasticsearchto Eland APIs
or a string containing the host to connect to:importelandased# Connecting to an Elasticsearch instance running on 'http://localhost:9200'df=ed.DataFrame("http://localhost:9200",es_index_pattern="flights")# Connecting to an Elastic Cloud instancefromelasticsearchimportElasticsearches=Elasticsearch(cloud_id="cluster-name:...",basic_auth=("elastic","<password>"))df=ed.DataFrame(es,es_index_pattern="flights")DataFrames in Elandeland.DataFramewraps an Elasticsearch index in a Pandas-like API
and defers all processing and filtering of data to Elasticsearch
instead of your local machine. This means you can process large
amounts of data within Elasticsearch from a Jupyter Notebook
without overloading your machine.➤Eland DataFrame API documentation➤Advanced examples in a Jupyter Notebook>>>importelandased>>># Connect to 'flights' index via localhost Elasticsearch node>>>df=ed.DataFrame('http://localhost:9200','flights')# eland.DataFrame instance has the same API as pandas.DataFrame# except all data is in Elasticsearch. See .info() memory usage.>>>df.head()AvgTicketPriceCancelled...dayOfWeektimestamp0841.265642False...02018-01-0100:00:001882.982662False...02018-01-0118:27:002190.636904False...02018-01-0117:11:143181.694216True...02018-01-0110:33:284730.041778False...02018-01-0105:13:00[5rowsx27columns]>>>df.info()<class'eland.dataframe.DataFrame'>Index:13059entries,0to13058Datacolumns(total27columns):#   Column              Non-Null Count  Dtype----------------------------0AvgTicketPrice13059non-nullfloat641Cancelled13059non-nullbool2Carrier13059non-nullobject...24OriginWeather13059non-nullobject25dayOfWeek13059non-nullint6426timestamp13059non-nulldatetime64[ns]dtypes:bool(2),datetime64[ns](1),float64(5),int64(2),object(17)memoryusage:80.0bytesElasticsearchstorageusage:5.043MB# Filtering of rows using comparisons>>>df[(df.Carrier=="Kibana Airlines")&(df.AvgTicketPrice>900.0)&(df.Cancelled==True)].head()AvgTicketPriceCancelled...dayOfWeektimestamp8960.869736True...02018-01-0112:09:3526975.812632True...02018-01-0115:38:32311946.358410True...02018-01-0111:51:12651975.383864True...22018-01-0321:13:17950907.836523True...22018-01-0305:14:51[5rowsx27columns]# Running aggregations across an index>>>df[['DistanceKilometers','AvgTicketPrice']].aggregate(['sum','min','std'])DistanceKilometersAvgTicketPricesum9.261629e+078.204365e+06min0.000000e+001.000205e+02std4.578263e+032.663867e+02Machine Learning in ElandRegression and classificationEland allows transforming trained regression and classification models from scikit-learn, XGBoost, and LightGBM
libraries to be serialized and used as an inference model in Elasticsearch.➤Eland Machine Learning API documentation➤Read more about Machine Learning in Elasticsearch>>>fromsklearnimportdatasets>>>fromxgboostimportXGBClassifier>>>fromeland.mlimportMLModel# Train and exercise an XGBoost ML model locally>>>training_data=datasets.make_classification(n_features=5)>>>xgb_model=XGBClassifier(booster="gbtree")>>>xgb_model.fit(training_data[0],training_data[1])>>>xgb_model.predict(training_data[0])[0110100010]# Import the model into Elasticsearch>>>es_model=MLModel.import_model(es_client="http://localhost:9200",model_id="xgb-classifier",model=xgb_model,feature_names=["f0","f1","f2","f3","f4"],)# Exercise the ML model in Elasticsearch with the training data>>>es_model.predict(training_data[0])[0110100010]NLP with PyTorchFor NLP tasks, Eland allows importing PyTorch trained BERT models into Elasticsearch. Models can be either plain PyTorch
models, or supportedtransformersmodels from theHugging Face model hub.$eland_import_hub_model\--urlhttp://localhost:9200/\--hub-model-idelastic/distilbert-base-cased-finetuned-conll03-english\--task-typener\--startThe example above will automatically start a model deployment. This is a
good shortcut for initial experimentation, but for anything that needs
good throughput you should omit the--startargument from the Eland
command line and instead start the model using the ML UI in Kibana.
The--startargument will deploy the model with one allocation and one
thread per allocation, which will not offer good performance. When starting
the model deployment using the ML UI in Kibana or the ElasticsearchAPIyou will be able to set the threading options to make the best use of your
hardware.>>>importelasticsearch>>>frompathlibimportPath>>>fromeland.commonimportes_version>>>fromeland.ml.pytorchimportPyTorchModel>>>fromeland.ml.pytorch.transformersimportTransformerModel>>>es=elasticsearch.Elasticsearch("http://elastic:mlqa_admin@localhost:9200")>>>es_cluster_version=es_version(es)# Load a Hugging Face transformers model directly from the model hub>>>tm=TransformerModel(model_id="elastic/distilbert-base-cased-finetuned-conll03-english",task_type="ner",es_version=es_cluster_version)Downloading:100%|██████████|257/257[00:00<00:00,108kB/s]Downloading:100%|██████████|954/954[00:00<00:00,372kB/s]Downloading:100%|██████████|208k/208k[00:00<00:00,668kB/s]Downloading:100%|██████████|112/112[00:00<00:00,43.9kB/s]Downloading:100%|██████████|249M/249M[00:23<00:00,11.2MB/s]# Export the model in a TorchScrpt representation which Elasticsearch uses>>>tmp_path="models">>>Path(tmp_path).mkdir(parents=True,exist_ok=True)>>>model_path,config,vocab_path=tm.save(tmp_path)# Import model into Elasticsearch>>>ptm=PyTorchModel(es,tm.elasticsearch_model_id())>>>ptm.import_model(model_path=model_path,config_path=None,vocab_path=vocab_path,config=config)100%|██████████|63/63[00:12<00:00,5.02it/s]