Package: pupu-elasticsearch-dbapi
ElasticSearch DBAPIelasticsearch-dbapiImplements a DBAPI (PEP-249) and SQLAlchemy dialect,
that enables SQL access on elasticsearch clusters for query only access.On Elastic Elasticsearch:
Uses Elastic X-PackSQL APIOn AWS ES, opendistro Elasticsearch:Open Distro SQLThis library supports Elasticsearch 7.X versions.Installation$pipinstallelasticsearch-dbapiTo install support for AWS Elasticsearch Service /Open Distro:$pipinstallelasticsearch-dbapi[opendistro]Usage:Using DBAPI:fromes.elastic.apiimportconnectconn=connect(host='localhost')curs=conn.cursor()curs.execute("select * from flights LIMIT 10")print([rowforrowincurs])Using SQLAlchemy execute:fromsqlalchemy.engineimportcreate_engineengine=create_engine("elasticsearch+http://localhost:9200/")rows=engine.connect().execute("select * from flights LIMIT 10")print([rowforrowinrows])Using SQLAlchemy:fromsqlalchemyimportfunc,selectfromsqlalchemy.engineimportcreate_enginefromsqlalchemy.schemaimportMetaData,Tableengine=create_engine("elasticsearch+http://localhost:9200/")logs=Table("flights",MetaData(bind=engine),autoload=True)count=select([func.count("*")],from_obj=logs).scalar()print(f"COUNT:{count}")Using SQLAlchemy reflection:fromsqlalchemy.engineimportcreate_enginefromsqlalchemy.schemaimportTable,MetaDataengine=create_engine("elasticsearch+http://localhost:9200/")logs=Table("flights",MetaData(bind=engine),autoload=True)print(engine.table_names())metadata=MetaData()metadata.reflect(bind=engine)print([tablefortableinmetadata.sorted_tables])print(logs.columns)Connection Parameters:elasticsearch-pyis used to establish connections and transport, this is the official
elastic python library.Elasticsearchconstructor accepts multiple optional parameters
that can be used to properly configure your connection on aspects like security, performance
and high availability. These optional parameters can be set at the connection string, for
example:elasticsearch+http://localhost:9200/?http_compress=True&timeout=100will set transport to use gzip (http_compress) and timeout to 10 seconds.For more information on configuration options, look atelasticsearch-py’s documentation:Transport OptionsHTTP tranportThe connection string follows RFC-1738, to support multiple nodes you should usesniff_*parametersFetch sizeBy default the maximum number of rows which get fetched by a single query
is limited to 10000. This can be adapted through thefetch_sizeparameter:fromes.elastic.apiimportconnectconn=connect(host='localhost')curs=conn.cursor(fetch_size=1000)If more than 10000 rows should get fetched thenmax_result_windowhas to be adapted as well.TestsTo run unittest launch elasticsearch and kibana (kibana is really not required but is a nice to have)$docker-composeup-d
$nosetests-vSpecial case for sql opendistro endpoint (AWS ES)AWS ES exposes the opendistro SQL plugin, and it follows a different SQL dialect.
Using theodelasticsearchdriver:fromsqlalchemy.engineimportcreate_engineengine=create_engine("odelasticsearch+https://search-SOME-CLUSTER.us-west-2.es.amazonaws.com:443/")rows=engine.connect().execute("select count(*), Carrier from flights GROUP BY Carrier")print([rowforrowinrows])Or using DBAPI:fromes.opendistro.apiimportconnectconn=connect(host='localhost',port=9200,path="",scheme="http")curs=conn.cursor().execute("select * from flights LIMIT 10")print([rowforrowincurs])Opendistro (AWS ES) Basic authenticationBasic authentication is configured as expected on the , fields of the URIfromsqlalchemy.engineimportcreate_engineengine=create_engine("odelasticsearch+https://my_user:my_password@search-SOME-CLUSTER.us-west-2.es.amazonaws.com:443/")IAM AWS Authentication keys are passed on the URI basic auth location, and by settingaws_keysQuery string keys are:aws_keysaws_regionfromsqlalchemy.engineimportcreate_engineengine=create_engine("odelasticsearch+https://<AWS_ACCESS_KEY>:<AWS_SECRET_KEY>@search-SOME-CLUSTER.us-west-2.es.amazonaws.com:443/?aws_keys=1&&aws_region=<AWS_REGION>")IAM AWS profile is configured has a query parameter nameaws_profileon the URI. The value for the key provides the AWS regionfromsqlalchemy.engineimportcreate_engineengine=create_engine("odelasticsearch+https://search-SOME-CLUSTER.us-west-2.es.amazonaws.com:443/?aws_profile=us-west-2")To connect to the provided Opendistro ES ondocker-composeuse the following URI:odelasticsearch+https://admin:admin@localhost:9400/?verify_certs=FalseKnown limitationsThis library does not yet support the following features:Array type columns are not supported. Elaticsearch SQL does not support them either.
SQLAlchemyget_columnswill exclude them.objectandnestedcolumn types are not well supported and are converted to stringsIndexes that whose name start with.GEO points are not currently well-supported and are converted to stringsAWS ES (opendistro elascticsearch) is supported (still beta), known limitations are:You are only able toGROUP BYkeyword fields (newexperimentalopendistro SQL already supports it)Indices with dots are not supported (indices like 'audit_log.2021.01.20'),
on these cases we recommend the use of aliases