SQL-lise Elasticsearch Query For Data Analysis Using Docker, SQL Client, And Python | by Aakash Rathor | Towards Data Science

About Elasticsearch:
Elasticsearch is a No SQL database provided as an opensource as well as a commercial product by a company called Elastic. It is used for storing and performing big data analytics on data like web server logs, Event data from any app or sensor data in IoT.
What makes elasticsearch popular is the simplicity to install and scales from few to hundreds of nodes without any extra software packages. It uses built-in REST API to perform queries on data.
So you heard all the cool things Elasticsearch can do and you decided to used it in your project to store and analyze your logs or events or any other data. But you find it cumbersome to write a complex, nested query in JSON just to perform a simple analysis of your data stored in Elasticsearch.
For example, a simple filter query in Elasticsearch looks like this:
GET /_search{ "query":{    "bool":{      "filter":[                {"term":{"status":"running"}},                {"range":{"last_updated_date":{"gte":"2019–12–31"}}}              ]      }    }}
The same query can be written in SQL like this:
SELECT * FROM   sample_table WHERE  status = "running"        AND last_updated_date > "2019–12–31"
Now, since starting with Elasticsearch 6.3 release, one of the plugin X-Pack is open and integrated by default as a feature in the Elasticsearch stack(ELK).
We can use the features provided by the X-Pack plugin to analyze our data in Elasticsearch using SQL like queries instead of writing complex and nested JSON ones.
We will look at 2 different approaches:
Using a universal database client(Dbeaver): To connect to our Elasticsearch server and run our query in SQL dialect.Using Python module(elasticsearch-dbapi and Sqlalchemy): To connect to our Elasticsearch server and query data into pandas dataframe for performing further analytics and visualization.
Using a universal database client(Dbeaver): To connect to our Elasticsearch server and run our query in SQL dialect.
Using Python module(elasticsearch-dbapi and Sqlalchemy): To connect to our Elasticsearch server and query data into pandas dataframe for performing further analytics and visualization.
Setup and Requirements:
Ubuntu OS: I am using Ubuntu for demonstration, but all the tools and modules required are available for other OS too.Dbeaver DB client: This is a universal DB tool that supports No SQL too. You can download it from here.Docker: For quickly spinning ELK cluster in my local for demo. You can download and install it from here. If you already have an Elasticsearch instance running then you can skip this.Jupyter notebook: For the second approach where we analyze data in python after fetching.
Ubuntu OS: I am using Ubuntu for demonstration, but all the tools and modules required are available for other OS too.
Dbeaver DB client: This is a universal DB tool that supports No SQL too. You can download it from here.
Docker: For quickly spinning ELK cluster in my local for demo. You can download and install it from here. If you already have an Elasticsearch instance running then you can skip this.
Jupyter notebook: For the second approach where we analyze data in python after fetching.
First Approach- Analysing Elasticsearch data from DB client :
First I will clone one of the free git repository which has all the required docker files for spinning an ELK cluster in local. You can also download/clone this repository from here.
Now get in the directory where the project is cloned and run below docker command:
$ sudo docker-compose up -d
You will get the below output if everything works fine:
Let's check if all the services are up or not, using below command:
$ sudo docker-compose ps
We saw that out Elasticsearch service is running at port 9200 and Kibana at port 5601.
I have added some sample data for demo purposes. If you are not working on your own data then you can also do the same using home page of Kibana as shown below:
First, let's fetch the data using Elasticsearch REST API format. I am using curl to run below query on Flight sample data:
curl -X GET "elastic:changeme@localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'{"query": {  "bool": {   "filter": [    { "term":  { "DestCountry": "IT" }},   { "range": { "timestamp": { "gte": "2020-03-06T21:08:31.000Z" }}}      ]    }  }}'
Below is the output:
Let's run the same query in the SQL language. For that open the Dbeaver client and click on add connection:
Now add credentials and host details:
Dbeaver will automatically download and install the required driver for making a connection to the Elasticsearch server.
If you face any issue with the driver version, please download the required version from here. And add it in Edit Driver settings and click OK.
Now click on Test Connection If everything works fine, you will get a success message as shown below:
Now click Finish. You will see the connection to Elasticsearch server on left side:
Now click on New SQL Editor:
Now we will write the same query using SQL language:
SELECT *FROM kibana_sample_data_flightsWHERE "DestCountry" = 'IT' AND "timestamp" > '2020-03-06T21:08:31.000Z'
And Voila!! You will get output in a beautiful tabular format:
Second Approach- Query data into pandas dataframe:
The old way of fetching data from Elastic search into python is by using Python Elasticsearch client(Read doc here), which uses the same REST API query format that we saw in Curl.
We will simplify this by using simple SQLAlchemy and elasticsearch-dbapi modules.
Below is the sample code:
##Importing required libraryfrom sqlalchemy import create_engineimport pandas as pd## Creating connection to our Elasticsearch server## format of connection string is: ## "dialect://username:password@host:port/"connection_es=create_engine("""elasticsearch+http://elastic:changeme@localhost:9200/""")##Fetching Data using SQL language and Pandas read_sql functionresult_data=pd.read_sql("""SELECT *FROM kibana_sample_data_flightsWHERE "DestCountry" = 'IT' AND "timestamp" > '2020-03-06T21:08:31.000Z'""", connection_es)
In above code we used the same SQL query that we ran previously on our SQL client.
Finally we will get the output as a pandas dataframe, you can use this dataframe for further analysis and visualization
Conclusion:
In this article, we saw how we can analyze our data present in Elasticsearch using simple SQL language instead of complex and nested REST API Query. We saw 2 different ways to query data from Elasticsearch one with DB client and another with python.
I hope you like my article, If you want more information on this topic, you can follow and message me on Instagram or LinkedIn.