Creating and Managing Elasticsearch Indices with Python | by Niels D. Goet | Towards Data Science

Elasticsearch (ES) is a distributed search engine that is designed for scalability and redundancy. It is fast, and it is suited for storing and handling large volumes of data for analytics, machine learning, and other applications. ES has gained traction in recent years, and is an important technology for any data scientist’s (and data engineer’s) toolbox. In this blog post, we’ll provision our own (free) ES cluster with Bonsai, and use the Python Elasticsearch Client to set up an index, and write and query data. All code for this blog post is available on GitHub.
In a first step, we’ll set up an Elasticsearch cluster. You can do this through a managed Elasticsearch service provider such as AWS’ Amazon Elasticsearch Service, Bonsai, or Elastic.co. How you decide to provision, deploy, and manage your cluster (and where) depends on what kind of solution you’re building and what features are important to you.
Here, we’ll go with Bonsai, which gives us a free “Sandbox” tier with 125mb of storage and a 35,000 documents storage limit. To get started, sign up for an account on the Bonsai and follow the steps on the website to set up and deploy your cluster. If you already have an account on Heroku, you can also provision an ES cluster for your app directly by selecting Bonsai as an add on in the Resources tab of your app, without the need for signing up for an account on Bonsai (see below).
After you’ve set up your Bonsai account and deployed your cluster (or provisioned your Bonsai cluster for your app via Heroku), nagivate to the “Credentials” page in in the left sidebar of your Bonsai Cluster overview dashboard and copy your host URL, access key, and access secret. We’ll use this information later to set up a connection using the Python Elasticsearch Client.
Bonsai’s Sandbox tier is not a production-level deployment and comes with several limitations. For example, the Sandbox tier does not allow you to set up users with limited permissions (e.g. read access only), so keep that in mind if you want to use this setup for an application that is exposed to the Internet. Elastic.co offers a 14-day free trial that you can use instead if you’d like to explore the full set of features that ES has to offer (including, e.g., cluster and index privileges).
After you’ve completed the above steps, you can inspect your cluster health and indices from the console using Elasticsearch’s Compact and Aligned Text (CAT) APIs. The CAT APIs are designed to give you human-readable output. They are therefore not intended to be used by applications; rather they should be used from terminal or in the Kibana console.
On the console (both on Kibana and on Bonsai directly), the following GET request will return information on cluster health (a “green” status means that all your shards are assigned):
Now that we have a cluster up and running, let’s look at the fundamentals of data and indices in ES.
Data in Elasticsearch is in the form of JSON objects called “documents”. A document contains fields, which are key-value pairs that can be a value (e.g. a boolean or an integer) or a nested structure. Documents are organised in indices, which are both logical groupings of data that follow a schema as well as the physical organisation of the data through shards.
Each index consists of one or more physical shards that form a logical group. Each shard, in turn, is a “self-contained index”. The data in an index is partitioned across these shards, and shards are distributed across nodes. Each document has a “primary shard” and a “replica shard” (that is, if the index has been configured to have one or more replica shards).
Data that resides in different shards can be processed in parallel. To the extent that your CPU and memory allow (i.e., depending on the kind of ES cluster you have), more shards means faster search. Note however that shards come with overhead, and that you should carefully consider the appropriate number and size of the shards in your cluster.
Now that we’ve gone over the basics of ES, let’s take a stab at setting up an index of our own. Here, we’ll use data on Netflix shows available on Kaggle. This dataset is in CSV format and contains information on movies and series available on Netflix, including metadata such as their release date, their title, and cast.
We’ll first define a ES mapping for our index. In this case, we’ll go with relatively straightforward field types, defining all fields as text, with the exception of release_year (which we’ll make an integer).
While Elasticsearch is able to infer the mapping of your documents when you write them, using dynamic field mapping, it does not necessarily do so optimally. Typically, you’ll want to spend some time on defining your mapping because the field types (as well as various other options) impact the size of your index and the flexibility you’ll have in querying data. For example, the text field type is broken down into individual terms upon indexing, allowing for partial matching. The keyword type can be used for strings too, but is not analysed (or rather: “tokenised”) when indexing and only allows for exact matching (in this example, we could have used it for the type field, which takes one of two values — “Movie” or “TV Show”).
Another thing to keep in mind here is that ES does not have an array field type: if your document includes a field that consists of an array of, say, integers, the ES equivalent would be the integer data type (for a complete overview of ES field types, see this page). Take for example the following document:
{    "values": [1,2,3,4,5]}
The underlying mapping for this data would simply define the values field type to be an integer:
mapping = {    "mappings": {        "properties": {            "values": {                "type": "integer"            }        }    }}
Now that we’ve defined our mapping, we can set up our index on ES using the Python Elasticsearch Client. In the Python code below, I’ve set up a class called EsConnection, with an ES client connection attribute (es_client). Note that I’m storing my access credentials in environment variables here for convenience (and to avoid accidentally sharing them when, e.g., pushing code to GitHub). For real applications you’ll have to use a more secure approach, like AWS Secrets Manager.
The EsConnection class also includes two methods, one to create an index based on a mapping (create_index), and another (populate_index) to take a CSV file, convert its rows to JSONs, and write each entry (“document”) to an ES index. Here, we’ll only use the first function, to create our netflix_movies index:
We can now inspect the mapping for our index using the following code, which should return the mapping that we have just written:
Now that we’ve defined our mapping, we can write the Netflix movies data to our ES index, using the following code (note that this code expects the Netflix data from Kaggle discussed earlier in this post to be available in a subdirectory called “data"):
While this code is running, we can check whether the data is coming through by reviewing the logs in your Bonsai dashboard. The logs should show POST requests to the netflix_movies index (see below).
We can also check the number of entries in the netflix_movies index using the count method, e.g.:
Finally, we can use the CAT indices API in your Kibana console to check the number of documents in your indices, their size, the number of primary and replication shards, and their health status:
In the output shown above, the store.size field shows the size taken by your primary and replication shards combined, while pri.store.size only shows the size of your primary shard(s). The store.size in this case will be 2x the primary shard size, since our shard health is “green”, which means that the replica shards were properly assigned.
The Python Elasticsearch client can also be used directly with the CAT API, if you’d prefer to use Python throughout.
Now that we have set up our cluster and written our documents, let’s take a brief look at querying data from Elasticsearch. ES queries are written in Elasticsearch Domain Specific Language (DSL). As described in the ES documentation, DSL is based on JSON and should be thought of as an Abstract Syntax Tree (AST) of queries. DSL queries consist of two types of clauses:
leaf query clauses that look for a specific value in a specific field (e.g. a match or range); andcompound query clauses that are used to logically combine multiple queries (such as multiple leaf or compound queries) or to alter the behaviour of these queries.
leaf query clauses that look for a specific value in a specific field (e.g. a match or range); and
compound query clauses that are used to logically combine multiple queries (such as multiple leaf or compound queries) or to alter the behaviour of these queries.
When you run a query against your index, ES sorts the results by a relevance score (a float) that represents the quality of the match (this value is reported in the _score field associated with a “hit”). ES queries have a query and a filter context. The filter context — as the name implies — filters out documents that do not match the conditions in the syntax. However, it will not affect the relevance score. The match in the bool context does contribute to the relevance score.
Let’s look at an example of a query with a query and a filter context. The example below filters on release year, but runs a match against the movie type.
{    "query": {        "bool": {            "must": [                {"match": {"type": "TV Show"}},            ],            "filter": [                {"range": {"release_year": {"gte": 2021}}}            ]        }    }}
We can run this query on our index directly using our ES Console, or through Python. The example shown below uses the latter approach. This code first sets up the ES client. Subsequently it defines the query, and finally, it runs the query against the netflix_movies ES index. The last line of the code prints the titles of the movies that match our query.
Keep in mind that there is a maximum of 10,000 hits per query for a regular search. If you need to extract more than that in a single query (i.e. “deep pagination”), you can use the search_after option (you could also use scroll search, but this is no longer recommended by ES). Also note that since we know that “TV Show” is the value that the type field can take, we may as well have used the filter context. I’ll leave these and other intricacies of DSL for a future blog post.
And that’s it! We’ve provisioned an ES cluster, inspected our cluster health, defined an index, written documents, and run a simple query against our index. Stay tuned for future blog posts on Elasticsearch, where I’ll dive into the details of DSL and of index store size optimisation.
Thanks for reading!
Support my work: If you liked this article and you’d like to support my work, please consider becoming a paying Medium member via my referral page. The price of the subscription is the same if you sign up via my referral page, but I will receive part of your monthly membership fee.
If you liked this article, here are some other articles you may enjoy:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Disclaimer: “Elasticsearch” and “Kibana” are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. Description and/or use of any third-party services and/or trademarks in this blog post should not be seen as endorsement for or by their respective rights holders.
Please read this disclaimer carefully before relying on any of the content in my articles on Medium.com.