Creating and Managing Elasticsearch Indices with Python
By: Niels D. Goet
Tags: Data, Data Science, Database Development, Data Engineering, Elasticsearch

Defining your ES index mapping

Now that we have a cluster up and running, let’s look at the fundamentals of data and indices in ES.

Data in Elasticsearch is in the form of JSON objects called “documents”. A document contains fields, which are key-value pairs that can be a value (e.g. a boolean or an integer) or a nested structure. Documents are organised in indices, which are both logical groupings of data that follow a schema as well as the physical organisation of the data through shards.

Each index consists of one or more physical shards that form a logical group. Each shard, in turn, is a “self-contained index”. The data in an index is partitioned across these shards, and shards are distributed across nodes. Each document has a “primary shard” and a “replica shard” (that is, if the index has been configured to have one or more replica shards).

Data that resides in different shards can be processed in parallel. To the extent that your CPU and memory allow (i.e., depending on the kind of ES cluster you have), more shards means faster search. Note however that shards come with overhead, and that you should carefully consider the appropriate number and size of the shards in your cluster.

Now that we’ve gone over the basics of ES, let’s take a stab at setting up an index of our own. Here, we’ll use data on Netflix shows available on Kaggle. This dataset is in CSV format and contains information on movies and series available on Netflix, including metadata such as their release date, their title, and cast.

We’ll first define a ES mapping for our index. In this case, we’ll go with relatively straightforward field types, defining all fields as text , with the exception of release_year (which we’ll make an integer ).

While Elasticsearch is able to infer the mapping of your documents when you write them, using dynamic field mapping, it does not necessarily do so optimally. Typically, you’ll want to spend some time on defining your mapping because the field types (as well as various other options) impact the size of your index and the flexibility you’ll have in querying data. For example, the text field type is broken down into individual terms upon indexing, allowing for partial matching. The keyword type can be used for strings too, but is not analysed (or rather: “tokenised”) when indexing and only allows for exact matching (in this example, we could have used it for the type field, which takes one of two values — “Movie” or “TV Show”).

Another thing to keep in mind here is that ES does not have an array field type: if your document includes a field that consists of an array of, say, integers, the ES equivalent would be the integer data type (for a complete overview of ES field types, see this page). Take for example the following document:

{

"values": [1,2,3,4,5]

}

The underlying mapping for this data would simply define the values field type to be an integer:

mapping = {

"mappings": {

"properties": {

"values": {

"type": "integer"

}

}

}

}

Now that we’ve defined our mapping, we can set up our index on ES using the Python Elasticsearch Client. In the Python code below, I’ve set up a class called EsConnection , with an ES client connection attribute ( es_client ). Note that I’m storing my access credentials in environment variables here for convenience (and to avoid accidentally sharing them when, e.g., pushing code to GitHub). For real applications you’ll have to use a more secure approach, like AWS Secrets Manager.

The EsConnection class also includes two methods, one to create an index based on a mapping ( create_index ), and another ( populate_index ) to take a CSV file, convert its rows to JSONs, and write each entry (“document”) to an ES index. Here, we’ll only use the first function, to create our netflix_movies index:

We can now inspect the mapping for our index using the following code, which should return the mapping that we have just written:

Writing data to ES

Now that we’ve defined our mapping, we can write the Netflix movies data to our ES index, using the following code (note that this code expects the Netflix data from Kaggle discussed earlier in this post to be available in a subdirectory called “data"):