Optimising Disk Usage in Elasticsearch | by Niels D. Goet | Towards Data Science

Elasticsearch (ES) has gained traction in recent years because it offers a robust and scalable engine for storing and analysing large volumes of data with low latency. If you’re a data engineer or data scientist working with large (and fast-growing) volumes of data, you’ll know that optimising for storage is a key component of building high-quality solutions. In this post, I discuss three strategies for optimising disk usage when using ES. Replication code for this blog post is available on GitHub.
Before we start exploring Elasticsearch (ES) storage optimisation, let’s review some ES fundamentals.
If you ask, say, four developers to describe what ES is, you’ll likely get as many different answers. And that’s not surprising: ES can be used as a storage engine, for applying machine learning to model data in real time, and for analysing events and logs, to name but a few applications. Further, ES can work with different data types (both structured and unstructured), while its distributed nature allows it to scale with large volumes of data. It is with reason, therefore, that ES has become the solution of choice for a growing number of companies across a variety of sectors, including the likes of Airbnb, Uber, Walmart, Cisco, and eBay.
To appreciate the power of ES, we need to understand its fundamental principles. Elasticsearch is a distributed search and analytics engine built on top of Lucene. When you set up and deploy an Elasticsearch cluster, you can add different nodes (servers). Both your data and the queries that you run against the data in your ES indices are distributed across those nodes — all done automatically to offer scalability and high availability.
The data in ES is organised in “indices”. Elasticsearch indices are logical groupings of data that follow a schema, but the term also covers the actual physical organisation of the data through shards. Each index consists of a “logical grouping of one or more physical shards”. Each shard, in turn, is a self-contained index.
The data that is stored in ES indices are JSON objects called “documents”. These documents consist of fields (key-value pairs), and are partitioned among “shards”, and the shards are distributed across nodes. Each document belongs to a “primary shard” and to one or multiple “replica shards” (provided you have configured replica shards for your index). As such, Elasticsearch is built for redundancy through a design that consists of nodes and shards, with primary shards and replicas.
In what follows, I’ll focus on three strategies to optimise data storage in ES: i) designing your indices well; ii) optimising your index mapping: and iii) using hot-warm-cold architecture. We’ll be using the netflix_movies index that I set up in my earlier blog post on Elasticsearch. This is a dataset of metadata on Netflix shows available on Kaggle. We’ll optimise this index step by step.
If you’d like to learn more about provisioning an ES cluster and setting up indices, please read my post on Creating and Managing Elasticsearch Indices with Python.
towardsdatascience.com
Before we start, please note that the data used in this blog post contains only 7787 “static” entries and cannot be classified as “Big Data” by any stretch of the imagination. However, the principles discussed in this post can be generalised to Big Data and streaming in general.
First, at risk of stating the obvious: you have to design your Elasticsearch indices well. This starts with deciding what goes into your indices, and more importantly, what does not. For example, do you need to store all your data in Elasticsearch? Or, do you only need to use Elasticsearch as a low-latency store for a subset of your data that needs to be accessible for an application, while the rest of the data can be stored elsewhere? If the answer is yes to the latter question, you can likely identify some “quick wins” easily and remove redundant data.
Once you’ve decided what data needs to be available in Elasticsearch, make sure you define your mappings well. Elasticsearch is able to infer the mapping of your data using dynamic field mapping. This means that ES adds the field type to your mapping dynamically whenever it detects a new field in a document. But that does not mean that it does so in a way that is optimal for data storage or for the purposes that you have in mind.
Dynamic field mapping (when the dynamic parameter is set to true) will for example index string fields as both text and keyword. You’ll typically only need one of these two. The text field type is broken down into individual terms upon indexing and allows for partial matching. Conversely, the keyword type is not analysed (or rather: “tokenized”) when indexing and only allows for exact matching (in our netflix_movies example, we can use the keyword field type for the type field, which takes one of two values — “Movie” or “TV Show”).
To establish a baseline to compare our optimisation strategies against, we’ll first write the Netflix movies data to a new ES index without defining our own mapping, using dynamic field mapping. ES infers the following mapping for our netflix_movies data:
{ "netflix_movies_dynamic_field_mapping": {  "mappings": {   "properties": {    "cast": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "country": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "date_added": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "description": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "director": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "duration": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "listed_in": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "rating": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "release_year": {     "type": "long"    },    "show_id": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "title": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    },    "type": {     "type": "text",     "fields": {      "keyword": {       "type": "keyword",       "ignore_above": 256      }     }    }   }  } }}
Note that our aim is to reduce the disk usage of our indices, that is, the store.size parameter. This parameter represents the storage size of your primary and replication shards for the index on your cluster. You can inspect the store size of your indices using the CAT indices API in your Kibana console. Using dynamic field mapping, we get a baseline store size of 17.1 MB (see screenshot below).
Can we do any better than our baseline of 17.1 MB by explicitly defining a mapping? Let’s start off with the simple (and unoptimised) mapping from my earlier blog post. We make all fields of the text type with the exception of release_year (which we’ll define as an integer):
The index store size for netflix_movies is 9.3 MB, a reduction of 7.8 MB compared to our baseline mapping (a reduction of nearly 46 perc.).
The second strategy to reduce your indices’ store size involves optimising your mapping. There are several choices you can make to reduce store size when you define a mapping. The tradeoff here is between search capabilities and limiting the size of your indices: typically, the more you optimise for storage, the less flexibility you will have in querying your data.
First, choose the right data types for your fields. In our netflix_movies example, we can make type, country, and rating fields into keyword field types, since these fields can only take on a number of predefined values and we can expect not to have to use partial matches.
Also make sure to check whether your variables can be fit into “smaller” data types. For example, we can reduce the size of release_year further by using the short instead of the integer field type. The short field type is a 16-bit integer. Our improved index looks as follows:
This optimised index gets us down to 8.7mb compared to our baseline of 17.1 MB (a 49.1 percent reduction). This represents a 6.5 percent reduction in disk usage compared to our unoptimised mapping (9.3 MB). However, if you’re working with truly large volumes of data even these small percentages can represent significant savings.
Second, you can disable indexing for fields that you don’t need to filter on. Indexing is the organisation of data for fast retrieval. In ES, text fields are for example stored in an inverted index (i.e.: documents — terms → terms — documents), while numeric field types, for example, are stored in BKD trees.
In our netflix_movies example, potential candidates to exclude from indexing are duration and description. The first is not defined consistently between series and movies (the former is measured in seasons, the latter in minutes). I’d likely further process this data and create a new field instead if I wanted to filter on duration. description, in turn, may be useful for partial matches, but let’s assume (for illustrative purposes) that we’re only interested in identifying movies by their metadata, and read their description afterwards.
To switch off indexing for specific fields, we set the index option to False:
Writing our netflix_movies data with this new mapping produces an index that has a total size of 7.5 MB (a 3.0 percent reduction compared to our previous iteration).
Third, you can drop normalisation factors for text fields if you don’t need the relevance scores. Normalisation factors are stored in the index to use for scoring, that is, the process of attaching a numeric value to the relevance of a document to the query defined by the user. These normalisation factors take up considerable amount of disk space (i.e. a byte per document per field, including for documents that don’t contain the field in question).
Let’s assume we’re not interested in the relevance scores, and that we can drop the normalisation factors for all our text fields. Our updated mapping looks like this (note the addition of “norms”: False):
Removing normalisation factors for all text fields brings us down to 7.3 MB (another 1.2 percent reduction compared to our previous optimisation step).
In summary, for our different optimisation options we see the following index store sizes and disk usage reduction for the netflix_movies data:
A third strategy to optimise disk usage (or rather: to reduce the cost of disk usage) is to use a hot-warm-cold architecture, using index lifecycle management (ILM) and automate index rollover.
The creators of the Elastic (ELK) Stack (that is: Elasticsearch, Kibana, Beats, and Logstash) announced better support for lower-cost cold tiers in November 2020 for their Elasticsearch Service. The official ES Service now supports cold tiers that can live in object stores, including AWS’ Simple Cloud Storage (S3), Google Cloud Storage, Azure Blob Storage, Hadoop Distributed File Store (HDFS), and Shared filesystems such as NFS. Compared to the warm tier, cold storage should cut your cluster storage by up to 50 percent.
The cold tier functionality is part of Elastic’s item lifecycle management, and you can define your own criteria for data transition rules. This means that you can store your older data more cheaply on S3 as “searchable snapshots”, while maintaining query-ability. Search performance should be comparable to searching a regular index.
Using hot-warm-cold architecture with ES involves configuring a lifecycle policy to define hot, warm, and cold phases for your indices. You can for example configure your index to roll over from hot to warm after a set number of days, or after the index has reached a certain size limit. You can repeat this pattern for warm-to-cold rollover.
Lifecycle policies can be configured via the Kibana console. Alternatively, you can use the create or update policy API. The example below defines a policy that rolls over the index and starts writing to a new index if either the data is older than 30 days, or if the index’s size exceeds 20 GB. Subsequently, the policy moves the current index (the one that was rolled over) to the warm phase after 60 days, and to the cold phase after 75 days.
PUT _ilm/policy/netflix_movies_lp{  "policy": {    "phases": {      "hot": {        "actions": {          "rollover": {            "max_size": "20gb",            "max_age": "30d"          }        }      },      "warm": {        "min_age": "60d"      },      "cold": {        "min_age": "75d",        "actions": {          "searchable_snapshot": {            "snapshot_repository": "my-repository"          }        }      },      "delete": {        "min_age": "90d",        "actions": {          "delete": {}        }      }    }  }}
In a second step, we can associate the above policy with any document that goes into the netflix_movies index:
PUT _index_template/netflix_movies_template{  "index_patterns": ["netflix_movies"],                     "data_stream": { },  "template": {    "settings": {      "number_of_shards": 1,      "number_of_replicas": 1,      "index.lifecycle.name": "netflix_movies_lp"         }  }}
Subsequently, any document pushed to the index will be subject to the netflix_movies_lp lifecycle policy, and progressively moved from hot to warm to cold storage.
A final side-note on the topic of ILM and hot-warm-cold architecture: Elastic has announced support for frozen tiers, which will also be powered by searchable snapshots. Data in a frozen tier cannot be changed and is meant for documents that are accessed rarely (further information on Elasticsearch’s data tiers is available here). However, at the time of writing this feature is in an experimental phase and cannot yet be used on the Elasticsearch Service.
There are a bunch of other strategies to optimise for storage in ES that I have not discussed in this post, such as applying the best compression codec, using larger shards, and reducing the number of shards (using the shrink API). Check out some of the links below if you’d like to learn more about any of these options.
Thanks for reading! What other ES storage optimisation strategies have you come across that you’d recommend? Please leave your suggestions in the comments!
Support my work: If you liked this article and you’d like to support my work, please consider becoming a paying Medium member via my referral page. The price of the subscription is the same if you sign up via my referral page, but I will receive part of your monthly membership fee.
If you liked this article, here are some other articles you may enjoy:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
If you’d like to get into the details of some of the aspects and features of ES described in this post, here is an overview of some of the sources that I’ve found useful, organised by topic (all sources used for this post are linked throughout the text).
A general introduction to ES
On nodes, shards, and indices
On dynamic field mapping
An overview of both disk and data storage optimisation strategies for Elasticsearch, with benchmarks
On optimising disk usage
On automating rollover with ILM
On different available Elasticsearch tiers on Elastic’s Elasticsearch Service
On Elasticsearch Service’s “cold tier” and searchable snapshots
On Elasticsearch Service’s data lifecycle management with data tiers
Disclaimer: “Elasticsearch” and “Kibana” are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. Description and/or use of any third-party services and/or trademarks in this blog post should not be seen as endorsement for or by their respective rights holders.
Please read this disclaimer carefully before relying on any of the content in my articles on Medium.com.