elasticsearch/elasticsearch — Issue #1488: Scalable Sharding
Author: folke
Created: 2011-11-23 11:33:38+00:00
State: open

The sharding that is currently possible with Elasticsearch is already great. But if you want to scale to really very big datasets, then you can still experience performance issues.

Suppose that you have 1 big index spread over 10 shards. In this case, Elastic still needs to perform searches on 10 shards (ideally on 10 different servers in parallel).

Another approach could be to have some kind of user configurable sharding in place. For example, in our case (we are a social media platform), we do not use Elastic sharding, but have our own custom sharding on top where we create indices for every topic and every month. This means that for a search of the last 30 days, at most, we need to search on the shard for this month and the shard for last month.

The same approach could be done on text fields. Suppose that we have setup 10 shards and do sharding on the field "contents". Instead of pushing the data to one of the shards, we could first perform an analysis step on the "contents" field to break up in words and then for each word, determine the servers responsible for that word. On those servers, we then index the document. The result of this would be that there would be more redundance on storing the documents (and as such adding documents will also take longer), but when searching, we can immediately determine based on the query what servers need to be searched (only those responsible for the data ranges that contain the analyzed words in the query). (key distribution could be done based on hashing and hashing ranges per server, similar to how for example Redis or Memcache work)

Multiple sharding keys should als be possible.

With this custom sharding, it should be possible to scale datasets to billions of documents. There will be redundant storage, but queries will be alot faster. For example in a 100 server setup, one search with 100 shards could potentially take a long time. With the custom sharding, only a few servers would need to be queried.