Does Elasticsearch lie? How does Elasticsearch work?
By: Maciej Szymczyk
Tags: Elasticsearch, Elastic Stack, Spark, Apache Spark

Elasticsearch surprises us with its capabilities and speed of action, but does it return the correct results? In this post, you’ll learn how Elasticsearch works under the hood, and why returned aggregations are some kind of approximation.

Elasticsearch under the hood

Indices, shards and replicas

Let’s start with how Elasticsearch organizes data. You already know the concepts of index, it looks like a table in a relational database. Each index consists of at least one shard and any number of replicas.

Shards divide the data in the index. Such shard can be allocated on different nodes in the cluster. Thanks to this we can distribute data processing. Generalizing, based on routing, the added document will go to the first, next to the second, etc.

On the other hand, replicas play the role of protection against failure and support readings (you can only write to primary shards, you can read from shards and replicas). Suppose you have an index with a configuration of 3 primary shardy and 2 replicas. How many indexes (primary + replica) will the index have? 9 (1, 1 ′, 1 ″, 2, 2 ′, 2 ″, 3, 3 ′, 3 ″).

Primary shard and replicas within the same data form a replication group. When a new record goes to primary shard, it is responsible for sending it to replicas.

Queries and aggregations

When we ask for a specific record, the query is forwarded to the node with the shard where it is located. The node knows this based on the metadata (the node with the master role is responsible for it).

When we ask for a set of documents or aggregation, the query goes to all nodes containing shards of the questioned index. These return the result to the node that aggregates the results and returns them to the client.

Estimation

Why is aggregation just an estimate? Let’s look at Terms Aggregation, equivalent to GROUP BY + COUNT in SQL.

Suppose we have 3 shards, each with some records. We ask the cluster to count all the repeating values of a field. Based on the previous section, we know that such a query goes to each of the nodes. Let’s assume that it is like in the picture below.

I forgot to mention that we wanted TOP 3 values. So each node takes its TOP 3 and returns it to the node that coordinates the query.

Change of decision. We want TOP 4. We ask the cluster about the same and get different results. Not only that X has more records, but A and Z swapped places.

Test

In previous stories I described the collection, processing and analysis of public transport data in Warsaw. Let’s see if the records will be different based on this dataset. For comparison, I used Apache Spark together with the elasticsearch-spark library. If you want to connect it to Spark, take a look at this post.

In previous stories I described the collection, processing and analysis of public transport data in Warsaw. Let’s see if the records will be different based on this dataset. For comparison, I used Apache Spark together with the elasticsearch-spark library. If you want to connect it to Spark, take a look at my blog post (it’s in Polish, but Google Translate works quite well).

DataFrame in Apache Spark

In the Elasticsearch index with bus data, the date field does not have a standard format, so I had to set the es.mapping.date.rich option to false.



.option("es.nodes.wan.only","true")

.option("es.port","9200")

.option("es.net.ssl","false")

.option("es.nodes", "

.option("es.net.http.auth.user","elastic")

.option("es.net.http.auth.pass","xxxxxxx")

.option("es.mapping.date.rich","false")

val ztm = es.load("ztm") val es = spark.read.format("org.elasticsearch.spark.sql").option("es.nodes.wan.only","true").option("es.port","9200").option("es.net.ssl","false").option("es.nodes", " http://hadoop06 ").option("es.net.http.auth.user","elastic").option("es.net.http.auth.pass","xxxxxxx").option("es.mapping.date.rich","false")val ztm = es.load("ztm")

Top 20 Lines

Apache Spark version:

ztm.groupBy("lines").count().orderBy(desc("count")).show()

Elasticsearch version:

POST ztm/_search

{

"size": 0,

"aggs": {

"num_of_lines": {

"terms": {

"field": "lines",

"size": 20

}

}

}

}

There are many differences, but mainly due to the rows shift. Count is the same for most records. How will it look like with ascending order instead of descending?

By the way, we learned what stupidity sometimes we get from the API. Once again, you can see how important data quality is, and hence the quality control.

Top 10 vehicleNumber

Apache Spark version:

ztm.groupBy("vehicleNumber").count().orderBy(desc("count")).show()

Elasticsearch version:

POST ztm/_search

{

"size": 0,

"aggs": {

"num_of_vehicle_number": {

"terms": {

"field": "vehicleNumber"

}

}

}

}

In this case, hardly anyone agrees. Only the record from the first place as such can be counted.

Can we fix it?

The larger the size in the Elasticsearch query, the closer the results will be to the correct ones, but we will have to wait longer for them. In addition to the size parameter, you can use shard_size. This parameter defines how many records the counting node should return to the coordinating node, and its default value is (size * 1.5 + 10). There is also an option to display the error value for the worst case by the show_term_doc_count_error parameter (helps in determining shard_size).

Conclusions

As you can see, sometimes Elasticsearch can fool us. Is it wrong? It all depends on our expectations. There is one parameter that I have not mentioned yet. Time to complete queries. Elasticsearch responded immediately. I had to wait a bit for the Apache Spark result.

As the saying goes: life is a matter of choice 🙂 either quickly or exactly