What We Learned Using Elasticsearch as a Time Series Database
By: Thousandeyes Engineering
Tags: Time Series Database, Mapping, Elasticsearch, Metrics, Impact Baseline Testing

by Gaurav Mishra, Software Engineer II at ThousandEyes

Introduction

At ThousandEyes, we are constantly working to add new capabilities to our platform, which can help our customers find meaningful insights from the network metrics collected by our agents. One such functionality required us to explore using Elasticsearch for storing our time series data and run aggregations on them. Using Elasticsearch was a journey for us, where we started with a very basic setup and kept discovering new things and improving our system until we got to our targets. In this post, I will describe where we started and the improvements that we made that helped us to query and aggregate on a dataset of 5 billion documents in under 1 second.

This post is intended for anyone who has a basic knowledge of Elasticsearch and is trying to use it as a time series database.

Background

Data Model and Queries

Depending on the kind of test, our agents collect different kinds of data, but all those data points follow a similar skeleton. They have some meta fields and some metrics fields. Meta fields are the dimensions on which users can filter/aggregate data and metrics fields are the actual measurements, which vary depending on the type of data.

{

meta: {

testId: 123,

agentId: 234,

timestamp: 1234567

},

metric1: 30,

metric2:15,

metric3: 145,

metric4: 15,

}

From network layer data, where the agents collects metrics around latency, loss, jitter, etc., users can ask the following two questions:

Timeline API — For the last 30 days, what are the average latencies for a given test bucketed into 1 min buckets? This query is used to build a timeline graph on the test UI, which shows trends for a selected metric.

Here users can use the agent dropdown to either see timeline data for one particular agent or aggregated timeline data for all agents.

Single Interval API — For any given test and a time interval, what are the aggregated values for all metrics? Here the aggregation operation can be average(), max(), min(), etc., depending on the metrics. This data is displayed in the bottom section of the test UI.

In both of these queries, we are searching on `meta.testId` and `meta.timestamp` and aggregating on `meta.agentId`. For the remainder of this post, I will be referring to these queries by the terms “Timeline API” and “Single Interval API.”

Online vs. Offline Aggregations

As you can see, the above queries not only require searching on the data but also running aggregations on them. During the aggregation, one or more meta fields need to be collapsed and an aggregation operation — like average() — needs to be applied to the documents that satisfy the given search conditions. Until now, we have been doing such aggregations using a service called Aggregator Service. This service reads from a stream of data points submitted by agents, does the required aggregations in memory and stores the aggregated documents in a Mongo collection. Our APIs can just query the appropriate Mongo collection for the aggregated data when needed.

As you might have noticed by now, the above approach puts restrictions on which fields we can do aggregations. In our current system, most data types just support aggregations on `meta.agentId`. Aggregating across more fields will require us to create a bunch of new mongo collections and pre-store more data. Also, the current system does not give users the ability to select a subset of agents to run aggregate upon. The selection can either be all agents or a single agent.

Clearly, the above described approach of offline aggregations wasn’t going to scale for data types that have lots of fields to aggregate on. So we had to look at how we can do these searches and aggregations on the fly. For this, we decided to use Elasticsearch.

Online Aggregations Using Elasticsearch

Because the purpose of this post is not only to list the key improvements that we did, but also to highlight the impact made by each of those improvements, I will start off with explaining our initial setup and what kind of performance we got from that setup. In this section, I will also go through our first index definition, our cluster setup, target response times for the two types of queries explained earlier, Performance Test setup, and finally the performance test results for the initial setup. Then, in each of subsequent sections, I will explain the changes we made on both code and infrastructure side and then will show the performance test results after making that change.

Index and Mappings

Because we were only interested in the last 30 days of data, it made sense for us to use daily indexes to store our data. This allows us to delete any data older than 30 days. The first version of mapping for our index looked like this:

{

"index_patterns": [

"cea-ping-v1-*"

],

"settings": {

"index": {

"refresh_interval": "30s",

"number_of_replicas": 1,

"number_of_shards": 6

}

},

"mappings": {

"properties": {

"meta": {

"properties": {

"agentId": {

"type": "long"

},

"testId": {

"type": "long"

},

"timestamp": {

"type": "long"

}

}

},

"metric1": {

"type": "long"

},

"metric2": {

"type": "long"

},

"metric3": {

"type": "float"

},

"metric4": {

"type": "float"

}

}

}

}

Sample Queries

In the previous section, I explained the two types of queries that we do against our data: Single Interval query and Timeline query. This is how those queries look in Elasticsearch Syntax.

Get all available metrics for a single 3 minute interval:

{

"size": 0,

"query": {

"bool": {

"must": [

{

"range": {

"meta.timestamp": {

"from": 1589081040000,

"to": 1589081220000,

"include_lower": true,

"include_upper": false,

}

}

},

{

"terms": {

"meta.testId": [

123

],

}

}

],

}

},

"_source": false,

"aggregations": {

"weighted_avg#metric1": {

"weighted_avg": {

"weight": {

"field": "metric2"

},

"value": {

"field": "metric1"

}

}

},

"sum#metric3": {

"sum": {

"field": "metric3"

}

},

"avg#metric4": {

"avg": {

"field": "metric4"

}

}

}

}

Get aggregated timeline for a particular metric (metric1) for 5 days bucketed into 1 min bins:

{

"size": 0,

"query": {

"bool": {

"must": [

{

"range": {

"meta.timestamp": {

"from": 1589072400000,

"to": 1589504400000,\

"include_lower": true,

"include_upper": false,

}

}

},

{

"terms": {

"meta.testId": [

123

]

}

}

]

}

},

"_source": false,

"aggregations": {

"date_histogram#timeline": {

"date_histogram": {

"field": "meta.timestamp",

"interval": "60s",

"offset": 0,

"order": {

"_key": "asc"

},

"keyed": false,

"min_doc_count": 0

},

"aggregations": {

"sum#metric1": {

"sum": {

"field": "metric1"

}

},

}

}

}

}

Search Space and Target Times

For network layer metrics, our data pipeline processes about 2k data points per second. That means each of the daily indexes will hold about 170 million data points. For the Timeline API, we will be querying 30 such daily indexes, which means we will be searching and aggregating on a database of about 5 billion data points.

Based on our current load on production, we had the following target latencies for both our APIs:

For Single Interval API, P95 latency should be under 250ms at 40 requests/sec

at 40 requests/sec For Timeline API, P95 latency should be under 1000ms at 5 requests/sec

Performance Test Setup

All our performance tests were run against production Elasticsearch where the writer service was constantly writing. We had a single pod of Reader Service deployed. We also built a tool based on Jmeter for generating load for the Reader Service, which would in turn make calls to Elasticsearch. The above mentioned target P95 latencies were for the Reader Service.

Baseline Test

The following is a summary of our setup when we ran our performance tests for the first time:

3 Master Nodes (c5.large)

6 Worker Nodes (m5.2xlarge) with 32 GB RAM

27 days worth of data

No Routing

Wildcard index names used in both APIs

Mix of Single Interval and Timeline Requests sent from load generator in ratio 2:1

Following are results:

As you can see, the performance wasn’t where we had hoped it would be, even at such a low request rate. It’s hard to say with 100% certainty what was causing such a bad performance, but there was some evidence that disk IO was our bottleneck. In this setup, we were not using routing to partition our data within any given index and, therefore, for any given testId the data would be spread across all shards. Also, each shard was pretty big (in this setup, each shard was holding roughly 30 million documents).

Elasticsearch runs its query in two phases: query phase and read phase. Big shards would mean bigger data structures to search ,which would mean search times would be relatively high. Also, since the shards are big, the number of segments in each shard would also be big, which means that during the read phase Elasticsearch coordinator would have to read data from a lot of files on disk.

Also, the amount of RAM used in this setup was probably pretty low, and that meant that caching done by Elasticsearch was not very good. The following graph shows the number of IO operations happening during the time tests were running.

Improvements and Their Impacts

More RAM and SSDs

RAM on each worker node was increased from 32GB to 64GB

Each worker node was using SSDs now

Used disk optimized ec2 instance for worker nodes ie i3.2xlarge

This test run was just for the timeline query

Following are the results:

This time we did slightly better. We were able to go up to about 3 req/sec in contrast to less than 1 req/sec in the previous setup. Increasing the RAM also means the file system cache capacity increased. And that did help us with reducing the number of disk IO ops as evident from the graph below. But P95 was still not where we wanted it to be. Also, at such a low rate of requests, we started seeing queuing at the worker nodes, which meant we needed to improve the performance of individual operations now.

Use Routing

Routing is a feature of Elasticsearch that allows partitioning of data within an index. So in contrast with the default setup, where any datapoint can go into any of the shards if we use routing we can make sure that data that will be queried together is being written to the same shard. We used `meta.testId` field for partitioning our data since this field was a required parameter in both our APIs. Using routing did involve re-indexing all existing data points into new indexes.

Other than using routing to partition our data, we also increased the number of shards to make sure the size of each shard became smaller in hope that smaller shards would improve search times.

To recap, following changes were made for this test:

Use `meta.testId` to partition our data within daily indexes

Increased number of shards for each index from 6 to 32

Increased number of worker nodes from 6 to 10

Results for the Timeline API tested at 12 reqs/sec:

Results for the Single Interval API tested at 12 reqs/sec:

As we can see from these results, we got a big improvement for both our APIs. We were able to go to a much higher request rate while keeping the latency reasonable. Some possible explanations are:

Now the data for each `meta.testId` resides in a single shard. So in contrast to the old setup where we were searching for our data in all shards for each daily index, now we are searching in just one shard of an index. Each shard is much smaller now. Holding about 5 million shards documents on average compared to about 30 million documents in previous setup. This might have helped with improving search times. Localized nature of data also means during the read phase of the query there will be significantly less number of files to read the data from the disk. This also must have improved the query times. Increasing the number of worker nodes also means we added more RAM to the cluster, which means caching is improved, too.

This was a big win for us in terms of moving the query times significantly closer to our goals under reasonable loads. However, Single Interval API P95 was still above 1 second, which was much higher than we were aiming for so more improvements were still required. Also, we noticed that at the above load we almost saturated our clusters’ CPUs, which was not a good sign.

No More Wildcard Indexes

Until this point in time, we were using wildcard index names for both types of queries. This means that for making a call to Elasticsearch, we were using URLs that looked like: `<es_url>/cea-ping-v2-*/_search?routing=416720` .

This basically tells Elasticsearch to look for the data in any index that matches the pattern `cea-ping-v2-*` . For any given request, the Coordinator node for that request will figure out which indexes to query, which shard to query within each index, and which nodes these shards live on. Now the coordinator node will fan out the initial search query into multiple smaller queries and wait for the results from other nodes and finally merge the results and return back to the client.

For our Single Interval API, we were interested in data for a very small interval of time, typically 5–15 minutes, but we were searching our entire database (all 30 daily indexes). This was a big waste of resources. So instead of using a wildcard index name, using the exact index name, which would contain the given time interval like `cea-ping-v2-2020-04-27` , made perfect sense in this case.

For our Timeline API, we were querying 30 indexes, so using individual index names would not reduce total work done by worker nodes. But for the node that is acting as coordinator node for any given query, it would reduce the amount of work to be done. The coordinator node will no longer have to do the fan out of requests and merge the results. But this also meant that our service layer code had to do more work now. The following is a rough pseudo code for our Timeline API.

timelineAPI(30dayRequest)

{

[indexNames, oneDayTimeRanges] = split(30dayRequest)

promises = [indexNames,oneDayTimeRanges].foreach(esClient.request)

results = wait(promises)

return mergeResults(results)

}

Here’s a summary of changes made for this test:

Use the exact index name for Single Interval APIs

Split the 30-day Timeline request for Elasticsearch into 30 one-day Timeline requests and issue them in parallel.

Note: For all these performance tests a single pod for service layer code was used.

Results for Timeline API at 7 reqs/sec:

Results for Single Interval API at 15 reqs/sec:

As you can see, these are pretty significant improvements as compared to the previous test.

P95 for Single Interval API has come down from 1000–1500 ms to 50 ms for similar request rate.

to for similar request rate. P95 for Timeline API also improved. Even though it’s hard to compare the results shared in the previous test compared to this test’s results because both were done on different request rates. Under similar load P95 for the previous test was around 600–700 ms, which improved to 250–400 ms. Since we just had a single service pod running and we offloaded a lot of work from coordinator node to service pod, for this test that single pod became the bottleneck. But it’s much easier to scale the service layer compared to Elasticsearch cluster. CPU usage on worker nodes were well under 25% mark, which shows there was a lot of room to increase the load if we had more pods during performance tests.

Overall, this change brought the P95 latency for both our APIs under the limits that we were chasing. We could have stopped here, but as you can see there were some big spikes in P99 times and this meant for some users this could have seen very big load time on the Data Visualization page. So we went ahead and tried one final thing in hopes to improve the P99 latency for both these APIs.

Merged Indexes

Every shard in Elasticsearch is a lucene index, and each of these lucene indexes are composed of one or many segments. These segments are basically a mini-lucene index. For any given search request Lucene has to look for the data in all of its segments. Also during the read phase of any query Elasticsearch coordinator has to read data each of these segments. Elasticsearch does allow us to force these segments to be merged into one big segment. The only thing we need to be careful about is we should only be force merging indexes, which are read only. Due to data retention policies in some of our datastores it was possible to get data up to five days old. Therefore, we merged all indexes that were more than five days old.

Results for Timeline API at 9 reqs/sec:

Results for Single Interval API at 30 reqs/sec:

As you can see here, P95 was better for both APIs even at higher request rates. The bigger impact of this change was on P99 of both APIs. The big spikes that we were seeing in the previous test calmed down a bit. There were still some spikes, but the peak of these spikes were much lower as compared to the previous setup.

Conclusions

Here are some key takeaways from this whole journey:

Pay close attention to the index mapping and use the right data types for each field depending on how those fields are used in queries.

Because it’s very difficult to get the cluster setup right in one attempt, invest some time up front in creating some kind of framework to do some performance testing.

Use routing to partition your data if your query patterns allow that.

Avoid wildcard indexes wherever possible. This might involve some extra effort on the implementation side but the performance benefits might be worth it.

Force Merge any index that has become read-only.

If solving complex challenges like this interest you, check out our Careers page to see open positions on our team.