Search at Wattpad via Elasticsearch: Upgrading and Updating
By: John Finnson
Tags: AWS, Elasticsearch, Software Engineering, Wattpad, Search Engines

Part A of this blog post is here. It covered Elasticsearch at Wattpad, problems we had, and important symptoms to look for.

Part B of this blog post is here. It covered how to diagnose Elasticsearch cluster issues, strategies for fixing them, and general research for Elasticsearch cluster design.

Upgrading 9 production clusters

Our clusters serve many different purposes and experience different loads. We decided on the following instance types:

r5.large.elasticsearch for small general clusters

for small general clusters r5.xlarge.elasticsearch for medium general clusters

for medium general clusters r5.2xlarge.elasticsearch for large general clusters

for large general clusters I3.2xlarge.elasticsearch for large i/o heavy clusters

for large i/o heavy clusters And eventually m5.4xlarge.elasticsearch for large thread heavy clusters

Which clusters should get which configuration?

Luckily, our clusters are already in production. All that was required was observing current configurations, resource loads, and then choosing a configuration based on current load. See Metrics observed in part A and B.

Cluster upgrades with AWS Managed Elasticsearch are ideally very simple:

Update cluster’s terraform file Run `terraform apply` Magic!… and hopefully not a fireball Observe and monitor changes

As the upgrades are applied, visibility is limited from AWS managed elasticsearch and CloudWatch dashboards. Node count will jump, as well as other metrics such as total free storage space, search latency, search rate, etc. These metrics will look confusing during the upgrade because they include all nodes (new and old).

External monitoring will also echo these metrics. These external monitors are helpful because they tell an interesting story about latency changes during an upgrade.

As we upgraded the first 8 clusters, each upgrade triggered a temporary spike in latency during the upgrade. The larger the cluster, the larger and longer the spike.

Why does this happen? It’s a “blue/green” deployment! Specifically it states in AWS’s documentation: “Blue/green typically refers to the practice of running two production environments, one live and one idle, and switching the two as you make software changes. In the case of Amazon ES, it refers to the practice of creating a new environment for domain updates and routing users to the new environment after those updates are complete.”

This is the part where Cerebro really became helpful!

Watching the upgrade from Cerebro gives a real time, lower level view of what’s happening. As the upgrade proceeds we are able to see:

Nodes joining and leaving the existing cluster Shards being moved between nodes

Later revealed as this command:

_cluster/settings -d ‘{“transient”: {“cluster.routing.allocation.exclude.di_number”: “”}}’

Reallocation of shards can be observed in Cerebro via a notification status at the top of the first column, and from green vs purple shards (squares). Solid squares are primary shards, dotted squares are replica shards. Load spikes on affected nodes

This revealed a very interesting detail about AWS’s “blue/green” approach.

Their documentation says “…creating a new environment for domain updates and routing users to the new environment after those updates are complete”, but its not truly a new environment. AWS is reusing the same cluster, adding nodes, and moving data (shards) to the new nodes.

This is a large distinction because the process of adding nodes and moving data can consume a lot of resources on existing nodes and results in a large amount of node-to-node network traffic. This is hinted to from this excerpt in the documentation “It is important to maintain sufficient capacity on dedicated master nodes to handle the overhead that is associated with these blue/green deployments.”

The combination of the factors above can result in higher latency during cluster upgrades, as well as possible downtime.

There are a few other important things to note with updating AWS Managed Elasticsearch clusters via the “blue/green” approach.

No clean and quick abort or revert. A new configuration will have to be applied again. This can be quite an issue if a bad configuration is applied and takes a long time to finish.

Shard balance is important for stability and load balance. If a new configuration will have an imbalance of shards, this will likely cause performance issues.

The rebalancing of shards is the riskiest part of a new configuration. I recommend creating a new index, with the new shard configuration , so it’s fast and easy to revert (via alias swap) to the previous index.

, so it’s fast and easy to revert (via alias swap) to the previous index. For scaling down, I recommend creating a new index with the desired shard balance. Switching to the new index to test stability. Then update the cluster node configuration.

For scaling up, it’s usually safe to apply the new configuration, then create a new index with the desired shard balance.

The latency spikes, and risk of downtime was small enough that we proceeded to upgrade most of our clusters.

Last and Largest

After upgrading the majority of our clusters, we finally came to our final large cluster, cluster X. Cluster X has issues from older “unknown” factors. The cluster is mysterious due to its high load but relatively low number of documents and requests. It is essentially the problem child of the cluster family.

This cluster was using 12 c4.8xlarge.elasticsearch data nodes. Compute optimized instance types for Elasticsearch? Anthropology aside, this felt very off. Historically the cluster had issues with indexing and was already using “io1” EBS volumes. We conservatively updated the cluster to use 10 i3.2xlarge.elasticsearch nodes. Or so we thought…

Updating the cluster to this configuration (similar node count, same memory per node, faster storage), promptly took down our primary search service. Initially we thought it was a bad case of that latency spike we mentioned, but the load did not reduce. This is where we noticed the thread pool rejection count was increasing very quickly.

We doubled the nodes and updated the cluster. No success. Increased instance type to i3.4xlarge.elasticsearch. Service was brought partially back online, but some nodes were still at 100% load. The nodes with more load had more shards. We added 4 more nodes (24 total) to help balance the shards per node. This solved the problem and brought our service back online!

Upon further investigation we learned the following things:

I/O operations were not high and were not a limiting factor

Before upgrade CPU usage was higher than other clusters (20%-60%)

(20%-60%) Number of vCPUs were the limiting factor!

Shard balancing on nodes is important for cluster stability

We later chose to change the instance type to r5.4xlarge.elasticsearch, then again to m5.4xlarge.elasticsearch. 24 m5.4xlarge.elasticsearch nodes brought a balance of vCPUs (16) vs memory (64GB) and at half the cost of c4.8large.elasticsearch. Why did we not choose to go back to c4.8large.elasticsearch? Because at 60GB of memory for 12 nodes, system memory was capped at 100% usage (causing swap to be used).

Issues post-upgrade

Smooth sailing post upgrade, for the most part…

Only issue observed were “seemingly” random latency spikes. We observed latency increases of 4–10X for approximately 5 min at a time. We first noticed them in cluster X, but later found the same pattern less often in smaller clusters.

Largest cluster’s latency post upgrade (cluster X)

Compared to the next largest cluster (cluster Y):

Digging into logs and metrics did not help much until the following correlation was made: the latency spike correlated with Elasticsearch garbage collection!

Following is the JVM memory pressure for cluster Y

Latency and JVM Pressure overlapped for cluster Y:

Latency and JVM Pressure overlapped for cluster X:

Why is JVM maximum memory pressure so high for cluster X? Why does it go into GC so often? It’s because this is a metric of the maximum pressure across all 24 nodes. Below is a graph of all nodes’ JVM memory pressures (each colour is a separate node).

Basically from saturation, there is always a node about to go into garbage collection. What’s unacceptable is that we get a latency spike for the entire service when any node goes into GC.

The correlation between latency and garbage collection is not 100%, however it is close enough to point a finger. At the moment we have a support ticket with AWS regarding this issue. This issue does not have a major impact on our users but should be resolved.

Next steps for us will be:

Dig into latency vs garbage collection issue more

Dig into what makes cluster X unique and make it more efficient/performant

Combine some smaller clusters to make better use of resources

Key Takeaways

How we research

Official AWS and Elasticsearch resources, and unofficial resources (forms and blogs like this!)

How we debug

Consider: latency, JVM pressure, system memory pressure, thread pool rejections, and shard node balance

Tools and Services that made our lives easier

Self-Hosted Elasticsearch vs AWS Managed Elasticsearch

AWS Managed Elasticsearch allows for very easy and fast cluster upgrades via terraform

AWS Managed Elasticsearch “blue/green” upgrade has risk of latency spikes and downtime

Cost Savings:

Optimizing the instance types allowed for substantial cost savings

Conclusion

Over approximately 2 weeks, Wattpad upgraded 9 AWS Managed Elasticsearch clusters. The clusters had a wide variety of issues: over/under provisioned, read/write latency, thread rejects, etc. We uncovered, researched, and debugged these issues using a variety of tools and technologies, including AWS Managed Elasticsearch, Terraform, and Cerebro.

In the process of upgrading the clusters we uncovered interesting issues and details about the tools we used, Elasticsearch DevOps, and our services.

Elasticsearch is a core component of our infrastructure at Wattpad. Watch for future blog posts on what’s NEXT at Wattpad!