Taming ElastiCache with Auto-discovery at Scale
By: Junyoung Kwak
Tags: Failover, Engineering, Tinder, Redis, Elasticache

Data Caching at Tinder

Our backend infrastructure at Tinder relies on Redis-based caching to fulfill the requests generated by more than 2 billion uses of the Swipe® feature per day and hosts more than 30 billion matches to 190 countries globally. Most of our data operations are reads, which motivates the general data flow architecture of our backend microservices (see Figure 1).

Figure 1 Cache-aside Approach: A backend service checks for requested data in a Redis cache before falling back to a source-of-truth persistent database store (Dynamo, Postgres, Mongo, etc.) and then upserts the value to Redis on a cache miss.

In the early days, Tinder ran Redis servers on Amazon Web Service’s (AWS) Elastic Compute Cloud (EC2) instances. Our applications then connected to these nodes via clients which implemented sharding by locally hashing keys to Redis instances (see Figure 2). This solution worked initially, but as Tinder’s popularity and request traffic grew, so did its number of EC2-hosted Redis instances, as well as the administrative overhead of maintaining them. The burden of 1) changing Redis cluster topology (e.g. scaling our clusters), and 2) orchestrating failovers sapped the time and resources of our engineers and led us to look for alternatives.

Figure 2 Sharded Redis Configuration on EC2: We maintained a fixed configuration of Redis topology (including # of shards, # of slaves, instance size), and services accessed the cache data through the sharded Redis client that implemented this configuration.

After exploring options for more scalable data caching, we settled on AWS’s ElastiCache (https://aws.amazon.com/elasticache/). Cluster-enabled Redis (Redis engine version 3.2.10) on ElastiCache allows us to horizontally scale with great ease. To scale up a cluster, all we have to do is to initiate a scaling event from the AWS developer console, and AWS takes care of data replication across any additional nodes as well as shard rebalancing for us.

Finally, AWS handles node maintenance during planned maintenance events with limited downtime and no intervention on our end. Excited to offload the infrastructural burden of Redis maintenance, we began migrating our backend services from EC2-hosted Redis instances to ElastiCache, and all seemed to be well until we noticed issues specific to some of our core Java applications directly downstream of ElastiCache.

Initial Java ElastiCache Client Implementation — Issues and Limitations

We researched ElastiCache extensively during the development of our original Java-based Redis client, however we did not find an AWS recommended Java library for Redis, or a widely adopted online convention for managing connections to cluster-enabled ElastiCache¹. Consequently, our first attempt was somewhat naive and did not take full advantage of the benefits ElastiCache offers.

In its first iteration, our Java-based Redis client used jedis (https://github.com/xetorthio/jedis), the most popular Java Redis library, to connect to our clusters via their primary endpoints. This simple solution was functional and allowed us to replace our legacy EC2 hosted Redis nodes, but we noticed connectivity issues during our clusters’ first AWS maintenance events.

A cluster maintenance event entails the patching of an underlying node, rendering it unusable for a short period of time. If the master node is affected, the maintenance event encompasses failing over a slave to become the new master (see Figure 3).

Figure 3 Description of Redis cluster failover of a primary node: In the event of a maintenance event on an ElastiCache cluster’s master node, one of the master’s slave nodes gets promoted to become the new master and the old master gets demoted to become a new slave before getting patched or replaced by a new replica. (Source: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoFailover.html).

Our jedis-based client seemed unable to appropriately respond to the topological swap of a master and slave node. We suspected that after a failover, our clients were still sending read and write requests to the old master node, even though it was inaccessible due to maintenance; if this was true, this misconfiguration following a failover could be causing the timeouts and exceptions that were breaking our backend services.

In addition to failover intolerance, our first Java Redis client generated inefficient CPU usage patterns across the nodes of our ElastiCache clusters, which was concerning. During maintenance events, the CPU utilization on our cluster’s slave nodes spiked, but at all other times it was at almost zero.

We reasoned that maintenance events would increase the CPU utilization across our cluster’s slave nodes as they resynced data from their masters. However, the lack of activity at all other times suggested that we were maintaining our clusters’ slave nodes for essentially no other use besides failovers.

Though our initial Java Redis client provided the core CRUD functionality necessary to replace our legacy EC2 hosted Redis nodes, it failed to take full advantage of all that ElastiCache offered us from the perspectives of failover tolerance and node utilization.

ElastiCache should have allowed us to add additional shards and replicas to our cluster without app downtime. Instead, our client seemed to form static connections to the cluster at application startup and it was unable to dynamically handle changes to the topology of the underlying cluster. Any previously connected backend services would lose their ability to communicate with the connected ElastiCache cluster when it underwent a maintenance event. Our developers would then have to manually intervene, and redeploy the affected services to restore functionality.

Figure 4 Impact of Cluster maintenance event on production cache read traffic: Impact of an ElastiCache maintenance event causing an app outage. Fortunately, quick developer intervention & redeployment of affected services kept the duration of production outage + service degradation relatively short.

In the worst case, unplanned maintenance events (due to hardware malfunction etc.) could break connected services in the middle of the night and full recovery could take hours (see Figure 4). Our backend infrastructure critically relies on Redis-based data caching — so much so, in fact, that our failure to dynamically process the cluster failover of AWS maintenance events was at one point the leading cause of app-wide outages at Tinder.