System design paradigm: Caching
Tags: Distributed Cache, Distributed Systems, Cache, System Design Interview

The Problem

Caching is one of the two ways(the other is replication) to scale read heavy applications. Assume we have an application that handles 100K/s read requests. After load balancing the HTTP request from clients, those read requests will still hit the query backend(traditional DB or some in memory storage). If the backend can handle only 10K/s QPS, what should we do?

Pareto distribution of the query

The queries can be abstracted as key lookup(most of the time, that’s exactly what they are). Then there is a distribution of the request number among the keys. This distribution is almost always the Pareto distribution(power law).

That means a small percentage(e.x. 15%) of keys consists of a large percentage of the requests(e.x. 90%). We can put the 15% most frequently used data in a cache who can handle bigger QPS. The App servers will first query the cache and only go to DB when the key is not there. In our example, that will happen to only 10% of the requests. Then only 10% of the QPS will reach DB. This is the reasoning behind caching.

Distributed look-aside cache

Distributed sits in front of DB and is shared by all App servers. The cache itself is usually an in-memory DB that handles reading very fast. Most common caches are Redis and Memcached.

To query key K, App server use the following procedure:

Query cache for K, if cache hit, return the result from cache Otherwise, query DB for K Store the result to cache Return the result to client

DB update will send a delete command to cache so that the cached value is invalidated. Delete is idempotent. Another way to keep cache consistent is to rely on expiration. Expiration is simpler design, but has higher inconsistency.

Handling thundering herd

One corner case is when multiple requests having the same cache miss. For example, suddenly 100K Twitter users reached the server from a link that went viral, but some of the contents are not in cache. Then the cache miss happens in parallel, all of such requests will try to query DB. DB will then face a load of 100K/s read.

The solution is a lease. The first cache miss will grant the app server a lease token. Only the app server having the token for a key can query DB and fill the cache. After a token is issued, all subsequent requests to the cache will be asked to retry after a period. The lease expires after a while to avoid deadlock.

Race condition between delete and set

If we depend on DB update to invalidate cache(delete), there will be a racing condition between cache miss read and DB update that could result in inconsistency:

B got a cache miss and queried DB to get V0 A updated DB value from V0 to V1 A sent delete to cache, which was an no-op B filled cache with V0

Lease can also solve the above issue. During update, A first writes DB, then invalidates both ongoing lease and current cache value. Then 4) will fail because its lease was invalidated by A.

Cascading failure

If cache layer fails, the traffic will hit DB causing cascading failure. The solution is to have a standby cluster handling failed traffic.

Full memory snapshot

If the data is small(can be sharded into 1~3 shards), all the data can be put in cache. Then there won’t be any cache misses. The update will be from DB update. Each update is buffered in a queue and a job will gradually fan out the update to each shard and replicas. This infrastructure is a separate paradigm(primary replica) by itself.

Read-your-write

To provide read-your-write consistency, the app server will directly invalidate the cache after DB update. The invalidation will unaunthenticate all the leases that are granted to store value for the given key.

Notice, app server can’t store cache directly because there will be race conditions resulting in setting stale value.

Local cache

Distributed cache will incur significant network traffic. An alternative is to have the cache co-deployed with the app server.

It can be a hybrid structure with both local and distributed cache. The local cache will duplicate hot keys on each app server. Less frequent keys are stored in distributed cache. It’s a challenge for the application to decide where to put the key.

Further reading

How Facebook scaled itself with memcached: paper