Database “Magic”​: Running Huge High Throughput-Low Latency KV Database with No Data In Memory
By: Zohar Elkayam
Tags: NoSQL, Redis, Database, Big Data, Aerospike

Database “Magic”​: Running Huge High Throughput-Low Latency KV Database with No Data In Memory Zohar Elkayam Follow Dec 20 · 4 min read

A couple of weeks ago I was talking to one of my oldest database colleagues (and a very dear friend of mine). We were chatting about how key/value stores and databases are evolving, and how they always seem to be revolving around in-memory solutions and cache. The main rant was how this kind of thing doesn’t scale well, while being expensive and complicated to maintain.

My friend’s background story was that they are running an application that uses a user profile with almost 700 million profiles (their total size was around 2TB, with a replication factor of 2). Since the access to the user profile is very random (meaning, users are fetched and updated without the application being able to “guess” which user it would need next) they could not use pre-heating of the data to memory. Their main issue was that sometimes they get peaks of over 500k operations per second of this mixed workload and that doesn’t scale very well.

User Profile use case summary

In my friend’s mind’s eye, the only things they could do is use some kind of a memory based solution. They could either use an in-memory store — which, as we said before, doesn’t scale well and is hard to maintain, or use a traditional cache-first solution, but lose some of the low latency required, because most of the records are not cached.

I explained that Aerospike is different. In Aerospike we can store 700 million profiles, 2 TB of data, provide the said 500k TPS (400k reads and 100k writes, concurrently) with a sub 1ms latency, but without storing any of the data in memory. The memory usage would then be very minimal — under 5 percent of the data for that use case.

My friend was suspicious: “ What kind of wizardry are you pulling here?!”

So since I am not a wizard (yet, I am still convinced my Hogwarts acceptance letter is on its way — I’m almost sure it’s just the owl delayed), I went ahead and created a modest demo cluster for them, just to show my “magic”.

Aerospike Cluster: Hybrid Memory Architecture

In this next screenshot, we can see the result: a 6 node cluster, running 500k TPS: 400k reads + 100k writes, storing 1.73TB of data but only utilizing 83.45GB of RAM.

Running a 6 node cluster: 1.73TB of data, 84 GB of RAM

This cluster doesn’t have a specialized hardware of any kind. It’s using 6 nodes of AWS’ c5ad.4xl (a standard option for a node), which means a total of 192GB RAM and 3.5TB of ephemeral devices, cluster-wide. From the pricing perspective it’s only about 1900$ a month, way less than what they pay now (and I price-tagged it before any discounts).

Obviously, if the cluster has a total of 192GB of DRAM, the data is not being stored fully in memory. In this case 0 percent of the data was fetched from any sort of cache — so, for 1.73TB of data, the memory usage was under 84GB (even though the Linux kernel would allow for some caching if needed! This makes things even better when using other access patterns like Common Records, or Read After Write).

The cool thing is the performance. Predictable performance is something every application needs — and for the peaks described earlier, we can see in the next screenshots a latency of under 1ms for both reads and writes!