Let’s talk about trade offs — How to choose the right database for the job — Part 1
By: Nimrod Parasol
Tags: NoSQL, Data Engineering, Database, Redis

There is no magic when it comes to databases. It would be amazing to find a database that could handle millions of ad-hoc queries every second over petabytes of data and do it without breaking a sweat on a small, cheap cluster of servers. Sadly, we are not magicians, we are engineers. Most of the time, when we look at a new database that promises great performance, it comes with some drawbacks. There might be some areas or functionalities that the designers of the database decided to cut back in order to achieve great results in other parts of the system.

Wizard Of Oz (1939), Warner Bros.

This means that every database is suitable for specific use cases. We might be able to implement other use cases with it as well, but it will come with increased cost — either in resources (which translates to money) needed to support our workloads, or in complexity of the system.

In this article series, we’ll go over some examples of databases that are referred to as “magical” these days. We’ll understand how they work under the hood and expose their pitfalls to understand what use cases are suitable for them and when we should look for other solutions.

Redis

It seems like everybody uses Redis these days. Some use it because it is very fast, others use it because it really simple and easy to interact with. For those who don’t know, Redis is an in-memory key-value store that supports using several data structures (such as hash tables, sets and lists) for values. Let’s explore two of the main pitfalls we might encounter when evaluating Redis for our use-case.

Persistency

The main thing that makes Redis so fast is the fact that it works entirely in memory. When we write or query data from the database, we will never have to wait for I/O from the disk. The risk when using memory to store data is its non-persistent nature. If for some reason the service crashes, all the data in memory will be lost. But don’t worry. Redis has two mechanisms to handle this pain point.

The first one is scheduled snapshots of the whole database to the disk. These snapshots are taken in the background to avoid affecting the performance of the database. If your database server crashes, you can recover the data from the snapshot.

But what about the data that changed after the last snapshot was taken? To handle this, Redis persists to disk a log of all the changes that have been made to the data. This log is called the “Append Only File” or “AOF”. Using this log, you can replay all the operations that happened from the moment the last snapshot had been taken. We said that Redis works entirely in memory, and this is still true. By default, Redis is configured to persist the AOF to disk asynchronously every second, which mean it won’t affect the performance of the user requests.

How do we recover data that has not been persisted to the disk yet? We don’t. This data is lost. This is the first pitfall. When using Redis, we should know that we might lose up to one second of data.

It’s important to note that we can configure Redis to persist the AOF synchronously after every operation. This is not recommended and will make the database considerably slower as it will cause every write to go to the disk. You can check this benchmark and reproduce it yourself to understand how impactful it is.

Scaling Model

Redis isn’t simple only in its user interface but also in its internals, which include its scaling model. Do you need to support more read requests? You can spin up replica instances that will replicate the data asynchronously and serve reads.

It gets a bit more limited when we talk about write operations. Replicas won’t help in that case as the write operations will always go to the main instance. If your bottleneck is with write operations or memory usage (remember that all the data sits in RAM?), you can use Redis in cluster mode. This means that you spin up multiple instances and each one takes responsibility over a subset of the data based on the keys. Every key and its data reside on a single instance (and its read-replicas) and all write operations against that key will be redirected to that instance.

Assuming our write operations are uniformly distributed between all the keys, they will have uniform distribution between the cluster instances as well. If a single instance could handle 1 million write operations per second, we could use 5 instances to handle 5 million write operations per second across the cluster. If the distribution isn’t uniform, but instead most of the write operations are against a single key, all of those will be redirected to the same instance, which is still limited to only 1 million operations per second. There is no way to increase the throughput of writes to a single key. If a single instance can’t handle the number of write requests for a single key, there is no way to scale it out using Redis.

Furthermore, if we are out of memory in a single instance, we can use cluster mode to divide the data between multiple instances. But even if a single key weighs so much that it fills the memory by itself, it will always stay in the same instance. This is true for simple values as well as complex ones (lists, sets, etc.). If we want to allow Redis to scale well, we need to make sure our data is distributed well between keys. Don’t get confused because Redis supports data structures as values: it is still a key-value store. Gigantic hashes, lists and sets are something that we should avoid at all cost.

Use Cases

To sum things up, Redis is really fast and simple to use. Its data structure mechanism is unique and fits into a lot of use cases. But we should use it only if:

We can tolerate loss of data

We won’t need to scale the writes to a single key

Every key’s value is fairly small

If these points align with your use case, go ahead, consider Redis. It’s a great product! Otherwise, you could revise your data model to fit into Redis’s limitations or look at the many other databases out there.

For example, we might want to avoid using Redis to keep track of account balances in a banking system as it can’t tolerate data loss. We would also skip it if we need a database to handle large media files uploaded by users.

One of the classic use cases for Redis is a cache over another persistent database. That way, we can utilize the great performance of Redis to increase the read throughput of the system and the persistency of the on-disk database to recover from failures in the cache.

Summary

Most of the things that are written in this post aren’t unique for Redis. When we evaluate a database for our system, we should look at its persistency level and scaling model and make sure that it copes with our use case. It doesn’t mean we should always look for full persistency and perfectly linear scaling model. Everything is about trade-offs; almost every feature has a drawback in another part of the system. So, understand your use-case, understand its requirements, and choose the database the fulfills those requirements with the fewest drawbacks you can find.