Reliability, Scalability, Maintainability
By: Tony Mu
Tags: System Design, Distributed Systems, System Design Interview, Data Intensive System, Software Engineering

This is the first part of a series where I jot down notes and write synopsis for each chapter of Designing Data Intensive Applications (the “hog” book). The reason why I am doing this is because of a blog post I read about how to read better. The plan is to write one or more posts for each chapter. That being said, knowing myself I will probably get distracted and abandon this project after only a few chapters, but we will see how far this leads us :)

Some background on why I chose this book: I was in a middle of a “System Design” interviews (those of you who had a couple of years of experience as a Software Engineer must know what I’m talking about!), and I found myself blanking out on the questions like “design a web scraper” or “design Google Doc”. What I managed to do was merely draw the high-level diagram of what components we need for the system to work. After that I would be out of words, no matter how many hints the interviewer kindly provided me — I cannot dive deep. One question standing out was during designing Google Docs during an interview, I was asked “how do you handle multiple people editing the same doc, and potentially the same place, and therefore conflicts?” I only came up with a solution of locking, which is not the optimal solution for an application like Google Docs.

This made me realized designing systems is a completely different beast from your daily work, and that I need to dedicate time to fully prepare for an interview format like this. Thus I decided to take some initiative, and hopefully finish and absorb this book that was recommended by multiple engineer friends of mine.

Let’s dive into it!

Photo by Tanner Boriack on Unsplash

Thinking about Data Intensive Systems

Data-intensive vs compute-intensive

Many applications have problems on the amount of data, the complexity of data, and the speed of data changing, as opposed to CPU power.

Reliability

The system should continue to perform correctly in face of adversity (hardware or software faults, human errors, etc.)

We shouldn’t confuse the term “fault-tolerant” as the system can tolerate any kind of fault. For example, most systems will not function properly when a blackhole swallows the earth. The term more precisely refer to “resilient”.

Fault isn’t the same as failure. Fault means one component of the system deviating from its spec, while failure is defined as the whole system stops working.

Sometimes, we want to increase the rate of fault by triggering them deliberately. Netflix’s Chaos Monkey does exactly this — it randomly terminates instances in production to ensure the services are implemented in a resilient way. Since most critical bugs are due to poor error handling, doing this can ensure the system is fault-resilient.

Hardware faults can be dealt with hardware redundancy. However, more applications are using more and more hardware as they grow, so the rate of hardware faults has been increasing. It is also common for some VM instances to fail silently in some cloud platforms (AWS). Hence there is a move towards system that can tolerate the loss of entire machines, by using software tolerance techniques. This also enables rolling-upgrade.

Software errors can cause cascading failures. Best prevented by using a sandbox when developing and deploying code.

Human errors are the leading cause of system faults. We can mitigate this by the following

Design systems that minimize opportunity for error. Design abstractions, interfaces, API well.

Decouple frequent errors from critical code that could cause failure

Test thoroughly

Quick and easy recovery

Detailed and clearing monitoring

Scalability

There should be reasonable way to deal with system growth.

We wouldn’t say “X is scalable” or “Y doesn’t scale”. Instead we should consider questions like “if the system grows in a certain way, what are our options to cope with the growth?” or “how can we add computing resources to handle the load?”

How to describe load? Use load parameters. It is different depend on the architecture of your system: i.e., QPS to a web server, RW ratio to a database, the number of simultaneous connections in a chat room, etc.

An example: Twitter timeline. Two approaches:

Using joins to search for tweets from people I follow, in a global collection of tweets

Each user has a timeline cache. When a new tweet is posted, insert into my timeline cache.

Twitter original used first approach but struggled to keep up with the number of timeline requests, so they switched to approach 2. However hot users tweets generate lots of cache writes. So they used a mixture of both.

How to describe performance? We can look at it in two ways:

Increase load parameter and keep system resources the same, how is performance impacted?

Increase load parameter and how much do I need to increase the resource, to keep the performance unchanged?

We use percentiles to describe performance. Tail latencies (high percentiles of response times) are important, because customers with slowest requests are often those with the most data, and thus the most valuable — Amazon shopper with long order history means they shopped much on Amazon. Amazon also discovered a 100s increase in response time reduces sales by 1%, and 1-second slowdown decrease customer satisfaction rate by 16%.

It only takes a small number of slow requests to slow down subsequent requests — head-of-file blocking.

Coping with load. Scaling up vs scaling out. Good architectures usually involve a mixture of both: using several fairly powerful machine could be simpler and cheaper than a large number of small VMs

Elastic systems can automatically add computing resources when a load increase is detected. Can be useful when load is highly unpredictable, but manual scaled systems are simpler and fewer operational surprises.

The architecture of a system is usually highly specific to the application — no one-size-fits-all, or magic scaling sauce. The architecture that scales well for a system is built around assumptions of which operations will be common and which will be rare. However, they are still built upon common building blocks and arranged in familiar patterns.

Maintainability

People should be able to work on the system “productively” over time, to both maintain current behavior and adapt the system to new use cases.

Operability, Simplicity, Evolvability.

“Good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations.”

One of the best tools for battling complexity is abstraction. For example, when programming in in high-level language, we are still programming in machine code, just not writing it directly.

Evolvability: Agile, TDD, refactoring