Trade-offs in distributed system (part 1)
Tags: Distributed Systems, Strong Consistency, System Design, Eventual Consistency

Motivation of this post

This post is intended for software engineers seeking deeper understanding of distributed system design. After reading this post and part 2, you should feel comfortable about this question:

What do we talk about when we talk about distributed systems?

We solve problems of the distributed system design everyday, like scheduling dinner with a group of friends. However, the literature is often intimidating with ambiguous concepts and vocabulary. Many concepts, like the CAP theorem, have specific meanings in history but are now widely misused.

In this post, I will try to reconcile and clarify the related concepts from database community, distributed system community and practical engineering vocabulary. The concepts are explained by the contextual problem they are created for. Having concise vocabulary for abstract concepts is critical for understanding the problem we are trying to solve. More often than not, once the problems are understood, the solutions are very intuitive.

The second part of this series consists of practical trade-offs described in the precise vocabulary.

Core concepts and vocabulary

Network Partition

Network partition is the least ambiguous concept. Partition means the hosts consisting of the distributed system are connected by an unreliable network. The connection between any two hosts will randomly fail.

Network partition is not some property our design can trade-off with. It’s the nature of any distributed system.

Serializability

Serializability means each concurrent operation(transaction) on the system must be able to execute as if it’s the only operation. Put it another way, when a set of operations(e.x. <O1, O2, O3, O4, O5>) are processed in parallel, the end result must be equal to some sequential(non-parallel) execution of the set. The sequence can be arbitrary, for example <O2, O1, O3, O5, O4>. Serializability is very close to isolation in AC(I)D from database literatures.

While in theory it’s possible to trade serializability for higher concurrency/throughput, I don’t know anyone who has done that in practice. Because that means often and permanently accepting wrong data. For example, x++ in most system is implemented as a transactions consisting of two commands:

t := x // read

x := t + 1 // write

Assume x begins with 0 and you execute x++ for five times, serializability guarantees that the value of x is 5 after all executions are finished. Notice that if you execute read concurrently with the five x++, the read result could be any value between 0 to 5. The behavior of concurrent events are not defined in any consistency models, not only strict serializability.

CAP-Availability

Availability in CAP means under network partition, any functioning host in the distributed system is able to process all valid requests from clients. For example, if Amazon’s order processing servers live in multiple servers, once your app is connected to one server, you should be able to buy from Amazon. Even if the server you connected to can’t connect to DB itself.

CAP-availability is directly meaningful for practical systems because we care about the availability provided to clients, not the behavior of our internal machines. Amazon can’t guarantee CAP-availability even their system is (correctly) regarded as ‘highly available’. When we think about availability in system design, we are in fact concerned about SLA-availability in below.

SLA-Availability

SLA-Availability means the percentage of valid requests can be served(for example, Cloud Spanner offers 99.999% availability). This is what we mean when making trade-offs in practical system design. Most user facing systems require very high availability because the alternative means immediately losing money and bad customer experience(also leads to losing money in the long run).

Latency

Latency(e.x. 10ms) is the time needed to process each request. Low latency is a very desirable property, sometimes a necessary feature for a practical system. Low latency is often one of the most important goals of many design trade-offs. Any user facing read requests should have end to end latency less than 1 second.

Latency SLA-availability are usually on the same trade-off side in that both are easier to achieve with more data replications.

Linearizability(CAP-Consistency)

Here begins the meaty parts of this post.

Linearizability is defined in a world where all operations(read/write on a register) are atomic. Operations happen instantaneously and the time of happening is a unique timestamp in the real world. No two operations share the same timestamp(concurrent operations).

Therefore, the operations in a system can be ordered chronologically by their timestamps. Linearizability requires that the ‘effect’ of the events must be observed in the same order. For example, if the events in a system can be ordered by their timestamp as below(R means read the value from register, W(x) means set the value to x)

<W(1), R, R, W(2), R, R>

Their corresponding results must be:

<null, 1, 1, null, 2, 2>

With replication, linearizability is hard to maintain. For example, the implementation of W(2) is not atomic because we need physical time to broadcast the new value(2) to all replicas. So the R after W(2) could return 1. Because the broadcasting is a process, it’s possible that the first R after W(2) returns 2, but the last R reads from a stale replica and returns 1(non monotonic read).

There are more bizarre ways of breaking linearizability but the above are the two most common concerns in distributed systems, namely not being able to ‘read my write’ and ‘monotonic read’.

Linearizability is what consistency means in CAP. CAP’s main point is that with network partition, linearizability and CAP-availability can’t both be satisfied. In fact, linearizability is very hard in distributed systems even if we don’t demand CAP-availability and low latency. Very sophisticated algorithms like Paxos and Raft are designed to provide strict serializability(see below) at the cost of latency and throughput.

Strict Serializable(Serializable + Linearizable)

Linearizability assumes all operations finish instantaneously. This is obviously not a reasonable assumption for the majority of software systems. Strict serializable extends linearizable and defines the behavior of operations that are not instantaneous.

In this world, each operation has a start time and end time and can be represented as [Os, Oe].

Concurrent operations

O1 and O2 are concurrent if

O1s <= O2s <= O1e || O2s <= O1e <= O2e

The strict serializable requirement for any concurrent operations are the same as serializable. It means their executions need to be isolated for each other. It doesn’t require any order for O1 and O2 to take effect.

Because concurrent relation is not transitive, undefined behavior of concurrent operations doesn’t make application programming harder.

Sequential operations

If O1 and O2 are not concurrent, they are sequential. By definition, they can be ordered by their finish time. Thus such operations can be seen as atomic(finishes instantaneously) when reasoning with each other. Strict serializable requires sequential operations to be linearizable under this atomic perspective.

In essence, strict serializable means:

All operations must be serializable

Sequential operations must be linearizable

The order of concurrent operations is undefined

Some literature use ‘strict serializability’, ‘linearizability’ and ‘strong consistency’ interchangeably. Google also coined an alias(external consistency) for strict serializable in their Spanner papers.

Further reading

[1] Please stop calling databases CP or AP is a good post explaining what does CAP really means and how does each popular database fare.

[2] Linearizability V.S. Serializability is a nice writing about those two concepts.

[3] Consistency models. Read it if you want to know all the bizarre ways to break consistency.

[4] Consistency models from CockroachDB is a very gentle introduction to same topics of my post.