Raft Consensus Protocol Made Simpler
By: Eileen Pangu
Tags: Tech, Consensus Algorithm, Distributed Systems, System Design Interview, Software Development

Related Posts:

Raft Consensus Protocol Made Simpler [this one] Raft Cluster Membership Change Protocol [link] Raft Consensus Protocol Implementations [link]

Background

For the past three decades, Paxos has been the golden standard for distributed consensus protocol. Paxos, seemingly straightforward at the surface, is actually incredibly difficult to reason about especially when you start looking into all kinds of edge case scenarios. With the advent of the big data era, researchers set out to discover a new generation of consensus protocols that are easier to understand and more practical to implement. Among them, Raft came out as one of the major winners and has been widely adopted in many real world systems (such as etcd in Kubernetes). Raft is invented by Diego Ongaro and John Ousterhout at Stanford, and has been thoroughly documented in Diego’s PhD dissertation. They conducted user study to show that Raft was in fact easier to grasp compared to Paxos, which met one of their main design goals — understandability. But make no mistake. Distributed consensus is such a complex topic that no design is going to magically make it trivial. What we can do is to try to study it from different angles over and over again until maybe one day we start to feel comfortable about it. Hence this blog post. I want to provide Raft learners an alternative to Diego’s 200+ pages dissertation. I think my way is simpler. But I’ll let you be the judge.

Raft deals with log replication. Intuitively, if you manage to get the exact same log sequence appear on a majority of machines, then you’ve managed to make that cluster of machines agree on something. Indeed, this is called total order broadcast. If you can get every entry of the log to show up in the same position among a cluster of machines, you can use that to implement consensus. For example, if one machine wants to propose a value and let’s assume its last log entry is at position i, it can try to get other machines to put this new value at i+1. After the majority of the machines in the cluster have replicated that value at i+1, we now call that value committed at i+1. That’s effectively the same as proposing a value and getting it accepted by others in Paxos’ terms.

Raft is a leader-based protocol. In its normal course of operation, one and only one leader will be elected by the cluster of nodes. The other nodes are followers. The leader takes client write requests and replicates them to the followers. It’s as simple as that. All the intricacy, however, arises when the leader fails, which I’ll get to in this blog post.

Simplification Attempts

In this blog post, I’m going to make three simplifications to the Raft protocol. They don’t affect the core algorithm, but they make a lot of things simpler to understand.

Raft talks about nodes in the cluster as state machines, and the log sequence is a series of commands to be passed to the state machines for execution. That has the generality of research, but it complicates the story. I’m going to simplify it and say we’re only concerned with getting the log sequence safely replicated in a majority of nodes, and don’t worry about the state machine part. This does not affect the protocol as the state machine execution is just something each node can do locally after the log sequence is correctly replicated on that node. Raft talks about the concept of committed index in the log sequence. I’ll first ignore this and come back later on it. Without it, nodes don’t know how much of the log sequence is safely replicated. They only know that it will be eventually replicated. This violates the liveliness requirement of a consensus protocol. But as we shall see later, the committed index is just a volatile state in each node, and they can get that easily from the leader of the cluster. The leader itself has a way to identify the committed index. Ignoring the committed index initially smooths the learning curve. Raft defines an AppendEntries RPC for the leader to send log entries to followers. I’m going to restrict it to only contain one entry at a time (i.e the singular AppendEntry). It’ll be fairly straightforward to see in this blog post that it does not affect the core of the protocol. It only makes it slower for lagging followers to catch up because now they can receive only one entry at a time from the leader.

Normal Course of Operation

Easy things first. I’ve alluded to this in the background section. To expand on the normal course of operation:

The leader accepts client write requests one at a time (requests may come in concurrently, but from the log perspective they will be serialized).

The leader first appends the write request to its own durable log sequence.

The leader then sends one AppendEntry RPC to each of its followers to instruct them to also append this write request to their log sequences.

Upon receiving the AppendEntry RPC, followers append that to their own log sequences, and then respond success to the leader. There will be some checks to decide whether to accept that entry but it’s only useful in failure scenarios, which I’ll get to later.

After getting success responses from a majority (n/2 + 1) of the nodes, the leader responds success to the client. At this point, the write request is safely replicated in the cluster.

Normal Course of Operation

The total number of nodes in the cluster is a static config known to all nodes. So the leader and frankly everyone else in the cluster can calculate the number needed for a majority. Changing membership (adding or removing nodes) will be covered in future blog posts. If for some reason, the leader doesn’t hear back from a follower for the AppendEntry RPC (follower crash or network interruption), the leader will retry that forever in the background. Since the leader only waits for success responses from a majority of the nodes, minority failures won’t block the system. Failed AppendEntry RPC will be retried in the same order as the log sequence. Lagging followers will eventually catch up and replicate the leader’s log sequence. That’s all there is to it. Life is so simple without those uncommon but hazardous complications.

Leader Election

It’s getting to deep water now. Leader election is needed in many cases. For example, at startup, the admin may not designate a leader and therefore the cluster needs to elect its leader. Or the leader may crash and come back as a follower, in which case there is again no leader. Or network partition may happen and isolate the leader from the majority nodes. A new leader needs to be elected to keep the cluster functioning but when the old leader comes back, we need to reconcile that.

Here comes the most important concept in Raft: term. Just like the president serves his term, a leader is “scoped” to a term. Term — a monotonically increasing integer in implementation — provides a critical boundary for a leader’s authority and it’s used as a conflict resolution mechanism to avoid the multiple leader problem and old/new leader competition. In a nutshell, leaders/nodes with a higher term number prevail. Let’s look at the details. By the way, the concept of term is also present in many other modern leader-based consensus protocols. It seems to be a good design pattern.

First, how does the cluster detect that it needs a leader election? The aforementioned AppendEntry RPC is sent regularly from the leader to followers. Even without a client write request, the AppendEntry RPC is still being sent (just without any entry in it). That serves as a heartbeat. If a follower does not hear the heartbeat from the leader after some configured election timeout, it assumes the leader is dead and starts an election to run for the new leader itself (the follower becomes a candidate so to speak). Random election timeouts will be used to avoid too many followers jumping on nominating themselves all at once, which is likely to lead to split votes and no winner. The protocol does have a mechanism to retry election even if that happens as we shall see later.

The candidate first increments its current term number from t to t+1, and sends out RequestVote RPC to all other nodes. Interjection: so far, I mentioned two RPCs: AppendEntry and RequestVote. In fact, those are the only two RPCs you need to know in Raft. Note that a node’s current term number must be persisted in durable storage. Each receiving node checks if they have already voted for others for term t+1, if not, they return success to this candidate. This ensures each node only votes for one candidate in any given term. Obviously, this information is also required to be stored in durable storage to survive crashes. When the candidate receives success from a majority of the nodes, it converts itself to leader and starts sending out AppendEntry RPC to all other nodes. Other nodes, upon receiving AppendEntry RPC, know that a leader is elected for t+1. If the candidate fails, or there are two candidates that split votes, no one will become the leader. Other nodes will try to run for the leader themselves if they don’t receive AppendEntry RPC after their election timeout.

There is another important element to make this work robustly. Every node will put their current term number in all the RPC requests they send out and all the responses they return. Here is a universal rule: whenever a node receives a term number from another node (either in incoming requests or returned responses), if the term number is larger than the node’s current term number, the node immediately converts itself to a follower regardless of its current state (leader or candidate), and updates its current term to this new higher term. To see why this is crucial consider the following examples:

The leader sends AppendEntry RPC to its followers. The followers will sync up to the leader’s term. In the normal course of operation, the leader’s term number is no less than any follower.

If a leader in term t is segregated from the majority, another leader will be elected for term t+1. This is because a candidate must have incremented its term from t to t+1 to run for the new leader. When the old leader comes back, it tries to talk to the majority and some node is guaranteed to return term t+1 because a majority must have been reached to elect the new leader for term t+1. The old leader can then step down.

If the candidate fails or two candidates split the votes, other nodes are either in term t or term t+1 if they have been contacted by the candidates. One of those nodes in term t may run for the leader for term t+1. It may succeed if the majority of nodes’ current terms are still t. Otherwise only nodes in term t+1 will succeed because they will increment their current term to t+2 to run for the leader.

There is one last bit before we finish the leader election. So far I may have given you the impression that any candidate with the right term number can win the election. That’s not true. Think about this: if a candidate in the right term but with a lagging log sequence wins the new leader status, it’ll start replicating its log sequence to other nodes, effectively overriding others’ log sequences that may have been safely replicated. This breaks the promise to the client since the cluster responded success to those client writes. So we have to make one final rule: a node only votes for a candidate whose log sequence is more complete than its. “More complete” is defined as the last entry of the log coming from a higher term (yes, we do have to record the term number when the log entry is received from the client); if the term numbers are the same, then whichever log sequence being longer. The latter part is easy to understand. — in the normal course of operation, some nodes will follow the leader more closely than others. To understand the former, you’ll have to go back to our term definition and our universal rule. You’ll see that term defines a new era. In some sense it resembles the concept of freshness. Leader election respects the term number, and log entries with a higher term number are essentially “newer”.

Fixing Broken Log Sequence

Log sequence on a node may be broken due to frequent leadership changes, especially when the node itself becomes leader and gets thrown out multiple times. Such as the following figure:

The numbers outside the boxes are log entry indices. The numbers inside the boxes are the corresponding term number of those log entries. (t1) the first node is the leader in term 1 and replicates log at position 1 to all followers. (t2) the first node fails and the second node becomes the leader in term 2, and then fails before it can replicate its new log at position 2. (t3) the third node becomes the leader in term 3 after getting votes from itself and the first node. It fails before it can replicate its new log at position 2. (t4) the first node becomes the leader again in term 4 and it has two new log entries that it needs to replicate to followers.

In the leader election section, we’ve established that the winning leader contains the most complete log sequence. That’s the source of truth that we need to replicate to the entire cluster. Thus the leader is responsible for “resetting” its followers log sequences. The way it does that is by the same AppendEntry RPC. Two extra parameters will be included: previous_entry_index and previous_entry_term. They’re used to find the log sequence prefix that they share with the leader’s log sequence. The end of that log sequence prefix will be the starting point the leader’s log sequence overrides the followers’ log sequences. Note that the leader does not need to send its entire log sequence for comparison. Because of the orderly way log entries are appended and the single leader guarantee provided by term, if a log sequence at position i has matching term with another log sequence at position i, the two log sequences are guaranteed to have a common prefix up to position i. Therefore, AppendEntry only needs to include the extra previous_entry_index and previous_entry_term. If followers respond that they don’t match, the leader only needs to backtrack to i-1 in another round of AppendEntry to continue searching.

The Committed Index

In the simplification attempts section, I promised to return to this. Here we are. Followers receive the committed index from the leader in the AppendEntry. In the normal course of operation, when the leader receives success responses from a majority of nodes for AppendEntry at position i, it knows that position i is safely replicated (committed). And since the leader replicates the log entries in order, if position i is committed, then position i-1 must have been committed before. That’s why the leader only needs to communicate the last committed index and followers will know that their log sequences up to that point are all committed.

There is a caveat in leadership changes though. Consider the following events:

The numbers outside the boxes are log entry indices. The numbers inside the boxes are the corresponding term number of those log entries. (t1) the first node is the leader in term 1 then it gets network-partitioned away before it can replicate log entry at position 2. (t2) the third node becomes the leader in term 2 by getting votes from itself and the second node. It has a new log entry at position 2 but gets network-partitioned away before it can replicate it. (t3) the first node comes back and becomes the leader for term 3. It replicates its log entry at position 2 to the second server. But this log entry is not safe even though it’s currently replicated in two out of three nodes. Because (t4) could happen where the first node fails and the third node comes back to be the new leader in term 4. It’s only safe in the case of (t3’) where a new log entry at position 3 is replicated in a majority of nodes. By then even if the third node comes back it can’t win the election due to the log completeness rule in the leader election section.

Therefore, position i is committed only when it’s replicated in a majority of nodes and its term is the current term.

Closing

I hope that you’ve gained a basic understanding of the Raft consensus protocol by now. I may write a few more follow-up blog posts to talk about things like implementation and cluster membership change. So stay tuned. Hope to see you soon again.