Raft Consensus Protocol Implementations
By: Eileen Pangu
Tags: Consensus Algorithm, Big Data, Distributed Systems, Tech, Software Development

Related posts:

Raft Consensus Protocol Made Simpler [link] Raft Cluster Membership Change Protocol [link] Raft Consensus Protocol Implementations [this one]

In my previous Raft blog posts, I’ve promised to talk about Raft implementations. Now I’m fulfilling that promise. I want to talk about two implementations today. One is my own demo implementation. I believe if you can’t implement it nicely, you don’t understand it well enough. I want to go through some of the internals of that demo implementation to show you how to structure a consensus protocol in a toy distributed system. The other is etcd’s implementation. It’s widely adopted by many production systems. I want to briefly touch on its interfaces. One last thing before we dive in: you may want to read my previous blog posts (listed above) to get a better understanding of Raft. This is not a plug, but rather a genuine recommendation since without a solid understanding of the basic concept, this implementation part won’t make much sense.

Demo Implementation

The full repo is available at https://github.com/eileen-code4fun/SimpleRaft. It’s coded in Golang. In that implementation, I created an in-memory single machine implementation of Raft. Nodes can be run as Go routines (you can think of them as light-weight threads if you’re not a Go person). They talk to each other via in-memory Go channels, which are thread-safe async queues provided by Golang as a language primitive. With Go channels, threads can concurrently enqueue and dequeue messages. One interesting note is that a node has a single inbound Go channel. All the requests — AppendEntry, RequestVote, AddServer, RemoveServer, ClientRequest — implement the same request interface and can therefore be sent to that channel in a multiplexed way. A switch-case fetches the requests and dispatches them to the corresponding handlers. The request interface looks like the followings:

type request interface { getResponseChannel() chan response }

Another noteworthy point is that each request contains a response channel that the node can directly respond to. This model simulates the synchronous RPC request and response without introducing an monolithic outbound channel. I only defined one concrete response structure since the data returned by those RPCs are pretty much of the same schema.

In Diego’s Raft dissertation, he proposed a thread-based model for a node’s internal working. There are service threads, peer threads, timer threads, log threads etc. They all share the same lock-protected state. While that works, it greatly complicates the development. As we all know, concurrency is hard. I instead adopted Redis’s design philosophy, and coded the logic in a single-threaded event loop. If you’re interested in how Redis works internally, I have a Redis Internal Tricks post [link] you can check out. So, the backbone of the logic is now simplified to a for-loop:

for isRunning() { select { case req := <- inboundQueue: // Process request. case <-time.After(heartbeatTimeout): // Broadcasts heartbeats if it’s leader. // Starts election countdown if it’s follower. // Campaign if it’s candidate. } }

By the way, select-case is the idiomatic way to do event loop in Golang.

As for storage, even though it’s all in memory, some Raft variables are durable (such as logs, past votes); others are volatile (such as committed index, leader ID). I have a start / kill method pair that will reset the volatile variables but keep the durable variables intact. They have great use in tests.

The whole implementation, including comments and blank lines are just a bit over 500 lines, which are quite nice given the complex nature of distributed consensus. I also added thorough tests that cover various edge cases and a large simulation test that keeps sending the cluster client requests while simultaneously randomly killing and restarting nodes (including the leader node). Feel free to check it out and play around it yourselves. Be sure to send me Github issues if you discover bugs.

etcd’s Implementation Interface

etcd has a production grade implementation in https://github.com/etcd-io/etcd/tree/master/raft. Other implementations tend to have a monolithic design (even my own demo implementation), ectd’s Raft implementation, on the contrary, adopts a minimalist approach. It implements the core Raft consensus algorithm, while leaving the storage layer and the transport layer to the integrators to fill. Even the core part itself doesn’t run automatically. The integrators have to explicitly trigger the “tick”. “Tick” is its internal logical clock. Timeouts such as heartbeat and election are all in units of “tick”. Using etcd’s Raft interface will look like the following:

ticker := time.NewTicker(100 * time.Millisecond) for { select { case <-ticker.C: node.Tick() case rd := <-node.Ready() // Process rd. // node.Advance() } }

node.Ready() returns all the relevant information to the caller. That includes durable states (etcd’s Raft calls it the hard states), new log entries, committed log entries, snapshot, messages to be sent out by caller, etc. The caller is responsible for persisting the hard states, appending the new log entries to its local log storage, backing up the snapshot, sending out the messages to the node’s peers, and any other custom processing. When these are all done, the caller needs to call node.Advance() to let node know that the previous ready information is all taken care of. The node can then move on to the next stage.

Ending

I think I’m all done with Raft now. I won’t make any more promise about writing future blog posts about it. But the learning should never stop. I encourage you to explore more and write about it yourselves.