coreos/etcd — pull request #874 by bmizerany (2014-07-02 20:54:12+00:00)
Title: replace goraft

We - the etcd team - are working on a replacing goraft with a new raft implementation intended to live in the etcd codebase under etcd/raft. Our reasons for this are as follows:

After discussions with users and maintainers of the goraft library we decided that there were some core things that could only be achieved by starting fresh with a new API and tighter codebase:

* Simplicity - Currently the implementation we use has a large surface area and tangles "core raft" with other essential pieces like transports, storage, and timing.

Our proposed implementation abstracts the store, timing, and network away and leaves us with with a simple state-machine that can be used by other packages that implement their own transport and storage. Implementing a transport is an easier task when it isn't intertwined with the Raft state-machine.

The proposed implementation takes a new approach. It assumes messages can come out of order, so transports need not rely on careful bookkeeping. The new etcd/raft also avoids having a concept of a transport - it simply queues outgoing messages, and committable entires for the transport and storage systems to dequeue and save/send when ready.

* Testability - Proving the stability of any codebase is hard. But, it is easier with smaller interfaces between parts of the code.

By removing the dependency of wall-clock time we can write isolated, wait-free, and race-free unit tests that run in an amount of time measured in milliseconds.

Removing the knowledge of an explicit network allows us to - amongst other things - emulate network partitions and perform other types of fuzzing to prove correctness right in our unit tests as opposed to larger systems that require complicated test harnesess and iptables manipulation. That isn't to say the latter method isn't necessary for a complete picture, but being able to emulate most of the chaos you encounter in a real network in our unit tests allows us to catch regressions much, much faster.

Since Raft is so essential to etcd we felt that investing time and energy into making a better library with these two goals in mind is worth the effort. Please review the PR if you have the chance. We intend for this package to be useful for other projects, too. Also, if you have the need for a simple, pure Go Raft package in your application, please take a look.

I also want to give a huge thank you to Ben and Xiang who put in a ton of work to make github.com/goraft/raft happen. It has gotten etcd a long way. That said they both feel having a few implementations of such an important building block is OK and that new cleaner APIs are possible.