influxdb/influxdb — pull request #903 by benbjohnson (2014-09-03 22:21:57+00:00)
Title: WIP: Streaming Raft

## Overview ![status](http://img.shields.io/%20/work%20in%20progress.png?color=red)

This pull request adds a high performance, streaming implementation of the [Raft protocol](http://ramcloud.stanford.edu/raft.pdf). This implementation will be used to implement a distributed write-ahead log that will be fanned out to the rest of the nodes in the cluster.

## Performance Target

The performance target for streaming Raft is to be able to replicate 2 million small messages per second in a 3 node cluster. InfluxDB uses small messages so the payload size will be 32 bytes for our tests.

### Encoding & decoding performance

The following benchmark was run against b4acaaf:

```sh
$ go test -bench=.
BenchmarkLogEntryEncoderEncode_8b	20000000	114 ns/op	243.99 MB/s	102 B/op	1 allocs/op
BenchmarkLogEntryEncoderEncode_32b	20000000	135 ns/op	382.37 MB/s	179 B/op	1 allocs/op
BenchmarkLogEntryDecoderDecode_8b	20000000	154 ns/op	181.48 MB/s	40 B/op	1 allocs/op
BenchmarkLogEntryDecoderDecode_32b	20000000	172 ns/op	302.01 MB/s	64 B/op	2 allocs/op
```

At 32 bytes per message, we are able to encode 7.4M msg/sec and decode 5.8M msg/sec. Further optimizations may be necessary but this is a good start.

## TODO

This pull request is a work in progress. The following needs to be completed:

- [x] Package doc
- [x] LogEntryEncoder
- [x] LogEntryDecoder
- [ ] Log Heartbeat
- [ ] Log Entries
- [ ] Log RequestVote
- [ ] HTTPTransport
- [ ] HTTPHandler
- [ ] TCPTransport
- [ ] TCPListener
- [ ] Simulation testing
- [ ] Benchmarking