influxdb/influxdb — pull request #658 by pauldix (2014-06-17 22:52:41+00:00)
Title: Make wal per shard with repairs

This is just a start to this conversation about how we're going to restructure things with the WAL. The current implementation doesn't allow for backpressure on the clients. This means that under very high load situations servers can just fill up their disks writing WALs and everything will come to a screeching halt.

There's another problem where certain failure scenarios can cause an inconsistent state. Moving the WAL for each shard out to the servers that own the shard along with regular repairs in their replication group will help this.

Here are some ideas around initial TODOs
- [ ] Remove write_buffer
- [ ] Originating server should no longer log writes or assign RequestNumber, SequenceNumber or Timestamps
- [ ] Move WAL from the originating server to the replication group
- [ ] Originating server should proxy the write to any one of the servers in the replication group that own the shard
- [ ] The server proxied to should log the write to its WAL and write to LocalStorage and return success back to the originating server (can add quorum writes later)
- [ ] There should be a WAL per shard and the server proxied to will log the write request marking its server id and a monotonically increasing request number
- [ ] The server proxied to should attempt to replicate the write (with the origin server id and request number) to others in the replication group
- [ ] The other servers in the replication group that receive the write should log the request to their WAL for this shard, write to LocalStorage and return a success
- [ ] Every minute (user configurable) servers should ask others in their replication group for the last request seen (from 10-12 seconds ago (user configurable)) from every other server in the group. The reason for the 10-12 second old number is to make sure we don't try to replay data in flight
- [ ] If there is missing data it should get replayed to the server that asked for the read repair
- [ ] On startup, a server should ask all other servers in the replication group for data they missed since they were down (from some last known request number).
- [ ] If a server receives a request number with a gap in the sequence, it should ask for a replay from the server sending it
- [ ] We need to handle request number resets (server restart) or rollovers