coreos/etcd — pull request #572 by benbjohnson (2014-02-14 23:50:34+00:00)
Title: WIP(Proxies)

## Overview
> **This is a work-in-progress. Please do not merge!**

This pull request includes some basics for proxying nodes. I originally tried to add proxying to the Raft library but it made it far too complicated and it's not actually part of Raft. I ended up externalizing the committed entries through an event listener on the Raft server instead and that seems to remove some complexity.

Feedback welcome!

## How it works

The idea of the new API is that any nodes that join after the cluster is full are created as proxies. These proxies function as dumb replicas and are not part of the Raft quorum. These proxies also work as standby nodes so that if a node inside the actual Raft cluster (these are referred to as "peers") is inaccessible for too long then the cluster will swap out a proxy and move the dead node to be a peer.

### Joining the Cluster

When a node joins after the cluster is full then a proxy flag is returned after the join index. The return value from the `JoinCommand` is a non-standard uvarint so I'm appending the proxy flag afterward as a single byte (`0=proxy`, `1=peer`).

After a node is returned a proxy flag then it marks its `PeerServer.isProxy` as true and runs `PeerServer.startProxy()`.

### Proxy Management & The Committed Log

The `startProxy()` function kicks off a goroutine which randomly chooses a node in the cluster and begins to tail its committed log. The committed log is a new endpoint which simply writes out all committed Raft entries from a given index. This is just a single open stream that gets flushed after each entry.

The proxy goroutine receives these entries and runs `Command.Apply()` on them with a fake Raft context. This updates the state machine in the same way as a regular Raft peer.

### Scaling

Originally I was going to try for a fan out algorithm to determine which proxy should tail from which node but that gets complicated if nodes start to go down. Instead I opted for randomized attempts to find a peer node to tail. This solves a few problems:

1. If a peer goes down then proxies can find a new peer to tail.

2. It gives peers the power to reject proxies if there are too many tailing.

3. It allows proxies to follow other proxies so we can have a replication hierarchy.

### TODO

Here the list of work to do. Checked items are complete.

- [x] Update the Registry to handle peers and proxies.
- [x] Automatically add proxies that join after the cluster is full.
- [x] Add Commit event type to Raft.
- [x] Write committed entries to the committed log endpoint.
- [ ] Handle snapshots in the committed log endpoint.
- [ ] Prepend current cluster configuration to committed log endpoint.
- [ ] Proxy promotion
- [ ] Replication from proxies

/cc: @philips @xiangli-cmu