benbjohnson/go-raft — pull request #52 by benbjohnson (2013-07-07 20:35:17+00:00)
Title: Refactor server event loop

@xiangli-cmu I still have clean up to do on this but the test suite is passing now. I haven't run the race detector either so I'll have to do that. I wanted to get this pushed in so you could start reviewing it as a sanity check.

So the basic idea is that the event loop is split out into 3 separate event loops depending on the current state. Those event loops feed off of a single channel (`Server.c`) so that processing of commands, AE/RV requests and responses all happen sequentially. It'll make debugging a hell of a lot easier. The one exception is that there is one goroutine for `Do()` that waits for the commit callback.

I was able to eliminate the `stop` channel and `stepDown` channel since the event loops check for state change after every event.

There are a couple issues that I know need to be addressed still:

1. The server mutexes need to be cleaned up. The term changes need to be wrapped in an `RWLock` since term and log index information is obtained from the peers.

2. I haven't touched anything with snapshots yet. Snapshot requests should be processed inside `leaderLoop()`.

3. The server needs to only allow one loop at a time.

4. The server needs to flush out events after stopping and prevent new events from being added before it is started up again.

5. The `TestServerPromote()` test case is being flakey. Not exactly sure why.

One more thing: there are now two levels of debugging: Trace & Debug (`traceln()` and `debugln()`, respectively). Use `traceln()` for anything lower level that is going to dump a lot of output. Use debug for any infrequent states or error states. Also, on the `Server` there is now a `Server.debugln()` and `Server.traceln()` which will prepend the server name to the output.

Sorry for the massive pull request!! It was a pretty huge change in how everything is processed.