basho/riak_ensemble — pull request #21 by jtuple (2014-05-13 03:00:46+00:00)
Title: Optimize handling of large ensemble count

This pull-request optimizes `riak_ensemble` to better handle a large number of ensembles. Prior to this pull-request, having more than 16 ensembles on a slow disk or 128 ensembles on a fast disk (eg. SSD) would lead to `riak_ensemble` overwhelming the I/O subsystem and using 100% CPU (as documented in #15).

This pull-request optimizes both how `riak_ensemble` writes to disk as well as reduces the number of messages sent by running ensemble leaders (the next bottleneck after fixing the disk related issues). After the changes in this pull-request, a single node can easily handle 2048 independent ensembles without issue.

This pull-request consists of a chain of logical commits. Each commit explains in detail a particular scalability issue and how that issue was resolved. Thus, see each commit for more details. Likewise, this pull-request is easiest to review commit-by-commit.

Resolves issue #15