basho/riak_core — pull request #301 by jrwest (2013-04-15 23:39:34+00:00)
Title: Ring Resizing

Ring Resizing enables `riak_core` applications to change the number of partitions (running vnodes) during normal operations, under load. Previously, a cluster was limited to always having `ring_creation_size` partitions. In order to change the number of partitions, a seperate cluster would need to be stood up along side the original and the data would be migrated between the two by external means.

The intended purpose of this feature is to support users who create a cluster with either too few or two many partitions and have a need to change this without all the hassle mentioned above. It *is not intended* as a scaling feature for clusters to add or remove
concurrent processing ability. Riak already has support for that via adding/removing nodes. However, since the number of partitions ultimately limits the number of nodes in the cluster, ring resizing can be used to increase capacity in that regard. In short, the feature is
intended for infrequent use in specific scenarios.

This feature is a work-in-progress that will undergo several more iterations. Some deatils on that work as well as issues with the current implementation are discussed below.

**For a more detailed description of the implemention see [here](https://github.com/basho/riak_core/blob/jrw-dynamic-ring/docs/ring-resizing.md).**

## Goals

Besides the obvious goal that the cluster should safely transition between the old and new rings -- ensuring requests for unmodified keys return the same data and new writes are reflected in bolth rings. The implemention was designed with the following goals in mind:

* During the operation requests for existing keys should not return `not_found`
* The minimum number of transfers to ensure safety should be made
* The implementation should be ignorant of the N values used to store keys in Riak
* The operation should be cancelable and support forceful replacement of nodes in the case that SHTF

## The Rest of the Story

There are some caveats and future work that are worth mentioning here.

* **Rehashing**: When the ring expands some existing partitions are not moved to new owners. In this case none of their data is removed. In the future, a "rehashing" operation will be submitted during the cleanup cycle that removes keys no longer owned by these existing partitions.
* **Repl**: replication does not support clusters of different ring sizes. The only available approach that currently works is to disable realtime until bolth cluster have been resized and then restablish the repl link, at which point a full-sync will probably be necessary.
* **Perf. Improvements**: There is a known optimization for how to determine if keys should be transferred that is not included here because it still has some kinks to work out. Performance improvements are also expected thanks to the many improvements coming to handoff/gossip in general.
* **Memory & Storage Considerations**: Like the old method of resizing (using two clusters), to safely transition to the new ring, in the worst possible case, the entire data set is copied (an ugly consequence of not cleaning up data until the new ring is installed). Clusters should have the available disk space (& memory if using bitcask) to account for this.