basho/riak_cs — pull request #723 by shino (2014-02-26 06:50:30+00:00)
Title: Multi-cluster support

Currently all Riak CS's internal data types are stored into single
Riak cluster. The idea is to distribute the keys among clusters
according to the data types.

From client, it looks single riak cs system as it is currently.

## Data Structure Change

Add **clusterid** entry into each bucket record, which indicates in
which cluster the object manifest is stored. Add (or Reuse)
**clusterid** entry into each object manifest, which indicates in
which cluster the blocks are stored.

This enables Riak CS to store object manifests and blocks into
separate clusters.

## How to choose cluster to store blocks

When PUT request came from users, CS node gets the bucket record
and retrieves clusterid to store the object manifest.

(Beforehand, when bucket being created Riak CS chooses the cluster to
store object manifests by the same logic from choosing cluster to
store the blocks => open question?)

Then Riak CS puts the object manifest into the cluster indicated by
the clusterid. Inside the object manifest, there are another clusterid
to put blocks, decided by the following manner:

- get estimated free space of each Riak cluster(*)
- filter out the cluster which doesn't have enough free space to store more objects
- probabilistically choose a Riak cluster depending on the amount of free space

Then kick a riak_cs_block_server.

(*) "free space of Riak cluster" can be defined in several ways.
One candidate is as follows:
- For each Riak node, decide `free percentage` from `disksup:get_disk_data()` by using the partition containing `paltform_data_dir`.
- For cluster's `free percentage`, take `minimum` of each node's `free percentage`.
Taking `minimum` rather than `average` is for safety to avoid disk full.