basho/riak_kv — pull request #526 by jrwest (2013-04-01 21:57:45+00:00)
Title: Expose Backend Size to Handoff for Progress Tracking

This PR exposes the necessary information for tracking handoff progress (added in https://github.com/basho/riak_core/pull/290) of kv_vnodes using the memory, level, or bitcask backends. Each backend exposes an upper-bound on the amount of data stored, but each one comes with some caveats. Although these caveats are inconvenient, the metrics still provide a significantly better indication of progress than having no indication. The API has been designed so more accurate (or less accurate even) metrics can be used, in the future, instead.

The bitcask backend uses the existing knowledge of keydir size to expose the total number of keys that will be transferred. This is an ok but not great indicator of progress because it doesn't account for varying object sizes (transferring 99 1k objects and one 20MB object).

The eleveldb backend uses the support added in https://github.com/basho/leveldb/pull/74 to directly query the number of bytes in sst files. This has the downside that the returned number may include overwritten keys whose bytes will not be transferred. However, this still means that we will always have made equal or more progress than what is indicated, never less, which is sufficient. If https://github.com/basho/leveldb/pull/74 is not used, tracking handoff progress in eleveldb is not supported.

The memory backend uses built in stats info in ets to return the number of keys that will be transferred. This also has the same drawback as the bitcask backend. In addtion, since the ets fold is not a snapshot, the backend uses the dynamic size feature to make sure progress is tracked correctly.

The multi backend is not supported.

Example output of `riak-admin transfers` with these changes: https://gist.github.com/jrwest/588d85bbeaebaf940e59

This PR also completes the move of `transfers/1` to `riak_core_console` started in https://github.com/basho/riak_core/pull/290