basho/riak_kv — pull request #479 by jrwest (2013-02-07 01:49:53+00:00)
Title: More Compact Riak Object Binary Format

This PR changes how `riak_object`s are serialized before storing them on disk, with the goal of reducing overhead. The current `term_to_binary` approach used results in a significant amount of unnecessary overhead due to the metadata dict, among other things. This is especially true for small objects. It also means we store the bucket and key twice for each object (once as the actual key in the backend, and once in the serialized object). The new format addresses these issues by building its own binary that more compactly stores the metadata dictionary in addition to not including the bucket and key.

To support the new format several other changes were made: `riak_kv_vnode` uses a new capability to determine when the new format is supported and encodes the object for writes using the version negotiated by capabilities. Several sub-systems, e.g. aae, backup, handoff were also changed to account for the new version. The commit notes have specific details on what was changed and why.

Additionally, in order for rolling downgrades to be possible after new objects have been written, `riak_kv_format` is included in this PR with the intention of it being run on a live node before starting the downgrade. A `riak_test` for this can be found [here](https://github.com/basho/riak_test/pull/197). `riak_kv_format` is general enough that it can be used to upgrade the node's data as well.

Finally, this PR includes some general cleanup of the `riak_object` module.

There are several possible improvements, such as some potential binary optimizations, that could be made before this PR is merged or in a separate PR. I figured those were best left to discussion with a reviewer.