basho/riak_kv — pull request #643 by evanmcc (2013-08-27 21:03:36+00:00)
Title: Add option for smaller bitcask keys

The kv bucket/key to bitcask key conversion uses term_to_binary. This results in 13 bytes of overhead per key.

This patch introduces a new key format:
```
|--|---bucket---|---key--|
^---- bucket_len
```
The first two bytes specify the length of the bucket, allowing kv to translate the compact binary back to the bucket/key pair that it needs. In order to read old-format values off of disk, this code allows the insertion of a translation function, so that keys in the keydir are in a consistent format. This saves 11 bytes per entry in the keydir.

When a large percentage of your keys on disk are in the old format, there is a small but noticable slowdown in startup time and fold time. This should go down over time as keys are replaced. If the slowdown is intolerable, adding `{small_keys, false}` to the bitcask section of your app.config will revert to the old format. This feature is currently on by default; whether this should be the case is up for debate.

This PR depends on a change in bitcask adding support for key transformation functions:
https://github.com/basho/bitcask/pull/103