basho/riak_kv — pull request #925 by engelsanchez (2014-04-23 00:23:20+00:00)
Title: Add Bitcask tombstone2 upgrade procedure

With the new tombstone changes in Bitcask (basho/bitcask#156), legacy tombtones will only be
removed on a merge when all files preceding the file containing the
tombstone has been merged already. Anticipating that many customers are
sensitive to disk not being reclaimed quickly enough, this change
triggers a migration that merges all Bitcask data files so that all the
legacy tombstones are completely removed.

At startup, the backend looks at the Bitcask app version and a version
written in the data directory, in version.txt. If the pair require an
upgrade, an upgrade file is written to the directory and a regular check
is scheduled to periodically check on its progress. When it determines
it has finished, the new version file is written and the update file
deleted.

Upgrading to Bitcask 1.7.0 schedules a merge for all existing files by
writing a merge.txt with the list of files in the data directory.
Bitcask's need to merge (starting in 1.7.0) will read that and use the
list for merging, deleting the file when all files have been merged.
The file contains empty lines every 5 lines, so that merges happen to at
most 5 files at a time.

Also:

Remove old migration code from when Bitcask directories were
timestamped.

Add a cuttlefish configuration parameter for the Bitcask merge check
interval.

Add jitter to bitcask merge check interval to avoid lockstep merging.
bitcask.merge_check_jitter is given as a percentage of the merge check
interval. So if bitcask.merge_check_interval is set to 10m and
bitcask.merge_check_jitter is set to 30%, merges will be spaced between
7m and 13m apart.

Add bitcask.max_merge_size config to cap the total size of the input
files to merge. This can help further throttle a complete merge.
Setting it to 10GB will make sure each merge doesn't read more than that
amount of data. Setting it too low may prevent all merges!

Skip merge check if KV service is not yet up. We've had problems
recently with merges slowing things down when Riak is just starting up
and having bad side effects.