Allow partial merge tombstone deletions

Until now, Bitcask tombstones were not safe to drop during a merge
because old values for the same key located in files not included in the
merge could make the value resurrect if Bitcask was closed and
re-opened. Resurrections have been a problem with Bitcask from the
beginning.

This introduces a new tombstone that includes the id of the file
containing the data it is deleting. Each tombstoned is paired up with
a file, so during a merge that sees both the original file and the
tombstone pointing it, the tombstone can be safely dropped without
risking resurrection of the data. Merges that see a tombstone but do not
include the original file now append that tombstone to that file, so
that it can be dropped by the next merge that includes that file. For
this to work with keys containing values in multiple files, a tombstone
has to be created every time a value is written for that key in a file
other than the file where the previous version was written.  That way
tombstones remain paired up with each file containing values.

Re-opening a file for append has some unfortunate side effects.
Hintfiles have a last record containing the CRC for its data. To append
more data, we need to remove that record and add it again when the merge
is finished. If a merge crashes without finishing, the re-opened files
will be missing the last record with the CRC, similar to a crash of the
writer process.  If Bitcask is configured to not open files without a
CRC, those files would not be available.

When merging old tombstones without file info, transform them into a
tombstone that contains the id of the containing file such that a later
merge can drop it eventually when all files < id have been merged.
Thought leadership on this brought to you by @jonmeredith

Other related fixes:

Added NIF to update stats properly when writing tombstones

Detect when merge includes a range of files starting with the oldest.
It is always safe to drop tombstones in that case. We take advantage
of this to make it possible to remove legacy tombstones.

When merging a consecutive set of files starting from the first file in
the bitcask directory, it is safe to drop all encountered tombstones.
We now detect this kind of "prefix" merge. Notice a merge containing a
"prefix" of the files plus some other later file should be able to drop
tombstones from files in the prefix, but we don't detect that. That is,
if we had files 2, 3, 4, 5, 6 and merged 2, 3 and 6 we could safely drop
tombstones found in files 2 and 3.

Allow merge input files to be specified by creating a merge.txt file in
the cask directory. bitcask:needs_merge() picks the contents of this
file to merge and will delete merge.txt when all files in it have been
deleted or marked for deletion. Leaving empty lines will delimit
batches, such that needs_merge returns one batch of files at a time.
This would prevent long merges including every single data file.

A cap on merge input size can now be passed to needs_merge, so that it
produces merges in batches. This can help with merging large amounts of
files.

Bumped version to 1.7.0. This will be used in the KV side to prepare a
data migration that schedules a merge of all files.

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch feature/partial-merge-tombstone-removal
# Changes to be committed:
#   (use "git reset HEAD^1 <file>..." to unstage)
#
#	modified:   c_src/bitcask_nifs.c
#	modified:   include/bitcask.hrl
#	modified:   src/bitcask.app.src
#	modified:   src/bitcask.erl
#	modified:   src/bitcask_file.erl
#	modified:   src/bitcask_fileops.erl
#	modified:   src/bitcask_nifs.erl
#	modified:   test/bitcask_pulse.erl
#	modified:   test/bitcask_qc.erl
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	log/