basho/riak_kv — pull request #516 by engelsanchez (2013-03-21 17:58:25+00:00)
Title: support for handling legacy sext encoding of 2i keys

This commit is a squashed version of the branch:
jrw-eas-2i-fix-keys-backport.

The version of sext used by riak 1.3 and prior did not
preserve the sort order of bigints when serializing erlang
terms resulting in 2i queries that returned missing or
invalid results (riak_kv#499). The updated version of sext fixes the issue
with an incompatible change to the encoding of bigints. Special
handling of this incompatability is necessary to provide a path
for indexes to be fixed (as they are written to disk).

A utility function for reformatting the on-disk indexes (per-node)
is provided. This allows indexes to be upgraded on a running node.
The utility folds over all indexes, per partition, and rewrites any
invalid index (those containing bigints encoded using the old version
of sext). In order to safely migrate indexes, until the fold that
rewrites the indexes in the new format completes, all riak_object
puts/deletes must delete the legacy index format in addition to the new format

Upon successful completion or if the partition is empty a flag is
written to the leveldb backend to ensure that future writes/deletes
to not inccur the additional cost of deleting the legacy index format

add support for downgrading back to legacy index format

Fix eleveldb fold_objects to start at 1st object

Also fixed special bucket option so it starts at the beginning of the
given bucket. But that doesn't appear to be used anywhere...

Add key namespace for special leveldb markers

My argument is that this may make it less messy down the road if others
start adding other special markers in leveldb partitions. They now have
a separate namespace, like objects and index keys do.

add console support for reformatting indices

Fix 2i bigint eq query during transition

This causes a double scan while in the transition from the legacy
(broken for big int) sext encoding for 2i keys. Once starting at the old
key value, later at the new one. Once the upgrade or
downgrade is finished, it should return to a normal scan.

add support for multi-backend index reformatting

the multibackend supports indexing on buckets assigned to sub-backends
that support indexing. LevelDB sub-backends must be reformatted.