basho/riak_kv — pull request #685 by jrwest (2013-10-04 05:35:21+00:00)
Title: add option to filter keys that are no longer in preflist during fold

In cases like reducing n-value and ring resizing Riak may leave additional
replicas on a partition that no longer is a member of a preflist for the
replica's key. Riak does not (yet) clean up this data, and even if it did,
there would be a window between the time the data was considered "straggling"
and when it was deleted. To prevent this a filter for user-facing folds (not handoff
folds -- via FOLD_REQ) has been added to check whether or not the key being folded
over has the partition as a member of its preflist.

This change, also, does not affect bucket folds because during these
folds the keys are not exposed and it would be expensive to verify if
at least one key in the bucket had the partition as a member of its
preflist.

This PR relies on https://github.com/basho/riak_core/pull/410

These changes are primarily meant to address a known issue with ring resizing (see rehashing in "The Rest of the Story [here](https://github.com/basho/riak_core/pull/301) and the last para of @jtuple's comment [here](https://github.com/basho/riak_pipe/pull/76#issuecomment-18715817)). Of course, it comes at a cost: primarily, hashing each key on every iteration of the fold. It is behind an option for this reason. We could perhaps make this better by storing the hash of the object in its metadata, but this PR does not include that.

A possible way to verify these changes is to:

1. populate Riak using [mapred_verify](https://github.com/basho/mapred_verify)
2. run [mapred_verify](https://github.com/basho/mapred_verify) tests to make sure they pass
3. run a ring resize to completion
4. run [mapred_verify](https://github.com/basho/mapred_verify) tests and watch some fail
5. `application:set_env(riak_kv, fold_preflist_filter, true)` on each node
6. run [mapred_verify](https://github.com/basho/mapred_verify) tests and watch them pass

Any other user facing fold functions like list keys or 2i would hit this as well.