basho/riak_test — pull request #646 by kellymclaughlin (2014-06-18 17:42:52+00:00)
Title: Add ability to handle identical siblings in rt:systest_read

Add optional parameter to `rt:systest_read` that provides the ability to
compare siblings and treat those with identical values and metadata
excluding the dot as a single value for the purposes of testing.

This is to facilitate testing with DVV enabled since there are several
known cases where use of DVV leads to siblings created internally by
Riak. One example of such a case is a write during handoff that is
forwarded by the vnode, but also executed locally.

The existing behavior of `rt:systest_read` is to fail when any siblings
are encountered with a `badmatch`. This behavior is maintained as the
default so existing tests maintain their current semantics without
explicit change.

## Related Issues

The following issues contain information about cases where these
siblings can occur and discussion around them:

* https://github.com/basho/riak_kv/pull/838
* https://github.com/basho/riak_kv/pull/627

## Testing

Failures due to lack of handling of the type of siblings described here have
been observed when running the `replication_object_reformat` and
`repl_aae_fullsync_custom_n` tests. In general, running either test in a loop for
20 iterations tends to result in the condition being hit at least once.
The `repl_aae_fullsyc_custom_n` test is more prone to it in my experience.

@andrewjstone noticed the failure when reviewing a change for `replication_object_reformat` [here](https://github.com/basho/riak_test/pull/640).