basho/riak_test — pull request #648 by kellymclaughlin (2014-06-19 21:08:52+00:00)
Title: Re-initiate fullsync after a number of failed checks for completion

## Overview

Re-initiate fullsync after 100 failed checks for completion. The
number of retries of the *start fullsync and then check for
completion* cycle is configurable using
`repl_util:start_and_wait_until_fullsync_complete/4` and defaults to 20
retries. This change is to avoid spurious test failures due to a rare
condition where the rpc call to start fullsync fails to actually
initiate the fullsync.

A very similar changed for the version of the
`start_and_wait_until_fullsync_complete` in the replication module
introduced in `0a36f9974c21427c306526442046749aba87326a` (#566) has had good
effect at avoiding this condition for v2 replication tests.

## Testing

This condition can be very difficult to replicate, but the best chance of seeing such a failure is running `replication_object_reformat` in a loop for 20-50 times. It is likely that one of those times the condition will be hit, but the downside is that testing consumes 10-25 hours of time. Running on one of the buildbot machines or a slower vm may also increase the likelihood of encountering the condition.

The failure manifests as the riak_test executing to a point where it outputs something like this:

```
12:00:39.190 [info] Waiting for fullsync count to be 1
12:00:39.190 [info] Starting fullsync on: 'dev1@127.0.0.1'
```

The test then hangs until the `wait_until` call times out and then fail.

For simple verification that the changes do not break the testing of fullsync I would recommend running a single execution of `replication2` and `replication_object_reformat` to ensure they pass. Both tests make use of `repl_util:start_and_wait_until_fullsync_complete` (though for `replication_object_reformat` the use is indirect via `repl_util:validate_completed_fullsync`).