basho/riak_test — pull request #529 by jrwest (2014-02-01 01:32:35+00:00)
Title: improve cluster_meta_basic

cluster_meta_basic has been intermittently failing [1]. This commit includes two improvements, the second of which addresses this intermittent failure.

The first change modifies the test to "wait_until_object_count" instead of reading the object count at a given moment and getting a possibly stale, or soon to be updated, value. This alone does not cause the test to pass reliably. However, it highlights that the underlying race condition is one where the object count will never reach the expected value.

The second change modifies the test to avoid the race, which was caused by resolving on two different nodes concurrently, each of which in turn wrote and broadcasted the resolved result. If an interleaving, such that both writes are allowed to succeed before one node is
notified of the modification on the other, occurs so does the failure. The test has been changed to only perform the write/broadcast on a single node ensuring that we eventually converge the the expected value and object count.

[1]
http://giddyup.basho.com/#/projects/riak_ee/scorecards/73/73-1556-cluster_meta_basic-centos-6-64/35530/artifacts/532185