basho/riak_core — pull request #345 by jrwest (2013-06-28 19:44:32+00:00)
Title: Incrememnt Ring Version when Force-Replacing during Resize

Because the claimant runs in a different "mode" the ring version may
not be incremented otherwise causing reconcilation during gossip to
fail. Seen in the wild and recreated periodically during riak_test:

Error looks like:
```
2013-06-27 20:05:42.812 [error] <0.150.0>@riak_core_ring_manager:handle_call:394 ring_trans: invalid return value: {'EXIT',{function_clause,[{riak_core_ring,'-reconcile_next/2-fun-0-',[{91343852333181432387730302044767688728495783936,'dev4@127.0.0.1','$resize',[],awaiting},{91343852333181432387730302044767688728495783936,'dev2@127.0.0.1','$resize',[],awaiting}],[{file,"src/riak_core_ring.erl"},{line,1529}]},{lists,zipwith,3,[{file,"lists.erl"},{line,385}]},{lists,zipwith,3,[{file,"lists.erl"},{line,385}]},{riak_core_ring,reconcile_ring,3,[{file,"src/riak_core_ring.erl"},{line,1578}]},{riak_core_ring,reconcile_divergent,3,[{file,"src/riak_core_ring.erl"},{line,1484}]},{riak_core_ring,internal_reconcile,2,[{file,"src/riak_core_ring.erl"},{line,1474}]},{riak_core_ring,reconcile,2,[{file,"src/riak_core_ring.erl"},{line,498}]},{riak_core_gossip,reconcile,2,[{file,"src/riak_core_gossip.erl"},{line,327}]}]}}
```

Originally seen during testing on AWS. I've been able to recreate this periodically during `riak_test` w/o this change, it seems to occur more often once mixed w/ the large cluster changes introduced in 1.4. With this change I have been unable to reproduce. Its also pretty easy to follow the code path from the error above and see we are entering `reconcile_next` when we should be calling `reconile_divergent_next`. This is b/c the `rvsn` for both ring's are equal. Also, notice in the error the next list items are identical except the owning nodes, this is what crashes `reconcile_next`'s pattern match.