basho/riak_kv — pull request #860 by engelsanchez (2014-03-04 14:45:57+00:00)
Title: Fix use of fsm timeouts in 2i AAE

NOTE: This is all the work that went into riak_kv from 1.4.7 to 1.48, forward ported to 2.0

Joe pointed out some problems with FSM time outs in this code. It is
possible to set a timeout for the next FSM state, then receive a message
passed to handle_info and lose that timeout if not set again. Basically,
they are useless.
Refactored code to use gen_fsm:send_event_after() instead and managed
timer for the AAE pid request manually.
Also, a zero timeout transition was avoided by refactoring the code
around the next_partition state, which is now a simple function
directly executed from other states.
Also, if a request for an AAE pid every failed, the whole process would
have died silently. Instead, that is handled now like any other
partition error. So 2i AAE can happily go finishing the other
partitions.

Testing notes: Joe and I tested this manually by modifying riak_kv_vnode to drop replies to the hashtree pid request and verifying that 2i AAE could finish. I don't have time right now to add that to verify_2i_aae, so if a kind sould could take care of that, it would be great.