basho/riak_core — pull request #300 by beerriot (2013-04-15 13:56:18+00:00)
Title: Fix worker pool races

Races in the vnode worker pool, which could cause the pool to become frozen, were described in #298. This PR resolves those races.

## Resolutions

The fix for the checkin race is for the worker to only checkin to riak_core_vnode_worker_pool, and allow riak_core_vnode_worker_pool to decide when to checkin to poolboy. If there is outstanding work in the queue, the worker will just be reused immediately instead of being checked in.

The fix for the DOWN race is for the worker to send a 'worker_started' message to riak_core_vnode_worker_pool, when the worker starts. While the worker is sending this message, poolboy is blocking, so riak_core_vnode_worker_pool can send poolboy a checkout message immediately, and still be guaranteed that it will be received after poolboy has put the worker in its pool.

## Testing

The new test `worker_pool_pulse` requires modules to be compiled with the `PULSE` macro defined, so it must be run on its own. A new make target, `pulse`, has been added to do just this. Before running the `pulse` target, ensure three things:

1. You have [Quviq's pulse_otp](https://github.com/Quviq/pulse_otp) on your path.
2. You have a copy of poolboy with PULSE annotations (currently in [the bwf-pool-race branch of basho/poolboy](https://github.com/basho/poolboy/tree/bwf-pool-race)
3. You have not yet built beams for `riak_core` or `poolboy` (so, either right after `./rebar get-deps` or `make clean`)

Running the tests using just the first commit in this PR should fail with EQC/PULSE complaining of `deadlock`. If it fails for another reason, you may have missed one of the three requirements above (or I only achieved WOMM status). Using both commits, the tests should pass.

Only `worker_pool_pulse` will run when using the `pulse` make target. After running this test, and before running other tests, you should clean and recompile, or the PULSE annotations may cause other tests to fail.