basho/riak_kv — pull request #408 by beerriot (2012-10-20 19:33:13+00:00)
Title: Correct abortion of MR input-sending logic

As of https://github.com/basho/riak_pipe/commit/276e90c7176329f650cde34306fc134254410a28 (released in Riak 1.1) using `riak_pipe:destroy/1` causes the `riak_pipe_builder` process to exit with reason `normal` instead of something abnormal. This means that the asynchronous MapReduce input sender spawned in `riak_kv_mrc_pipe:send_inputs_async/3` no longer receives an exit signal, despite being linked to the builder. So, an input-sender might continue running long after its target pipe has disappeared. This will fill riak logs with messages of the form:

```
riak_pipe_vnode:new_worker:766 Pipe worker startup failed:fitting was gone before startup
```

This affects MR input types of explicit bucket-keys, and certain kinds of `modfun` inputs.

Changing `riak_pipe_builder:destroy/1` to make the builder exit abnormally does not completely solve this case, because there are other points at which the builder decides to exit, but does so with reason 'normal' to avoid log spam from its supervisor.

In addition, not all MapReduce users use `riak_kv_mrc_pipe:send_inputs_async/3`; some use `:send_inputs/2,3` (non-async). These senders also need to notice that the pipe has vanished.

This PR takes a two-pronged attack:

1. Clauses of `riak_kv_mrc_pipe:send_inputs/3` now check the return value of their calls to `riak_pipe_vnode:queue_work/2`. When the pipe has closed, instead of returning `ok`, `:queue_work` will return `{error, [worker_startup_failed,...]}`. This error is raised for the endpoint to handle.

2. The `riak_kv_wm_mapred` and `riak_kv_pb_mapred` modules, handling HTTP and PB MapReduce requests, now explicitly kill the async sender process.

Messages about "fitting was gone before startup" may still appear in the log, but they should be limited to one input's N-value worth (since each of the primary vnodes in the preflist must fail before the error is raised). In addition to being protection against holes in this strategy, the explicit kill added to the HTTP and PB endpoints should help reduce the number of spam log messages further.

@evanmcc and @jonmeredith should both be interested in this PR.