basho/riak_core — pull request #630 by aberghage (2014-09-29 22:05:26+00:00)
Title: Refactored handoff sender max_concurrency to reduce overloaded meanings.

Addresses #564 [JIRA](https://bashoeng.atlassian.net/browse/RIAK-1189) — still needs tests, just opening this to get review started.

* Separated handoffs terminated for concurrency limit violations from those
terminated for real errors -- concurrency violations no longer cause
abnormal process exits.

* Split up the three sources of max_concurrency exits in handoff_sender:
1. vnode lock max_concurrency
2. remote node hitting its concurrency limit
3. concurrency limit lowered after handoffs already started

* Added a mechanism for marking a transfer as "rescheduled" -- considered
failed on completion.

This is used to notify the handoff manager that a given handoff, while
exiting normal, should not signal xfer_complete to the corresponding
vnode. NB this logic is basically working around the relatively tight
coupling between handoff_manager and vnode_manager; while there's some
groundwork laid for decoupling them in the form of the optional
`handoff_started/2', the handoff manager still special-cases a bit
just for vnode_manager.