basho/riak_kv — pull request #769 by buddhisthead (2013-12-18 03:08:49+00:00)
Title: Integrate Background Manager with Handoff And AAE Tree Rebuilds

NOTE: DO NOT MERGE YET (this PR requires code from #475 or https://github.com/basho/riak_core/pull/484)

The riak_core handoff portion is here: https://github.com/basho/riak_core/pull/484 (includes #475)
The riak_core background manager PR is here: basho/riak_core#475
The riak_repl PR is here: basho/riak_repl#488

The background manager allows subsystems to coordinate with locks (and tokens) in a non-blocking API when desired. In our case, we've seen several customer support issues related to a production cluster being overloaded by the combined weight of MDC fullsync replication in combination with either handoff, or AAE tree rebuilds (or all three!).

Each of these subsystems is being taught, for 2.0, to "take" a lock on the kv vnode/partition before it does a fold over that partition. Locks are returned when the process dies or exists normally. Each subsystem (at this point in time) is coded to keep working on other partitions if one partition is denied a lock via the return value of max_concurrency, which means progress is made even when one partition is busy.

Use of this feature can be bypassed in production. Besides a global skill switch on the background manager itself, the riak_core configuration supports two new parameters: aae_skip_bg_manager and handoff_skip_bg_manager, which defaults to false. Setting it to true will not call the background manager and proceed like it used to.

I chose not to separate the AAE and Handoff into two separate PRs because we already have too many, but if we determine that we want to exclude the handoff portion, I'll spit them up.