basho/riak_core — pull request #481 by buddhisthead (2013-12-13 23:47:46+00:00)
Title: Integration of background manager with Handoff and AAE tree rebuild

NOTE: DO NOT MERGE YET (requires #475 + a rebase to remove the bg_manager itself from this PR)

The riak_core background manager PR is here: basho/riak_core#475

The riak_repl PR is here: https://github.com/basho/riak_repl/pull/488

The background manager allows subsystems to coordinate with locks (and tokens) in a non-blocking API when desired. In our case, we've seen several customer support issues related to a production cluster being overloaded by the combined weight of MDC fullsync replication in combination with either handoff, or AAE tree rebuilds (or all three!).

Each of these subsystems is being taught, for 2.0, to "take" a lock on the kv vnode/partition before it does a fold over that partition. Locks are returned when the process dies or exists normally. Each subsystem (at this point in time) is coded to keep working on other partitions if one partition is denied a lock via the return value of max_concurrency, which means progress is made even when one partition is busy.

Use of this feature can be bypassed in production. Besides a global skill switch on the background manager itself, the riak_core configuration supports two new parameters: aae_skip_bg_manager and handoff_skip_bg_manager, which defaults to false. Setting it to true will not call the background manager and proceed like it used to.