[bp] implemented quick failover

NOTE: we still don't allow janitor runs during failover if any of
non-failed-over nodes are unhealthy. You still need to fail-over all
unhealthy nodes for failover to become effective.

Originally reviewed on http://review.couchbase.org/12349

Conflicts:

	src/ns_janitor.erl

Change-Id: Ib6d620239d77300e1ee40cc9569e85fc96fa38e3
Reviewed-on: http://review.couchbase.org/13955
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] note that vbuckets of vbm sup child vbucket ids are sorted

We already relied on vbuckets field being sorted.
ns_vbm_sup::set_nodes_replicas in fact heavily relied on it.

This change documents this assumption and fixes ns_vbm_sup:add_replica
that did not maintain vbuckets sortedness.

Few optimizations enabled by this sortedness are also implemented.

Originally reviewed on http://review.couchbase.org/12398

Change-Id: Id7f176e6a80531754fe35ff86fd51f9985098340
Reviewed-on: http://review.couchbase.org/13956
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] adapted failover safeness est. code to replicator-on-destination

Originally reviewed on http://review.couchbase.org/12761

Conflicts:

	src/menelaus_web.erl
	src/ns_heart.erl

Change-Id: I17a37188aca025b18f4242e01763b7758f108184
Reviewed-on: http://review.couchbase.org/13959
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] MB-4839 Don't ignore errors in failover safeness multicall.

Originally reviewed on http://review.couchbase.org/13637

Change-Id: Ibf4751dac5fea94fbe4bb019add5305bfaf41eb5
Reviewed-on: http://review.couchbase.org/13960
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Handle empty result list in misc:multicall_result_to_plist_rec.

Change-Id: I35fa5006dcd9a07d39d0d44ee5d47981c7123918
Reviewed-on: http://review.couchbase.org/13962
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Generalize ns_vbm_sup to cb_gen_vbm_sup.

ns_vbm_sup now is just a callback module (+ some API functions calling
cb_gen_vbm_sup). This will allow us to have almost the same codebase
for new and old style replications.

Change-Id: I968ae6099752ba08c7c10571eb0df2d00f004aca
Reviewed-on: http://review.couchbase.org/13963
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Rename ns_vbm_sup to ns_vbm_new_sup.

Change-Id: I42b6b60de0e87dd82be68656c671ed8b79df215b
Reviewed-on: http://review.couchbase.org/13964
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reintroduce ns_vbm_sup module.

Now it's just a callback module for cb_get_vbm_sup.

Change-Id: I445866c87a203a077e0a7a2800b386c3f18e37c4
Reviewed-on: http://review.couchbase.org/13965
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Choose between old and new replication styles.

cb_replication process (running on master node) keeps track of
replication modes for each bucket. Replication mode can be switched
only in ns_janitor during cleanup. The mode stored in cb_replication
is used only on master node (to avoid switches during rebalance). All
the other nodes determine which mode to use only based on
heartbeats. This means that certain lags are possible. But this seems
to be acceptable.

Change-Id: I36f22df51f722d73f32e17e8e63c02c76ddbb434
Reviewed-on: http://review.couchbase.org/13966
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Old replication info where old nodes expect to find it.

In particular broadcast 'replication' property via heartbeats and
implement ns_rebalancer:buckets_replication_statuses.

Change-Id: I9d93a55673a89e2a9fe13c33f3f34405ab2aafd5
Reviewed-on: http://review.couchbase.org/13967
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Estimate old nodes failover safety as well.

Change-Id: I5a7690ab3e76d3da1eba713163aacc32f2d239e7
Reviewed-on: http://review.couchbase.org/13968
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>