[bp] initial replicator-on-destination implementation

We now run ebucketmigrator on destination node rather than
source. This will allow us to do quick failover because with pull
replication it's not necessary to wait for node being failed over to
stop push replication.

NOTE: this is not backwards-compatible with 1.8.0. This will be
addressed as part of future 1.8 work.

Originally reviewed on http://review.couchbase.org/12348

Conflicts:

	src/ns_janitor.erl

Change-Id: Ie06189d6e436aeced58f4c3d82b44968d53872a8
Reviewed-on: http://review.couchbase.org/13954
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] implemented quick failover

NOTE: we still don't allow janitor runs during failover if any of
non-failed-over nodes are unhealthy. You still need to fail-over all
unhealthy nodes for failover to become effective.

Originally reviewed on http://review.couchbase.org/12349

Conflicts:

	src/ns_janitor.erl

Change-Id: Ib6d620239d77300e1ee40cc9569e85fc96fa38e3
Reviewed-on: http://review.couchbase.org/13955
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] note that vbuckets of vbm sup child vbucket ids are sorted

We already relied on vbuckets field being sorted.
ns_vbm_sup::set_nodes_replicas in fact heavily relied on it.

This change documents this assumption and fixes ns_vbm_sup:add_replica
that did not maintain vbuckets sortedness.

Few optimizations enabled by this sortedness are also implemented.

Originally reviewed on http://review.couchbase.org/12398

Change-Id: Id7f176e6a80531754fe35ff86fd51f9985098340
Reviewed-on: http://review.couchbase.org/13956
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] adapted failover safeness est. code to replicator-on-destination

Originally reviewed on http://review.couchbase.org/12761

Conflicts:

	src/menelaus_web.erl
	src/ns_heart.erl

Change-Id: I17a37188aca025b18f4242e01763b7758f108184
Reviewed-on: http://review.couchbase.org/13959
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] MB-4839 Don't ignore errors in failover safeness multicall.

Originally reviewed on http://review.couchbase.org/13637

Change-Id: Ibf4751dac5fea94fbe4bb019add5305bfaf41eb5
Reviewed-on: http://review.couchbase.org/13960
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Handle empty result list in misc:multicall_result_to_plist_rec.

Change-Id: I35fa5006dcd9a07d39d0d44ee5d47981c7123918
Reviewed-on: http://review.couchbase.org/13962
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Generalize ns_vbm_sup to cb_gen_vbm_sup.

ns_vbm_sup now is just a callback module (+ some API functions calling
cb_gen_vbm_sup). This will allow us to have almost the same codebase
for new and old style replications.

Change-Id: I968ae6099752ba08c7c10571eb0df2d00f004aca
Reviewed-on: http://review.couchbase.org/13963
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Rename ns_vbm_sup to ns_vbm_new_sup.

Change-Id: I42b6b60de0e87dd82be68656c671ed8b79df215b
Reviewed-on: http://review.couchbase.org/13964
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reintroduce ns_vbm_sup module.

Now it's just a callback module for cb_get_vbm_sup.

Change-Id: I445866c87a203a077e0a7a2800b386c3f18e37c4
Reviewed-on: http://review.couchbase.org/13965
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Choose between old and new replication styles.

cb_replication process (running on master node) keeps track of
replication modes for each bucket. Replication mode can be switched
only in ns_janitor during cleanup. The mode stored in cb_replication
is used only on master node (to avoid switches during rebalance). All
the other nodes determine which mode to use only based on
heartbeats. This means that certain lags are possible. But this seems
to be acceptable.

Change-Id: I36f22df51f722d73f32e17e8e63c02c76ddbb434
Reviewed-on: http://review.couchbase.org/13966
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Old replication info where old nodes expect to find it.

In particular broadcast 'replication' property via heartbeats and
implement ns_rebalancer:buckets_replication_statuses.

Change-Id: I9d93a55673a89e2a9fe13c33f3f34405ab2aafd5
Reviewed-on: http://review.couchbase.org/13967
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Estimate old nodes failover safety as well.

Change-Id: I5a7690ab3e76d3da1eba713163aacc32f2d239e7
Reviewed-on: http://review.couchbase.org/13968
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Merge branch 'backports-into-18' to master

* backports-into-18:
  [bp] MB-4839 Don't ignore errors in failover safeness multicall.
  [bp] adapted failover safeness est. code to replicator-on-destination
  [bp] note that vbuckets of vbm sup child vbucket ids are sorted
  [bp] implemented quick failover
  [bp] initial replicator-on-destination implementation

diff is empty because merging back backports

Conflicts:
	src/menelaus_web.erl
	src/ns_heart.erl

Change-Id: I76494c3a0ce72f1b33241cb5ad104beca75697fb
Merge branch 'branch-18'

* branch-18:
  Estimate old nodes failover safety as well.
  Old replication info where old nodes expect to find it.
  Choose between old and new replication styles.
  Reintroduce ns_vbm_sup module.
  Rename ns_vbm_sup to ns_vbm_new_sup.
  Generalize ns_vbm_sup to cb_gen_vbm_sup.
  Handle empty result list in misc:multicall_result_to_plist_rec.

Conflicts:
	src/ns_heart.erl
	src/single_bucket_sup.erl

Change-Id: Icc5e89ab6f62da914c5cde14fb49c711b1fbcecc
Merge remote-tracking branch 'origin/master' into dp4-adaptation

* origin/master:
  Estimate old nodes failover safety as well.
  Old replication info where old nodes expect to find it.
  Choose between old and new replication styles.
  Reintroduce ns_vbm_sup module.
  Rename ns_vbm_sup to ns_vbm_new_sup.
  Generalize ns_vbm_sup to cb_gen_vbm_sup.
  Handle empty result list in misc:multicall_result_to_plist_rec.
  [bp] MB-4839 Don't ignore errors in failover safeness multicall.
  [bp] adapted failover safeness est. code to replicator-on-destination
  [bp] note that vbuckets of vbm sup child vbucket ids are sorted
  [bp] implemented quick failover
  [bp] initial replicator-on-destination implementation