Report failure to connect to memcached correctly.

Seen in MB-5080.

Change-Id: I22adf90a583b7dcdd0a987fbfa86d792649a120d
Reviewed-on: http://review.couchbase.org/14902
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Handle rebalance request correctly when it's already running.

Seen in MB-5080.

Change-Id: Id7a64c724003f8a5b916e5d2379785f738b4fad9
Reviewed-on: http://review.couchbase.org/14903
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5060 Remove bucket from remaining_buckets upon deletion.

If user request bucket deletion when janitor run is in progress, we
shut janitor down but we used to keep remaining_bucket list intact. It
means that in certain circumstances subsequent janitor run would try
to cleanup the bucket that had just been deleted.

Change-Id: Ic5a12b4a320ea7fe547d0e54ee0d8c695848e06b
Reviewed-on: http://review.couchbase.org/14957
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
don't allow attempts to "rebalance out" unknown nodes.MB-5062

It was previously possible to ask rebalancing out of node that's not
part of cluster anymore. This fix will return nodes mismatch if
somebody tries to rebalance out unknown node.

Actual fix is verifying that EjectedNodes are subset of KnownNodes.

Change-Id: Ie8a6713070850e5b705f91a33d6a8abd31ccd07c
Reviewed-on: http://review.couchbase.org/14969
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
force-refresh pool details before creating bucket. MB-5019

We're generally supposed to have correct and automagically updated
pool details all the time. But there are some issues correctly
spotting that all node's status is updated after bucket
deletion. We're often seeing pool details being updated after first
node completes bucket deletion, and subsequent updates do not
currently initiate pool details update.

Simple workaround is to force update pool details right before
creating bucket. Simple and effective.

Change-Id: I5a94251e6e1e0b44cc0871723c57b6148c744e34
Reviewed-on: http://review.couchbase.org/14971
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
do enter gen_server loop when no vbuckets are ready.MB-5052

Otherwise changing filter doesn't work.

Change-Id: I26895af89d8cc17ea1349c6343f777c763cb92bc
Reviewed-on: http://review.couchbase.org/14970
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
added "not ready vbuckets" ebucketmigrator master event.

As can be seen in MB-5052 we're having one of them here but it should
not happen. Also currently ebucketmigrator events log only ready
vbuckets (i.e. vbuckets actually used) without any way to communicate
full set of vbuckets. Now we have both ready and not ready (if any)
vbuckets visible through master events.

Change-Id: Ie868803c2c7d7383c438c7ec2deb0f32f9d8f415
Reviewed-on: http://review.couchbase.org/15005
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Unexport confusing and unused ns_config:reload. MB-5110

Change-Id: I2fed607a4d5ffff6fd8389a1a1b94d43195a17ab
Reviewed-on: http://review.couchbase.org/15006
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Merge remote-tracking branch 'membase/branch-181' into merge

* membase/branch-181:
  Unexport confusing and unused ns_config:reload. MB-5110
  added "not ready vbuckets" ebucketmigrator master event.
  do enter gen_server loop when no vbuckets are ready.MB-5052
  force-refresh pool details before creating bucket. MB-5019
  don't allow attempts to "rebalance out" unknown nodes.MB-5062
  MB-5060 Remove bucket from remaining_buckets upon deletion.
  Handle rebalance request correctly when it's already running.
  Report failure to connect to memcached correctly.
  MB-5087 - moved statstore URL
  Special case buckets key when logging config changes.
  Convert ns_config_log to gen_server and remove unused code.
  Remove to verbose logging from ns_bucket_sup.
  MB-5000 Eject master as late as possible.

Conflicts:
	src/ns_rebalancer.erl

Change-Id: I0309238026a397f4201231fb27c96e0ae89490fd
do force bucket shutdown on failover.MB-5118

Even though we keep data files we still should shutdown bucket as soon
as we can. This fixes regression introduced in this "keep data files
after failover" story implementation.

Change-Id: I83f3a34d0ea2f44a33ecd010e54ffae63a434598
Reviewed-on: http://review.couchbase.org/15048
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
delete all node's unused bucket db files before rebalance.MB-5115

As noted in MB-5115 we were not treating partially failed over node as
'new' when rebalancing back in, which could lead to failure to delete
old bucket's db files.

Correct fix seems to be to get rid of _any_ unused db files on all
nodes at the beginning of rebalance.

Change-Id: Ic1e5af53063f34d6791c238669461cd690ac6a3f
Reviewed-on: http://review.couchbase.org/15051
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
extracted executing_on_new_process into misc.erl. MB-5118

This is needed for futher work on MB-5118 fix

Change-Id: Ib3409d8e36feaa39df4b946bf4a4b76ffd00d096
Reviewed-on: http://review.couchbase.org/15052
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
extracted multicall_ignoring_undefined. MB-5118

We're going to use it in next patch

Change-Id: I17909b96089fe1f0749b7d8694dd3ee722ead082
Reviewed-on: http://review.couchbase.org/15053
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
wait for old bucket's shutdowns on nodes rebalanced in. MB-5118

Prior to actually starting rebalance and prior to deleting database
files we're now waiting for all pending bucket shutdowns to
complete. So that we don't try to reuse old bucket and if we have
problems shutting down some old buckets, we'll report it correctly
rather than wait_for_memcached failed

Change-Id: I1ec633c22761f3226150692d23f5181c18eec3aa
Reviewed-on: http://review.couchbase.org/15054
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Merge remote-tracking branch 'membase/branch-181' into merge

* membase/branch-181:
  wait for old bucket's shutdowns on nodes rebalanced in. MB-5118
  extracted multicall_ignoring_undefined. MB-5118
  extracted executing_on_new_process into misc.erl. MB-5118
  delete all node's unused bucket db files before rebalance.MB-5115
  do force bucket shutdown on failover.MB-5118

Conflicts:
	src/ns_vbm_sup.erl

Change-Id: I0bf328ba23ef85a863d275b6929cb3fadd9af851
Merge remote-tracking branch 'couchbase/branch-18'

* couchbase/branch-18: (49 commits)
  wait for old bucket's shutdowns on nodes rebalanced in. MB-5118
  extracted multicall_ignoring_undefined. MB-5118
  extracted executing_on_new_process into misc.erl. MB-5118
  delete all node's unused bucket db files before rebalance.MB-5115
  do force bucket shutdown on failover.MB-5118
  Unexport confusing and unused ns_config:reload. MB-5110
  added "not ready vbuckets" ebucketmigrator master event.
  do enter gen_server loop when no vbuckets are ready.MB-5052
  force-refresh pool details before creating bucket. MB-5019
  don't allow attempts to "rebalance out" unknown nodes.MB-5062
  MB-5060 Remove bucket from remaining_buckets upon deletion.
  Handle rebalance request correctly when it's already running.
  Report failure to connect to memcached correctly.
  MB-5087 - moved statstore URL
  Special case buckets key when logging config changes.
  Convert ns_config_log to gen_server and remove unused code.
  Remove to verbose logging from ns_bucket_sup.
  reimplemented named tap fix for branch-18. MB-4366
  MB-5000 Eject master as late as possible.
  disallow automatic failover during rebalance.MB-5050
  ...

Conflicts:
    priv/public/js/app.js
    priv/public/js/buckets.js
    src/auto_failover_logic.erl
    src/menelaus_event.erl
    src/menelaus_stats.erl
    src/menelaus_web.erl
    src/misc.erl
    src/ns_bucket.erl
    src/ns_cluster.erl
    src/ns_memcached.erl
    src/ns_storage_conf.erl
    src/stats_collector.erl

Change-Id: Iaed7b3e194922fb9e2c7bd0d5a98986c2f7e5a5c
Log all rebalance errors to user visible log.

Change-Id: Ie32377ff9eb462a79c3bd2d6759937b662c14f3e
Reviewed-on: http://review.couchbase.org/15071
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Handle ns_vbm_new_sup:local_change_vbucket_filter result correctly.

Change-Id: I49f6065480f7b663fcff061668b6b63f7fe49bbf
Reviewed-on: http://review.couchbase.org/15064
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Log stacktrace when unable to start replication.

Change-Id: I506443315aa101816513129533bef78a1ce5329f
Reviewed-on: http://review.couchbase.org/15065
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
made config saver failures more visible.

Previously we ignored saver failures silently. But that means some
issues may've be unnoticed. Added to trace down MB-5110

Change-Id: I6705457a70d309ef006eec9448a6f99f342c39a2
Reviewed-on: http://review.couchbase.org/15050
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>