upgrade config for vbucket map history with current balanced maps

So that failover/add back for existing installations can be optimized.

Change-Id: Id26c1ee57d2c59069bc84c8b2c1cd23db47e23a6
Reviewed-on: http://review.couchbase.org/14199
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
using debug level for debug-only ebucketmigrator messages

Change-Id: I5f47a9dff2286737c7b1c5d795bbe535da739ca1
Reviewed-on: http://review.couchbase.org/14198
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
skip moves for completely same new/old chains

Change-Id: I36ec7cc471dbbdeac92602dbbd8919aa238cb2de
Reviewed-on: http://review.couchbase.org/14191
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
use proc_lib:spawn_link to spawn single vbucket mover

Change-Id: I547724669cc5ed46c058d7a0e6d15df9db93ba62
Reviewed-on: http://review.couchbase.org/14189
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
reset node's cookie before joining cluster.MB-4476

Change-Id: I958731daad0f85deb28990b1b1a2040e3c65fc04
Reviewed-on: http://review.couchbase.org/14186
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
try not creating bucket if it's running on any cluster nodes

We'll do best effort check if ns_memcached for this bucket still
exists anywhere and refuse creating it.

Best-effort here means that we'll try to reach other nodes, but with
sane timeout and if we cannot we'll assume it's fine. It is consistent
with current behavior and don't want to change that.

It is intended to prevent issues when somebody tries to re-create
bucket too soon after it was deleted. And to protect us from any
issues in ep-engine/bucket-engine that may delay bucket deletion. See
for example http://www.couchbase.com/issues/browse/MB-4329

Change-Id: I8d66cb5dc9603b4da8ad9ca430a0cd112e2a422f
Reviewed-on: http://review.couchbase.org/14182
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
don't register/unregister ns_memcached too late/soon

There's seeminly no need anymore. And it allows us to check if bucket
is still running by looking at registered processes.

Change-Id: Iecef1bed6690e20cc14d3f8caed2dbdcd27bf6ab
Reviewed-on: http://review.couchbase.org/14179
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
handle errors_500 from do_bucket_create in setup default call

Change-Id: I0f005254ce8aef10e1c7b501e4310ef2e565d21d
Reviewed-on: http://review.couchbase.org/14183
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
fixed filter_ready_buckets support of memcached buckets

Without this fix memcached buckets that are not yet "rebalanced" will
not be actually started on new nodes and that will cause periodic
crashes of hot keys keeper.

Change-Id: I3c366efc3e75d7ab2e9f2c4f1eef3318d14f8d4e
Reviewed-on: http://review.couchbase.org/14184
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
don't crash but return 400 when new bucket name is missing

Change-Id: I8ec90b8a00891183efb6772e4eb615de58abb3f2
Reviewed-on: http://review.couchbase.org/14185
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
use past vbucket maps to optimize vbucket movements

New code picks closest of past but matching vbucket maps and newly
generated vbucket map.

Unit tests particularly ensure that failover/add-back case is handled
with minimal movements. Also simultaneous removal of N nodes and
addition of N other nodes performs minimal movements as well (transfer
of vbuckets between this nodes).

Change-Id: Iccaafb26e0d75cedfa161b741847a82bd02bdf90
Reviewed-on: http://review.couchbase.org/14187
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
log distance between current and future maps

Change-Id: If4f97b496ea07678373f81370f8f6beab240f957
Reviewed-on: http://review.couchbase.org/14188
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
log more janitor state when setting dead state

Change-Id: Ic82b033b050f5b350111f96e32034e8018316f72
Reviewed-on: http://review.couchbase.org/14190
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
make sure we've replicated latest config before leaving

ns_config_rep:synchronize is not enough because ns_config_rep may yet
to see latest config. We need to sync on config announcements,
queueing config replication, first. And then wait for all config
replications to be done.

Change-Id: Iac70fced38cc70c52667c257b6cd560519eb8fb6
Reviewed-on: http://review.couchbase.org/14192
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
actually delete buckets when node is failed over

Not just shutdown ns_memcached. Right now we shutdown ns_memcached,
delete bucket without force option but leave data files. Those files
are deleted when node is finally ejected, but if user does add-back
instead it leads to lengthy warmup of those data files. Essentially
not allowing rebalance without good reason.

Change-Id: Iba2a82e35471b496e04ec2c433a3868a4f886176
Reviewed-on: http://review.couchbase.org/14180
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
try to delete data files even if delete bucket failed

So that if memcached crashes/dies when we're force deleteting bucket
we still delete data files.

Change-Id: I2ab6d7212091351f4f04cc0d2b81ef4455176237
Reviewed-on: http://review.couchbase.org/14181
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
backfill all new replicas and master before doing vbucket takover

It has effect of limiting backfills on source and destination
side. But most important reason for doing this is that by switching to
built replicas we have more reliable rebalance. It should be possible
to stop it at any time and things will be in much safer shape now
compared to previous implementation.

Change-Id: I6c9c8aef6197178558c61940598e4aa43715df0c
Reviewed-on: http://review.couchbase.org/14193
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
deregister tap names after using them for building replicas

Change-Id: Ica75699760f9c0a73216d80fea7c5353e5384eb4
Reviewed-on: http://review.couchbase.org/14194
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
delete old copies of vbucket after vbucket movemenet

So that memory is freed earlier.

After config update, we're doing sync on config replication first to
play extra safe. And then we're deleting vbucket on old nodes.

Change-Id: I15693d791a781dfcf0c523fb2bb6f5d4e6c01ed2
Reviewed-on: http://review.couchbase.org/14195
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
log planned moves instead of vbucket map

Change-Id: I010fb62480732a207251db9ba172696947e81b46
Reviewed-on: http://review.couchbase.org/14196
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>