only reuse tap name when changing vbucket filter.MB-4366

Now for all ebucketmigrators we're killing named tap before
establishing connection, except as noted below.

Basically we're trying to avoid named tap reuse as much as
possible. And there's only one case where it's worthwhile reusing
it. That is our quite frequent change of vbucket filter for most
replication connections during rebalance. If we're not using named tap
and only relying on checkpoints, it'll cause excessing resending of
already received data, because our checkpoints are normally quite
large.

The approach is the following:

* when we detected vbucket filter may be changed

* we send erl_eval lambda (via rpc:call) to node that hosts some
  ebucketmigrator

* that lambda checks if it support this node supports vbucket filter
  change and if it does calls local vbucket filter change code

* that code spawns separate process where it's work is performed

* that process is registered in vbucket_filter_changes_registry. This
  process represents 'vbucket filter change transaction'. And
  everything is linked carefully so that any crash will not leave any
  resources orphan.

* vbucket filter change then proceeds with finding current
  ebucketmigrator for stream we want change filter for

* it then calls start_vbucket_filter_change/1 on it

* which is handled inside ebucketmigrator by creating new downstream socket

* checking if current downstream socket is alive (by use of confirm_sent_messages)

* and if all is well replying with that new downstream, original
  ebucketmigrator enters 'zombie' state then where it'll only handle
  EXIT messages

* vbucket filter changing code then shuts down original
  ebucketmigrator process through it's supervisor (terminate & remove
  child)

* it then spawns new child for which it adds extra option:
  downstream_retriever

* function that's value of this option is called by new
  ebucketmigrator to check if downstream is being passed to it

* implementation of this function then consults
  vbucket_filter_changes_registry and if there's still 'transaction
  process', it is asked for downstream via gen_server:call

* if that succeeds new ebucketmigrator is then using it instead of
  opening it's own downstream _and_ only in this case tap name is not
  killed before sending tap_connect command

As can be seen this is quite complicated procedure. Why?

* erl_eval lambda sending trick is part of our handling of old
  nodes. Old nodes _will_ reuse named tap and quite obviously do not
  support special vbucket filter change operation. For them we fall
  back to old sequence of stopping child and starting it back (with
  updated vbucket filter).

* we need to handle ns_vbm_sup restarting our ebucketmigrators with
  same arguments we used. Thus instread of just passing downstream as
  one of options/parameters we're using that trick with process
  registry.

* also because vbucket filter is used as part of process's child_id
  inside vbm_sup, we seemingly really need this dance with shutting
  down old process and creating new

* and we're creating downstream in old process because:

  a) we cannot afford, even temporarily, two ebucketmigrators for same
  name. Otherwise some crash in the middle may cause supervisor to
  actually (re)spawn them both.

  b) we want to make sure nothing can happen with downstream memcached
  in between

  So we have this complex procedure of creating new downstream while
  old downstream is alive and only after old downstream passed check
  we're passing new dowsntream up where it's then passed into new
  vbucketmigrator

Change-Id: I36fd85c25d5c36c96f822dc2dbb8169f2d683dae
Reviewed-on: http://review.couchbase.org/14555
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
focus password when passwords fields don't match in wizard

Change-Id: I14b261636c60ac2941a80bbb89a5847a2619e808
Reviewed-on: http://review.couchbase.org/14645
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
silenced false dialyzer warnings for ns_pubsub tests

Change-Id: I406cafd0cbdcc76852999fee5c4ff21e950ccf4e
Reviewed-on: http://review.couchbase.org/14647
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] Handle badrpc errors in multicall_result_to_plist.

This is a backport of combination of http://review.couchbase.org/13637
and http://review.couchbase.org/13962.

Change-Id: I3c4c4ba299815f7ae8ca585364ad4ed6d3bb9e9c
Reviewed-on: http://review.couchbase.org/14673
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Remove impossible case in menelaus_web.

Empty username and password are rejectect by validate_settings
function. Hence I'm removing the case to handle this to avoid
confusion.

Change-Id: Ida978fcc107591ec270f59439ca304f3fcf62509
Reviewed-on: http://review.couchbase.org/14674
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Be more verbose about file deletion errors in misc:rm_rf.

Namely, log error about certain file deletion error and not the error
about non-empty containing directory.

Change-Id: Id15484fe0c9228515e9ef0d4a4f69e091160587c
Reviewed-on: http://review.couchbase.org/14675
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-4564 Don't delete database files when node leaves cluster.

To cover all the corner cases the following approach used. We delete
bucket files in ns_memcached only when this bucket is being explicitly
deleted from all the nodes. We don't delete bucket files at all when
node is removed or failed over from a cluster. Thus we need to handle
two following cases:

  * New cluster is started on the removed node.

    Here we remove all the old buckets when new cluster becomes
    provisioned or when new bucket is being created when cluster is
    not provisioned.

  * The node is added back to a cluster.

    In this case we just clean up all the buckets before rebalancing
    the node in.

Change-Id: I3c2c60104358759bd42603dd5d1a7fbfced4532f
Reviewed-on: http://review.couchbase.org/14628
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Remove menelaus_event:watchers API.

It seems to be outdated and is not used anywhere.

Change-Id: I309462ab8860e6e41eccf0b939290a70f5eccf37
Reviewed-on: http://review.couchbase.org/14676
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Force long polling reply when master changes.

alk: This fixes one potential issue with fix of MB-5020

Change-Id: Ia5dece278b70b042a95dd98f4d52e1fa0a34b0a5
[alk@tut.by: added ticket reference]
Signed-off-by: Aliaksey Kandratsenka <alk@tut.by>
Reviewed-on: http://review.couchbase.org/14677
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
implement ns_config setting for default vbuckets count.MB-4967

couchbase_num_vbuckets_default ns_config key if present defines number
of vbuckets for all new buckets.

We don't provide official way of setting it, but it's possible to
set/alter value via /diag/eval.

Change-Id: Iae4d88d7d4186f4043dfa05ea41a5930951a72cd
Reviewed-on: http://review.couchbase.org/14688
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
log getting rid of old data files into user-visible log

Change-Id: I92f0831326b5e7c9a24b344521343bba08231417
Reviewed-on: http://review.couchbase.org/14717
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
on starting mb_master check if current master needs to be killed

The idea is that joining new node will not complete until all
supervisors are done starting services. So we're delaying new
mb_master startup while checking if we need to take over old master's
mastership.

Note we're doing it from start_link rather than init so that mb_master
is not yet registered which prevents deadlock in other nodes trying to
do same sync check.

Change-Id: I0225a437d2be03d9d930ad9c7721b0b0b2d063d3
Reviewed-on: http://review.couchbase.org/14689
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
log mastership takeover through user-visible logger

Change-Id: Ia8bdcad772204a6207d3f0ed408b16ff13a3c29f
Reviewed-on: http://review.couchbase.org/14716
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
dispay mem_used & eviction watermarks as stats.MB-4960

Change-Id: I5233931dfe1eceaf799aed15b6fec59b209be18a
Reviewed-on: http://review.couchbase.org/14719
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5033 Handle messages with undefined code in ns_log correctly.

If some module doesn't define ns_log_cat message and logs something to
ns_log_sink, then an undefined code will be assigned to this
message. This undefined code is changed to 0 just before
logging. Previously it was done right in ns_log:handle_cast (do_log
clause). But this is not correct since do_log messages are abcasted to
all the nodes. Some of them can be of old version. On old nodes
messages with undefined codes will be just ignored when shown in the
UI (because of the catch around corresponding io_lib:format). And they
will crash diag_handler because of no catch around io_lib:format used
there.

Change-Id: I3ebeb0b2d0e434eeb0d55689d74748ccfd7a767f
Reviewed-on: http://review.couchbase.org/14723
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Update ns_memcached comment on bucket files deletion.

Change-Id: Idb83c3fe862288edafbcdfdc0073fa6a08c5e1e4
Reviewed-on: http://review.couchbase.org/14724
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5008 Retry net_kernel:start several times on duplicate name.

epmd unregisters old node names when a socket that was used to
register this name is closed. This is what happens when
net_kernel:stop is called. But there's no guarantee that subsequent
register name request (issued by net_kernel:start) will be served
after the old name has been unregistered. Because of this we can get
{error, duplicate_name} on net_kernel:start (even though it seems to
be quite unlikely). With this change we will retry net_kernel:start
several times if we get this duplicate_name error.

Change-Id: I2fab246897dcff1fcf73cebc65aef6ff4247286f
Reviewed-on: http://review.couchbase.org/14728
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Include debug log file into diag by default.

The same holds for /sasl_logs and different flavors of cbbrowse_logs.

The reasoning is as follows. It's better to have more than less
information by default. And also prevents cases when one just forgets
to specify that debug info is required, but when it's figured out,
the state of the cluster has already changed significantly.

Change-Id: I85c6924cbaa27cf45d2906c220a1f3834973280b
Reviewed-on: http://review.couchbase.org/14729
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
killed impossible branch in janitor's do_sanify_chain

Somehow we missed that last branch won't be ever taken. Because
previous branches cover master being active or zombie and negation of
master being active or zombie. Thus last branch would never be taken.

Change-Id: I3df3b8e352e74a04ae835b8d1615f8eba834f995
Reviewed-on: http://review.couchbase.org/14757
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
complete chain switch in janitor if new master is active.MB-4988

When rebalance is stopped and future master is already active
previously janitor would use new master but mess up replicas.

We're conservatively detect if active vbucket is on future master as
seen on fast forward map. And if all nodes in fast forward chain are
replicas. In that case we can safely assume takeover was done and we
simply update vbucket map.

Change-Id: Id268b701c548277d47a765f788033cecd09840ac
Reviewed-on: http://review.couchbase.org/14758
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>