MB-6951: don't log subscribed/unsubscribed messages in ns_pubsub

Change-Id: I2845aa57674d07e002120b836a77322c8762c575
Reviewed-on: http://review.couchbase.org/21732
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-6494: handle empty ready vbuckets set in ebucketmigrator

We cannot send empty vbucket filter to ep-engine. If we do, ep-engine
will send all vbuckets, instead of none.

Thus when all vbuckets are not ready to be replicated from, we have to
nothing. We cannot die, because it will cause crashes above.

We had such behavior previously, but apprently recent code for
handling new-style vbucket filter change or something else removed it.

Note, there's also effect on vbucket filter change. When we hit this
condition, we force old-style vbucket filter change. That's because
there's no existing tap producer with our tap name.

Because no tap stream is going to be used while we do nothing (waiting
restart message), we kill tap stream with our name (if any).

Change-Id: Ic868a059fddf5055290b783882031453e3fb2ceb
Reviewed-on: http://review.couchbase.org/21733
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-6953: anticipate exit from parent while waiting index update

Doing blocking janitor agent call for waiting index update implies we
were insensitive to shutdown signal from parent. Which means if
indexing is too slow or deadlocked somewhere, we cannot really stop
rebalance.

We mitigate it by using tried-and-tested spawn_and_wait function of
single vbucket mover ther performs work on new process while waiting
for both this process and exit from parent.

Change-Id: Ic5ed1b7f30b624473ce1c300327782b937bec2a9
Reviewed-on: http://review.couchbase.org/21751
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6955: only react on our bucket mccouch events

Quite embarrassingly we were reacting on any bucket's mccouch
event. Refresh vbuckets event is harmless, but vbucket deletion will
obviously cause _other_ bucket's capi_set_view_manager to delete this
vbucket from it's indexes. Lame indeed :)

Fix is trivial.

Change-Id: I3f3a8be0ce9233b4268df97041a805c450f76eee
Reviewed-on: http://review.couchbase.org/21752
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6919: expose more XDCR stats to UI

This change-set provides more XDCR stats on UI to help QE, performance
team and customers understand the XDCR operations better. We add 3 new
outbound operation stats, namely, 1) data replicated, which is amount
of data XDCR have replicated to the remote data center through inter-
cluster network; 2) active vb replications, which shows the number of
ongoing active replications to the remote clusters; 3) waiting vb
replications, which is the number of vb replications in the waiting
pool to wait for turn to start replication. These stats will be useful
to performance monitoring and diagnosis.

Change-Id: If9d42cde5dd715404f0b1bcf26ea143493ae9f75
Reviewed-on: http://review.couchbase.org/21613
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-6638: disable async threads

Our empirical evidence so far suggests that those weird crashes are
caused by async threads which is somewhat experimental erlang VM
feature.

Thus we turn it off. We compensate by generously bumping up number of
scheduler threads.

Change-Id: I3ff3da7ec8190b719df8e7a06e40bdf2154f360d
Reviewed-on: http://review.couchbase.org/21884
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-6934: separate timestamp from error msg within XDCR

Today we include timestamp as part of error msg grabbed by ns_server.
For better msg filtering based on time stamps, it is better to return
them separately as {Timestamp, Error}, where Timestamp is the raw
time returned by erlang:localtime(). Another commit is needed to sort
and filter error msgs on UI.

Change-Id: Ic80121d53782379d746e6104b985b14c1a456e6d
Reviewed-on: http://review.couchbase.org/21694
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6944: clean statistics if replicator terminated

When vb replicator terminated, we need to clear the statistics
num_change_left, docs_changes_queue and size_changes_queue, and
update them to the bucket replicator since the changes queue for
this particular vb replicator no longer exists. The vb replicator
supervisor will start a new vb replicator and populate these
stats to bucket replicator.

Change-Id: I4eb5796a43f6cf89b40e9ce80cf96c73f403401f
Reviewed-on: http://review.couchbase.org/21794
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6939: increase XDCR checkpoint interval to 30 min

XDCR will use priority checkpoint intead of regular checkpoints. To
address the overhead concern, we increase the checkpoint interval from
5 min to 30 min.

Change-Id: I128a2c826e3823084912bb6049a6390e2aec25d6
Reviewed-on: http://review.couchbase.org/21799
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6925 Don't try to cleanup old indexes in vbucket databases.

It's meaningless because there won't be any indexes.

Change-Id: I1fde741456f932a37244b7573b49f96f3af1d22d
Reviewed-on: http://review.couchbase.org/21885
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6993 Log PID even for named processes.

Change-Id: Ie06994641093cf218cb95e6337b1a60a1fe8d24c
Reviewed-on: http://review.couchbase.org/21898
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-6939: use priority checkpointing in ensure_full_commit

Today we use regular ep_engine checkpoint to persist XDCR checkpoint
which in most cases failed because the disk drain queue build up over
time. The failed XDCR checkpoints caused XDCR to pay restart time per
vb replicator and XDCR may have to resume replication from the last
successful checkpoint which was persited long time back. This change-
set uses priority ep_engine checkpoint which is less reliant on the
disk write queue to improve the chance of successful XDCR checkpoint.

Change-Id: I929274533a0586f6e6ca584593bb5da7228c4f56
Reviewed-on: http://review.couchbase.org/21730
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
MB-6934: show xdcr errors in less panic color

Part of 6934 improvements suggested by Dipti is to remove red color
from last 10 errors link.

Change-Id: Ic93e1fabb7e80218bb80aeabf04378c199af3bbf
Reviewed-on: http://review.couchbase.org/21903
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-6934: omit too old xdcr errors

XDCR keeps errors forever. Even if errors happened last year. We want
to omit too old errors. By default, stale error is error that happened
more than 2 hours ago.

Change-Id: I06223fa328a9fee201ed9336bbf685f6b0f16d71
Reviewed-on: http://review.couchbase.org/21904
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-6974: set reasonable ERL_FULLSWEEP_AFTER

Erlang's default leads to massive memory blowup. Our perf developers
found that 512 is a reasonable value.

Change-Id: I50822ecb752d6667af3a6f8b9bac283e3d5aef3a
Reviewed-on: http://review.couchbase.org/21902
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Typo - COUCHBASE_NUM_VBUCKET(S) to set vbucket cnt

Instead of COUCHBASE_NUM_VBUCKET

Change-Id: I1b7f1eca03924089a02ac631787532148793017d
Reviewed-on: http://review.couchbase.org/21813
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Steve Yen <steve.yen@gmail.com>
MB-6998: remove heavy logs which block stats request from UI

In previous fixes we logged msgs from couchdb, which turns out to
be a flooding of msgs for bucket replicator to dump. Consequently
he bucket replicator is too busy to repond to UI stats request and
causes UI stat unaccessible.

Change-Id: Ib8155a43d0411254cc9f19e05fe510168eea2fa0
Reviewed-on: http://review.couchbase.org/21932
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-6994: use only useful xdcr stats in xdcr tasks building

XDCR tasks do not need to have tons of stats. If somebody needs stats
there's API for stats. Tasks by design only expose 'progress' of
task. Thus we remove everything that we don't need for that.

Plus our code actually depended in a really hard way on those stats
attributes making hard backward-compat constrain for us.

Change-Id: I2f9dff24af8740453df62b6a6581446b4dd81a18
Reviewed-on: http://review.couchbase.org/21906
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Paulau <pavel.paulau@gmail.com>
MB-6995 Bump couch_app supervisor shutdown timeout to infinity.

Shutdown timeouts are not supposed to be finite for supervisors.

Change-Id: I55c4a29977e039e75a0ed1528e430b212ff1b385
Reviewed-on: http://review.couchbase.org/21944
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-6995 Restart cb_couch_sup in a special way.

That's because we also need to restart other apps like couch_set_view,
etc.

Change-Id: Ica61751580d057c47bb9a3dc86c6930278e62422
Reviewed-on: http://review.couchbase.org/21945
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>