MB-6586: concurent throttle state clean up

Today when vb replicator crash abnormally we may not always clean
the state correctly and as a result risk token loss during the XDC
replication, which may block all replicators if running out of the
tokens. This change-set will fix the issue and recycle the token if
a vb replicator crashed in middle of replication.

Change-Id: Idc8b78ac14a830242e8333b6928e5bf145e33c61
Reviewed-on: http://review.couchbase.org/20682
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
MB-5307: reshuffled stats display for better clarity

As worked out together with Dipti.

Also renamed confusing 'Replications' link to 'XDCR'

Change-Id: I7c02564ec380be5099fb491f836598f3933e7a3f
Reviewed-on: http://review.couchbase.org/20807
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5308: avoiding fetching ddocs when it's not actually needed

I've found that we're fetching ddocs even when not on views
section. Apparently it was due to dependency on productionDDocsCell
and which has subscribeValue 'demand' pulling ddocs transitively all
the time.

Change-Id: I8a9977a9771ef61234cf918f593c73bf38443407
Reviewed-on: http://review.couchbase.org/20662
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-6619: remove unnecessary stats polling in ensure_fulll_commit

This change-set will make capi to be notified by ep_engine when
waiting for the persisted checkpoint id to catch up with the open
checkpoint id it created. It removes repeated ep_engine statistics
polling from ns_server during waiting.

Change-Id: I7e7ee6c3d73f0cdcdf2a71a737f99f37873a45b2
Reviewed-on: http://review.couchbase.org/20768
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-5943: Add xdcr stats for work and commit times

Tracks and adds 2 stats to the xdcr stats UI, "secs working" and
"secs committing".  The "secs working" is the cumulative wall clock
time spent checking and writing documents, and "secs committing" is
the cumulative wall clock time spent committing (ensuring all docs
written to a target are durably written to disk) and writing the
checkpoint documents.

Change-Id: Ie82d839e846d291063e0c6af57a7dcba247c86fd
Reviewed-on: http://review.couchbase.org/20708
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-6275: removed past replication section

Change-Id: Ie4fcfe9647f155cb975e90d8dd7e177b07b5ed3f
Reviewed-on: http://review.couchbase.org/20693
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5308: removed invalid Last Run column in XDCR UI

Change-Id: Ib596125747eab6a4039ed90eb7faeffc80b35100
Reviewed-on: http://review.couchbase.org/20920
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5308: removed broken status column on XCDR page

Change-Id: Ib29b7f69ff6abf714ecb56aa8860de1defaffefa
Reviewed-on: http://review.couchbase.org/20921
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-6637: make replication batch size pluggable

This change-set introuce a configurable parameter which allows user
to modify the size limit of batched documents in XDCR manager before
sending the packet to desinatoin cluster. The new parameter name is
"xdcr_doc_batch_size_kb" and it is defined in kilobytes. The default
value is 512, which means the size limit of batched doc is 512KB.

Change-Id: I4ba80659b43df49139d1bab1d0b67c3d049fb063
Reviewed-on: http://review.couchbase.org/20945
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-5307: fix dialyzer warnings on R15

Apparently R15 is verifying we're returning expected values in
gen_server callbacks, and in few places we specifically return invalid
values to cause crash. This places were fixed in order to crash
explicitly.

Change-Id: I69696f23a15833cac0fba5fa9eaa0a8d6ce2895d
Reviewed-on: http://review.couchbase.org/20919
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
MB-6560: handle absense of bucket on UI node in views

Because in order to list ddocs we contact local ddocs service and
because that service won't be even started on pending-add nodes,
previously we'd crash with noproc.

We also had similar problem with random key where we in fact need at
least single vbucket hosted by UI node.

Now both server side and UI handle this condition somewhat.

Change-Id: I56b3c6f7ec496bd196b7d3ebf8edad9064b538c9
Reviewed-on: http://review.couchbase.org/20663
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: made atomic_write_file be more atomic

I.e. it now fsyncs before rename.

Change-Id: Ic05cfae5d7a32042c4b3f66193b7eec14a9d5b14
Reviewed-on: http://review.couchbase.org/20772
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
MB-5170: implemented disable_traffic command support

Change-Id: I50d421ec1488800337f4499d04eb62b8f5033bd6
Reviewed-on: http://review.couchbase.org/20773
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
MB-5170: implemented sync vbucket deletion

Default is still async as it's too late to switch to sync vbucket
deletion completely.

Change-Id: Idd7e3c48b2b3a578aedaa18c90773170366a8756
Reviewed-on: http://review.couchbase.org/20774
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
MB-5170: extracted this_node_bucket_dbdir

And cleaned up bucket config generation code.

Change-Id: Iae761238e83b071739a58b035d42e3f83eef341d
Reviewed-on: http://review.couchbase.org/20775
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
MB-5170: implemented reliable cluster-wide flush

New approach is the following:

* we begin by disabling traffic on all nodes. If it fails, we fail
  entire bucket flush

* if it succeeds we 'commit' (so essentially, 2 phase commit) bucket
  flush by bumping flushseq counter in bucket configuration.

* after that there's no way back. There's no way janitor can enable
  back traffic without completing flush on all nodes. Each
  janitor_agent maintains (on disk) it's local applied flushseq
  counter. Local flushseq being smaller means janitor_agent needs to
  complete flush. Querying vbucket states which is done at the
  beginning of janitor runs will check that and if needed it'll
  complete failed bucket flush.

* we normally complete flush by syncing on config replication and doing
  complete_flush call on all node's janitors.

* each node janitor will shutdown all replicators, remove all vbuckets
  from indexes and then delete all vbuckets.

* after we're done on all nodes we simply ping janitor which will
  create all vbuckets and restore everything.

Change-Id: I3e73bcaf047a3adebb9af484b99d30e68b641750
Reviewed-on: http://review.couchbase.org/20776
Reviewed-by: Liang Guo <liang@couchbase.com>
Tested-by: Liang Guo <liang@couchbase.com>
MB-5308: enabled flush button on UI

Change-Id: Ie75a119acaac062c591173e69dce3c957f2bb29d
Reviewed-on: http://review.couchbase.org/20777
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
MB-6595: avoid requesting remote stats for cluster storage totals

We're using this particularly in pool details and gathering stats from
all the nodes is not what we really need there.

New implementation is taking this ram and disk usage stats from
heartbeats instead of reaching for fresh data all the time.

Change-Id: I4d488fabbe148ddafbf520695f5f9d3e18e73d1c
Reviewed-on: http://review.couchbase.org/20883
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307: stop using orddict in stats collection

We found this elephant is too slow.

Change-Id: I67c92139e31047a44616b16428e92712d6ff7a67
Reviewed-on: http://review.couchbase.org/20884
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5623: hibernate processes serving streaming clients

For significant savings in memory usage

Change-Id: I1e69a2e83525dfcabe406d93db1965d7e02cfd81
Reviewed-on: http://review.couchbase.org/20885
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Pavel Blagodov <stochmail@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>