Merge remote-tracking branch 'couchbase/2.0.2'

* couchbase/2.0.2:
  MB-7701: latency optimized XDCR

Change-Id: Ic6f45c909b12b3aa17167806b53b9b7fab103efb
MB-7730 Master event for forced view compaction during rebalance.

Change-Id: Ibcfc920204cf0eece902fb7df0266a232c18c982
Reviewed-on: http://review.couchbase.org/24572
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-7731 Count moves from 'undefined' nodes in total_in_flight.

If moves schedule contains only moves from 'undefined' nodes, we'll
immediately decide that rebalance is done. This is because 'undefined'
moves are not counted in total_in_flight. We don't expect that in
ns_vbucket_mover:on_backfill_done which causes badmatch error. Which
in turn causes rebalance to fail.

Change-Id: I5b1a31b5f51b99bf5ea2e0bf4a2a93ca0a421bc2
Reviewed-on: http://review.couchbase.org/24569
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
MB-7432: Add XDCR latency stats

This commit adds two new stats in outbound XDCR stat section,
namely, 1) meta ops latency and 2) doc ops latency, which measure
the latency of metadata operations and document update operations
in XDCR. The unit is millisecond.

Both stats are computed as the weighted average of the stats
from all vb replicator worker processes. In particular, each
vb replicator worker process will monitor two latency stats
and use the number of checked docs (replicated docs) as weight
of meta ops latency (doc ops latency, respectively). Both
the aggregated latency and its weight are passed all the way
up to vb replicator, bucket replicator, node stat collector,
and cluster-wise stats collector. The finally aggregation is
computed as the weighted average (the total aggregated latency
divided by total weights).

Change-Id: Id57614ea98890bb03ba3ae305e0652bdb1c921f0
Reviewed-on: http://review.couchbase.org/24399
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-7432: add XDCR percentage of completeness stat

Yet another new stat required by customer and supporting engineers,
probably the last one under this bug. This stat tracks the progress
of XDCR by computing the percentage of work done out of all. Note
it is computed as the number of items checked, instead of items that
have been replicated, divided by all items (items checked plus to-
be-replicated). That means, if source and destination clusters
are already synced, nothing will be replicated and the completeness
will be 100%.

Change-Id: I2d284bf8e1da8e0001f6c6778abc098e59e1be05
Reviewed-on: http://review.couchbase.org/24583
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-100: un-write-only-ify scripts/visualize-rebalance.rb

I've found previous version quite hard to understand and to
fix/extend. I belive that's likely due to "spread"-ness of state
there.

New code is "local-state" as much as possible with code flow
reflecting timeline state changes of every nodes.

Hopefully it will be much less write-only :)

Change-Id: I7f7a6a0b7192f04ec2197f0cf024af9129c411ad
Reviewed-on: http://review.couchbase.org/24576
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Merge remote-tracking branch 'origin/2.0.1'

* origin/2.0.1:
  MB-7731 Count moves from 'undefined' nodes in total_in_flight.

Change-Id: I838e7552b215699b3feaf00848d7982038844e4c
MB-100: correct typespec and silence dialyzer

Change-Id: I3d82a6e0a4bf0921f8aa5d957ad5718f4359895c
Reviewed-on: http://review.couchbase.org/24588
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-7743: always initialize ns_config_ets_dup data

Previously it was working because of ns_config:reannounce as part of
ns_config_rep initialization. But there's clear race between
initialization of ns_config_ets_dup and continuation of startup which
does ns_config_rep initialization.

So we're asking for explicit config reannounce to populate ETS table
now.

Change-Id: I798a1bf0e818f876a7e1c9833e161724249db257
Reviewed-on: http://review.couchbase.org/24606
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Merge remote-tracking branch 'origin/2.0.1'

* origin/2.0.1:
  MB-7743: always initialize ns_config_ets_dup data

Change-Id: I6efc1680b2d4da42b6d15c307c9029eaa9792194
MB-100:explicitly forbid long un-supported joining of 1.7.x nodes

Change-Id: I00629d059257c5a1c496c34ae5d9b0179443c782
Reviewed-on: http://review.couchbase.org/24608
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-7799 Use precomputed value in local_bucket_disk_usage.

Pre-2.0 nodes do rpc:multicall to
ns_storage_conf:local_bucket_disk_usage whenever someone request
bucket info. So if this is done often our local file_server_2 might
get overloaded by the requests. To prevent this we just use the value
from couch_stats_reader's ets table which gets updated periodically.

Change-Id: I372dd41895d487da78ae5133b255162a5577fac0
Reviewed-on: http://review.couchbase.org/24882
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Merge remote-tracking branch 'couchbase/2.0.1'

* couchbase/2.0.1:
  MB-7799 Use precomputed value in local_bucket_disk_usage.

Change-Id: Ifd841255db2ee9ee94e17c68d24c05e9cc95f1de
MB-7837: continuously tunable optimistic XDCR

This commit introduces "xdcr_optimistic_replication_threshold",
which is 256 bytes by default. This paraemter is used by XDCR to
split docs list into two: a list of big docs, whose docs are all
bigger than the threshold parameter,  and a list of small docs
whose docs are no greater than it. For small docs, we skip all
revs_diff operations and optimistically send them directly to the
remote cluster. For big docs, we still send revs_diff first and
then only send those docs surviving conflict resolution at remote
node.

By this way, users are able to continuously adjust the parameter
to determine which docs should be replicated optimistically.

The deletion, however, is always treated as small doc, regardless
of its doc size, because there is no benefit to send revs_diff for
deletions at all.

The corresponding environment parameter is:

"XDCR_OPTIMISTIC_REPLICATION_THRESHOLD"

and users can always override the ns_server parameter using the env
parameter.

The ns_server parameter "xdcr_optimistic_replication" and its env
parameters will both retire with this commit since the optimistic
replication is naturally encoded in the new parameter, which can
be set to a big number to cover all docs.

Change-Id: I739b2c63cce64e31b18387938acf1c844f023cc6
Reviewed-on: http://review.couchbase.org/24934
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
MB-7824 Convert messages to binary before sending to sinks.

Disk sinks do this anyway. So it will save us some work there and
unneeded copies between processes. Besides this, we now that logging
long strings takes a lot of space and erlang is quite reluctant about
garbage collecting it. So it's better to get rid of strings before
message reaches long-living sink processes. The caller is more likely
to be short-living process. So it's likely that memory will be
released sooner.

Change-Id: Ib2fb805194fb46ed3a09284f2cfe289266a44531
Reviewed-on: http://review.couchbase.org/24909
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-7824 Hibernate ns_log and ns_log_sink.

ns_log expects messages to be strings. And it's not easy to change it
to work with binaries because of backward compatibility issues. So
instead we just hibernate ns_log and ns_log_sink. Since we don't frequently
log to the ns_log, this should not cause any performance issues.

Change-Id: I1a1fbf2cea9ce4ef300846c86eb2f714c26b5710
Reviewed-on: http://review.couchbase.org/24910
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-7786 Group concurrent remote bucket requests.

When XDCR replication is created, every vbucket replicator calls
remote_cluster_info:get_remote_bucket_by_ref about the same time. At
this only one remote node is known. Namely, the one that was used to
create remote cluster reference. Previously remote_clusters_info did
not attempt to group all these concurrent calls into a single request
to the remote node. Which could lead to the overload on the
destination node. This would produce an error. Replicators would
retry and most likely hit cached value. So replication would start
without any issues. But the error is reported to user and creates
certain confusion.

To lower the probability of overloading any of destination nodes,
remote_clusters_info will not make more than one outstanding request
per node per remote bucket. So for the case described earlier, only
one remote request will be performed for all the simultaneous
get_remote_bucket calls.

Change-Id: I9ca0bb0dba5670c545f824d00ebaff9cf331ac06
Reviewed-on: http://review.couchbase.org/24986
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307 Properly validate usernames and passwords.

Change-Id: Id46f42f20cde19a1733fece9de27d16316302013
Reviewed-on: http://review.couchbase.org/24998
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307 Handle colon in passwords correctly.

Change-Id: Id9f23a8f3a58906c1fc393f0469d5d27d3d6663d
Reviewed-on: http://review.couchbase.org/24999
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-100: render visualize-rebalance timeline in real scale

So that it's easy to compare different runs.

Change-Id: I08282039cf7e5fc1895f8648d5e767843d256575
Reviewed-on: http://review.couchbase.org/25002
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>