MB-5307: support simple way to adjust some timeouts at runtime

Particularly timeouts affecting our communication with memcached.

This also makes few timeouts larger. Particularly memcached protocol
level timeouts (timeouts speaking to usually local memcached) are
increased.

As well as ns_memcached "outer" level (i.e. including potential queing
delay) timeouts of heavy operations.

Timeouts can be adjusted by settings node's config. Efficently reading
those timeout values is achieved by maintaining public named ets table
with all key-value pairs.

Example key is {node, node(), {timeout, ebucketmigrtor_connect}}.

Change-Id: I4c878213b8cc2710d03ecd56ff4f02080c2684c3
Reviewed-on: http://review.couchbase.org/16571
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5203 Include diag in collect_info

Also, get rid of cbupgrade dependency.

Change-Id: I77046d30aaadbf8e7cbc8f21feb22cd0bb405490
Reviewed-on: http://review.couchbase.org/16614
Tested-by: Bin Cui <bin.cui@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5402: avoid system_stats_collector crashing other services

System stats gathering is a cool thing. But MB-5402 points out that
sometimes we want to count some stats quite before
system_stats_collector configured ets table for stats. We're just
ignoring any exceptions from trying to update those stats now.

Change-Id: Ifd6b0a0e985e8213a0cfc12090fe6beb7df24607
Reviewed-on: http://review.couchbase.org/16632
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307: allow just grabbing current archive of master events

It's useful to be able to just get current history without infinitely
streaming. That's particularly useful for tools that want to analyze
previous rebalance, because otherwise they need some heuristic to stop
streaming.

Change-Id: Ia2621d0a0a4a6f9d46e9d43361e392f5fb6aab07
Reviewed-on: http://review.couchbase.org/16633
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307: unbreak cli ebucketmigrator

Change-Id: Ie9e09ac62085cc9ffee4cb807a90d34fe63cdf1e
Reviewed-on: http://review.couchbase.org/16767
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:ebucketmigrator:dont ask password when not given username

Change-Id: I4156d2be4314f553e74e80e56b185ccc39a61f1b
Reviewed-on: http://review.couchbase.org/16768
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: made ns_vbm_sup:apply_changes only touch affected nodes

Previously it would go find children list on all nodes, apply changes
and do set replicas call with big list of mostly same replicas.

Change-Id: Id7430e44abf30ffbf32ddec37cd8b016a95bc8a7
Reviewed-on: http://review.couchbase.org/16769
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: support slow network simulation for cluster_run

This is quite obviously linux-only. And requires quite recent iproute
tools and reasonably recent kernel.

First part of commit allows ./cluster_run to prepend something to
arglist.

This allows for command lines like this:

./cluster_run -n4 --prepend-extras ./scripts/run_in_ns.rb

Second part is run_in_ns script. Which sets up VDE networking with
round-trip delay around 2.4 milliseconds from host to host and
bandwidth limit of 30 megabytes per second. It spawns nodes inside
newly created network namespaces which makes kernel use "full" routing
between erlang nodes (and memcacheds). netns per node also makes
running per-port-moxi possible as netns-es don't intersect
w.r.t. bound ports even on 0.0.0.0.

Change-Id: I5a6d435565c280592ba7a2da2c4837dbddcb52d6
Reviewed-on: http://review.couchbase.org/16770
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: dont hibernate gen_events

Which seems to have negative effect on performance during rebalance,
where we're changing vbucket map quite a lot.

Change-Id: I866ee10eecdfab9f67601e02b7bb1b2a1ff944d0
Reviewed-on: http://review.couchbase.org/16771
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: menelaus_web:handle_streaming now eats all notifications

I've noticed positive effect on CPU during rebalance because we have
significant amount of config change due to vbucket map updates. And it
makes sense to consume batch of changes before sending out updated
stuff.

Change-Id: If4e85f6d25367283226534c6e190030d7e1865e9
Reviewed-on: http://review.couchbase.org/16773
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: don't compute saslBucketsStreaming twice on each update

menelaus_web:handle_streaming is calling streaming function twice per
iteration. Once with 'stable' argument and second time with
'normal'. This is done in order to remove 'volatile' fields from value
that's used to decide if change needs to be streamed. But
saslBucketsStreaming doesn't have difference for stable and normal
responses.

So new rule is if 'stable' invocation returns {just_write, JSON} we'll
use JSON without calling streaming function second time. And
saslBucketsStreaming is now using it.

Change-Id: I08aa427b1e34d3d1d6ec84c50ffa7886c1a1c83b
Reviewed-on: http://review.couchbase.org/16774
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:saslBucketsStreaming now avoids sending whole node status

Because moxi doesn't need it and grabbing/sending it requires
work. All moxi needs per bucket's node is hostname and ports.

This was found to have positive effect on CPU consumption during
rebalance.

Change-Id: I7ad1091f31d642f5c1da75f4731b9aefe76416ef
Reviewed-on: http://review.couchbase.org/16775
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:refactored ns_port_init to avoid excessive recomputations

Previously it considered any ns_config event as potential change of
port servers config. That's obvious waste of CPU.

New code only ignores irrelevant events and also avoids talking to
supervisor unless reconfiguration is really needed.

Change-Id: Ia1c348ea59837c4742ea389b17f99e81a871c408
Reviewed-on: http://review.couchbase.org/16776
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: skip ebucketmigrator sasl auth step for default bucket

Because default is..., well, default

Change-Id: I15d738b5a12b2a60b55855fdf18ba65e3a3e5194
Reviewed-on: http://review.couchbase.org/16777
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: impl. more efficient check for ebucketmigrator timeout

Profile was showing quite a lot of activity in timer service. Quite a
lot of it was due to ebucketmigrator seting up timeout timer. Which we
don't normally need. erlang module timers are known to be more
efficient.

Change-Id: I28bcccbe5939f9b85393889d0525fdae655effb2
Reviewed-on: http://review.couchbase.org/16778
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:ebucketmigrator: use prim_inet instead of gen_tcp

Change-Id: Id0a6eef83bf30e69f0cc344289948f0eb993a79b
Reviewed-on: http://review.couchbase.org/16779
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:implemented more efficient ebucketmigrator initialization

Particularly we're dealing with larger set of vbuckets, so we don't want
very chatty exchange with memcacheds for things like finding out
open/closed checkpoints and setting destination vbuckets to replica
state.

Particularly:

a) instead of doing a bunch of sequential set_vbucket requests we
first check which vbuckets needs to be set to replica state

b) instead of doing a bunch of sequential get_last_closed_checkpoint
requests we're doing single stats request. And we're using quick stats
to get this.

c) we're using quick stats for getting list of open checkpoints

Change-Id: I99b4bcfa74bd64974c306ee7ab386302595d0b05
Reviewed-on: http://review.couchbase.org/16780
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307: optimized ebucketmigrator network delays

It was found that Naggle algorithm delays are slowing down rebalances
quite a lot. On the other hand just naively setting TCP_NODELAY option
does not work, because ebucketmigrator is using send per tap
message. And I've found it to slow down takeovers that require sending
some data volumes.

Alternative approach of permanently having TCP_NODELAY on upstream
socket and briefly enabling TCP_NODELAY on downstream socket for
pushing of previously sent messages seems to work well. The idea is
that erlang & memcached will batch enough stuff that we'll receive as
one piece in handle_info. And then pass to process_data. Process data
will do (potentially a bunch of) send calls. If after that we don't
have full frame transmission may be delayed according to Naggle's
algorithm. So we "push" this stuff through.

Something better perhaps can be done. I.e. we could detect messages
that we need to deliver asap (most likely with ack flag set) and only
push them. But implemented behavior seems to be good enough. My
thinking is that memcached is seemingly doing 10 messages per send
syscall. Which for short messages is already imperfect. And memcached
is using TCP_NODELAY. So we should be matching their questionable
behavior. But hopefully not making it significantly worse.

Change-Id: Ibc36bd8b010f3052c6731e78db656cf2cd7369bb
Reviewed-on: http://review.couchbase.org/16781
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307: implemented exponential backoff for replicas building

I.e. so that empty rebalance is quick and longer rebalance doesn't
waste time in needless stats polling.

Change-Id: I77d10fd903dd8d0a624e6f55a7f25a24c80e5158
Reviewed-on: http://review.couchbase.org/16782
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5434: don't add ejected nodes back to bucket's servers list

It's possible that ejected node(s) is not part of bucket's node list
anymore or yet. One possibility is if it was already rebalanced out
for this bucket.

If that is the case we should not add it back to server's list at the
beginning of rebalance, which would cause unnecessary waiting for
bucket startup on this node.

Change-Id: I70e1ed622b5246aca0f32000d51f67bcc08c6da4
Reviewed-on: http://review.couchbase.org/16783
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>