MB-5307: support slow network simulation for cluster_run

This is quite obviously linux-only. And requires quite recent iproute
tools and reasonably recent kernel.

First part of commit allows ./cluster_run to prepend something to
arglist.

This allows for command lines like this:

./cluster_run -n4 --prepend-extras ./scripts/run_in_ns.rb

Second part is run_in_ns script. Which sets up VDE networking with
round-trip delay around 2.4 milliseconds from host to host and
bandwidth limit of 30 megabytes per second. It spawns nodes inside
newly created network namespaces which makes kernel use "full" routing
between erlang nodes (and memcacheds). netns per node also makes
running per-port-moxi possible as netns-es don't intersect
w.r.t. bound ports even on 0.0.0.0.

Change-Id: I5a6d435565c280592ba7a2da2c4837dbddcb52d6
Reviewed-on: http://review.couchbase.org/16770
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: dont hibernate gen_events

Which seems to have negative effect on performance during rebalance,
where we're changing vbucket map quite a lot.

Change-Id: I866ee10eecdfab9f67601e02b7bb1b2a1ff944d0
Reviewed-on: http://review.couchbase.org/16771
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: menelaus_web:handle_streaming now eats all notifications

I've noticed positive effect on CPU during rebalance because we have
significant amount of config change due to vbucket map updates. And it
makes sense to consume batch of changes before sending out updated
stuff.

Change-Id: If4e85f6d25367283226534c6e190030d7e1865e9
Reviewed-on: http://review.couchbase.org/16773
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: don't compute saslBucketsStreaming twice on each update

menelaus_web:handle_streaming is calling streaming function twice per
iteration. Once with 'stable' argument and second time with
'normal'. This is done in order to remove 'volatile' fields from value
that's used to decide if change needs to be streamed. But
saslBucketsStreaming doesn't have difference for stable and normal
responses.

So new rule is if 'stable' invocation returns {just_write, JSON} we'll
use JSON without calling streaming function second time. And
saslBucketsStreaming is now using it.

Change-Id: I08aa427b1e34d3d1d6ec84c50ffa7886c1a1c83b
Reviewed-on: http://review.couchbase.org/16774
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:saslBucketsStreaming now avoids sending whole node status

Because moxi doesn't need it and grabbing/sending it requires
work. All moxi needs per bucket's node is hostname and ports.

This was found to have positive effect on CPU consumption during
rebalance.

Change-Id: I7ad1091f31d642f5c1da75f4731b9aefe76416ef
Reviewed-on: http://review.couchbase.org/16775
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:refactored ns_port_init to avoid excessive recomputations

Previously it considered any ns_config event as potential change of
port servers config. That's obvious waste of CPU.

New code only ignores irrelevant events and also avoids talking to
supervisor unless reconfiguration is really needed.

Change-Id: Ia1c348ea59837c4742ea389b17f99e81a871c408
Reviewed-on: http://review.couchbase.org/16776
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: skip ebucketmigrator sasl auth step for default bucket

Because default is..., well, default

Change-Id: I15d738b5a12b2a60b55855fdf18ba65e3a3e5194
Reviewed-on: http://review.couchbase.org/16777
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: impl. more efficient check for ebucketmigrator timeout

Profile was showing quite a lot of activity in timer service. Quite a
lot of it was due to ebucketmigrator seting up timeout timer. Which we
don't normally need. erlang module timers are known to be more
efficient.

Change-Id: I28bcccbe5939f9b85393889d0525fdae655effb2
Reviewed-on: http://review.couchbase.org/16778
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:ebucketmigrator: use prim_inet instead of gen_tcp

Change-Id: Id0a6eef83bf30e69f0cc344289948f0eb993a79b
Reviewed-on: http://review.couchbase.org/16779
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307:implemented more efficient ebucketmigrator initialization

Particularly we're dealing with larger set of vbuckets, so we don't want
very chatty exchange with memcacheds for things like finding out
open/closed checkpoints and setting destination vbuckets to replica
state.

Particularly:

a) instead of doing a bunch of sequential set_vbucket requests we
first check which vbuckets needs to be set to replica state

b) instead of doing a bunch of sequential get_last_closed_checkpoint
requests we're doing single stats request. And we're using quick stats
to get this.

c) we're using quick stats for getting list of open checkpoints

Change-Id: I99b4bcfa74bd64974c306ee7ab386302595d0b05
Reviewed-on: http://review.couchbase.org/16780
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307: optimized ebucketmigrator network delays

It was found that Naggle algorithm delays are slowing down rebalances
quite a lot. On the other hand just naively setting TCP_NODELAY option
does not work, because ebucketmigrator is using send per tap
message. And I've found it to slow down takeovers that require sending
some data volumes.

Alternative approach of permanently having TCP_NODELAY on upstream
socket and briefly enabling TCP_NODELAY on downstream socket for
pushing of previously sent messages seems to work well. The idea is
that erlang & memcached will batch enough stuff that we'll receive as
one piece in handle_info. And then pass to process_data. Process data
will do (potentially a bunch of) send calls. If after that we don't
have full frame transmission may be delayed according to Naggle's
algorithm. So we "push" this stuff through.

Something better perhaps can be done. I.e. we could detect messages
that we need to deliver asap (most likely with ack flag set) and only
push them. But implemented behavior seems to be good enough. My
thinking is that memcached is seemingly doing 10 messages per send
syscall. Which for short messages is already imperfect. And memcached
is using TCP_NODELAY. So we should be matching their questionable
behavior. But hopefully not making it significantly worse.

Change-Id: Ibc36bd8b010f3052c6731e78db656cf2cd7369bb
Reviewed-on: http://review.couchbase.org/16781
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307: implemented exponential backoff for replicas building

I.e. so that empty rebalance is quick and longer rebalance doesn't
waste time in needless stats polling.

Change-Id: I77d10fd903dd8d0a624e6f55a7f25a24c80e5158
Reviewed-on: http://review.couchbase.org/16782
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5434: don't add ejected nodes back to bucket's servers list

It's possible that ejected node(s) is not part of bucket's node list
anymore or yet. One possibility is if it was already rebalanced out
for this bucket.

If that is the case we should not add it back to server's list at the
beginning of rebalance, which would cause unnecessary waiting for
bucket startup on this node.

Change-Id: I70e1ed622b5246aca0f32000d51f67bcc08c6da4
Reviewed-on: http://review.couchbase.org/16783
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-5307: only replicate locally made config changes

Previous behavior was to push any config change to all nodes. Even if
this config change is due to incoming config replication.

NOTE: push_key accumulation corner cases are manually tested by
observing ns_server system stats and by lowering retries count. We
actually hit occasional retries in practice from time to time.

Change-Id: I7b407fa780fa5f66082e348e293a0e5ae53b4513
Reviewed-on: http://review.couchbase.org/16772
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Merge remote-tracking branch 'origin/branch-181' into branch-18

* origin/branch-181:
  MB-5307: only replicate locally made config changes
  MB-5434: don't add ejected nodes back to bucket's servers list
  MB-5307: implemented exponential backoff for replicas building
  MB-5307: optimized ebucketmigrator network delays
  MB-5307:implemented more efficient ebucketmigrator initialization
  MB-5307:ebucketmigrator: use prim_inet instead of gen_tcp
  MB-5307: impl. more efficient check for ebucketmigrator timeout
  MB-5307: skip ebucketmigrator sasl auth step for default bucket
  MB-5307:refactored ns_port_init to avoid excessive recomputations
  MB-5307:saslBucketsStreaming now avoids sending whole node status
  MB-5307: don't compute saslBucketsStreaming twice on each update
  MB-5307: menelaus_web:handle_streaming now eats all notifications
  MB-5307: dont hibernate gen_events
  MB-5307: support slow network simulation for cluster_run

Conflicts:
	src/ns_rebalancer.erl

Conflict was minor.

Change-Id: Ib96fecf1cdd4936304248caca7a9cd409bfafaa6
MB-5307: don't allow duplicate nodes in apply_changes

This fixes bug introduced by the following merge
c58f1d76c012d61ecb298c50238ae4605ad4b273.

It wasn't failing anything visibly, but was applying vbucket filter
changes multiple times.

Change-Id: Icf9aca8cfbd127fe6be9efeb3a06cec67175fae7
Reviewed-on: http://review.couchbase.org/16830
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>