MB-12642:bp:MB-10437:prefer oldest waiter in concurrency_throttle

Old behavior could choose same pid(s) over and over again. Which
caused particular problem for initial replication under slight udpates
load. I.e. it would repeatedly pick already mostly up-to-date vbuckets
over vbuckets that still need to be replicated. Which would cause very
short batches which in turn would cause very slow rebalance.

By picking oldest waiter (modulo system clock jumps which should be
infrequent) we avoid this problem. It should also help at least
somewhat for the case of ongoing replication under somewhat heavy but
not too heavy load, due to better avoidance of too short replication
batches.

Change-Id: Ie6dcafabd510367015c1e5897a620e37eb7a17f5
Orig-Reviewed-on: http://review.couchbase.org/38368
Reviewed-on: http://review.couchbase.org/43227
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-12643:bp:MB-11631 Ensure terminate is called if ensure_bucket succeeds.

This is backported as part of backporting MB-12156.

Change-Id: Ib03c6034c8d63e7c6a1d027f34992c9c63608183
Orig-Reviewed-on: http://review.couchbase.org/39197
Reviewed-on: http://review.couchbase.org/43228
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-12643:bp:MB-12156 Don't delete buckets when changing datapath.

Change-Id: I0039e0bfa7353cd784791e6f7aca7d73a89778b7
Orig-Reviewed-on: http://review.couchbase.org/41332
Reviewed-on: http://review.couchbase.org/43229
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-12643:bp:MB-12156 Don't trap exits while change disk paths.

It's not needed and prevents us from a situation where process gets
stuck and wakes up much later (potentially after node became part of a
cluster) and thus invalidates the check that node is not provisioned.

Change-Id: Ic0d5b9a05088b0d376002338109d80cabc44f0ed
Orig-Reviewed-on: http://review.couchbase.org/41333
Reviewed-on: http://review.couchbase.org/43230
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-12644:bp:MB-12238: set timeout of 10 minutes on xmem requests

As another measure against possible issues when NAT between DCs
forgets about our connections. With this patch, if this happens, xdcr
will still get stuck, but it won't last too long due to timeout. When
timeout hits, affected xdcr replicators will crash and be restarted
have a decent chance of working.

Orig-Reviewed-on: http://review.couchbase.org/41588

Conflicts:
	src/pooled_memcached_client.erl
	src/xdc_vbucket_rep_xmem.erl

Change-Id: I3c8a96d71bcb983be5e804c3e6e8eaaf7be4e7b4
Reviewed-on: http://review.couchbase.org/43231
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-12644:bp:MB-12238: set SO_KEEPALIVE on xmem and ssl proxy sockets

As another guard against possible issues when NAT forgets about
connections between datacenters.

bp note: While I don't think it'll actually affect MB-12238 at all
given that keepalive doesn't work for non-idle sockets and idle
sockets are not kept long by connections pool, I don't want to cause
additional discussion with folks who monitor backports.

Orig-Reviewed-on: http://review.couchbase.org/41589

Conflicts:
	src/memcached_clients_pool.erl

Change-Id: Ica2f6e03b4929b730ba98a4b8621e321870d025d
Reviewed-on: http://review.couchbase.org/43232
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Merge commit 'cc8cca2b4df9e58568289984de9da15daf073e1b' into rel-3.0.0++

* commit 'cc8cca2b4df9e58568289984de9da15daf073e1b':
  MB-12643:bp:MB-12156 Don't trap exits while change disk paths.
  MB-12643:bp:MB-12156 Don't delete buckets when changing datapath.
  MB-12643:bp:MB-11631 Ensure terminate is called if ensure_bucket succeeds.
  MB-12642:bp:MB-10437:prefer oldest waiter in concurrency_throttle

This is empty merge of backported commits.

Conflicts:
	src/concurrency_throttle.erl

Change-Id: Ied5dd16f7896636ea3d9a39abb616c61a0925b1e
Merge branch '2.5.1.1' into rel-3.0.0++

* 2.5.1.1:
  MB-12644:bp:MB-12238: set SO_KEEPALIVE on xmem and ssl proxy sockets
  MB-12644:bp:MB-12238: set timeout of 10 minutes on xmem requests

Conflicts:
	src/memcached_clients_pool.erl
	src/pooled_memcached_client.erl
	src/xdc_vbucket_rep_xmem.erl

This merges backport of MB-12238 into 2.5.1.1. Approval for 2.5.1.1
made MB-12238 automatically approved for 3.0.2.

Change-Id: I59a3199bec546963538a626997a7ec59115bd719
Merge branch 'rel-3.0.0++' into rel-3.0.1

* rel-3.0.0++:
  MB-12644:bp:MB-12238: set SO_KEEPALIVE on xmem and ssl proxy sockets
  MB-12644:bp:MB-12238: set timeout of 10 minutes on xmem requests
  MB-12643:bp:MB-12156 Don't trap exits while change disk paths.
  MB-12643:bp:MB-12156 Don't delete buckets when changing datapath.
  MB-12643:bp:MB-11631 Ensure terminate is called if ensure_bucket succeeds.
  MB-12642:bp:MB-10437:prefer oldest waiter in concurrency_throttle

This is empty merge of backports.

Change-Id: I8cbf2fe70cf74406d15c6787137888f7d4165496
Merge branch 'rel-3.0.1'

* rel-3.0.1:
  MB-12644:bp:MB-12238: set SO_KEEPALIVE on xmem and ssl proxy sockets
  MB-12644:bp:MB-12238: set timeout of 10 minutes on xmem requests
  MB-12643:bp:MB-12156 Don't trap exits while change disk paths.
  MB-12643:bp:MB-12156 Don't delete buckets when changing datapath.
  MB-12643:bp:MB-11631 Ensure terminate is called if ensure_bucket succeeds.
  MB-12642:bp:MB-10437:prefer oldest waiter in concurrency_throttle

This is empty merge of backports

Change-Id: Iddf29188543cd6eae27ff7c45531353efc1fed49