allow non-power-of-2 vbuckets count

There's no real need to limit vbuckets count to powers of 2. And some
other values seem preferrable. So lets enable experimentation while
keeping old default.

Change-Id: Ib3607b0e2fdfa30ccbc3317932f4e175d9ede95a
Reviewed-on: http://review.couchbase.org/13798
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
log normal and not scary message when retry_not_ready_vbuckets

Currently in rebalancing clusters with replica count more then 1 you
can quite often seen "crashes" which are not crashes in fact. But
voluntary death of ebucketmigrators.

This voluntarily death happens if some vbuckets we need to replicate
from are not yet backfilled. In such cases we do some progress with
ready vbuckets and terminate after 30 seconds. This commit changes
crash to normal exit and friendly info message.

Change-Id: Id6bc8d71e6cdcaaa77f02394caafeb1dc7887abd
Reviewed-on: http://review.couchbase.org/13799
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Merge remote-tracking branch 'origin/branch-18' into master

* origin/branch-18:
  log normal and not scary message when retry_not_ready_vbuckets
  allow non-power-of-2 vbuckets count

Change-Id: I90b1ea645419d5b714c5cacafdda5d12251415a1
[bp] added ns_bucket:get_bucket/2

Originally reviewed on http://review.couchbase.org/12760

Change-Id: I0f9dd80d578c9d8f5839a6a3cc4621b829ac5bbd
Reviewed-on: http://review.couchbase.org/13958
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] got rid of unused ns_vbm_sup:kill_dst_children

Originally reviewed on http://review.couchbase.org/12345

Change-Id: Ie4869f2d3eaaf9e93ab76b55c5bf6549ffc53149
Reviewed-on: http://review.couchbase.org/13951
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] added description of misc:multicall_result_to_plist_rec/2

Originally reviewed on http://review.couchbase.org/12758

Change-Id: I191f5ba039d7938f9fab39c6b1138127e50ce2e9
Reviewed-on: http://review.couchbase.org/13950
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] optionally refresh replication graph svg

Originally reviewed on http://review.couchbase.org/12344

Change-Id: I3d26273aff5abe667c72366bf97b43bc3b50a83a
Reviewed-on: http://review.couchbase.org/13952
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] moved vbucket map visualization code out of janitor

Originally reviewed on http://review.couchbase.org/12346

Change-Id: I80817673927efff3d86a442f56ddbd85f5316e48
Reviewed-on: http://review.couchbase.org/13953
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] extracted ns_doctor:get_node/2

Originally reviewed on http://review.couchbase.org/12759

Conflicts:

	src/ns_doctor.erl

Change-Id: I9a7395ffabd1eabebf0282d6d1e70695e1534c21
Reviewed-on: http://review.couchbase.org/13957
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
[bp] fixed long polling timeouts during rebalance

2.0 rebalance performs incremental vbucket map updates this is causing
constant stream of menelaus_event notifications. Old
handle_pool_info_wait could delay long polling reply infinitely when
this happens. This was visible as "Lost connection to..." thing on UI
due to long polling request timing out.

Originally reviewed on http://review.couchbase.org/11933

Change-Id: I26b187f0c0d97acd7e11255d1e78846afad78a03
Reviewed-on: http://review.couchbase.org/13961
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Merge remote-tracking branch 'origin/branch-18' into master

* origin/branch-18:
  [bp] fixed long polling timeouts during rebalance
  [bp] extracted ns_doctor:get_node/2
  [bp] moved vbucket map visualization code out of janitor
  [bp] optionally refresh replication graph svg
  [bp] added description of misc:multicall_result_to_plist_rec/2
  [bp] got rid of unused ns_vbm_sup:kill_dst_children
  [bp] added ns_bucket:get_bucket/2

Diff is empty as we're merging back backports

Change-Id: I297082f854075beffa78fd1f992a72c6362b79e6
implemented set_param command and fixed bucket resizing.MB-4872

Also note that ns_memcached will now crash if resizing command didn't
succeed. Part of our problem previously was that error was really
silently dropped on the floor.

Change-Id: I5c4c37086aa74c4a724358a240c120889688b98a
Reviewed-on: http://review.couchbase.org/13842
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Srinivas Vadlamani <srinivas@couchbase.com>
Merge remote-tracking branch 'origin/branch-18' into master

* origin/branch-18:
  implemented set_param command and fixed bucket resizing.MB-4872

Conflicts:
	include/mc_constants.hrl
	include/ns_common.hrl
	src/mc_client_binary.erl

Change-Id: I3a84937294cfc407bcc2efd3b67871cfa4cd8f07
fixed Cell.wrapWithArgsResolving and cover it with tests

The problem was that Cell.resolveToValue that it is calling is
automagically adding body as last arg and it's causing issues with
functions that have optional last arg. One such function is couchReq.

Change-Id: I3a111b98ce9f489d8724cd6883df0e9e3d796c9f
Reviewed-on: http://review.couchbase.org/13402
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
fire node_disco event when nodes_wanted is updated

It was previously possible to see some node in nodes() first and only
then on nodes_wanted(). In such case ns_node_disco_events event won't
be fired. This commit fixes it by forcing notify_clients check on
each update of nodes_wanted.

Change-Id: I66d84987e2451cfd676c44d31e75d21510d3c416
Reviewed-on: http://review.couchbase.org/13832
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
unexport externally unused function

So that there's less confusion on what's API and what is not.

Change-Id: I8f89c2c9a7639dc01010f74f228511efb759af3c
Reviewed-on: http://review.couchbase.org/13835
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Connection pooling of ns_memcached instances

Currently there's only one ns_memcached data process per bucket.
However, with xdcr, there will likely be many concurrent replication
streams coming into a given node, where they'll end up getting
serialized due to only one ns_memcached process serving their
requests. This change set makes it so that there can be multiple
processes for serving data requests. Their number is configurable by
means of an environment variable.

Change-Id: Ie81fb223324f3ca789f54568d09853fffcb9eb2e
Don't use num_items_for_persistence during xdcr checkpointing

ep-engine flusher decrements num_items_for_persistence before
actually persisting an item, so relying on its value becoming zero
before updating the checkpoint document is not safe. Instead, we wait
for the current open checkpoint id to be persisted. If it gets
persisted before we timeout, the checkpoint document is updated.
Otherwise, we don't checkpoint.

Change-Id: I1e86be03a67d60f9a22a36d5ef052fae53b414e9
Explicitly trigger doc replication upon startup

Due to a race condition, it may so happen that after a new node is
added to a cluster, the design/replicator docs on existing nodes do
not replicated to the new node. This change set explicitly triggers
the replication of all existing docs by sending the
replicate_newnodes_docs message to all the existing nodes.

Change-Id: I3d3f47d3a2546334dfdb3b4052019e06b99c33f2