MB-7902: Log additional info when nodedown is seen

Request net_kernel to supply a reason when sending nodedown
messages and log them, to aid debugging unexplained disconnects
during rebalance.

Change-Id: Ic4ef710b393e63c03385776e834d424c2119c396
Reviewed-on: http://review.couchbase.org/25297
Tested-by: Sriram Melkote <siri.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Merge remote-tracking branch 'origin/2.0.1'

* origin/2.0.1:
  MB-7902: Log additional info when nodedown is seen

Conflicts:
	src/ns_node_disco.erl

Change-Id: If353f8c3e2d88fd025e6e4cd8137ddf1720b1341
MB-5307 Don't do unneeded list copies.

Change-Id: I2315e8ed62cabc2d86fb4e8a71843817c4b7a3b9
Reviewed-on: http://review.couchbase.org/25298
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307 Don't increment counter when autofailover is unsafe.

And to prevent spamming someone's mailbox we keep a flag showing if
we've already reported this condition.

Change-Id: I312283d1f8a2a979570b152f1faaa69039673cc3
Reviewed-on: http://review.couchbase.org/25299
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307 Don't report that auto failover max reached every tick.

Change-Id: I78918b0b9caf3b8f9a373f6ece194db89b40c0bf
Reviewed-on: http://review.couchbase.org/25300
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307 Unify failover and try_autofailover type specs.

They used to return different replies in case rebalancing was going
on. Now both of them just fall through to the default rebalancing
handler that returns rebalance_running. Additionally, I added the code
to actually handle these return values.

Change-Id: Ic0a33afd7ecf770b208614f0d4ea71374d778d0c
Reviewed-on: http://review.couchbase.org/25238
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5307 Race-free handling of rebalance in auto_failover.

Change-Id: I7bc5e30fd92dae36d46bb045b67680c1c4e03e6d
Reviewed-on: http://review.couchbase.org/25301
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
CBD-816 Export apply_new_bucket_config_new_style.

It will be used for recovery. Additionally I made it return more
detailed error.

Change-Id: I4b0a502039c6e81e465035184f8c50876b10f5c6
Reviewed-on: http://review.couchbase.org/25239
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
CBD-816 Export ns_rebalancer:maybe_cleanup_old_buckets.

It will be used for recovery. I also refined the way it handles errors
so that it would be more convenient to use it outside rebalancer.

Change-Id: Ifaf1cbe7cfe08f1cde3fa1900c3b03f82cb5aa1c
Reviewed-on: http://review.couchbase.org/25240
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
CBD-816 Recovery mode support.

When a membase (couchbase) bucket has some vbuckets missing it can
be put into a recovery mode using startRecovery REST call:

 curl -sX POST -u Administrator:asdasd \
      http://lh:9000/pools/default/buckets/default/controller/startRecovery

In case of success, the response looks as follows:

 {
     "code": "ok",
     "recoveryMap": [
         {
             "node": "n_1@10.17.40.207",
             "vbuckets": [
                 33,
                 34,
                 35,
                 36,
                 37,
                 38,
                 39,
                 40,
                 41,
                 42,
                 54,
                 55,
                 56,
                 57,
                 58,
                 59,
                 60,
                 61,
                 62,
                 63
             ]
         }
     ],
     "uuid": "8e02b3a84e0bbf58cbbb58919f1a6563"
 }

So in this case replica vbuckets 33-42 and 54-63 were created on
node n_2@10.17.40.207. Now the client can start pushing data to
these vbuckets.

All the important recovery URIs are advertised via tasks:

 curl -sX GET -u 'Administrator:asdasd' http://lh:9000/pools/default/tasks

 [
     {
         "bucket": "default",
         "commitVbucketURI": "/pools/default/buckets/default/controller/commitVBucket?recovery_uuid=8e02b3a84e0bbf58cbbb58919f1a6563",
         "recommendedRefreshPeriod": 10.0,
         "recoveryStatusURI": "/pools/default/buckets/default/recoveryStatus?recovery_uuid=8e02b3a84e0bbf58cbbb58919f1a6563",
         "stopURI": "/pools/default/buckets/default/controller/stopRecovery?recovery_uuid=8e02b3a84e0bbf58cbbb58919f1a6563",
         "type": "recovery",
         "uuid": "8e02b3a84e0bbf58cbbb58919f1a6563"
     },
     {
         "status": "notRunning",
         "type": "rebalance"
     }
 ]

 - stopURI can be used to abort the recovery
 - recoveryStatusURI will return information about the recovery in the
   same format as startRecovery
 - commitVBucketURI will activate certain vbucket

   This call should be used after the client is done with pushing
   the data to it. VBucket is passed as a POST parameter:

    curl -sX POST -u 'Administrator:asdasd' \
         http://lh:9000/pools/default/buckets/default/controller/commitVBucket?recovery_uuid=8e02b3a84e0bbf58cbbb58919f1a6563 \
         -d vbucket=33

    {
        "code": "ok"
    }

All the recovery related REST calls return a JSON object having a
"code" field. This (together with HTTP status code) indicates if the
call was successful.

Here's a complete list of possible REST calls replies.

 - startRecovery

   +-------------+-------------------+------------------------------------+
   | HTTP Status |       Code        |              Comment               |
   |             |                   |                                    |
   +-------------+-------------------+------------------------------------+
   |          200|        ok         |Recovery started. Recovery map is   |
   |             |                   |returned in recoveryMap field.      |
   +-------------+-------------------+------------------------------------+
   |          400|    unsupported    |Not all nodes in the cluster support|
   |             |                   |recovery.                           |
   +-------------+-------------------+------------------------------------+
   |          400|    not_needed     |Recovery is not needed.             |
   +-------------+-------------------+------------------------------------+
   |          404|    not_present    |Specified bucket not found.         |
   +-------------+-------------------+------------------------------------+
   |          500|   failed_nodes    |Could not start recovery because    |
   |             |                   |some nodes failed. A list of failed |
   |             |                   |nodes can be found in the           |
   |             |                   |"failedNodes" field of the reply.   |
   +-------------+-------------------+------------------------------------+
   |          503| rebalance_running |Could not start recovery because    |
   |             |                   |rebalance is running.               |
   +-------------+-------------------+------------------------------------+

 - stopRecovery

   +-------------+---------------+------------------------------------+
   | HTTP Status |     Code      |              Comment               |
   |             |               |                                    |
   +-------------+---------------+------------------------------------+
   |          200|      ok       |Recovery stopped successfully.      |
   +-------------+---------------+------------------------------------+
   |          400| uuid_missing  |recovery_uuid query parameter has   |
   |             |               |not been specified.                 |
   +-------------+---------------+------------------------------------+
   |          404| bad_recovery  |Either no recovery is in progress or|
   |             |               |provided uuid does not match the    |
   |             |               |uuid of running recovery.           |
   +-------------+---------------+------------------------------------+

 - commitVBucket

   +-------------+------------------------+------------------------------------+
   | HTTP Status |          Code          |              Comment               |
   |             |                        |                                    |
   +-------------+------------------------+------------------------------------+
   |          200|           ok           |VBucket commited successfully.      |
   +-------------+------------------------+------------------------------------+
   |          200|   recovery_completed   |VBucket commited successfully. No   |
   |             |                        |more vbuckets to recover. So the    |
   |             |                        |cluster is not in recovery mode     |
   |             |                        |anymore.                            |
   +-------------+------------------------+------------------------------------+
   |          400|      uuid_missing      |recovery_uuid query parameter has   |
   |             |                        |not been specified.                 |
   +-------------+------------------------+------------------------------------+
   |          400| bad_or_missing_vbucket |VBucket is either unspecified or    |
   |             |                        |couldn't be converted to integer.   |
   +-------------+------------------------+------------------------------------+
   |          404|   vbucket_not_found    |Specified VBucket is not part of the|
   |             |                        |recovery map.                       |
   +-------------+------------------------+------------------------------------+
   |          404|      bad_recovery      |Either no recovery is in progress or|
   |             |                        |provided uuid does not match the    |
   |             |                        |uuid of running recovery.           |
   +-------------+------------------------+------------------------------------+
   |          500|      failed_nodes      |Could not commit vbucket because    |
   |             |                        |some nodes faileed. A list of failed|
   |             |                        |nodes can be found in the           |
   |             |                        |"failedNodes" field of the reply.   |
   +-------------+------------------------+------------------------------------+

 - recoveryStatus

   +-------------+---------------+------------------------------------+
   | HTTP Status |     Code      |              Comment               |
   |             |               |                                    |
   +-------------+---------------+------------------------------------+
   |          200|      ok       |Success. Recovery information is    |
   |             |               |returned in the same format as for  |
   |             |               |startRecovery.                      |
   +-------------+---------------+------------------------------------+
   |          400| uuid_missing  |recovery_uuid query parameter has   |
   |             |               |not been specified.                 |
   +-------------+---------------+------------------------------------+
   |          404| bad_recovery  |Either no recovery is in progress or|
   |             |               |provided uuid does not match the    |
   |             |               |uuid of running recovery.           |
   +-------------+---------------+------------------------------------+

Recovery map generation is very simplistic. It just distributes
missing vbuckets to the available nodes and tries to ensure that
nodes get about the same number of vbuckets. It's not always
possible though, because after failover we often have quite
unbalanced map. The resulting map is likely very unbalanced too. And
recovered vbuckets are not even replicated. So in a nutshell,
recovery is not a means of avoiding rebalance. It's suitable only
for recovering data. And rebalance will be needed anyway.

Change-Id: I84a80b3d5643133fea3f84add06dcc3c8eb81d7b
Reviewed-on: http://review.couchbase.org/25241
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
CBD-816 Basic UI support for recovery mode.

When cluster is in recovery mode, servers section will have
additional button to abort the recovery. Rebalance, fail over and
other similar buttons will be disabled.

Change-Id: I88db22066dade6a97b2acf38b10aa04d370cf7e9
Reviewed-on: http://review.couchbase.org/25242
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
CBD-816: bump mb_master_advertised_version to [2, 0, 2]

So that 2.0.2 node takes over 2.0.1.

Change-Id: Ib0d22e34aeebdab9872ff4dad070847c744b6338
Reviewed-on: http://review.couchbase.org/25323
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
CBD-816: log dropping of recovery status into ns_log

It deserves to be very visible.

Change-Id: I92d1db7f5d30b686e0b195d6bed7cd72f6e6887b
Reviewed-on: http://review.couchbase.org/25324
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
CBD-220: establish ns_memcached connection asyncrhonously

So that shutdown during that phase is possible

Change-Id: I98d9c78328e3c1c8b297dca3a05c1f2f9aed166c
Reviewed-on: http://review.couchbase.org/25325
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-7961: don't try to create dbdir inside ns_bucket:config_string

Because config_string is assumed by it's callers to be pure.

Change-Id: Iec68afe519168d3cacbf0dab7378636a782e2aa0
Reviewed-on: http://review.couchbase.org/25326
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
switched ns_server_sup to supervisor2

So that we don't have to rely on supervisor_cushion for compaction
daemon.

Change-Id: I1199636cbf1142a4742423120b443ae7cec09de9
mark

Change-Id: Id4f1e6185726a2e9040b88400a99e38d44390bb0
some work on child_erlang

Change-Id: I18164d654b491b9888418a567a37eee506235bf1
amend

Change-Id: Ia36f124e907931bfa2360c924d88a8da2c7d6344
mark

Change-Id: I284e155a510c7e6f5c1e45cf780ccf370ee3f5ef