Merge branch 'master' into 2.0.2

* master:
  MB-7493: allow last active server if it's unhealthy
  MB-7211: add default port to hostname
  MB-8009: couchbase logo update on UI
  MB-8229 Disallow renaming of nodes that are part of a cluster.
  MB-100 Don't enclose new address in excessive double quotes.
  MB-100 Don't log ns_rebalance_observer state.
  MB-100 Don't log state after every update_stats message.
  MB-8201 Don't update docs estimate if we can't grab it.
  MB-7863 Don't link to waited process in wait_for_process.
  MB-8126: start babysitter as hidden node
  MB-8189 Start child_erlang in a intermediate supervisor.
  MB-8189 Start ns_babysitter application as permanent too.
  MB-8189 Start ns_server application as permanent.
  MB-8196 Fix badmatch in handle_call_via_servant.
  MB-100:visualize-rebalance.rb: handle 2 checkpoint waitings/vb
  MB-8190: fixed race in janitor_agent:handle_call_via_servant
  MB-5307 Use define for ns_server logger category.
  MB-8034: impl. some ui for xdcr_optimistic_replication_threshold
  MB-8163: don't attempt to track stats if not all nodes are 2.0.2
  MB-100: nuked long unused _replicator_infos ddoc and related APIs
  MB-8127 Handle diag_failed when grabbing new per node diags.
  MB-8157: update couchbase favicon.ico
  MB-8013 Mention detailed rebalance progress feature in CHANGES.
  MB-8013 Detailed rebalance progress UI.
  MB-8013 REST API for detailed rebalance progress.
  MB-8013 Detailed rebalance progress backend.
  MB-8013 Notify rebalance start/end for memcached buckets too.
  MB-100: fixed dialyzer found bug
  MB-8132:cbcollect_info: don't use try..except..finally
  MB-100: returned +MMmcs option for ns_server VM
  MB-100: lowered babysitter memory usage
  MB-100: added listing of view index directory into cbcollect_info
  MB-100: added lvm diagnostics to cbcollect_info
  MB-8066: expose second checkpoint waiting in master events
  MB-100: fixed drop of rebalance-out nodes in visualize-rebalance
  MB-100: added another rebalance visualization
  MB-100: added scripts/analyze-rebalance-waiting.rb
  MB-100: capture all bucket's ddocs in cbcollect_info
  MB-100: added file fragmentation info to cbcollect_info output
  MB-100: try to avoid using 127.0.0.1 as cbcollect_info directory
  MB-7636: removed include_docs from UI
  MB-7988: Updated sparjquery.sparkline.js to version 2.1.1
  MB-8009: Couchbase Logo updated on UI
  MB-8009: removed unnecessary images
  MB-8028: updated copyright at footer
  MB-7994: don't cbcollect hash stats
  MB-100: added cbcollect_info option to grab just product diag
  MB-100: automagically append .zip to cbcollect_info output name
  MB-8101: changed cbcollect_info to use information from dump-guts
  MB-8101: implemented dump-guts script
  MB-8101: allow overriding of logs reports dir
  MB-8101: save full erlang init:arguments to initargs file
  MB-8101: added nodefile to static_config
  MB-100: fixed dialyzer reported typo
  MB-8098: delete ip_start when ip is updated
  MB-8088: prevent max_restart_intensity_reached during node rename
  MB-8085: detect x86_64-unknown-linux-gnu as GNU/Linux
  MB-100: don't consider failed over nodes for recovery
  MB-100: fixed dialyzer failure by correcting ns_log:log spec
  MB-8056: always set babysitter's cookie
  MB-8056: log reason of kill_memcached failures
  MB-100: mention MB-8046 in CHANGES
  MB-8046: don't make $prefix/var/lib/couchbase world-readable
  MB-100: fix potential parallel make breakage
  MB-8047: pass babysitter cookie to ns_server securely
  MB-100: start babysitter node with nocookie
  MB-100: use erlang:now for timer time
  MB-100: feed binaries to ns_log for some memory savings
  MB-8045: bumped default number of moves before compaction to 64
  MB-3321: skip timer interval ticks on clock jumps forward
  MB-100: batch all outstanding crashes before saving them
  MB-100: save crash log atomically
  MB-100: correctly treat empty crash log as empty queue
  MB-100: sync babysitter logs before dying
  MB-100: fixed indentation
  MB-8032: stop unwanted replications before setting vbucket states
  MB-100: correctly categorize tap replication changes
  MB-100:ale: really silence default error_logger
  MB-100: hibernate ns_config_log
  MB-100: slashed CPU consumption of timer
  MB-100: implemented quicker way of grabbing vbucket db infos
  MB-100: removed unused ns_watchdog and cucumberl
  MB-100: use much larger chunk in misc:raw_read_file
  MB-100: killed addr_util.erl
  MB-100: fixed build error due to bad .PHONY and rule interaction
  MB-100: provide api for testrunner for messing with memcached
  MB-7398: added CHANGES entry for hostname management feature
  CBD-220: run memcached and ns_server from babysitting VM
  CBD-220: [ale] allow syncing all log messages to all sinks
  CBD-220: switched ns_server_sup to supervisor2
  MB-100: added typespec to ns_pubsub:subscribe_link
  MB-7398: don't allow hostnames without dot
  MB-100: moved samples_loader_tasks earlier in ns_server_sup list
  MB-5307 Handle failure to save address after rename.
  MB-7398 Rename added node to what was specified as its address.
  MB-7398 Ability to specify server hostname in wizard.
  MB-7398 REST call to rename a node.
  MB-7398 Handle ip_start file in dist_manager.
  MB-7398 Move is_good_address to misc.
  MB-5307 Try to stop net_kernel on dist_manager start.
  MB-5307 Remove references to non-existent path-box class.
  MB-7604:cbcollect_info: curl /diag passing credentials properly
  MB-100: mention that old stats are lost on upgrade from mnesia
  MB-100: tolerate unknown nodes in master_events formatting
  MB-100: report replication docs and design docs in /diag
  MB-7961: don't try to create dbdir inside ns_bucket:config_string
  CBD-220: establish ns_memcached connection asyncrhonously
  CBD-816: log dropping of recovery status into ns_log
  CBD-816: bump mb_master_advertised_version to [2, 0, 2]
  CBD-816 Basic UI support for recovery mode.
  CBD-816 Recovery mode support.
  CBD-816 Export ns_rebalancer:maybe_cleanup_old_buckets.
  CBD-816 Export apply_new_bucket_config_new_style.
  MB-5307 Race-free handling of rebalance in auto_failover.
  MB-5307 Unify failover and try_autofailover type specs.
  MB-5307 Don't report that auto failover max reached every tick.
  MB-5307 Don't increment counter when autofailover is unsafe.
  MB-5307 Don't do unneeded list copies.
  MB-7902: Log additional info when nodedown is seen
  MB-5307 Don't log state when skipping janitor run.
  MB-5307 Replace ns_config:search by search_quick.
  MB-5307 Don't set vbucket state to dead before deleting.
  MB-7885 Don't do unneeded unquotes in xdc_rep_utils.
  MB-100: gather some numa stats in cbcollect_info
  MB-100: improve linux-specific info gathering
  MB-5307:mention rebalanceMovesBeforeCompaction setting in CHANGES
  MB-5307: dont crash doctor if somebody asks about unknown node
  MB-5307: allow 'rebalance' with specified vbucket moves
  MB-5307: bucket warming up on some node(s) is not error
  MB-5307: avoid parsing value that we already know
  MB-7758: implemented faster misc:dir_size
  MB-100: use quick version of cfg access to check rebalance_status
  MB-100: introduced ns_config:search_quick/1
  MB-100: clean up timer's kill_after timers
  MB-7670: don't save config on empty changes
  MB-7670: avoid key order order change in ns_config:update
  MB-100: fixed stats timestamp logging
  MB-100: stop messing with timer_server on stats messages
  MB-100: render visualize-rebalance timeline in real scale
  MB-5307 Handle colon in passwords correctly.
  MB-5307 Properly validate usernames and passwords.
  MB-7786 Group concurrent remote bucket requests.
  MB-7824 Hibernate ns_log and ns_log_sink.
  MB-7824 Convert messages to binary before sending to sinks.
  MB-7837: continuously tunable optimistic XDCR
  MB-7799 Use precomputed value in local_bucket_disk_usage.
  MB-100:explicitly forbid long un-supported joining of 1.7.x nodes
  MB-7743: always initialize ns_config_ets_dup data
  MB-100: correct typespec and silence dialyzer
  MB-100: un-write-only-ify scripts/visualize-rebalance.rb
  MB-7432: add XDCR percentage of completeness stat
  MB-7432: Add XDCR latency stats
  MB-7731 Count moves from 'undefined' nodes in total_in_flight.
  MB-7730 Master event for forced view compaction during rebalance.

Change-Id: Id2074e63718497888a6e86dd20c2758d6309f99f
MB-8263 Start ns_bucket_{worker,sup} in a dedicated supervisor.

With previous approach if both ns_bucket_worker and ns_bucket_sup
crash at about the same time, it was possible to lose the work that
would start single_bucket_sup's. To prevent this both ns_bucket_worker
and ns_bucket_sup are now started in a dedicated supervisor with
one_for_all strategy. So that if any of them crashes, the other gets
restarted and correctly reinitialized.

Change-Id: I288676eed3f016c1708d8f79b295a1f109c83835
Reviewed-on: http://review.couchbase.org/26331
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-8013 Don't update stats when we know nothing new.

Change-Id: I8068a69d0f8bfb37fd41c02eb9fc276cc437bec6
Reviewed-on: http://review.couchbase.org/26332
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-8013 Show details about ingoing transfers during rebalance.

Rebalance task updated accordingly, e.g.:

 {
        "detailedProgress": {
            "bucket": "default",
            "bucketNumber": 1,
            "bucketsCount": 1,
            "perNode": {
                "n_0@10.17.45.204": {
                    "ingoing": {
                        "activeVBucketsLeft": 0,
                        "docsTotal": 0,
                        "docsTransferred": 0,
                        "replicaVBucketsLeft": 0
                    },
                    "outgoing": {
                        "activeVBucketsLeft": 0,
                        "docsTotal": 1024,
                        "docsTransferred": 816,
                        "replicaVBucketsLeft": 0
                    }
                },
                "n_1@127.0.0.1": {
                    "ingoing": {
                        "activeVBucketsLeft": 0,
                        "docsTotal": 1024,
                        "docsTransferred": 816,
                        "replicaVBucketsLeft": 208
                    },
                    "outgoing": {
                        "activeVBucketsLeft": 0,
                        "docsTotal": 0,
                        "docsTransferred": 0,
                        "replicaVBucketsLeft": 0
                    }
                }
            }
        },
        "perNode": {
            "n_0@10.17.45.204": {
                "progress": 72.91666666666667
            },
            "n_1@127.0.0.1": {
                "progress": 100.0
            }
        },
        "progress": 86.45833333333334,
        "recommendedRefreshPeriod": 0.25,
        "status": "running",
        "type": "rebalance"
    }

Change-Id: I045341eaafd9e6c884dd7e149871763ce0e26724
Reviewed-on: http://review.couchbase.org/26333
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-8199 Throttle REST and CAPI requests.

There're three internal settings that control this:

 - limit on the number of REST connections (everything that goes
   through 8091 port)
 - limit on the number of CAPI connections (everything that goes
   through 8092 port)
 - memory threshold that when exceeded causes us to drop any incoming
   connection

Change-Id: Ib79c8068056193beecda536436572ca66b74f31f
Reviewed-on: http://review.couchbase.org/26334
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-100 Grab babysitter processes as part of diag.

Change-Id: I0cf14713b3b39ed0cca79d11ddeaf798485b7a18
Reviewed-on: http://review.couchbase.org/26336
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-100 Log couchdb messages only to the dedicated file.

Sometimes couchdb may be particularly loud which causes logs to be rotated
much faster than we would want.

Change-Id: I4020010e1175ce03d9b17d43002b5a141ff8b7bb
Reviewed-on: http://review.couchbase.org/26348
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-100: use proc_lib:spawn_link

So that crashes can be diagnosed easier.

Change-Id: I9ccf0762db41d215a0d6f2a94dd32a7fb1b6bb1c
Reviewed-on: http://review.couchbase.org/26352
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
MB-100 Update internalSettings in doc/api.txt.

Change-Id: I6729cdeb53711f6f3cbde7a39c8ef570a7f072c6
Reviewed-on: http://review.couchbase.org/26354
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-8199 Allow keeping request throttling settings intact.

So that if certain setting is just omitted, then the old value is
used. But if the value is empty, then we set it to unlimited.

Change-Id: I5c339ed50c98b07343c12e8c2790720b3d7b5cc3
Reviewed-on: http://review.couchbase.org/26355
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-8199 Prevent request_throttler from crash if it restarts.

Some of our REST endpoints restart almost all the supervision
tree. Previously note_request_done of such request would be handled
already by restarted process. This would cause it to crash and would
fail the request. To prevent this note_request returns throttler pid
as part of response. Then note_request_done uses this pid. So that new
instance of request throttler is unaffected. Additionally,
note_request_done now uses gen_server:cast instead of gen_server:call
to prevent request handlers from random request_throttler failures.

Change-Id: I9890d586fdfc25364de9153ae54925cb639a0c82
Reviewed-on: http://review.couchbase.org/26356
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
bp:MB-8242: fix outbound XDCR lazy computed stats

Change-Id: I589627d669639e6083c5d7376a4ca704a87872bb
Orig-Change-Id: If444c0369b832972b1f809f001e7126a577ac712
[alk@tut.by: "backported" to 2.0.2]
Signed-off-by: Aliaksey Kandratsenka <alk@tut.by>
Reviewed-on: http://review.couchbase.org/26360
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-100 Add a list of commands cbcollect_info depends on.

Change-Id: I9d27fe9faad5f67dead6f788df3175bc4edf7866
Reviewed-on: http://review.couchbase.org/26359
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-8199 Add entry about request throttling to CHANGES file.

Change-Id: I940132e644b7d19d7c8949cf287fa264ecf99f47
Reviewed-on: http://review.couchbase.org/26361
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-100: replaced orddict:find with lists:keysearch

Which we already know is much faster. Overall effect on bucket stats
building appears to be shaving off about 1/3rd of time.

Change-Id: I6393cccf204eb8c16551bdda68ed86993f3b0651
Reviewed-on: http://review.couchbase.org/26362
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Merge remote-tracking branch 'origin/2.0.2'

* origin/2.0.2:
  MB-100: replaced orddict:find with lists:keysearch
  MB-8199 Add entry about request throttling to CHANGES file.
  MB-100 Add a list of commands cbcollect_info depends on.
  bp:MB-8242: fix outbound XDCR lazy computed stats
  MB-8199 Prevent request_throttler from crash if it restarts.
  MB-8199 Allow keeping request throttling settings intact.
  MB-100 Update internalSettings in doc/api.txt.
  MB-100: use proc_lib:spawn_link
  MB-100 Log couchdb messages only to the dedicated file.
  MB-100 Grab babysitter processes as part of diag.
  MB-8199 Throttle REST and CAPI requests.
  MB-8013 Show details about ingoing transfers during rebalance.
  MB-8013 Don't update stats when we know nothing new.
  MB-8263 Start ns_bucket_{worker,sup} in a dedicated supervisor.

merge is trivial

Change-Id: Ie876a97bd92c7e0615f2ecc4ac7f9d0e4f0e353b