MB-8244: Add dispatcher, worker thread stats

- Enables dispatcher stats
- Prints reader-writer thread stats
- Add condition to return immediately without
printing stats if engine in shutdown mode
- Changes based on previous comments included
- Adding a unittest: test_worker_stats
- Formated dispatcher stats (sample)

 iomanager_worker_0
    runtime: 6ms
    state:   running
    task:    Running a flusher loop

- Add stats' details in doc/stats.org

Change-Id: I034dc6ea04df0e7e018849e1ebe8c1f6a02d9246
Reviewed-on: http://review.couchbase.org/27121
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: abhinav dangeti <abhinav.dangeti@gmail.com>
MB-8578: Add stats uuid

Change-Id: I348f644e810711297700b5c94de075e3ee3d154d
Reviewed-on: http://review.couchbase.org/27356
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
MB-7935: Disabling setting vbuckets to dead state

- During warmup, ns_server blocks connections to vbuckets,
so ep-engine doesn't need to set the vbucket states to dead.
- master branch (for next release)

Change-Id: I095adfe0253c28212d8b46b6193c47771d5bee78
Reviewed-on: http://review.couchbase.org/26763
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8600: Add all args to log message to prevent possible segfault

Change-Id: I4a60431fd775670ef1353e7cf36884e7f1436b4c
Reviewed-on: http://review.couchbase.org/27431
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-8643: Remove registered tap client functionality

Change-Id: I510a96e46391bc684d6320853392c972a45d0217
Reviewed-on: http://review.couchbase.org/27336
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Merge remote-tracking branch 'gerrit/2.1.1'

Change-Id: Ib3a623d970d4c686b8d5c66d845bdf880f810d25
MB-8486 Create a new CAS for an item with corrupted CAS in warmup

If the CAS value of a given item is corrupted during warmup, we
can simply create a new CAS value for that item instead of
aborting the process.

Change-Id: I02b4889c6f71d27f98946aeeb9529f90bdbff53b
Reviewed-on: http://review.couchbase.org/27376
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8012 Store time of deletion in expiry field

Using this timestamp the compactor can purge delete records once they
reach a certain age.

Change-Id: Ib541de9e1a5c14674eaabbea6c03629f9f713870
Reviewed-on: http://review.couchbase.org/27427
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100 make ep-engine compile

Change-Id: I72a1c766415c1e458d625f4d1f58fa56048ed10d
Reviewed-on: http://review.couchbase.org/27521
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8698: Fixed conditional check bug in warmup/loadingAccessLog

When loading access log during warmup, make sure we correctly
check for duplicate items and skip them. The bug caused all
items to be skipped.

Change-Id: I26f4452a9862ea69ab496b3d43758bb824412559
Reviewed-on: http://review.couchbase.org/27685
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8012: store deletion timestamp in correct byteorder

Change-Id: Ia2d3f1257cc530e7f81fbe610f44595bd96544b4
Reviewed-on: http://review.couchbase.org/27721
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
MB-8012 Fix to unit test failures in set/add/del/WithMeta ops.

We recently made a change in persisting a deletion so that we
put the current epoch timestamp of the deletion in the
expiration field of the key's meta data.

This change fixes the unit test failures due to the above change.

Change-Id: I15ffa887c5464c7185176173a68ba86a6a9833df
Reviewed-on: http://review.couchbase.org/27698
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Check for the existence of the file before opening it

I had to add the following test to avoid ep-engine to
dump core on SmartOS.

Change-Id: I139acd69de51547e71ea0f787e4c6f903f5e4ba4
Reviewed-on: http://review.couchbase.org/27922
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8581 Continue warmup after loading access_log.

Added a loadingData step after loadingAccessLog to meet
warmup watermarks. In order to avoid loading items already
loaded earlier in loadingAccessLog phase, we check if an
item is resident or not during loadingData phase.
A method "isLoaded" is added to LoadingStorageKVPairCallback
to perform the check.

Added a test case "warmup_accesslog" for this feature.

Change-Id: I5eeb86b43b4e05a87df97b103c253a2db79db5ed
Reviewed-on: http://review.couchbase.org/28038
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8657 fix total number of threads allocated for MRW

Upon this fix the total number of threads allocated for MRW
workers matches the max thread number specified by users via
UI.

Certain setting such as 4, 5, 7 may break the workload ratio (2:1)
btw numbers of readers and writers but will conform to the user
input thus avoid any confusion.

Change-Id: I3e45ebc38903d37df6746a9f20b9a1e8f6919e03
Reviewed-on: http://review.couchbase.org/28096
Tested-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-7483: Remove duplicate stats holding bucket size

ep_max_data_size (removed)
ep_max_size

Change-Id: Ia29539cdef796961a240064e95375a3020dce760
Reviewed-on: http://review.couchbase.org/26945
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-8581 Continue warmup after loading access_log."

This reverts commit 06c086ccb69109fb1a89c17a8bf6c509ea03a9c8.

Change-Id: I2354ff0e7e66fff46f0c78ecab6b150a0e03d706
Reviewed-on: http://review.couchbase.org/28141
Tested-by: Philip Labee <philip@couchbase.com>
Reviewed-by: Philip Labee <philip@couchbase.com>
MB-100: Fix build error on windows.

On windows, std::tr1 doesn't have definition for
static_pointer_cast

Change-Id: If978d165bd55833fde34e03a4e650eeb94166db5
Reviewed-on: http://review.couchbase.org/28149
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Revert "Revert "MB-8581 Continue warmup after loading access_log.""

This reverts commit b74fdcc82e895d36f974e8e5ac3554532a3cdb90.

Conflicts:
	src/couch-kvstore/couch-kvstore.cc

Change-Id: If7aae2ee894911038410eee82aa6ec5a1474fbc0
Reviewed-on: http://review.couchbase.org/28152
Tested-by: Bin Cui <bin.cui@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8679 Add pendingVbs queue to bgfetcher

This queue allows bgfetcher to visit only those vbuckets
in the given shard that have pending bgfetch requests.
This optmizes scheduling bgfetch requests in more efficient
fifo way.

Rebalance should benefit the above enhancement most, and initial
test shows below performance increase.

<Reb-1-out-3-buckets>

                2.1.0-718-rel ||  2.1.1-with-this-fix

Time hrs        4.48          ||  3.83

Rebalance
Time (sec)      3095          ||  2578

Sets (90-95-99)
(ms)            11 – 17 – 53  ||  6 – 9 – 18

Gets (90-95-99)
(ms)            13 – 20 – 208 ||  9 – 14 – 164

Ops (avg)       2.40k         ||  1.48k

Change-Id: Iadbbcb069701da947aac2a8cbc0d8b30762a13a4
Reviewed-on: http://review.couchbase.org/27178
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>