MB-8486 Create a new CAS for an item with corrupted CAS in warmup

If the CAS value of a given item is corrupted during warmup, we
can simply create a new CAS value for that item instead of
aborting the process.

Change-Id: I02b4889c6f71d27f98946aeeb9529f90bdbff53b
Reviewed-on: http://review.couchbase.org/27376
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8012 Store time of deletion in expiry field

Using this timestamp the compactor can purge delete records once they
reach a certain age.

Change-Id: Ib541de9e1a5c14674eaabbea6c03629f9f713870
Reviewed-on: http://review.couchbase.org/27427
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100 make ep-engine compile

Change-Id: I72a1c766415c1e458d625f4d1f58fa56048ed10d
Reviewed-on: http://review.couchbase.org/27521
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8698: Fixed conditional check bug in warmup/loadingAccessLog

When loading access log during warmup, make sure we correctly
check for duplicate items and skip them. The bug caused all
items to be skipped.

Change-Id: I26f4452a9862ea69ab496b3d43758bb824412559
Reviewed-on: http://review.couchbase.org/27685
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8012: store deletion timestamp in correct byteorder

Change-Id: Ia2d3f1257cc530e7f81fbe610f44595bd96544b4
Reviewed-on: http://review.couchbase.org/27721
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
MB-8012 Fix to unit test failures in set/add/del/WithMeta ops.

We recently made a change in persisting a deletion so that we
put the current epoch timestamp of the deletion in the
expiration field of the key's meta data.

This change fixes the unit test failures due to the above change.

Change-Id: I15ffa887c5464c7185176173a68ba86a6a9833df
Reviewed-on: http://review.couchbase.org/27698
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Check for the existence of the file before opening it

I had to add the following test to avoid ep-engine to
dump core on SmartOS.

Change-Id: I139acd69de51547e71ea0f787e4c6f903f5e4ba4
Reviewed-on: http://review.couchbase.org/27922
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8581 Continue warmup after loading access_log.

Added a loadingData step after loadingAccessLog to meet
warmup watermarks. In order to avoid loading items already
loaded earlier in loadingAccessLog phase, we check if an
item is resident or not during loadingData phase.
A method "isLoaded" is added to LoadingStorageKVPairCallback
to perform the check.

Added a test case "warmup_accesslog" for this feature.

Change-Id: I5eeb86b43b4e05a87df97b103c253a2db79db5ed
Reviewed-on: http://review.couchbase.org/28038
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8657 fix total number of threads allocated for MRW

Upon this fix the total number of threads allocated for MRW
workers matches the max thread number specified by users via
UI.

Certain setting such as 4, 5, 7 may break the workload ratio (2:1)
btw numbers of readers and writers but will conform to the user
input thus avoid any confusion.

Change-Id: I3e45ebc38903d37df6746a9f20b9a1e8f6919e03
Reviewed-on: http://review.couchbase.org/28096
Tested-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Revert "MB-8581 Continue warmup after loading access_log."

This reverts commit 06c086ccb69109fb1a89c17a8bf6c509ea03a9c8.

Change-Id: I2354ff0e7e66fff46f0c78ecab6b150a0e03d706
Reviewed-on: http://review.couchbase.org/28141
Tested-by: Philip Labee <philip@couchbase.com>
Reviewed-by: Philip Labee <philip@couchbase.com>
MB-100: Fix build error on windows.

On windows, std::tr1 doesn't have definition for
static_pointer_cast

Change-Id: If978d165bd55833fde34e03a4e650eeb94166db5
Reviewed-on: http://review.couchbase.org/28149
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Revert "Revert "MB-8581 Continue warmup after loading access_log.""

This reverts commit b74fdcc82e895d36f974e8e5ac3554532a3cdb90.

Conflicts:
	src/couch-kvstore/couch-kvstore.cc

Change-Id: If7aae2ee894911038410eee82aa6ec5a1474fbc0
Reviewed-on: http://review.couchbase.org/28152
Tested-by: Bin Cui <bin.cui@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8679 Add pendingVbs queue to bgfetcher

This queue allows bgfetcher to visit only those vbuckets
in the given shard that have pending bgfetch requests.
This optmizes scheduling bgfetch requests in more efficient
fifo way.

Rebalance should benefit the above enhancement most, and initial
test shows below performance increase.

<Reb-1-out-3-buckets>

                2.1.0-718-rel ||  2.1.1-with-this-fix

Time hrs        4.48          ||  3.83

Rebalance
Time (sec)      3095          ||  2578

Sets (90-95-99)
(ms)            11 – 17 – 53  ||  6 – 9 – 18

Gets (90-95-99)
(ms)            13 – 20 – 208 ||  9 – 14 – 164

Ops (avg)       2.40k         ||  1.48k

Change-Id: Iadbbcb069701da947aac2a8cbc0d8b30762a13a4
Reviewed-on: http://review.couchbase.org/27178
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8453 Maintain the TAP connection list for each vbucket

When we receive a mutation event for a given vbucket, we
should check if any of its replicator TAP connections is
in a paused state, and notify that event to the paused
TAP connections, so that they can immediately send it
to their replicas.

To address the above issue, this change maintains a
separate list of TAP connections for each vbucket.
The list gets updated when an TAP connection is closed or
an TAP connection's vbucket list is changed.

Change-Id: I5ca49fe17fbeb406da3ebcc046285ddc164735eb
Reviewed-on: http://review.couchbase.org/27972
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8798 Remove dead conns by iterating the conn list one time.

Change-Id: I04a9e64462391b7b93249828d1ecbef3dfd60175
Reviewed-on: http://review.couchbase.org/28107
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8846 Use std::set instead of unordered::set.

The current mingw build env doesn't support unordered::set.

Change-Id: Ib4c6e635015ff464442430e138169d1fb067b063
Reviewed-on: http://review.couchbase.org/28257
Tested-by: Bin Cui <bin.cui@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-8828: Check memory condition before returning enomem for withmeta ops

If we run into a memory condition when we are trying to put something
into the hashtable then we return enomem even though we might actually
be able to free memory. This change makes sure we check the memory
condition before actually returning an error code.

Change-Id: I3d10641764653c971aaeb9e8089fb21fe891a699
Reviewed-on: http://review.couchbase.org/28269
Reviewed-by: David Liao <david.liao@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-8825: Add stat to keep track of conflict resolution fails

This stat will help with xdcr debugging

Change-Id: Ie2ea77f51f40581ab60c823cf40c5276b4e27845
Reviewed-on: http://review.couchbase.org/28271
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Merge branch '2.1.1'

* 2.1.1:
  MB-8798 Remove dead conns by iterating the conn list one time.
  MB-8453 Maintain the TAP connection list for each vbucket
  MB-8679 Add pendingVbs queue to bgfetcher
  Revert "Revert "MB-8581 Continue warmup after loading access_log.""
  MB-100: Fix build error on windows.
  Revert "MB-8581 Continue warmup after loading access_log."
  MB-8657 fix total number of threads allocated for MRW
  MB-8581 Continue warmup after loading access_log.
  Check for the existence of the file before opening it
  MB-8012 Fix to unit test failures in set/add/del/WithMeta ops.
  MB-8012: store deletion timestamp in correct byteorder
  MB-8698: Fixed conditional check bug in warmup/loadingAccessLog
  MB-100 make ep-engine compile
  MB-8012 Store time of deletion in expiry field
  MB-8486 Create a new CAS for an item with corrupted CAS in warmup

Conflicts:
	src/common.h
	src/couch-kvstore/couch-kvstore.cc
	src/ep.cc
	src/scheduler.cc
	src/tapconnection.cc
	src/tapconnection.h
	src/vbucket.h
	src/warmup.cc
	src/warmup.h
	src/workload.h

Change-Id: If473196d3813efc5ffe1da6dfbb021395bb84bee
MB-8861: Release bucket lock before queueing a dirty item

When we queue a dirty item we notify any tap connections that
are streaming items from the vbucket the item was added to. If we
don't release the lock then we can cause a deadlock on the
notifySync and queueLock variables in the tap replicator.

Change-Id: Ia8fc31f5c0ac33cf6a79ef4dd4337f8a6459d6ef
Reviewed-on: http://review.couchbase.org/28319
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>