MB-6952 SetWithMeta & GetWithMeta do not set nru bit

Items being replicated by XDCR should not set reference bit (nru)
by default.

Basic unit test has been added to the existing bg_meta_stats
case.

Change-Id: I135b9878cf7060715277c221cfdb37af6c91e723
Reviewed-on: http://review.couchbase.org/21729
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6963: List all stats in stats.org file

Change-Id: Ibf375ed8d8b304bca65e4371de7cfbb5ebc484b3
Reviewed-on: http://review.couchbase.org/21756
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-6976 Make checkpoint_flush_timeout adjustable at runtime

VBucket flush prioritization has an interal timeout on checkpoint
persistence. The timeout needs to be adjusted at runtime
in order to give more flexibilities to the external components
that request the prioritization. In the large cluster tests,
we see that more than 30 - 40% of checkpoint persistence requests
have 10 secs - 2 minutes range timings when view indexing and
querying are being executed during rebalance.

The internal timeout is dynamically adjusted between 10 and 60
secs by considering the lastest checkpoint persistence timing.

Change-Id: I4352333a932c7e0f2bcc56829e3b2b1ef26f30eb
Reviewed-on: http://review.couchbase.org/21857
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-6991 Don't resume TAP producer in case of shutdown

If the engine is supposed to shut down, we don't need to resume
any TAP producers that were temporarily suspended. Otherwise,
there could be a race condition between TAP producer removal
and resume tasks, which causes memory reference errors.

Change-Id: I1a92487ae90935ae20921b519c1dcbfcc2e3ce5e
Reviewed-on: http://review.couchbase.org/21887
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
MB-6961 Fix validation of numbers as JSON

Change-Id: Ibe14f516283ccdfd26795a0b9e9659d3b5cf8932
Reviewed-on: http://review.couchbase.org/21901
Reviewed-by: Chris Anderson <jchris@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
MB-7006 Add tests for JSON validation

Change-Id: I7ece614adf3dc6c8be92d6168d3e80f4aae7418c
Reviewed-on: http://review.couchbase.org/21905
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Tested-by: Steve Yen <steve.yen@gmail.com>
MB-7008 Remove vbucket reference overhead in the flusher loop

There is unnecessary vbucket reference in the flusher loop,
which causes the reference count tracking overhead.

Change-Id: I3b1ab0487b13ba25ab54cf8f6a0b76fe78f0eed3
Reviewed-on: http://review.couchbase.org/21975
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-7022 Remove the empty flush queues for non-existent vbuckets

We maintain a separate flush queue per vbucket, but should
remove the empty flush queue if its vbucket doesn't exist
anymore on a given node.

Change-Id: Ie43ccf06d5b8c8c26efddbbabef65e1acbff8e2d
Reviewed-on: http://review.couchbase.org/21976
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-7023 Fix to race conditions between vbucket delete and recreate

resetVBucket() internally deletes the vbucket from memory and
disk, and recreate the empty vbucket in memory and disk. If there
are any dirty items for that vbucket between these two
operations, we might not persist those dirty items.

Change-Id: I41d5f210978545e491056a075531b16a1c92979d
Reviewed-on: http://review.couchbase.org/21978
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-7026 Drop disk backfilled items filled by memory backfill

If an item backfilled from disk is resident in memory hash
table and has the same seq num and cas value as the ones in
memory, that disk backfilled item can be simply ignored because
the same item was already (or will be) replicated by the memory
backfill operation.

Change-Id: I881872ed23bad52b1a7324a10c760be97d98a012
Reviewed-on: http://review.couchbase.org/21981
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-7028 Unit test for MB-7026

This change adds the simple unit test for

MB-7026 Drop disk backfilled items filled by memory backfill

Change-Id: I482fe22e61ad14157910e6f9d90db59963695709
Reviewed-on: http://review.couchbase.org/21997
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-6992 Add more informative logs to checkpoint prioritization

Change-Id: I328fa8b4b6c1cd6e115125b18e9995f5dafc270e
Reviewed-on: http://review.couchbase.org/22000
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-7023 Fix to race conditions in vbucket delete / recreate

Due to the cluster rebalance, a given vbucket can be
removed and recreated in memory and disk in a short time.

In this case, the dirty items in the vbucket queue, which
belong to the deleted vbucket should be simply skipped.
In addition, persisting incoming dirty items, which belong to
the newly created vbucket, should be blocked until the new
vbucket database is created on disk.

Change-Id: If733a9e758993066344a4ed3314fcb89bb354b24
Reviewed-on: http://review.couchbase.org/22001
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-6992 Control the flusher execution by the transaction size

We recently removed the transaction size limit, which determines
how long the flusher can be executed by the RW thread without
being preemted by other higher priority tasks. However, this
allows the flusher to be run continuously without preemption
until its outgoing queue becomes empty. Consequently, other
high priority tasks such as vbucket deletion, vbucket snapshot,
should wait for a long time in the thread queue.

To resolve this issue, we bring back the transaction size limit
to the flusher, so that the flusher can yield the thread
when it reaches to the transaction size limit.

Change-Id: I15f2cb0b6855b1bc1553398db7c7e0349f2640c5
Reviewed-on: http://review.couchbase.org/22022
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
Revert "MB-7026 Drop disk backfilled items filled by memory backfill"

This reverts commit c9f4470890c9d57cef22d0822aeca7d5569760a0.

Change-Id: I9d5fbe62f5cba8cb699f9ed192fd96d3dee86529
Reviewed-on: http://review.couchbase.org/22031
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Revert "MB-7028 Unit test for MB-7026"

This reverts commit f9bfdb3502c3a42999d2e881573464fc525227bd.

Change-Id: I54daab29ea1c6ac060bf8f8f85757e2742577851
Reviewed-on: http://review.couchbase.org/22032
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-7050 check pending fetch request right before going into snooze

Change-Id: I0141309c88b7d9c14b408b355efd213bd0b67a32
Reviewed-on: http://review.couchbase.org/22062
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-6494 Add more logs to vbucket reset and deletion functions

Change-Id: I7f8ece7b42423753bb1698bc0e8c32c76045e176
Reviewed-on: http://review.couchbase.org/22064
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-7017 Separate entry per high priority checkpoint flush request

The flusher can receive multiple high priority checkpoint
persistence requests for the same vbucket within a short time
window. Therefore, it should maintain a separate entry per
request with the corresponding upstream connection cookie, and
notify either SUCCESS or TIMEOUT to that connection cookie.

Change-Id: I9263d0161363ca88b55ece99a38da81a5f87656f
Reviewed-on: http://review.couchbase.org/22105
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-7045 Start a new transaction for high priority VB checkpoints

This change makes sure that a new transaction should be started
for persisting high priority vbucket checkpoints.

Change-Id: Ia738db340125000a4015096903cfceee80b61c7d
Reviewed-on: http://review.couchbase.org/22106
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Jin Lim <jin@couchbase.com>