MB-6662 Ignore the item's expiration time for SET_WITH_META

If a given item was already expired, but receives SET_WITH_META
with a positive cas value, that item should be still updated by
using the new value and meta data received as long as the cas
values are matched.

Change-Id: I2523040311247418ad079eac0dfaf91f3d67e2b2
Reviewed-on: http://review.couchbase.org/20927
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-6628: Seperate meta data bg fetch stats from bg fetch stats.

Change-Id: I0e80db7d4386ba4b9b872c158cec7fea74d78477
Reviewed-on: http://review.couchbase.org/20896
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-6614 Do not send CREATE flag when updating existing document

CouchKVStore saveDocs module for set/get operation should not
set the CREATE option for couchstore open db call.
This can cause a race condition in between creating
new revision doc and removing old revision doc by compactor.

Change-Id: I8fa5e75897c147dbeda8bbb9ad81739710fbdb30
Reviewed-on: http://review.couchbase.org/20933
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-6661: The delete packet returns a cas value

This is neccessary for clients to detect an update after a delete
when using the observe command.

Change-Id: Ifb0a3426baa143b2be9da8e901cdbd2583f9f2d3
Reviewed-on: http://review.couchbase.org/20854
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-6616 schedule each unique key in the access log for data loading

Fix the logic that detects duplicate key items in the access log,
so it correctly schedules each unique item for bgfetch (getMulti)

Change-Id: I38afeada422fd420a2a247c3226b49966806525b
Reviewed-on: http://review.couchbase.org/20764
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Jin Lim <jin@couchbase.com>
Add shutdown force & no force test cases with pending flusher items

Test cases newly added: test shutdown with force
                        test shutdown without force

Change-Id: If0a316a98033fe9e1fd06fa220a04793bfddbd89
Reviewed-on: http://review.couchbase.org/20806
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Jin Lim <jin@couchbase.com>
Fix minor memory leaks in ep_testsuit

Memory allocated from createPacket() must be release after
the completion.

Change-Id: If2cfc7a17a53e0fae21973371e32279cc4259e68
Reviewed-on: http://review.couchbase.org/20808
Tested-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
MB-6647 Reduce the num of items in "duplicate items on disk" test

We saw the sporadic timeout in "duplicate items on disk" unit
test because too many items are fetched from disk.

Change-Id: I6b3c7efb7dcc7b86a57a89cf41f903f269c7bdf6
Reviewed-on: http://review.couchbase.org/20842
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Jin Lim <jin@couchbase.com>
CBD-377: reset conn. engine-specific after sync vbucket deletion

Otherwise next sync vbucket deletion on same connection would be
completely ignored.

Change-Id: I47c4f7ba01d5eefd1374c692925cd35af009d00d
Reviewed-on: http://review.couchbase.org/20771
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Peter Wansch <pwansch@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-6614 log invalid docinfo error and continue bgfetch

Log error message when couchstore_get_docinfos_by_sequence() returns
invalid docinfo (unknown seq id) and continue bgfetch initiated by
getMulti().

Change-Id: I2ac334f682cd8159e5a3f69551298554caffe046
Reviewed-on: http://review.couchbase.org/20761
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-6711 Do not create new db file with old revision number

When COUCHSTORE_FILE_CREATE options is set for opening a db
file, first check if the requested file exists with a newer
revision number. Otherwise create one with the default revision
number.

Change-Id: I1cdef60564abe7e143f747d19fdaf02ecde6c5f7
Reviewed-on: http://review.couchbase.org/21056
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-6714 Wake up the flusher after creating a checkpoint by force

Whenever a new checkpoint creation is requested from the external
components (e.g., consistent view support, XDCR synchronization),
we should wake up the flusher if its disk write queue was empty,
so that the flusher can update the last persisted checkpoint id
immediately.

Change-Id: Ib224aa5f57e4a6d7e5bec686d9da07bfe2001f0f
Reviewed-on: http://review.couchbase.org/21067
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
MB-6714 Create a new checkpoint for the non-empty open checkpoint

We recently made some changes in checkpoint implementation such
that a new checkpoint creation requested from the external
components is always executed even if the current open checkpoint
is empty.

Due to another change (ac6c438d35cb433d30c139cc48266188e738c350),
we actually don't need to force a checkpoint creation if the
open checkpoint is empty.

Change-Id: I3c0d23cdc8c410471148a24ffd4208d05efe7c36
Reviewed-on: http://review.couchbase.org/21068
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
MB-6676 Don't abort for communication failures with mccouch

If CouchKVStore receives an unexpected error from mccouch
during shutdown, it should not abort. Otherwise, it would cause
data loss for soft shutdown.

Change-Id: I6bb01bbca3b8e62361e6a8c6da5001d6d9236c99
Reviewed-on: http://review.couchbase.org/20982
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
Fixed build errors

Change-Id: I558a446f50c528728f6eece0800a5e626a50c3df
Reviewed-on: http://review.couchbase.org/21081
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-6714 Persisted chk id is included in checkpoint_creation reply

The response to checkpoint creation request includes the
persisted checkpoint id in addition to the open checkpoint id.
The value format is 8 bytes (open checkpoint id) + 8 bytes (
persisted checkpoint id).

Change-Id: Ifbc5bb7b0a713e1aab8763ba4cd1d2d02c589bb1
Reviewed-on: http://review.couchbase.org/21086
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-6751: Fix vkey not found in cbstats script

Change-Id: I90946dd36079a01ea08f0165f1e9797551d1b27d
Reviewed-on: http://review.couchbase.org/21160
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
MB-6753 Provide the timestamp of TAP backfill start

This stat is used to debug when the backfill is scheduled.

Change-Id: I50713f00419c6d20ff023c2b4ce0e0eb1fe96b70
Reviewed-on: http://review.couchbase.org/21148
Reviewed-by: Chisheng Hong <chisheng@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chisheng Hong <chisheng@couchbase.com>
MB-6770 Add the recreate option to KVStore::delVBucket()

VBucket reset operation should delete a vbucket database file
and then recreate the empty database file. As the vbucket reset
operation internally invokes KVStore::delVBucket API, this API
should support the recreate option.

Change-Id: I70f429038a9ed572ddc5cc4e9ad046a257a27f66
Reviewed-on: http://review.couchbase.org/21189
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Karan Kumar <karan@couchbase.com>
MB-6764: cbepctl displays extended help

Change-Id: I6a92fe1544c6c19eb233e012bfa65ca90ba18646
Reviewed-on: http://review.couchbase.org/21246
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>