MB-5102 Don't assert for dead vbuckets in CouchKVStore::saveDocs

When rebalance out is performed, there could be a short time
window on existing nodes, which dead vbuckets on those nodes are
reactivated immediately before the ns-server janitor clears them
up. Consequently, assertion should not be generated for dead
vbuckets in the underlying KV store while persisting items into
those vbuckets.

Change-Id: Ibf2d615dd4b62a019b19f13afe865e338a5166ff
Reviewed-on: http://review.couchbase.org/15104
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5138 Restore the value length during the first phase warmup

After warmup is partially done and ns_server allows get/set
operations, a bgfetch can cause the value length mismatch assert
while the warmup thread restores data for a given key from disk.
This change fixes the above assertion by popualting the value
length field of a key when the first phase of warmup loads the
key into the hashtable.

Change-Id: I46ce6a96f1aa9a53c7f158a768dccef7987ab95f
Reviewed-on: http://review.couchbase.org/15167
Tested-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5125 Revert "Remove unnecessary dedup checks from beginFlush()"

This reverts commit 3c8d17ff067f36e148efef96cab4ffb451e93391.

We observed the seg fault crash from std::sort() if we sort
QueuedItem instances by std::string. We even replaced RCPtr with
shared_ptr, but still had the same issue.

Using std::set is more expensive, but we observed that it has
little impact on the frontend performance or write througput.

Change-Id: Iec3a6bbedd119e37c72e54de2de7cf844e1982ce
Reviewed-on: http://review.couchbase.org/15205
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
Revert "MB-5125 Revert "Remove dedup checks from beginFlush()""

This reverts commit 7eeab643e0ac01120aa093cd67c25f75b82ca419.

Change-Id: I2b8f1528966e114bbb68ec82df0aab942a6ca370
Reviewed-on: http://review.couchbase.org/15212
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-5125 Preserve strict weak ordering in compareing QueuedItems

std::sort requires a comparison function that preserves strict
weak ordering.

Change-Id: I7bd02df3d6cb6294916e530c9b8b549d639925a0
Reviewed-on: http://review.couchbase.org/15213
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
Revert "MB-5138 Restore a value size at the first phase warmup"

This reverts commit de4bdeee7a913a3bdc5817f1542d374e11250386.

This change still doesn't work with the sqlite-based multi phase
warmup.

Change-Id: I38c0b9a8d570d9c442ab46e883c3be1a3543bcbb
Reviewed-on: http://review.couchbase.org/15214
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
MB-5138 Return the value size as 0 for partially loaded items.

Change-Id: I723ee5823c9772c00eae921dc1843e4e78de2229
Reviewed-on: http://review.couchbase.org/15215
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Jin Lim <jin@couchbase.com>
Remove additional hash table lookup in PersistenceCallback

Change-Id: I07fcd81c1f50085f4fd16139f1199e7f505560b9
Reviewed-on: http://review.couchbase.org/15216
Tested-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
[MB-5107] Return correct metadata for deleted documents

A non zero length was being returned upon doing a get_meta for deleted
documents.

Change-Id: I801b512891b7d7f0be4ac18b7d89682b4bb65cd6
[MB-5072] Fix bgfetch of metadata to return ENGINE_SUCCESS

Both ENGINE_SUCCESS and ENGINE_KEY_ENOENT are a success for the fetch
of a non-resident key. In both these cases, we should return
ENGINE_SUCCESS so that a blocked memcached connection could be
unblocked and the get-meta request retried.

Change-Id: Ib40ab4c29c8a4e6c75616582467a4d57346b07b6
[MB-5074] Clear id in SET if the key's metadata bgfetch is pending

While the background fetch for a non-resident/non-existent key is
underway, if we receive a SET that will overwrite the temporary item
created for the background fetch, we must also clear the "id"
attribute of the associated StoredValue object as it no longer
constitutes a temporary item.

Change-Id: Id46fc6991b23d8919c36f2a670419d4f458732a4
[MB-5073] Check whether temp item is clean before restoreMeta()

While the bgfetch of a non-resident/non-existent key is underway,
there might be intervening mutations to the key in memory. In such a
case we must not restore the metadata of the key. We verify this by
checking whether the temp item is still clean.

Change-Id: Id5447af8ed5a78b03068eabb6a596c87634386f0
Fix the message logged when bgfetch of metadata fails

Change-Id: Ia98fd037686f1ac5e188ee28de016c52a87ac6f9
Encode expiration time instead of length in item revision ids

Change-Id: Ic24ce9946131f886cc554d17baefdbda2a9946eb
Do not set expiration time for temporary items

Previously, after the bgfetch for a temporary item completed, we used
to set it to expire after 1 hr. However, with the recent change of
encoding expiration time instead of length in an item's revid, it is
no longer safe to do this.

The new approach is as follows: After the bgfetch for metadata
completes, we set its seqno, cas, expiration time and flags
attributes. These are then used to service requests for get_meta.
Anytime either the item pager or the expiry pager runs, we look for
look for temporary items and delete them from memory. If CAPI does not
finish conflict resolution during this time, it will retry.

Change-Id: Ie4bc662c39e085c4bde73f279b845de5cd1723a3
Revert "Add a new engine parameter for expiring temp items"

This reverts commit d76c319ff4b5f76eb3e561c37771ade12930a880. This
parameter is no longer needed as we do not set the expiration time
for temporary items.
[MB-5088] Update cas, flags and exptime in delete-with-meta

In case of delete-with-meta, we are not updating the target's revid
with the source's cas, flags and expiration time attributes. This
change set fixes that problem.

Change-Id: Idb5ada5cb1851401a518661def3f83ff2360962c
Correctly initialize item seqno during warmup

Change-Id: I7d12131a936991c5670b8908f8d958fc81a9745d
Add unit tests for get_meta

The following cases are covered by these unit tests:
1. get_meta for an existent item
2. get_meta for a deleted item
3. get_meta for a nonexistent item

A new function add_response_get_meta() has also been added to allow
decoding the flags attribute in the response to a get_meta request.
The flags attribute contains information about whether the item being
queried is deleted or not.

Change-Id: I866464983e6076932d53e4c6aa81d8c1502378fd
Add unit tests for delete_with_meta

The following cases are covered:
1. delete_with_meta for a deleted item
2. delete_with_meta for a non existing item

delete_wit_meta for an existing item was already part of the unit
tests.

Change-Id: I2913c556ec828a95b83c0410fd1888599ea67a04