MB-4898 Rebuild mutation log with the database if the log is empty

There is a case where the database has some data, but the mutation
log is empty when the warmup starts (e.g., cluster upgrade).
In this case, the mutation log should be reconstructed from
database during warmup. Otherwise, data loss can happen when the
node is restarted, because the mutation log might not contain all
the entries.

Change-Id: I2fdbb9458c54f6e074b15fb18f7d757234bcc3c9
Reviewed-on: http://review.couchbase.org/14007
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Merge branch 'branch-18'

* branch-18:
  MB-4898 Rebuild mutation log with the database if the log is empty

Change-Id: Iac13d9c31cd7c169f324a8f88a9614292055057b
Track max seqno of deleted items per vbucket

In order to guarantee monotonically increasing seqno values even for
recreated items i.e., items that are deleted and reintroduced using
an add or a set command, we track the max seqno of all deleted items
seen so far. Then, when an item is reintroduced, we pick this max
seqno value, increment it by 1 and assign it to the "new" item.
Although conservative, this guarantees that if an item is thus
reintroduced, it gets a seqno that's higher than its previous value.

Change-Id: Ibb30fc5a90db21e05bf1043bfcb1c1050f3913ce
Reviewed-on: http://review.couchbase.org/11901
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Load the per vbucket maxDeletedSeqno values during warmup

Change-Id: I5b5ca992d90f25328791ede70827efcefe319daf
Reviewed-on: http://review.couchbase.org/11902
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
delete-with-meta implementation in ep-engine

Change-Id: I879aa4f4d4a71b6080613673f6c72e7408d0ee4c
Reviewed-on: http://review.couchbase.org/12007
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Introduce static consts to track StoredValue states

This is a refactoring change set and introduces a few static consts
to track the different states of the id attribute of a StoredValue
object.

Change-Id: I9224a55f1e025ec48e77762e79ea838c1fda3b98
Reviewed-on: http://review.couchbase.org/12450
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Add new stored value states for tracking deleted items

For servicing getMetaData() requests for possibly deleted items, we
create a temporary item, add it to the hashtable and schedule a
background fetch of its metadata from the persistent store. After the
fetch completes, we restore its metadata in memory (if it was found
in the persistent store) or mark it as non existent. We also set it
to automatically expire after a predetermined but configurable amount
of time. As such, in order to correctly track the background fetch,
the following new stored value states are required:
1. deleted_key: after the fetch completes, if the item was found
   deleted in the persistent store, it is marked as deleted_key.
2. non_existent_key: after the fetch completes, if the item was not
   found in the persistent store, it is marked as having never
   existed. engine_key_noent will be returned if the item is queried.

This change set introduces these states. The actual logic of handling
deletes as described here is part of subsequent change sets.

Change-Id: I7511e24b6e3ae51f1ba5b06a92b0b83ec0a2b160
Reviewed-on: http://review.couchbase.org/12204
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Add a new engine parameter for expiring temp items

This configurable parameter's value will be used to automatically
expire temporary items created in response to getMetaData() requests
for keys that aren't found in memory.

Change-Id: I5abb4d86c2832247682034426081b789cdf15f58
Reviewed-on: http://review.couchbase.org/12206
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Add a resetValue() method to StoredValue class

Change-Id: I6b60a2629c82dfa96969e86a8233036a34c0b4d1
Reviewed-on: http://review.couchbase.org/12451
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Add an unlocked version of the add() method

This is a refactoring change set and will be used subsequently for
adding temporary items to the hash table while servicing get-meta
requests.

Change-Id: I0e9f969a8ebaad24f9b468781878fed5ba279bc5
Reviewed-on: http://review.couchbase.org/12452
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Introduce addTempDeletedItem() method to StoredValue

This will be used to add temporary items to the hash table during
get-meta processing.

Change-Id: I7b5856285bfd1abcbab455aff1d4a141f996ac9a
Reviewed-on: http://review.couchbase.org/12453
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Rename restoreValue() to unlocked_restoreValue()

This function assumes the applicable hashtable bucket is already
locked. Hence, renaming the function accordingly.

Change-Id: Ie83668888d730a0356272ce881b3531de5396c8b
Reviewed-on: http://review.couchbase.org/12679
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Introduce unlocked_restoreMeta() method in StoredValue

Similar to the unlocked_restoreValue() method, this method will be
called upon completion of a background fetch of a temporary item's
metadata. After restoring the metadata, this method also sets it to
expire after a predetermined amount of time (default is 60 secs). Any
number of requests within this window will be served with the fetched
metadata.

Change-Id: I7153c64bd35a8b57c0e4d4186008c505fdfd357f
Reviewed-on: http://review.couchbase.org/12207
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Add support for get-meta in mc-engine

The get() method and GetResponseHandler() class in mc-engine now
check for whether a partial load has been requested. If so, a getmeta
request is issued to mccouch and the metadata returned back to the
caller.

Change-Id: Ie85f74aa47168cac9a0f50935c8419906a6a0638
Reviewed-on: http://review.couchbase.org/12454
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Add method to GetValue to specify fetch of metadata

This change set adds a setter method for the partial attribute of the
GetValue class.

Change-Id: Idd13e81bbac9e0ca95520ad37fecdeaaca2fba4a
Reviewed-on: http://review.couchbase.org/12455
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Schedule background fetch for non-existent items in getMetaData()

If the key being queried doesn't exist in memory, create a temporary
item and schedule a background fetch for its metadata from the
persistent store. Until the fetch completes, the temporary item will
be in meta_pending state during which further invocations of this
function will be blocked. After the fetch completes and the state has
transitioned to either meta_available or non_existent, the correct
state will be returned.

NOTE: We currently return ewouldblock in the meta_pending state This
isn't optimal from a performance point of view. This will change
shortly -- we will implement an asynchronous mechanism so that the
connection doesn't have to block until the fetch completes.

Change-Id: I9a761b916920ffe00291251873f6c26bcf96dfed
Reviewed-on: http://review.couchbase.org/12456
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Expire temporary items created in response to get-meta requests

Upon restoring deleted items from the persistent store, we set them
to expire after a predetermined amount of time (default is 60 secs).

This change set modifies the Deleter class to also look for temporary
items and delete them when the following conditions are met:
1. They are either in the meta_available or non_existent states i.e.,
   the background fetches for these items have completed. If a fetch
   is in process, we let it finish.
2. They have expired.

Also, upon deleting temporary items, we don't put them on the check-
point queues as don't need to be either persisted or replicate to
another node.

Change-Id: Ife51b5b37e72f63db817f4c7519f5e4da3cd8eaa
Reviewed-on: http://review.couchbase.org/12211
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Return CAS along with enoent in get_meta()

When we receive a get_meta request for a non-existent key, we create
a temporary item and schedule a background fetch to check whether the
item exists as deleted in Couch. After the background fetch completes
we return either the deleted revid or enoent (if the key never wasn't
found in Couch). This change set also returns the CAS initially
assigned to the temporary item. In case of XDCR, CAPI will use the
CAS to issue a subsequent add_/delete_with_meta command.

Change-Id: Ice47710713dc05e4d2cd25fbc9865ae97a9f2328
Reviewed-on: http://review.couchbase.org/13754
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Fix delete-with-meta check

The extlen parameter has a value of 4 for delete-with-meta command
rather than 12.

Change-Id: I0b6aa20725bf623980140e0048f74db1356bd5af
Reviewed-on: http://review.couchbase.org/13756
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Remove object length check in setWithMeta()

Per the new CouchDB changes, we don't store the length attribute as
part of the revid anymore. After discussing with Damien, we decided
to remove this check as it doesn't apply anymore.

Change-Id: I80c09fa8cc454ef361c65f5ea1943647a97b06d4
Reviewed-on: http://review.couchbase.org/13757
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>