MB-9159 Batch read uses a key instead of a sequence number.

When an item is evicted from memory, we plan to evict all the
metadata fields including the item's sequence number in addiiton
to the item's value. This new eviction approach requires a
batch read to use a key instead of a sequence number when it
reads items from disk.

This change adapts the batch read, so that it can use a key to
fetch an item's metadata or value from disk.

Change-Id: Ief14fd0ef4f42b316e75c6719349cdb1c63fac80
Reviewed-on: http://review.couchbase.org/29181
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
CBD-1034 Support the full eviction as an optional eviction policy

When an item is evicted from the cache, its value is only
evicted from memory by default. To further reduce the memory
overhead, this change supports the full eviction as an additional
option, which removes an item's key, metadata, and value from the
cache. The eviction policy can be chosen when a bucket is
instantiated.

If the full eviction option is chosen, this results in affecting
lots of engine APIs and also causes more disk IO overhead.
For example, CAS for a non-resident item will require a backgroud
fetch because it should know the item's CAS value.

Change-Id: I2edbe2b12527ef1c1fb31992cb04afbb7c078dac
Reviewed-on: http://review.couchbase.org/29155
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
CBD-1040 Adapt warmup to not load all the keys with full eviction

When the full eviction is selected, we don't need to load all the
keys and their metadata during warmup. This means that the warmup
process skips "key_dump" phase and enters into
"loading_access_log" phase.

If the access log is corrupted or not found, then the warmup
process will enter into "loading_kv_pairs" phase.

Change-Id: Ia2a04e18d4f051ba5aac0a6b26d8e8f3073ca83c
Reviewed-on: http://review.couchbase.org/29702
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-9404 FlushAll should be invoked across all the shards

FlushAll(i.e., database reset) was only invoked with a single
shard before. This can result in an inconsistent state and
doesn't clean up the underlying storage states.

Change-Id: I6485e22e6234c488eeb87e648c42f1468d863648
Reviewed-on: http://review.couchbase.org/29707
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix minor space issues.

Change-Id: I2f10227d4b785dfd88ca06c9cdbdc707cb034cd9
Reviewed-on: http://review.couchbase.org/30272
Reviewed-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
Tested-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
MB-8988: Sequence numbers are assigned by the CheckpointManager

Change-Id: If3fdf08e1091ad36ea8bcd30b10cd3803db009ae
Reviewed-on: http://review.couchbase.org/29226
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix warnings in ep-engine

Change-Id: Idc21a4f79b1b927a8e7b14ac2cbf52ac47873229
Reviewed-on: http://review.couchbase.org/30280
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Adapt to changes in couchstore_last_os_error

Change-Id: I0ec035cf698c9312ab99207de35a6223ac6c8441
Reviewed-on: http://review.couchbase.org/30256
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Updates to add stream response api based on memcached changes

Change-Id: I46d0022f8cdd86be3399c646afadcba629ef6b56
Reviewed-on: http://review.couchbase.org/30273
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-9521: Flusher must retry vbucket if no snapshot

When a bucket is created, ep-engine sets the flag
isBucketCreation into the vbMap for all its vbuckets.
These flags are only cleared by the the snapshot task
which should run first on all the vbuckets.
But if a flusher task runs first, then it will simply skip
that vbucket, Even if it has dirty items to flush!
Flusher should be able to retry a vbucket if it's
isBucketCreation flag is set so that it can run again
after the snapshot task is done.
Change-Id: Id41d5bf1dcc4e2459af5bb3719ff161f240c03d8
Reviewed-on: http://review.couchbase.org/30269
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
UPR consumer add_stream command

The upr add stream command is used in order to force a
consumer to initiate a connection with a producer. When
this command is received by a consumer it will initiate
an upr handshake over the same connection. Upon handshake
completion or failure a response for the add stream
command will be sent indicating the status  of the stream.

Change-Id: I40332293a9ae5df0c366297041b4d8d3fa871668
Move Producer stats into TapProducer class

These stats all apply to tap and most of them will not be shared
with UPR.

Change-Id: I412ecf7a1cbef3bd6a1369955a79286bd67dd74b
Move flags field from Producer to TapProducer

Upr connections will not have this concept so it should be in the
tap producer class only.

Change-Id: I4b43a6ebc97ea3d6b91435b2b187fbfb2db5ed07
WIP

Change-Id: Ie63d9f3438b091de309b67b3ad7372bd2876c451