MB-12057: warmup must not block bucket creation

Bucket creation is a front end thread and warmup is done by
engine background threads. So if bucket creation waits for
warmup then a deadlock happens when there are few reader threads
Test Infrastructure expects vb0 created by ep-engine init
or warmup. This is not necessary for production, so moving
vb0 creation to test setup. Minor test code bug fixes.
Change-Id: I5eae9c8f36fe39d2a18cb1800dc48a29162eab28
Reviewed-on: http://review.couchbase.org/40892
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-12063: Lock the current snapshot mutex when flushing items

We need to lock the current snapshot mutex before grabbing items
because it is possible that an itme might be queued, but the
current snapshot might be updated.

Change-Id: Ieff85a4605041e764d6facf055c71c59e7e19955
Reviewed-on: http://review.couchbase.org/40996
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-12063: Properly save the max deleted seqno during vb snapshots

Change-Id: Ife9e3b26956638622de0427cd573cd4d37c7e443
Reviewed-on: http://review.couchbase.org/40997
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Fix build breakage

Change-Id: Ibeac0d57fc1ea3bc94bf488ee5e82cf7e9143d3a
Reviewed-on: http://review.couchbase.org/41012
Reviewed-by: Chris Hillery <ceej@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-12100: Don't expire items for replica vbuckets during warmup

Change-Id: I81095beb69c0996d819158e3cadc451a83653d2f
Reviewed-on: http://review.couchbase.org/41177
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I678740458ccc13cbf79be847369cf6561b5b678b
MB-12063 Don't delete expired items during warmup.

When a node is added back to the cluster, its vbucket states
can be different from its previous states (e.g., active -> replica).
Therefore, we should not remove expired items from active vbuckets
during warmup because the states of those vbuckets can be changed
after the warmup.

Change-Id: Ifec4fda80e424a8afa74bae66e17de77b2e0c36b
Reviewed-on: http://review.couchbase.org/41219
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
remove warmup wait dead code to prevent accidental deadlock

Since we should never have a front-end thread wait for background
thread, having a warmup wait logic will ensure that we do not use
it accidentally in the future and cause a deadlock

Change-Id: I28ec96dceaffa02810415402226df20f8eb44cc7
Reviewed-on: http://review.couchbase.org/41206
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Remove bgfetcher retry code

Change-Id: Ic87e1506a5ec5b8f7786c73064101871a63ddf36
Reviewed-on: http://review.couchbase.org/40959
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/3.0.1' into master

Change-Id: I09bbd9e506c65260feab6c345fc0a6c0fb5e42c2
Handle unlink/remove failures

The unlink/remove function can fail if there is another process that
has an open file handle on that file. In such cases, we need to retry
the unlink periodically in the flusher task until we get rid of those
files.

Change-Id: I4bfcf29b3fa866ec4946db658a245c722f3725ce
Reviewed-on: http://review.couchbase.org/41268
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Fix build breakage on Linux

Remove usage of constant strings for maintaining file deletion queue

Change-Id: I1992343ff65f923b6d3ffbf7a25932b936e3ff7c
Reviewed-on: http://review.couchbase.org/41291
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
MB-12137: Don't update the current snapshot during persistence

If we update the current snapshot only when we persist items to
disk then we may miss out on updates to the current snapshot
when there are vbucket state changes.

Change-Id: Iea3139afb669bdf32e6b4d98e8b3515cafe578dc
Reviewed-on: http://review.couchbase.org/41289
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I195452c3d0684dff5ff673efd00a18b9d70da3cf
Merge remote-tracking branch 'gerrit/3.0.1' into master

Change-Id: I0d53fc77023a7bdfa7fcda50272240a9d4141a30
Fix errors in Blob comments

Spotted a few typos / inaccurate comments in the Blob class. Fix them.

Change-Id: I285a21307b2332bc2bdd84dce3fbb6728eeb8404
Reviewed-on: http://review.couchbase.org/41391
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10496: Initial version of HashTable defragmenter

Create a new Defragmenter task, which visits all members of the
HashTable and defragments them. Currently the defragmentation
algorithm is very naive - it simply reallocates all Blob objects
encountered, regardless of how fragmented or not the pages they reside
on. This is obviously very wasteful (as we will move object in both
fragmented and not fragmented pages on each pass), but however is
sufficient to significantly reduce RSS of the memcached process, if
jemalloc is used as the memory allocator.

jemalloc is required as it prefers low addresses during re-use;
and so the act of allocating a new Blob object, copying the old to the
new and then free()ing the old Blob will result in the new objects
being packed with other objects in the same size class.

To obtain this behaviour, we need to disable jemalloc's thread cache
for the thread which is performning the defragmentation, for the
duration of that task, and hence this requires the use of new
functions from the allocator hooks API.

Change-Id: I8f0e8daaa5e5d7ea8fde58f1c8377bf30312c85a
MB-10496: run_defragmenter_task for ep_engine

Change-Id: Ieaeadfbd4186213e35a1e89bc0202d39ff27ab96
MB-10496: Unit test for degragmenter task

Basic smoketest for the defragmenter: Create a number of documents,
spanning at least two or more pages, then delete most (but not all) of
them - crucially ensure that one document from each page is still
present. This will result in the rest of that page being
'wasted'. Then verify that after defragmentation the actual memory
usage drops down to (close to) mem_used.

Change-Id: Ibfe8780fb02ad5eb504242b01e7583bc1e305184
MB-10496: Add LongTermStorage class, use for Blob objects

This makes use of the new memory allocator hooks to allocate certain
objects which are expected to be long-lived in their own region of
memory. Initally this is just used for Blob objects - i.e. document
bodies held in RAM.

Change-Id: Ia8950d153a296feb29e7d66dc8c3b3d089d089be