Truncate last unaligned bytes of access log

Truncating unaligned bytes of access log allows use of access log
even if last bytes are unaligned to blocksize.
Avoid throwing a ShortReadException which just disables the access log
Change-Id: I6e82fb0aa4b62e892e092c98654df71bee3f16d6
Reviewed-on: http://review.couchbase.org/36995
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
MB-12057: warmup must not block bucket creation

Bucket creation is a front end thread and warmup is done by
engine background threads. So if bucket creation waits for
warmup then a deadlock happens when there are few reader threads
Test Infrastructure expects vb0 created by ep-engine init
or warmup. This is not necessary for production, so moving
vb0 creation to test setup. Minor test code bug fixes.
Change-Id: I5eae9c8f36fe39d2a18cb1800dc48a29162eab28
Reviewed-on: http://review.couchbase.org/40892
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-12063: Lock the current snapshot mutex when flushing items

We need to lock the current snapshot mutex before grabbing items
because it is possible that an itme might be queued, but the
current snapshot might be updated.

Change-Id: Ieff85a4605041e764d6facf055c71c59e7e19955
Reviewed-on: http://review.couchbase.org/40996
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-12063: Properly save the max deleted seqno during vb snapshots

Change-Id: Ife9e3b26956638622de0427cd573cd4d37c7e443
Reviewed-on: http://review.couchbase.org/40997
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Fix build breakage

Change-Id: Ibeac0d57fc1ea3bc94bf488ee5e82cf7e9143d3a
Reviewed-on: http://review.couchbase.org/41012
Reviewed-by: Chris Hillery <ceej@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-12100: Don't expire items for replica vbuckets during warmup

Change-Id: I81095beb69c0996d819158e3cadc451a83653d2f
Reviewed-on: http://review.couchbase.org/41177
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I678740458ccc13cbf79be847369cf6561b5b678b
MB-12063 Don't delete expired items during warmup.

When a node is added back to the cluster, its vbucket states
can be different from its previous states (e.g., active -> replica).
Therefore, we should not remove expired items from active vbuckets
during warmup because the states of those vbuckets can be changed
after the warmup.

Change-Id: Ifec4fda80e424a8afa74bae66e17de77b2e0c36b
Reviewed-on: http://review.couchbase.org/41219
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
remove warmup wait dead code to prevent accidental deadlock

Since we should never have a front-end thread wait for background
thread, having a warmup wait logic will ensure that we do not use
it accidentally in the future and cause a deadlock

Change-Id: I28ec96dceaffa02810415402226df20f8eb44cc7
Reviewed-on: http://review.couchbase.org/41206
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Remove bgfetcher retry code

Change-Id: Ic87e1506a5ec5b8f7786c73064101871a63ddf36
Reviewed-on: http://review.couchbase.org/40959
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/3.0.1' into master

Change-Id: I09bbd9e506c65260feab6c345fc0a6c0fb5e42c2
Handle unlink/remove failures

The unlink/remove function can fail if there is another process that
has an open file handle on that file. In such cases, we need to retry
the unlink periodically in the flusher task until we get rid of those
files.

Change-Id: I4bfcf29b3fa866ec4946db658a245c722f3725ce
Reviewed-on: http://review.couchbase.org/41268
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Fix build breakage on Linux

Remove usage of constant strings for maintaining file deletion queue

Change-Id: I1992343ff65f923b6d3ffbf7a25932b936e3ff7c
Reviewed-on: http://review.couchbase.org/41291
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
MB-12137: Don't update the current snapshot during persistence

If we update the current snapshot only when we persist items to
disk then we may miss out on updates to the current snapshot
when there are vbucket state changes.

Change-Id: Iea3139afb669bdf32e6b4d98e8b3515cafe578dc
Reviewed-on: http://review.couchbase.org/41289
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/3.0' into 3.0.1

Change-Id: I195452c3d0684dff5ff673efd00a18b9d70da3cf
Merge remote-tracking branch 'gerrit/3.0.1' into master

Change-Id: I0d53fc77023a7bdfa7fcda50272240a9d4141a30
Fix deadlock in checkpoint persistence command

We need to release the hpChkMutex before notifying memcached in
order to prevent a deadlock. The deadlock occurs when the flusher
trys to notify memcached of checkpoint persistence at the same
time a memcached worker thread is trying to add a new checkpoint
persistence request.

Change-Id: Ida313f5b39ef0e063dee9882410cd0a19ce55292
Reviewed-on: http://review.couchbase.org/41363
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Fix errors in Blob comments

Spotted a few typos / inaccurate comments in the Blob class. Fix them.

Change-Id: I285a21307b2332bc2bdd84dce3fbb6728eeb8404
Reviewed-on: http://review.couchbase.org/41391
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Merge remote-tracking branch 'gerrit/2.5.0' into 3.0.1

Change-Id: I59c45877fc783e30c86dcfd7d303c731cde433fc
MB-11999: Preferential loading of active/replica vbuckets

During warmup, order the vbucket list such that
active vbuckets get 60% preference while replica
vbuckets get 40% preference.

Example:
In a 4 node cluster (DGM), approx. resident ratios:
1. Before warmup
    active:     36%     34%     34%     41%
    replica:    33%     35%     35%     27%
2. After warmup
    active:     40%     42%     41%     46%
    replica:    31%     29%     29%     24%

Change-Id: I60e0427bca58530247086d730135ebb4be70bb84
Reviewed-on: http://review.couchbase.org/41468
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>