Truncate last unaligned bytes of access log

Truncating unaligned bytes of access log allows use of access log
even if last bytes are unaligned to blocksize.
Avoid throwing a ShortReadException which just disables the access log
Change-Id: I6e82fb0aa4b62e892e092c98654df71bee3f16d6
Reviewed-on: http://review.couchbase.org/36995
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
MB-11601 Fix to the memory leak in processing UPR messages.

If UPR_MUTATION fails to be sent or a message type is UPR_DELETION,
then we should free an Item instance after processing an
UPR_MUTATION or UPR_DELETION message.

Change-Id: I61849a1074f829b00d5191eb1417ee60059d4716
Reviewed-on: http://review.couchbase.org/39065
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Revert "MB-11553 Reduce the min/default/max number of checkpoint items."

This reverts commit a72bea627fe0b9ca2d8f1e53947c9e554a8c67d5.

We observed some performance regression in view indexing building.

Change-Id: Ie7fb174c8d21353f21bc787574876f0445c485f5
Reviewed-on: http://review.couchbase.org/39090
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-11553 Reduce the min/default/max number of checkpoint items

This reverts commit af0f22545d9fd5f2ab675329de5d540ff5990c01.

It turns out that this change has nothing to do with view index
performance.

Change-Id: I416de3fbb125579cded468584e6ace98e1f34207
Reviewed-on: http://review.couchbase.org/39092
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10252 Add the descriptions of memory-related stats to stats doc

Change-Id: Ib0039cc6526eed54c32fb87c8584f8ac7e82c620
Reviewed-on: http://review.couchbase.org/39108
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Return clusterConfig in response, when NOT_MY_VB

Context:CompactDB

Change-Id: Iaabcd2f1a1f6bbe672f83cdf679b64f2a08cf9b2
Reviewed-on: http://review.couchbase.org/39144
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-11410: Use last persisted seqno as end seqno for disk only upr stream

This will allow upr streams to function properly when a consumer
requests a disk only snapshot and the flusher is paused. Note that
this is also more correct in terms of what the disk only stream is
going to send anyways.

Change-Id: If532a0d0b93a23adf76e7754eae127c0ae917a87
Reviewed-on: http://review.couchbase.org/39166
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11361: Implement upr noop command

The upr noop command is used to detect dead connections. It is sent
once every 3 minutes and waits 3 minutes for a response. If no
response comes in that time the connection is deemed to be dead. The
noop command is only sent when flow control is turned on by the
consumer.

Change-Id: I2eb9fc201eeb91878f9339c6b3b582cd2ca5da06
Reviewed-on: http://review.couchbase.org/38439
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11361: Make upr no-ops optional

Clients that are both on localhost do not need to use no-ops and
can choose not to turn this feature on for their connection.

Change-Id: I0d1186461e3aee5efde05cce0472f64655a809ab
Reviewed-on: http://review.couchbase.org/38991
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Move flow control logic into its own function

Change-Id: Iab3e1afdee4666ff3ea07cdda06445d1f9933df3
Reviewed-on: http://review.couchbase.org/39176
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11621: Grab streamMutex before reconnecting stream during rollback

When we reconnect a stream we notify the memcached layer that the
stream has been reconnected. The notification mecachism in the
consumer assumes that we already are holding the stream lock so we
need to make sure we grab it in order to avoid a race condition we
modifying the ready queue.

Also note that the opaqueMap relies on the streamMutex which is why
we grab it where we do.

Change-Id: Ia65c0ab2db3500f7480818f849000bc5ee3db0af
Reviewed-on: http://review.couchbase.org/39181
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-11658 compactor must purge tombstones if drop_deletes set

This must be done unconditionally. Current logic will not
purge any tombstones if ns_server does not set the drop_delete flag.
However in 2.5 the expectations is that if
drop_delete flag in compactor is set, delete all tombstones unconditionally
if drop_delete flag is not set, purge tombstones based on purge_seqno value
Change-Id: I5d6060fb3e4c65a7f6850f30db0cf61a9e1c3d92
Reviewed-on: http://review.couchbase.org/39183
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
MB-11629: In CompactVBTask, set completeBeforeShutdown to true

Possible race condition between handleDisconnect and notify from
CompactVBTask, causing the session counter to be
decremented for a compactVB task before it completes execution,
and when it does run later, results in an underflow of the
session counter.

Change-Id: Ie3681f84925df7bf4cf71672b6625aa43b1c6a4d
Reviewed-on: http://review.couchbase.org/39212
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-11573 Don't check the cached number of items in disk backfill

We should avoid checking the cached number of items from the
read-only kvstore instance because it is not always consistent
with the one on disk. In addition, there is a case where there
are only deleted items in the Couchstore file, which causes the
cached item counter to zero. In this case, we still need to read
deleted items from disk for UPR streaming.

Change-Id: I04c98414ccedb707dd171b050252f1c465c7795d
Reviewed-on: http://review.couchbase.org/39224
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Release a hashtable partition lock before UPR/TAP notification.

This change removes the unnecessary locking overhead of hashtable
partition lock while notifying UPR/TAP producer of an arrival of
a mutation.

Change-Id: I186dbb6a94438290557bf6456b1c5ccb8a2e4484
Reviewed-on: http://review.couchbase.org/39239
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-11676: Properly set the end seqno for disk only streams

It's possible that if there were no persisted disk items that we
would not schedule a backfill and go straight to the in-memory
phase which is incorrect for a disk only stream. This change also
makes sure that the end seqno is reported correctly in the stats
for a disk only stream.

Change-Id: I30c473149fb37f38e47f5c8c87c7d7db60608223
Reviewed-on: http://review.couchbase.org/39242
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-10962 Don't snooze the connection notifier unnecessarily.

There is a race in pausing and resuming the connection
notifier upon receiving a new mutation. To address this issue,
This change adds a flag that indicates if there is a pending
notification event for any TAP connections.

Note that this change was originally merged in 2.5.0 branch to
address the sporadic high latency issue in TAP-based replication.

We still need to address it in the UPR-based replication
separately.

Change-Id: Ia8d39220c46647a21254c63f37e7ce3ceaa2bc99
Reviewed-on: http://review.couchbase.org/39250
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Wake up the connection notifier iff it is in sleep state.

Change-Id: Ibd2579aeaa456fb14904b27ca00e453c56d33786
Reviewed-on: http://review.couchbase.org/39251
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Rename TAP-specific variables and functions to generic ones.

Change-Id: I02f4b3fa317e109c2ab6105a77a2379c3aa6ce83
Reviewed-on: http://review.couchbase.org/39252
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Move ConnMap::notifyVBConnections to TapConnMap.

ConnMap::notifyVBConnections function is only used by TapConnMap.

Change-Id: If0eec43dc07253fa9c757910bb3ff110512220b3
Reviewed-on: http://review.couchbase.org/39253
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>