MB-10962 Don't snooze the TAP connection notifier unnecessarily.

There is a race in pausing and resuming the TAP connection
notifier upon receiving a new mutation. To address this issue,
This change adds a flag that indicates if there is a pending
notification event for any TAP connections.

Change-Id: Ide00d835ef3d407087c7313f2346394b7f37c630
Reviewed-on: http://review.couchbase.org/36361
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Use spinlocks instead of conn mutex to guard UPR notifications
similar to the way TAP does. This will decrease lock contention
since conn mutex is no longer needed here.

We now need to manipulate vbconns structure for UPR too:
add and remove connection for a stream based on vbid.

Also Refactored ConnMap/TapConnMap/UprConnMap as needed.

Change-Id: I86ac07d29f3575d334f3b7f27530dc6019adf293
Reviewed-on: http://review.couchbase.org/35050
Tested-by: David Liao <david.liao@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10928: Correct the status check for rollback

- status to match PROTOCOL_BINARY_RESPONSE_ROLLBACK,
not ENGINE_ROLLBACK

Change-Id:I53f0d150b6d4b8267e5bb460d13f51076029b4ad
Reviewed-on: http://review.couchbase.org/36572
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Revert "Use spinlocks instead of conn mutex to guard UPR notifications"

This reverts commit 6003f370772d611fa7981e84bb135911761176f8.

Change-Id: I1affd06e7c02b43e3ad18f498b2f891fea9466c4
Reviewed-on: http://review.couchbase.org/36696
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-10652: Uncomment the notifier test

I resolved the issue that was causing this test to fail in
memcached.

Change-Id: Ia6b336375d40cc57b443cbcb3ade614808ec00e5
Reviewed-on: http://review.couchbase.org/36656
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Check if datatype is supported for Meta commands

For the following commands:
- SET_WITH_META
- ADD_WITH_META
- SET_RET_META
- ADD_RET_META
We need to inspect the document for JSONness
only if datatype isn't supported for the connection.

Change-Id:I51331552b21d97243ec5a94549efb917b65e179e
Reviewed-on: http://review.couchbase.org/36738
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Always check opaque in upr consumer response handler

Change-Id: I0912d5615c8297a1b921205837b70e40a4f29f39
Reviewed-on: http://review.couchbase.org/36553
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Improve logging for upr by having a standard place for the vbucket

This logging change makes sure that all upr log messages contain
the log header and also the vbucket id at the beginnning of the
message. This helps with debugging by allowing easier grep'ing
of these logs.

Change-Id: I6c504ca2baa0971823d3b5584a1455ed31740715
Reviewed-on: http://review.couchbase.org/36554
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
CBD-485: Remove couch-kvstore TMPFAIL & retry logic

with compaction moved to ep-engine completely, we no longer
need this logic.

replaced it with asserts instead.

Change-Id: I23d98baaebd52728a77730942bb4cfb5551c834b
Reviewed-on: http://review.couchbase.org/36716
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
MB-10856 Position the cursors correctly in collapsed checkpoints

Change-Id: I362f232640458420122f477943c6a6fe9da9ef0f
Reviewed-on: http://review.couchbase.org/36821
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
MB-11036: Validation of the session cas

Validate the session_cas for all the following
ns_server requests:
- CMD_DEL_VBUCKET
- CMD_SET_VBUCKET
- CMD_SET_PARAM
- CMD_DEREGISTER_TAP_CLIENT
- CMD_CHANGE_VB_FILTER
- CMD_SET_CLUSTER_CONFIG
- CMD_COMPACT_DB

Change-Id: I6863566e60e2d1b442018315cc282878dc1b904b
Reviewed-on: http://review.couchbase.org/36736
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Disable intra-cluster replication flow control

This should be disabled temporarily in order to all qe to make
progress on testing while we debug an issue in flow control.

Change-Id: I37bfac8511c4380362ba870d4268f1653ff64526
Reviewed-on: http://review.couchbase.org/37000
Tested-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix sporadic unit test failure in upr end stream test

Because we now se flow control the actual end stream reasponse is
processed by another thread. As a result it is possible that the
other thread hasn't processed this item when we check the stats. As
a result we need to wait until the stat changes.

Change-Id: Icdb71d61e5842da462b9668321f444a6868bee05
Reviewed-on: http://review.couchbase.org/37015
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Fix broken unit test "test rollback to zero on consumer"

This fixes a segfault on ubuntu and centos.

Change-Id: Ie0d889e1d85764e31155e868ac2698ebd38559cc
Reviewed-on: http://review.couchbase.org/37001
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Add items to retry queue when vbucket is in deletion state

It is possible that ep-engine may create, delete, and then recreate
a vbucket in a short timespan during vbucket takeover. When this
happens both the vbucket creation and deletion flag will be set an
this means that it is possible that the last created vbucket will
be visited by the flusher. If this happens the flusher will drop the
items from the vbucket and result in items not getting persisted to
disk. This change causes items that the flusher is trying to persist
to be re-queued for persistence when we are in this state.

Change-Id: I34ea9208abddad34d9506c20ee7e693f957d2de9
Reviewed-on: http://review.couchbase.org/37018
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Set the persistence seqno only when the reject queue is empty

We should only set the seqno a vbucket has persisted up to if
the reject queue is not empty in order to ensure that we don't
miss persisting any items at a lower seqno.

Change-Id: I5d7296897a5c60b5789e2111e3288c2c3137a9ef
Reviewed-on: http://review.couchbase.org/37019
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Set the number of backfill items only when we actually begin backfill

If we get the number of items in then constructor then it is
possible that their might be more items when we actually begin
backfill. This causes a problem when the num items was 0 in the
constructor and then non-zero when we act.

Change-Id: I81b039e0a78ecfe289904bbe13e60c927eaa39cc
Reviewed-on: http://review.couchbase.org/37020
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-10640 Improve UPR rollback during failover.

Stream request now has snap_start_seqno and snap_end_seqno.
(high_seqno is removed) which can be used to deal with
partial snapshot. failover check is based on start_seqno,
snap_start_seqno and snap_end_seqno.

Snapshot Marker now has start_seqno, end_seqno and flag.
start_seqno/end_seqno indicates sequence number ranges
of the stream.

The flag indicates if a stream is from disk or memory.
The current value are:
0: memory
1: disk

Change-Id: I4f7528f453e1f0c3021ab51f2173e111ed4a17e9
Reviewed-on: http://review.couchbase.org/35198
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Add a module test for the failover table

This change intends to introduce a place for module testing the
failover log. It is not intended to be a complete set of tests.

Change-Id: I75dabff76547f9d090870fabae9cacaee773db16
Reviewed-on: http://review.couchbase.org/36729
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11072 Don't check the expiration time for temp items.

We don't need to check the expiration time for temp items when
we fetch an item from the hash table.

Change-Id: I627eb296bedf48fa2868e838ac787fb03e54eb08
Reviewed-on: http://review.couchbase.org/37075
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>