MB-11726: QueueDirty when item expired on warmup

The return value for unlocked_softDelete is
NOT_FOUND even when an expired item is deleted.
Thus in the deleteItem scenario, we will need to
queue a dirty item, if v is not NULL.

Change-Id: I78bfa5f082aebb0e2760d761f00fd0a37744e0c1
Reviewed-on: http://review.couchbase.org/39423
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-11749: Prevent GetLocked response from masking NOT_MY_VB

Change-Id: Ibd1e8fa8a1635746c838a753b46d3e93096fd5b8
Reviewed-on: http://review.couchbase.org/39450
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-9471: add atomic replace operation

Add an atomic replace operation

Change-Id: I3fa454301ac3745d960dbef0295996161ee1a8dd
Reviewed-on: http://review.couchbase.org/39109
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
MB-11434 distributed sleep on taskQ

Make shared thread pool workers sleep inside their respective
TaskQueue mutexes as opposed to a single global mutex to reduce
the CPU overhead of putting threads to sleep and waking them up.
We still try to wake only as many threads as ready tasks.
First try waking local threads, if still more tasks are ready,
then visit other TaskQueues to wake their sleeping threads

Results in 15% improvement in view operations local testing
Lowered scheduling overhead with larger number of threads

Change-Id: I8ca746a696b4b2166f327385125b8510c0567407
Reviewed-on: http://review.couchbase.org/39210
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
MB-11698: Release reference to all connections in upr upon shutdown

Change-Id: I5759f89dcf1faaa5f7902fdb8fa08604a34f5814
Reviewed-on: http://review.couchbase.org/39514
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-11731: Vbucket-level locking as opposed to shard-level

For flusher tasks:
- flushVBucket
- compactVBucket
- snapshot task
- vbDelete

Change-Id: I0f17ed7dacdc23556f110e271048836ed6701685
Reviewed-on: http://review.couchbase.org/34501
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-11731 remove shard level locking from ExecutorPool

we no longer need to block threads at shard level since
locks are acquired at the vbucket level

Change-Id: Iebc368d243499b76b5cf1f4f0d50c487866960dc
Reviewed-on: http://review.couchbase.org/39502
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
MB-11760 track max purge seqno on compactions' deletes

Also update cachedVBStates post compaction

Change-Id: I30dd24ee56ddd9ac3af0f6b2677ed595ec808f59
Reviewed-on: http://review.couchbase.org/39506
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Return want more when noop or flow control is successful

We want the memcached layer to call into our upr connection if there
is any sort of success on the previous call.

Change-Id: I092f92ef5a1de937b015a2f91e17c4306943d023
Reviewed-on: http://review.couchbase.org/39554
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11405 MB-11434 limit max writers to 8 (configurable)

In systems having 40+ cores, spawning 20 threads causes many
issues like heavy CPU and memory use.
Limit number of writers to 8, readers to 16
This change adds dynamic configurability to the shared thread pool.
It is now possible to tune a running system dynamically by setting limits
on the number of readers, writers, auxio and nonio threads using cbepctl
For example,
cbepctl set flush_param max_num_writers 1 (1 writer thread across buckets)
cbepctl set flush_param max_num_auxio 4
cbepctl set flush_param max_num_nonio 1
cbepctl set flush_param max_num_readers 1
Change-Id: Ie1314c2658792d6bd80d5f95fa57c94feb35b32b
Reviewed-on: http://review.couchbase.org/39475
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-11592: Use a queue_item in a MutationResponse instead of and Item

This will allow us to pull queued_items out of the CheckpointManager
and into and ActiveStream without adding any memory overhead. This
also simplified the cleanup of a MutationResponse because we no
longer need to seperately delete the item contained inside this
class.

Change-Id: I0fc67ac66da6bd0c0ad0cd0369ee0a377d2b3bd9
Reviewed-on: http://review.couchbase.org/39283
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11592: Create snapshots when getting items from the chk queues

This change introduces a new flag to indicate whether or not
ep-engine should create a new checkpoint for this snapshot. This
flag is used internally by ep-engine and only ep-engine. We are
also changing the flags values for snapshot markers since they
weren't really flags previously. Below are the new values.

Disk 0x01
Memory 0x02
Checkpoint 0x04

Change-Id: I8a262eb7856c97e11e95570fbdaef82f3f79054a
Reviewed-on: http://review.couchbase.org/38779
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11721: Send acks for snapshot markers during the takeover phase

By sending acks for snapshot markers we can make sure that all items
hve actually been processed on the new master before sending any
set vbucket state messages. This change will reduce the time that
a vbucket might be unavailable due to state changes.

Change-Id: I70b64035c4684754bfcdb632f29cb83fecf161a1
snap start/end seqno stuff in memory only

Change-Id: I7c9430e0b24ef01cc427cb03b518f4c41a55d875