Update due to refactor of memcached_ntohll

Change-Id: Ib657c2f5813a2ea7ab33d86c9407a96b23e39be9
Reviewed-on: http://review.couchbase.org/29241
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
CBD-1097: GIO can Limit num of threads on a Queue

GIO Manager can now put a limit on the number of threads
which can pick up Tasks from specific queues.
This feature is currently used to ensure that at most 1 thread
ends up picking up tasks from the AuxIO queue so that we don't
end up in a case where all the threads are working
on the long running cron jobs of the AuxIO queue while there are
other higher priority tasks waiting to run.

Change-Id: I27ba74f0d9f3e0acf03c79604f36b26a53f60f83
Reviewed-on: http://review.couchbase.org/29136
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-9038: AuxIO dispatcher tasks to the global IO

GIO to schedule following tasks to auxIOQueue
- BackfillDiskLoad
- AccessScanner - VBucketVisitorTask
- TapBGFetchCallback
- WarmupStepper

Change-Id: If5196af6996c1dc7ffde041ae4029ee47d171d87
Reviewed-on: http://review.couchbase.org/29014
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-9173: Refactoring code in IOManager module

Change-Id: Ide088a95a85012e82dad51927d9228b3128a30ff
Reviewed-on: http://review.couchbase.org/29178
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Remove the checkpoint extension command

This was part of maintainence mode in 1.8.x and is not longer
supported in the 2.x series of Couchbase.

Change-Id: Iedbc87af2d006915aa811f6eac32c5282673ccc9
Reviewed-on: http://review.couchbase.org/29192
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-9179 Shard count made configurable

This change makes the number of shards configurable when a bucket
is instantiated and sets its default number to four. As part of
the global IO manager optimization, the workload policy (i.e.,
read-heavy, write-heavy, and mixed) parameter, which is used in
determining the number of shards, is no longer necessary.

As part of further enhancements, we plan to investigate
differentiating the number of shards depending on a bucket
priority.

Change-Id: I88f1dd0b681aaa562415c2748f8a0d9d9b1a26e3
Reviewed-on: http://review.couchbase.org/29234
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100: Fix the engine name in the readme since we've evolved.

Change-Id: Icdd348f34a85bafaa24c24f18c9370dff3e93fc7
Reviewed-on: http://review.couchbase.org/29305
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100: Add a note about versioning, tags to deconfuse the situation.

Change-Id: If585bccfe24aa09f4de1779f32185819171b6900
Reviewed-on: http://review.couchbase.org/29306
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
mb-9202: Fix bucket deletion hang

Cancel Access scanner task on shutdown
scheduler must cancel tasks if it is dead regardless of return type
order the #include in alphabetical order to fix make test failure
Change-Id: Id995e15cca925a78d9588167c59201ff92e2af96
Reviewed-on: http://review.couchbase.org/29343
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Cast mc_ssize_t to ssize_t

Change-Id: Ie8fa31ef74eebe558aaa9f5c6488ef5aa7af552e
Reviewed-on: http://review.couchbase.org/29351
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
MB-9237: Fix bucket creation hang with 3 replicas

With new shard based thread change, it can happen that the
total number of threads does not match the total number of
workers set on each bucket.

Hence fix startWorkers() to exit immediately if we have
hit the max limit on the number of IO worker threads

Change-Id: I9884d3edb1ef3e38360ce961f7799594ecaacc7a
Reviewed-on: http://review.couchbase.org/29396
Reviewed-by: abhinav dangeti <abhinav.dangeti@gmail.com>
Tested-by: abhinav dangeti <abhinav.dangeti@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-8606: Parallelizing warmup as a READER task

Improve the efficiency of warmup, by having multiple
threads catering to it, rather than a single thread.

Taking into considering GIO, the warmup task will now
be made a READER_QUEUE task rather than an AUX_QUEUE
task.

The advantage will be with spliting up the keyDump,
LoadingKVPairs and LoadingData state_tasks during
the warmup process into shard level tasks, and have
multiple reader threads cater to each of the sub-
scheduled tasks.

This shouldnt affect any other READER_QUEUE pertinent
task, as traffic is disabled during warmup anyway.

Change-Id: I74fbe3fd97ed34e5af9a1df121ec1988b88be651
Reviewed-on: http://review.couchbase.org/29313
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-9098: Acquire lock to retrieve list size

- Need for synchronizing access to shared data
to avoid segmentation faults and possible
memory corruption.

Change-Id: If02dcf0d61a397960f0e7bd0b738fe1bb5f93cbb
Reviewed-on: http://review.couchbase.org/29477
Reviewed-by: abhinav dangeti <abhinav.dangeti@gmail.com>
Tested-by: abhinav dangeti <abhinav.dangeti@gmail.com>
MB-9306 Fix v->isDirty assertion due to race

BGFetcher must Validate CAS values before queuing dirty
item to disk when exptime is updated

Relax assertion since downside is just another set operation

Change-Id: Icfcce79371bb867052f12c8a8574c8a318d71ace
Reviewed-on: http://review.couchbase.org/29461
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-8606: Unit test fix with multi threaded warmup

Change-Id: I6e86ae1cc8dbee62a9045c937dea931f1f7d89d4
Reviewed-on: http://review.couchbase.org/29556
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Refactor how items are retrieved from the Checkpoint Manager

Change-Id: I98d34ee9720142c7c5d8524836bbe68284330b33
Add a cursor index type

Change-Id: I15f0d62228bbcc6d399f1fbd7bd190086f344687
Only log message if we are acutally removing a tap cursor

There is a place in the code that calls this function for each
vbucket and this causes the logs to be flooded with messages
about removing tap cursors from vbuckets even when none are
removed. If we log the message after we check to see if the
cursor exists then the problem goes away.

Change-Id: Ie910422e7cca5e1da05a467f68aac78f6db12c53
MB-9124: Don't let CheckpointManager move cursors internally

When removing unreference checkpoints and when adding a new
checkpoint the CheckpointManager attempts to move cursors if they
are at the end of the open checkpoint. This case should be rare
should not make much difference in terms of performance. The
removing checkpoint cursors case would have a potential to make a
difference, but it appears that the current code will not actually
do anything since it doesn't take into account the fact that there
is a checkpoint end message that was recently inserted into the
checkpoint. As a result the cursors will never move to the new
open checkpoint. The other place we do this is when adding a new
checkpoint and in that case we only try to move the persistence
cursor. I don't see a benefit here since we will not be removing
any checkpoints at that time and we can just wait for the
persistence cursor to grab the items.

This change will also simplify tracking which items have been
persisted because we will only have one place in the code that can
actually move the cursors forward and that will only happen when
someone grabs items from the checkpoints.

The new UPR changes to the CheckpointManager will also render this
optimization obsolete because we will not allow cursors to be in
open checkpoints.

Change-Id: I1137997a81ee502857fc0517b29cabfe1a1a92c9
MB-9124: Use persistence cursor last chk id to mark persisted items

Change-Id: Iac5c53b8594bd4e1611b164a7c925e1acd1e7d6c