MB-10382: change upr-related stats format

The vbucket-seqno stats will be showed in a way consistent
with existent ep-engine stats, for example:

`vbucket-seqno 1`:
vb_1:high_seqno: 107
vb_0:vb_uuid 2723958720985

Change-Id: I8d500cb2c2247229584a855e103cd6c91f6dd397
Reviewed-on: http://review.couchbase.org/34647
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: David Liao <david.liao@couchbase.com>
MB-10476: Use cb_assert instead of assert

If compiled with -DNDEBUG assert is an empty macro

Change-Id: I266ddf8a8fe3f1a388a9d8ce3d21f28614c35315
Reviewed-on: http://review.couchbase.org/34579
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10474: Avoid assigning a NULL pointer to string

- Assigning a char pointer to a string would cause a
segmentation fault if the char pointer were NULL.

- Fix: Placing a check before this assignment.

Change-Id: Ia43649bab628fc548e86ef9f3c663519032eded0
Reviewed-on: http://review.couchbase.org/34656
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Clear last upr op in testsuite before adding latest op data

Not all of the upr fields are used for each message so it is
possible to have stale data sitting around that can cause tests
to verify behavior incorrectly. Similarly if no new message is
received after calling step then the old data will still be
sitting around.

Change-Id: I5736fc3c2835a5a8968bc94d4920ab65fa8b9468
Reviewed-on: http://review.couchbase.org/34571
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Fix to the bug in repositioning checkpoint cursors.

This bug can cause the data loss issues in TAP / UPR.

Change-Id: I1eda9f2d49a71ab5fe456d9b1e106794c4b50bce
Reviewed-on: http://review.couchbase.org/34667
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10474: In case of an empty failover log

- Create a new entry in case of an empty failover log,
during warmup.

Change-Id: I42d9c5eccbb93f6679c21628228813c5e4d83fcb
Reviewed-on: http://review.couchbase.org/34702
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10507 Skip a deleted item for TAP BG fetch task.

If a given item is deleted on disk while the TAP BG fetch job is
being executed. We can simply ignore it because the deletion
will be replicated separately through a checkpoint cursor.

Change-Id: I8b9bd60596fbbf658d2d297e89cc04de6683d06b
Reviewed-on: http://review.couchbase.org/34694
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: David Liao <david.liao@couchbase.com>
Iterate over select vbs only for flushOneDeleteAll

- Iterate over only the vbuckets that belong to the
particular shard when flushOneDeleteAll invoked.

Change-Id: Ia990dfef4596c3aa101209e4e3ce9482be548aea
Reviewed-on: http://review.couchbase.org/34653
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: David Liao <david.liao@couchbase.com>
MB-10259: Increase max number of threads AtomicQueue supports.

Increase the limit from 100 to 500 to handle large number of CPU
cores. Also added an assert if this limit is exceeded.

Change-Id: If9445a8c91c0daf9475a711145cccc7de8986e53
Reviewed-on: http://review.couchbase.org/34699
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10513: compaction must honor drop deletes flag

This is because passing the flag onto couchstore compaction
api is not valid if there is a callback hook also in place

Change-Id: I76875c75dff6c3c2155a30814907b6dc03095427
Reviewed-on: http://review.couchbase.org/34745
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Sundararaman Sridharan <sundar@couchbase.com>
MB-10489 Register the seq_num based replication cursor correctly.

Change-Id: I511ff81b5b519602d262984c370952f0074d0788
Reviewed-on: http://review.couchbase.org/34747
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Added CMake PROJECT setting

Change-Id: Id5f3918eb66311dbb246f211ea269c3aed24e535
Reviewed-on: http://review.couchbase.org/34764
Reviewed-by: Chris Hillery <ceej@couchbase.com>
Tested-by: Chris Hillery <ceej@couchbase.com>
MB-10520 Fetch a deleted item for TAP BG fetch / backfill.

As the metadata fields for a deleted item should be replicated,
the deleted item should be fetched for TAP BG fetch or backfill.
Note that this is a regression from the recent change in the
master branch.

Change-Id: I24eae8e596ed70083875a948aec478f27bfbd68a
Reviewed-on: http://review.couchbase.org/34758
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: David Liao <david.liao@couchbase.com>
MB-10488: Add a notifier upr connection type

The upr notifier connection is used to get updates through an upr
stream about when new items are available to stream.

Change-Id: I276e558bf36c43f427d221351495587ee5063154
Reviewed-on: http://review.couchbase.org/34572
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-10488: Add a notifier upr connection type"

This will revert commit ba01f68086c3d9b62bbeed9629674efece3fc61d.

Change-Id: Ife7c4bb2c4bbcad4932e4cac53a004bbe90f25e0
Reviewed-on: http://review.couchbase.org/34859
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-10164: Set numShards based on disk type

In configuration.json, numShards is initialized to zero.
If this value is greater than zero, this will be used
as numShards, otherwise using sigar we determine the
disk access, based on which numShards gets set to either
4 or 2. If sigar also fails, set numShards to 4 by
default.

SSDs: numShards=4, HDDs: numShards=2

Change-Id: Ie79348816fbd4bc11648c825014e6670b451bd99
Reviewed-on: http://review.couchbase.org/34744
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Use array delete for ids in CouchKVStore saveDocs function

Change-Id: Id5caf22db29e334ff15d05260a95cbf1008aeef7
Reviewed-on: http://review.couchbase.org/34865
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-10489 Don't use std::map::insert() in updating the cursor pos

Using std::map::insert() to update the cursor position will fail
if the cursor alreay exists in the checkpoint manager. Instead,
we should overwrite it with the new cursor object. This is a
regression from the changes that we made in the master branch.

Change-Id: Icf59763fb7d3fd685768373ea9f0636619ee99cb
Reviewed-on: http://review.couchbase.org/34870
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10164: log reasons for the shard count choice

Change-Id: Iea57eea88167e1f00949a35051d632c09ec8db5e
Reviewed-on: http://review.couchbase.org/34869
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Fix build break with C++11

The std::atomics in C++11 don't have a copy constructor so
you need an explicit copy constructor

Change-Id: I833058285c72e4071efa32ddaf12a8f54f7e964f
Reviewed-on: http://review.couchbase.org/34876
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>