Fix issue that caused items to be re-replicated through upr

We us the lowest sequence number in a checkpoint as the sequence
number for a checkpoint start message. During the setOpenCheckpointId
call we were resetting the sequence number to something higher and
this caused problems for sequence number replication. I also
refactored the sequence number registration so that we don't send
the item at the start sequence number.

Change-Id: Idc6555b09036c546fa58859947af09ca8100a38e
Reviewed-on: http://review.couchbase.org/34413
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: David Liao <david.liao@couchbase.com>
Reduce the amount of headers that are imported in .h files

This change removes imports of certain header files from all
other header files and into .cc files. The problem with
importing a header in another header file is that any change
causes all headers used in that header file to be recompiled.
In some cases changing one header file causes us to have to
recompile everyhting. This change moves some of these headers
to only .cc files and should reduce compile time.

File no longer imported from other headers:

conflict_resolution.h
failover-table.h
flusher.h
mutation_log.h
tap-throttle.h
upr-consumer.h
upr-consumer.h

Change-Id: Idd44bcaac874e294c1085da46760badf37a1ea05
Reviewed-on: http://review.couchbase.org/34395
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10441: Adapt item_pager test for WIN32

It seems like the scheduling of the tasks is different on windows
(at least on my virtual machines). Once we have all of the tests
passing for WIN32 we're going to start benchmarking it so we'll
figure out a better fix for this at a later time..
For now just spend some time waiting for it to bump the values

Change-Id: I4446cd7ab3a42356dff9605d6a694c9aaed7fa00
Reviewed-on: http://review.couchbase.org/34428
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10442: Windows timers are of a lower resolution

This is causing fast operations to happen within
the same clock resolution.

Change-Id: Ic142adcab750e9defe76c9ae956eb17acccb2063
Reviewed-on: http://review.couchbase.org/34429
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Lower log level indicating sending/receiving mccouch messages

Change-Id: I18ca7bb5491a66a70964ebea359e530f06fcdea1
Reviewed-on: http://review.couchbase.org/34414
Reviewed-by: David Liao <david.liao@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10372: Make estimateTime in warmup class atomic

Change-Id: Ie362b00f9d56bb94a62b562610dd9be2dfee68d5
Reviewed-on: http://review.couchbase.org/34376
Reviewed-by: David Liao <david.liao@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Add entry points for upr flow control api's

Change-Id: Iad313af354ebb9087563f9226156a1adfcda34c8
Reviewed-on: http://review.couchbase.org/34461
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: David Liao <david.liao@couchbase.com>
CBD-485: Fix cbcompact for testing

waitTillDone option is now incorrect since compaction is
synchronous. This is to unblock QE testing

Change-Id: I1c58e5e923fcc4768bba73f18755c85a6d64a1d1
Reviewed-on: http://review.couchbase.org/34459
Reviewed-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
Tested-by: Sundararaman Sridharan <sundar.sridharan@gmail.com>
MB-10378: stat vbucket-details should return correct value

When requesting the stat vbucket-details with a vbucket the
server doesn't have, it should return a not_my_vbucket instead
of an empty reply. This commit will reutrn not_my_vbucket error
in this case.

Change-Id: I84076156ffca361ba5fec9b54dec0a9bfcae8859
Reviewed-on: http://review.couchbase.org/34443
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-9993: make warning msg more readable

Today the warning msg returns generic I/O exception which does
not provide any insight of the consequence. This commit simply
modifies warning msg by specifying to which json stat doc we
fail to log the ep_engine stats.

Change-Id: Iba3facc57ebb4c55c8852b87607a5f4aea093a32
Reviewed-on: http://review.couchbase.org/34442
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-9105 Close the open checkpoint upon receiving snapshot request

If a given UPR client requests a snapshot, this change closes the
current open checkpoint and open a new checkpoint if necessary.
In this way, we can provide the snapshot upto a given closed
checkpoint for the UPR client.

We confirmed from all UPR producer related unit tests that they
close the open checkpoint correctly and stream items upto that
checkpoint.

Change-Id: Ie7134fcf15b5f7c954a3da774e4177e00c038285
Reviewed-on: http://review.couchbase.org/34482
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Use size_t offset type in CheckpointManager::putCursorsInChk()

Change-Id: I8fc218d7baa8cc5d87fb171694bb6748f822f64a
Reviewed-on: http://review.couchbase.org/34483
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-10463 Correct the rep cursor position of UPR client.

This change fixes the bug in registering the UPR replicaiton
cursor, so that the cursor can start with the item whose seq
number > start_seqno passed.

Change-Id: Ie2d87f872abb025f587fa4c40b0f64bd8724e0d7
Reviewed-on: http://review.couchbase.org/34485
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-100: fix a couple of compilation warnings

The warnings of comparison between signed and unsigned integers
come out everytime I recompile ep_engine code, not critical but
a bit annoying,  thus probably it is better to fix it.

Change-Id: I82525f68edabbeec7060bbe986a40a1d3a6c9ce4
Reviewed-on: http://review.couchbase.org/34568
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Don't start upr_event_t enum at 101

This was done originally because the tap and upr events might have
been passed in to the same variables. Starting at 101 would avoid
conflicts. We no longer use upr_event_t like this.

Change-Id: I80adaead6877ec7cd72a9b430a04b3797b25440f
Reviewed-on: http://review.couchbase.org/34462
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-10164: Boosting warmup speed

- Set dbFileRevMapPopulated to true, upon
populating the FileRevMap the first time to avoid
updating the map in every phase of the warmup
process.

- Test results (for the same scenario):

                    ep_warmup_time
2.5.x           :   632.835ms
3.0 master      :   2594.427ms
3.0 with fix    :   263.519ms

Change-Id: Ifbafecb25b175607c25e517e4870a8187f8de7d6
Reviewed-on: http://review.couchbase.org/34585
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10382: change upr-related stats format

The vbucket-seqno stats will be showed in a way consistent
with existent ep-engine stats, for example:

`vbucket-seqno 1`:
vb_1:high_seqno: 107
vb_0:vb_uuid 2723958720985

Change-Id: I8d500cb2c2247229584a855e103cd6c91f6dd397
Reviewed-on: http://review.couchbase.org/34647
Tested-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: David Liao <david.liao@couchbase.com>
MB-10476: Use cb_assert instead of assert

If compiled with -DNDEBUG assert is an empty macro

Change-Id: I266ddf8a8fe3f1a388a9d8ce3d21f28614c35315
Reviewed-on: http://review.couchbase.org/34579
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-10474: Avoid assigning a NULL pointer to string

- Assigning a char pointer to a string would cause a
segmentation fault if the char pointer were NULL.

- Fix: Placing a check before this assignment.

Change-Id: Ia43649bab628fc548e86ef9f3c663519032eded0
Reviewed-on: http://review.couchbase.org/34656
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Clear last upr op in testsuite before adding latest op data

Not all of the upr fields are used for each message so it is
possible to have stale data sitting around that can cause tests
to verify behavior incorrectly. Similarly if no new message is
received after calling step then the old data will still be
sitting around.

Change-Id: I5736fc3c2835a5a8968bc94d4920ab65fa8b9468
Reviewed-on: http://review.couchbase.org/34571
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>