MB-4772 Fix to handle backfill for closed checkpoint only tap

rollback() method was clearing the taplog items when a tap
connection is restablished in order to reset the cursor back to
the start of current checkpoint. Due to this condition, when
backfilled items are added to the taplog, it was getting cleared and
hence backilled mutations where never sent through the tapconnection
since it is empty. This patch adds a condition to handle backfill
with closed checkpoint only tap cursor.

Change-Id: I2fdcda9fe3702f8efd2147b799cfbab142babf85
Reviewed-on: http://review.couchbase.org/13092
Reviewed-by: Manik Taneja <mtaneja@zynga.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Add option -f to take full backup in mbbackup-incremental

This patch makes uses of backfill to take full backups from
membase. When -f option is used with mbbackup-incremental script,
the script will de-register the existing tap, register a new tap
with same tapname enabling backfill. It will create a new backup file
the full keys from membase and add all checkpoints from 1 to current
closed checkpoint in cpoint_state table.

Change-Id: Iee839e013e621aec3d7a3a1c0720ebbdb5bea4ca
Reviewed-on: http://review.couchbase.org/13093
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Removed sync configuration parameters

Change-Id: Iffde533f58d9e2e0f2a2d8d1103144e1f9fe3c53
Reviewed-on: http://review.couchbase.org/13767
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Remove "unused" variable for return value

Using the variable indicates that it might change somewhere
in the function, but we never change it..

Change-Id: I02cf6bdaa228956236b9103603ba7c7e6a4e9991
Reviewed-on: http://review.couchbase.org/13768
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
MB-4859 Add back the callback for CMD_LAST_CLOSED_CHECKPOINT

This line was removed in 26250d064c714898f12ed2536633c11cd70b9741

Change-Id: Ie7eafcd902aaeb3af6212851ae9908b91f63acd5
Reviewed-on: http://review.couchbase.org/13787
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Remove duplicate string comparison

Change-Id: I397cb4ea3d27ecc65584a20a492d539b1fc302d7
Reviewed-on: http://review.couchbase.org/13788
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Fix byte ordering for flags in tap mutations

Change-Id: I759dcefcd89d413610fc550455fc3cf2ac452126
Reviewed-on: http://review.couchbase.org/13717
Reviewed-by: Matt Ingenthron <matt@couchbase.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Move static stats calls out of StoredValue

We should just call the stats directly instead of having to go
through an intermediary class. This code also caused there to
be two different code paths to get or set these stats.

Change-Id: I1a8fd8aa526a33619e691b1f8c8b36711a66050d
Reviewed-on: http://review.couchbase.org/13782
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Dustin Sallings <dustin@spy.net>
Reviewed-by: Dustin Sallings <dustin@spy.net>
mb-4857 prevent to free object prematurely

InvalidItemDbPager is allocated in the constructor. And it will be
used right after warmup. And it is freed in the initialize(). If
dispatcher schedules the task ahead of flusher task, the original
object will be freed prematurely before it is referenced again after
warmup. We use share_ptr to keep honest of object pointer references.

Change-Id: Iaf59e113df58a5f5a53e9cf35b6cd38d9a2afc19
[alk@tut.by: whitespace-only M-x align]
Signed-off-by: Aliaksey Kandratsenka <alk@tut.by>
Reviewed-on: http://review.couchbase.org/13817
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Bin Cui <bin.cui@gmail.com>
Merge branch 'branch-20'

* branch-20:
  mb-4857 prevent to free object prematurely

Conflicts:
	ep.hh

Change-Id: Ibf16bc4390ca8dd889af4c030e6040de7e5c9128
Merge branch 'branch-17'

* branch-17:
  Add option -f to take full backup in mbbackup-incremental
  MB-4772 Fix to handle backfill for closed checkpoint only tap

Change-Id: Ic1378741f83c88dfeb4c34b14786b2fcabf958bb
Help valgrind to not report false positives for uninit reads

Running valgrind on my mac reports multipe uninitialized reads
caused by the mutexes / syncobjects. We're initializing these
objects with the pthread_xxx_init methods, but it seems to leave
some of the internals uninitialized (and probably read out a
word including these bytes). Add a --enable-valgrind so that we
can define VALGRIND and memset the variables in these cases.

Change-Id: Ief294963fe1f03ad18b5b011b27d19860fe50f9d
Reviewed-on: http://review.couchbase.org/13916
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Add more stats needed by ns_server

Change-Id: Id3305f69c542571f97768a360056cf764124bf96
Reviewed-on: http://review.couchbase.org/13844
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Refactor: Move the warmup logic out of the flusher

The warmup logic should not run within the flusher thread, but as it's
own dispatcher task with its own state machinery.

Change-Id: Id3ce08865627ad7fa6e7eb66b7f59579c6b66f33
Reviewed-on: http://review.couchbase.org/13876
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Add a make target to run the testsuite under valgrind

Change-Id: I8881e922d8f0578fcf3df9ebe433d7e607a899f4
Reviewed-on: http://review.couchbase.org/13918
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Track max seqno of deleted items per vbucket

In order to guarantee monotonically increasing seqno values even for
recreated items i.e., items that are deleted and reintroduced using
an add or a set command, we track the max seqno of all deleted items
seen so far. Then, when an item is reintroduced, we pick this max
seqno value, increment it by 1 and assign it to the "new" item.
Although conservative, this guarantees that if an item is thus
reintroduced, it gets a seqno that's higher than its previous value.

Change-Id: Ibb30fc5a90db21e05bf1043bfcb1c1050f3913ce
Load the per vbucket maxDeletedSeqno values during warmup

Change-Id: I5b5ca992d90f25328791ede70827efcefe319daf
delete-with-meta implementation in ep-engine

Change-Id: I879aa4f4d4a71b6080613673f6c72e7408d0ee4c
Introduce static consts to track StoredValue states

This is a refactoring change set and introduces a few static consts
to track the different states of the id attribute of a StoredValue
object.

Change-Id: I9224a55f1e025ec48e77762e79ea838c1fda3b98
Add new stored value states for tracking deleted items

For servicing getMetaData() requests for possibly deleted items, we
create a temporary item, add it to the hashtable and schedule a
background fetch of its metadata from the persistent store. After the
fetch completes, we restore its metadata in memory (if it was found
in the persistent store) or mark it as non existent. We also set it
to automatically expire after a predetermined but configurable amount
of time. As such, in order to correctly track the background fetch,
the following new stored value states are required:
1. deleted_key: after the fetch completes, if the item was found
   deleted in the persistent store, it is marked as deleted_key.
2. non_existent_key: after the fetch completes, if the item was not
   found in the persistent store, it is marked as having never
   existed. engine_key_noent will be returned if the item is queried.

This change set introduces these states. The actual logic of handling
deletes as described here is part of subsequent change sets.

Change-Id: I7511e24b6e3ae51f1ba5b06a92b0b83ec0a2b160