MB-4772 Fix to handle backfill for closed checkpoint only tap

rollback() method was clearing the taplog items when a tap
connection is restablished in order to reset the cursor back to
the start of current checkpoint. Due to this condition, when
backfilled items are added to the taplog, it was getting cleared and
hence backilled mutations where never sent through the tapconnection
since it is empty. This patch adds a condition to handle backfill
with closed checkpoint only tap cursor.

Change-Id: I2fdcda9fe3702f8efd2147b799cfbab142babf85
Reviewed-on: http://review.couchbase.org/13092
Reviewed-by: Manik Taneja <mtaneja@zynga.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Add option -f to take full backup in mbbackup-incremental

This patch makes uses of backfill to take full backups from
membase. When -f option is used with mbbackup-incremental script,
the script will de-register the existing tap, register a new tap
with same tapname enabling backfill. It will create a new backup file
the full keys from membase and add all checkpoints from 1 to current
closed checkpoint in cpoint_state table.

Change-Id: Iee839e013e621aec3d7a3a1c0720ebbdb5bea4ca
Reviewed-on: http://review.couchbase.org/13093
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Replica Read bug fix - invalid response after completion

This change fixed a bug where ep_engine wasn't returning a correct
response code, PROTOCOL_BINARY_RESPONSE_SUCCESS, upon the successful
completion of CMD_GET_REPLICA.

Per review recommendation I modified existing get replica test cases
so that each verifies its binary protocol response code after running
the CMD_GET_REPLICA command. Also I added a new test case that checks an
error response from running the command with invalid key id:

test_get_replica_invalid_key : expects
PROTOCOL_BINARY_RESPONSE_NOT_MY_VBUCKET
response

Change-Id: Ib7414424254495c8100b11902677d2f6223011e7
Reviewed-on: http://review.couchbase.org/13199
Tested-by: Dustin Sallings <dustin@spy.net>
Reviewed-by: Dustin Sallings <dustin@spy.net>
Refactored set functions in stored value

This change removes code duplication.

Change-Id: Ia54ecd8978f639122c4ec67effa9079f43f39400
Reviewed-on: http://review.couchbase.org/12603
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Parameterize engine parameters based on their types.

Dynamic engine parameters are classfied into three types and
can be configured by the command "CMD_SET_PARAM" and its type.

Change-Id: Ia810896b5239250fc10fa62c7d5e05f181d8ad6e
Reviewed-on: http://review.couchbase.org/12964
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Rearrange command opcodes and message formats in commmand_ids.h

This change simply puts command opcode definitions and their
binary message formats separately in commands_ids.h.

Change-Id: Ib0d38364a2b7d2bbbb37e0f500bba8f2d0df9db1
Reviewed-on: http://review.couchbase.org/13157
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
CMD_CREATE_CHECKPOINT command to create a new checkpoint by force

Change-Id: Ic98f335d65bda0b9517de7e7a43fbc775ba9149d
Reviewed-on: http://review.couchbase.org/13158
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
CMD_EXTEND_CHECKPOINT command to extend the open checkpoint.

If this new command is received with a positive value for a given
vbucket, its open checkpoint will be kept and a new checkpoint
won't be created. This command is used as part of supporting the
maintenance mode.

Change-Id: I9240c2dc2fefd18daaac5c2864c976de561ce9a3
Reviewed-on: http://review.couchbase.org/13159
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Provide a single vbucket's checkpoint stats through stats call.

Usage: ./cbstats 127.0.0.1:11210 checkpoint vbucket_id

Change-Id: I6f127a4ee54b26b1ae9500a13e0956cf614c094a
Reviewed-on: http://review.couchbase.org/13160
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Disable the checkpoint extension mode if the memory usage is high

The cluster manager enables the checkpoint extension mode for
active vbuckets that are affected by the maintenance mode, so that
the full backfill from those active vbuckets can be avoided if
possible when the node in the maintenance mode rejoins the
cluster.

However, if the memory usage is still above the high water mark
and there were few items evicted even after the item pager runs,
the extension mode on active vbuckets should be disabled, so that
their open checkpoints can be closed and their non-dirty items
can be evicted.

Change-Id: Id44b6a91f3f3843bb4716e9ddcc662b06c71819f
Reviewed-on: http://review.couchbase.org/13176
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Dump core immediately on mutex error

I was looking at a bug today and lost a lot of stack trace
info because we threw an exception which was caught so high
up the stack that I wasn't able to figure out exactly what
caused it.

Change-Id: I1d90ceeae2755b60f3dada201ec5df76d7a52176
Reviewed-on: http://review.couchbase.org/13205
Reviewed-by: Dustin Sallings <dustin@spy.net>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Removed observe test cases

The observe functionality was remove from the code a few
months ago. This commit removes its test cases.

Change-Id: I0419b5b2ba8ee15474ff020562ffe3cba14f603f
Reviewed-on: http://review.couchbase.org/13222
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Add more comprehensive log messages to checkpoint modules.

Change-Id: I2560f7073eefdb2c76b065e88b1934ddfe761ccb
Reviewed-on: http://review.couchbase.org/13239
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Add more comprehensive log messages to dispatcher modules.

Change-Id: I05d30db4fca4ac60a17e691c9c602087090978a1
Reviewed-on: http://review.couchbase.org/13240
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
refactor: Moved stat writing template generator stuff into a header file.

This makes it easier to delegate stat writing down to other stat
containers.

However, this has a nasty side-effect of identical duplicate template
functions being generated in separate compilation units and then
making the linker angry when bringing them back together.  I'm
generating them in different namespaces with a gross mix of templates
and cpp.

http://www.youtube.com/watch?v=ruhFmBrl4GM

Change-Id: I333ca02d42c78c50e71226aa82b73c99a12142b2
Reviewed-on: http://review.couchbase.org/13269
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Add more comprehensive log messages to TAP-related modules.

Change-Id: Ic02e5f4257670b9ad2b386b7d39eaf30dd6e3712
Reviewed-on: http://review.couchbase.org/12053
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
declare base add_casted_stat overload as inline and unbreak build

gcc with our paranoid warning levels complains that add_casted_stat
function (not template) is not declared before being defined.

Which kind of makes sense as duplicating trivial 'proxy' function in
all compile units is not what we need. Inline fixes it.

Change-Id: I28cef250b68ef4b4c93c3ba271953b2bd5bcb83f
Reviewed-on: http://review.couchbase.org/13337
Tested-by: Dustin Sallings <dustin@spy.net>
Reviewed-by: Dustin Sallings <dustin@spy.net>
SQLite low-level stats.

Documentation is not included here yet, but the seek stats are
approximately how far we asked the disk head to move from the position
we last left it.  The read and write sizes are how big the chunks of
data were that we were reading and writing.  The times are the amount
of time it took us to do one of the things it says we did.

Counters:

 close:       201
 lock:        1042
 open:        207
 sector_size: 512
 truncate:    0

And timings:

 delete (406 total)
    16us - 32us   : (  0.49%)   2
    32us - 64us   : (  0.74%)   1
    64us - 128us  : ( 30.30%) 120 ##########
    128us - 256us : ( 43.60%)  54 ####
    256us - 512us : ( 54.93%)  46 ###
    512us - 1ms   : ( 87.19%) 131 ##########
    1ms - 2ms     : ( 99.75%)  51 ####
    2ms - 4ms     : (100.00%)   1
 readSeek (3710601 total)
    0 - 2         : (  0.01%)     411
    16 - 32       : (  0.01%)      89
    64 - 128      : (  0.01%)       5
    256 - 512     : (  0.07%)    2098
    512 - 1KB     : (  0.07%)      16
    1KB - 2KB     : (  9.40%)  346284 ##
    2KB - 4KB     : ( 11.27%)   69202
    4KB - 8KB     : ( 13.46%)   81508
    8KB - 16KB    : ( 14.77%)   48473
    128KB - 256KB : ( 14.94%)    6146
    256KB - 512KB : ( 15.17%)    8698
    512KB - 1MB   : ( 15.18%)     171
    1MB - 2MB     : ( 97.41%) 3051513 ########################
    2MB - 4MB     : ( 98.35%)   34781
    4MB - 8MB     : ( 98.36%)     231
    8MB - 16MB    : ( 98.37%)     456
    16MB - 32MB   : (100.00%)   60519
 readSize (3710604 total)
    8 - 16        : (  0.06%)    2187
    16 - 32       : (  0.08%)     689
    64 - 128      : (  0.08%)       5
    1KB - 2KB     : (100.00%) 3707723 #############################
 readTime (3710597 total)
    0 - 1us       : (  0.04%)    1387
    1us - 2us     : ( 79.08%) 2932976 #######################
    2us - 4us     : ( 97.87%)  697021 #####
    4us - 8us     : ( 99.57%)   63127
    8us - 16us    : ( 99.88%)   11589
    16us - 32us   : ( 99.97%)    3467
    32us - 64us   : (100.00%)     865
    64us - 128us  : (100.00%)     127
    128us - 256us : (100.00%)      13
    256us - 512us : (100.00%)      13
    512us - 1ms   : (100.00%)      12
 sync (4780 total)
    64us - 128us  : ( 29.10%) 1391 #########
    128us - 256us : ( 45.63%)  790 #####
    256us - 512us : ( 47.55%)   92
    512us - 1ms   : ( 47.87%)   15
    1ms - 2ms     : ( 49.00%)   54
    2ms - 4ms     : ( 51.84%)  136
    4ms - 8ms     : ( 90.17%) 1832 ############
    8ms - 16ms    : ( 92.07%)   91
    16ms - 32ms   : ( 94.23%)  103
    32ms - 65ms   : ( 95.96%)   83
    65ms - 131ms  : ( 97.82%)   89
    131ms - 262ms : ( 99.31%)   71
    262ms - 524ms : ( 99.90%)   28
    524ms - 1s    : (100.00%)    5
 writeSeek (14502993 total)
    4 - 8         : ( 49.99%) 7249692 ##############
    8 - 16        : ( 49.99%)     269
    16 - 32       : ( 50.00%)     954
    128 - 256     : ( 50.00%)      88
    256 - 512     : ( 50.00%)     181
    512 - 1KB     : ( 50.00%)     271
    1KB - 2KB     : ( 78.26%) 4099250 ########
    2KB - 4KB     : ( 78.27%)    1388
    4KB - 8KB     : ( 78.28%)     177
    8KB - 16KB    : ( 78.28%)     122
    16KB - 32KB   : ( 78.28%)      72
    32KB - 64KB   : ( 78.28%)     178
    64KB - 128KB  : ( 78.29%)    1202
    256KB - 512KB : ( 78.29%)      89
    512KB - 1MB   : ( 78.29%)      88
    1MB - 2MB     : ( 99.35%) 3054955 ######
    2MB - 4MB     : ( 99.59%)   34789
    4MB - 8MB     : ( 99.59%)     248
    8MB - 16MB    : ( 99.59%)      90
    16MB - 32MB   : (100.00%)   58890
 writeSize (14503006 total)
    4 - 8         : ( 49.99%) 7249969 ##############
    8 - 16        : ( 50.01%)    2456
    16 - 32       : ( 50.01%)     954
    512 - 1KB     : ( 50.03%)    2189
    1KB - 2KB     : (100.00%) 7247438 ##############
 writeTime (14502983 total)
    1us - 2us     : ( 37.00%) 5366472 ###########
    2us - 4us     : ( 89.60%) 7628245 ###############
    4us - 8us     : ( 97.53%) 1149346 ##
    8us - 16us    : ( 99.73%)  320212
    16us - 32us   : ( 99.92%)   26762
    32us - 64us   : ( 99.97%)    8117
    64us - 128us  : ( 99.99%)    2997
    128us - 256us : (100.00%)     351
    256us - 512us : (100.00%)     324
    512us - 1ms   : (100.00%)     144
    1ms - 2ms     : (100.00%)       9
    2ms - 4ms     : (100.00%)       3
    4ms - 8ms     : (100.00%)       1

Change-Id: Ia8135bb810f242575c9498b559cc1e40906279cd
Reviewed-on: http://review.couchbase.org/13270
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Checkpoint maintains keys and subset of meta data fields only.

The previous checkpoint implementation requires each checkpoint
to maintain the list of keys, their values, and all of their
meta data fields, which can cause very high memory overhead
in case disk write throughput or replication is very slow.

Based on various use cases for checkpoint, it turns out that it
is not necessary to maintain the strong consistency across
checkpoints. This can simplify a checkpoint's item structure,
which allows us to have keys and subset of their meta data fields
only. Basically, a checkpoint only contains the information about
which keys have been mutated and their operation types.

With this simplication, the memory overhead of a single checkpoint
item (i.e., QueuedItem instance) is reduced from 88 bytes to 40
bytes. In addition, the closed checkpoint won't point to stale
Blob value instances anymore.

Change-Id: I8c0273f582a53014ea92dd94ca9194cb9f5bc387
Reviewed-on: http://review.couchbase.org/13286
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Minor refactoring in TapProducer class's member variables.

This simply reorders the member variables of TapProducer class
and does minor formatting in comments.

Change-Id: I527f5faec3449469b7a39a743947efa5a9c1fea0
Reviewed-on: http://review.couchbase.org/13341
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>