MB-4714: Reset connection if mccouch stops responding

If mccouch crashes after we send a request and before we receive
a response then ep-engine will wait forever for a response that
won't come. This change introduces a new parameter called
couch_response_timeout that will cause ep-engine to re-establish
a connection to mccouch if it a request doesn't come back for the
given amount of time specified in couch_response_timeout. By
default this parameter is set to 10 minute.

Change-Id: Iede592f378d3ed73c0f7aed1a2986148cfb25d0b
Reviewed-on: http://review.couchbase.org/12720
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
reverted part of bad 1.8 merge and fixed analyze_core. MB-4742

Applied result of

git diff HEAD..46d10cc05bf3f70ba28193fa35e647c26780354b^ -- analyze_core

Here's how gitk -- analyze_core looks http://i.imgur.com/5rM3W.png.

You can see that 1.8 only has membase->couchbase rename, that was also
made earlier on 2.0 branch. Individual commits
051aadb496f0916fb9f9f12035525bde26c992c0 (rename on 2.0) and
22dbf911296d7b5ed1227039455974c66812a665 (rename on 1.8) both look
sane. Merge on the other hand seems to introduce some unpexpected
stuff. Verify with:

git diff 46d10cc05bf3f70ba28193fa35e647c26780354b^..46d10cc05bf3f70ba28193fa35e647c26780354b -- analyze_core

And because there were no changes after merge, the fix is reverting
that extra stuff introduced during merge.

Change-Id: I48fa4e5f670574a23f1ccf41d7f89288ef7aaf0f
Reviewed-on: http://review.couchbase.org/12894
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Track a histogram counter by object size.

This is a counter and not a gauge, so it is used to accurately show
growth by value size.  It does not give you any idea what you
currently have stored, just what your object patterns look like.
Because it's a counter, you can measure how the object sizes change
over time, even in very small time windows.

Example stat capture:

 item_alloc_sizes (102920 total)
    0 - 1         : (  0.01%)    10
    1 - 2         : (  0.02%)     8
    2 - 4         : (  0.04%)    20
    4 - 8         : (  0.08%)    44
    8 - 16        : (  0.16%)    82
    16 - 32       : (  0.33%)   178
    32 - 64       : (  0.66%)   341
    64 - 128      : (  1.30%)   656
    128 - 256     : (  2.71%)  1455
    256 - 512     : (  6.22%)  3609 #
    512 - 1KB     : ( 16.71%) 10791 ####
    1KB - 2KB     : ( 35.54%) 19384 ########
    2KB - 4KB     : ( 68.34%) 33759 ##############
    4KB - 8KB     : ( 96.31%) 28789 ############
    8KB - 16KB    : ( 99.03%)  2794 #
    128KB - 256KB : (100.00%)  1000

Change-Id: Ic3e32f25bd5df4009cf002a6b33a3414fbdc550e
Reviewed-on: http://review.couchbase.org/12851
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Dustin Sallings <dustin@spy.net>
Fixed binary unl, cas was not converted to host byte order

Change-Id: I2097bcb856e60fa861c845db107bacecbaea8b4b
Reviewed-on: http://review.couchbase.org/12940
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-4619 Set the registered TAP custor to its checkpoint's begin()

When the registered TAP client reconnects to the server, its
checkpoint cursor should be always reset to the beginning of its
current checkpoint, so that the checkpoint_start meta message
should be first sent before streaming mutations.

Change-Id: I66acbe4aebd44385eeeec51f3960d3f9d871a784
Reviewed-on: http://review.couchbase.org/12918
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-4657 Don't schedule backfill if backfill age > current time.

Change-Id: I733af91a0c6d0edfa4204748f3754452c5130e09
Reviewed-on: http://review.couchbase.org/12919
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-4716 Clear checkpoints with open state upon backup completion

If there are any checkpoint state records with open state when
the incremental backup client is terminated, remove them from
checkpoint_state table on the incremental backup files.

Change-Id: I07c6d47e44b258b92461e7afec94c9d16636476c
Reviewed-on: http://review.couchbase.org/12920
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-4715 Fix to backup validation in backup_util.py

mbbackup-merge-incremental is unable to consistantly merge the
incremental backup files which are created by splitting
one checkpoint into multiple backup files. This patch changes
validate_incremental_backup_files() to return backup files in
stable sorted order by comparing checkpoints. If backup split
files of one checkpoint is given, validate_incremental_backup_files()
should return backup file list in the same order of backup file list
given as argument.

Change-Id: I0f1e154c2a007e6e7bbddf9667100fd435a87c11
Reviewed-on: http://review.couchbase.org/12817
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Add resetCursors() to CheckpointManager as a separate function.

Change-Id: Ie7cb4ec383b30a307b61e6491957a36a197a2a43
Reviewed-on: http://review.couchbase.org/12852
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-4707 Merge all checkpoints if chk id from the master is smaller

The checkpoint id from the new master could be smaller than the
open checkpoint id on the slave right after rebalance because the
new vbucket map generated for rebalance is sometimes inefficient
and promotes one of the replica vbuckets, which does not catch up
its original master, to the new master. In this case, the slave
simply removes checkpoints whose ids are larger than the
checkpoint id from the new master. However, delete operations in
the checkpoints removed won't be persisted.

To resolve the above issue, this change collapses the checkpoints
on the slave into a single checkpoint so that all delete
opeartions can be persisted and does not break any view
consistency.

Change-Id: Ie232d8da3ca217cda35e609d0a8a84bcbfd3321b
Reviewed-on: http://review.couchbase.org/12853
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Fixed unl test

I fixed a bug where we weren't converting the cas to host byte
order and forgot to also fix the unit test.

Change-Id: Id7a05244349f7ce2841ac11766ee35fe96f78ad2
Reviewed-on: http://review.couchbase.org/12956
Tested-by: Dustin Sallings <dustin@spy.net>
Reviewed-by: Dustin Sallings <dustin@spy.net>
Added histogram to represent the time objects are paged out.

We track the time an object is evicted and update the histogram when
the object is needed again from disk.  Object replacement is not
considered here (e.g. if an object is evicted and then a new value is
set, we just ignore the time).

 paged_out_time (4 total)
    4s - 8s       : ( 50.00%) 2 ########################
    2m - 4m       : (100.00%) 2 ########################

Change-Id: I905eb3c763b1ff08a05b8809886137f625d95470
Reviewed-on: http://review.couchbase.org/12957
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Dustin Sallings <dustin@spy.net>
Move tap-related config param settings to setTapParam().

Change-Id: I721ec646465c99fe9d730bbe5c1e28d08de0b600
Reviewed-on: http://review.couchbase.org/12962
Reviewed-by: Dustin Sallings <dustin@spy.net>
Tested-by: Dustin Sallings <dustin@spy.net>
New script that allows us to change tap config params at runtime.

Example usage:

% cbtapctl 127.0.0.1:11210 set tap_keepalive 600

Change-Id: Ie068b1f526f78188386a0b5dc6fa90369ff64862
Reviewed-on: http://review.couchbase.org/12963
Tested-by: Dustin Sallings <dustin@spy.net>
Reviewed-by: Dustin Sallings <dustin@spy.net>
Add primary keys to split backup in mbbackup-merge-incremental

Change-Id: Ie46d6e3fb2a0470986fbadf3ab2f5b0841efc3de
Reviewed-on: http://review.couchbase.org/13001
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Merge branch 'branch-17'

* branch-17:
  Add primary keys to split backup in mbbackup-merge-incremental
  MB-4715 Fix to backup validation in backup_util.py
  MB-4716 Clear checkpoints with open state upon backup completion
  MB-4657 Don't schedule backfill if backfill age > current time.
  MB-4619 Set the registered TAP custor to its checkpoint's begin()

Change-Id: I8d2d75ee17d30851d0036aea17f0f69a858ab3ef
New binary protocol command for replica read.

Add the new retrieval command, CMD_GET_REPLICA, that works exactly
like the existing "get" command except it returns data only for
vbuckets that are in replica state.

Add new ep-engine unit tests that run the CMD_GET_REPLICA command
against all four vbucket states - active, pending, dead, replica:

- replica read fails returning data for vbuckets that are in active
- replica read fails returning data for vbuckets that are in pending
- replica read fails returning data for vbuckets that are in dead
- replica read returns data for vbuckets that are in replica

Change-Id: Ib71593859133da262a93493e1ee2aa64025fe3fa
Reviewed-on: http://review.couchbase.org/12792
Reviewed-by: Dustin Sallings <dustin@spy.net>
Tested-by: Dustin Sallings <dustin@spy.net>
Some doc updates.

There are still a few completely undocumented classes -- some of which
should probably just be marked as DETAILS so they don't show up in the
docs, but it's good to at least get a bit captured.

Change-Id: I606595f6b7c66f5acc4348fe74688a8e9c2695d7
Reviewed-on: http://review.couchbase.org/13048
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Avoid double memcpy of key and fix compile error

Change-Id: I3b546dddc3696c378ebdb6ce1e695b0450b894ca
Reviewed-on: http://review.couchbase.org/13050
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Don't reschedule checkpoint items only for the registered TAP.

If the TAP connection is for the registered client that is only
interested in items from the closed checkpoints, only checkpoint
items should not be rescheduled because its cursor is going to
be reset to the beginning of its current checkpoint upon
the reconnection. On the other hand, other TAP opaque messages
should be rescheduled for the registered client.

Change-Id: I7f0a267b935d7efe7bab5ca776fd111b74b96abb
Reviewed-on: http://review.couchbase.org/13069
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>