CBD-821 Extend an item's NRU value to two bits.

The current NRU scheme allocates only one bit per item in memory
cache and resets it to false whenever it is updated or referenced.
However, this scheme doesn't have the re-reference frequency of
each item and treats update and read operations with the same
weight in a cache management.

To support the easy and simple prediction of the re-reference
frequency, this change extends an item's NRU value to two bits,
so that it can have the re-reference hint for each item.

Basically, when a new item is inserted, its NRU value is set to
the intermediate value (i.e., two), and decremented as long as
the value is greater than zero whenever the item is referenced
later. The new item eviction algorithm (shown in another change)
uses an item's NRU value to collect the list of items with high
NRU values, which will be candidates for evictions.

Change-Id: Id863d68f1e0cfed6382a2ad53a2d7b089029df70
Reviewed-on: http://review.couchbase.org/24439
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
CBD-821 Adapt the item pager to use the extended NRU bits.

An item's NRU value is extended to two bits, which allows us
to have the range of 0 - 3 values. More frequently accessed
items will likely have zero NRU value as their NRU values are
decremented everytime they are referenced.

The item pager is adapted to leverage the extended NRU bits:

Phase 1:
1. Scan the next vbucket and collect items whose NRU score is 3
2. Evict those items with NRU score '3'
3. Check the memory usage and go to step 1 if the memory usage
   is still above the low watermark.

Once the phase 1 is completed, then the pager goes to the phase
2 if the memory usage is still above the high watermark.

Phase 2:
1. Scan the next vbucket and increment every item's
   NRU score by 1
2. If its NRU score is equal to '3', then generate a random
   number N whose range is from 0 to 1, and evict that item
   if N < P where P is the probability that is calculated based
   on the current memory usage and memory low watermark.
   Note that we also differentiate the probability P depending on
   the vbucket state (active, replica).
3. Check the memory usage and go to step 1 if the memory usage is
   still above the low watermark.

Change-Id: Idfcfda834e5fe89061cd49d5abaefba0621c6a42
Reviewed-on: http://review.couchbase.org/24526
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-7483: Remove duplicate stats holding bucket size

ep_max_data_size (removed)
ep_max_size

Change-Id: Ie0dd4f94b15917ad397c1902449cd6f193e73c89
Reviewed-on: http://review.couchbase.org/24638
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-7483: Remove duplicate stats holding bucket size"

This reverts commit 7e84fc9ce1488722e055a496382af302f655b417.

Change-Id: I889e1bb0411e0c991e061fa3eb7e9300e00e0230
Reviewed-on: http://review.couchbase.org/24669
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-7509: Skip merging checkpoint end items in chk queues

Change-Id: Idd73a04fc0ff8cce774ee956c006b879bfb1fec5
Reviewed-on: http://review.couchbase.org/24675
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7509 Removed item should be removed from keyIndex too.

Change-Id: I842be6f77606e99ca7b47a451fd5f03e4b95feb6
Reviewed-on: http://review.couchbase.org/24730
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: xiaoqin ma <maxiaoqin2005@gmail.com>
MB-7524 Schedule the stats snapshot before purging TAP connections

When the engine shuts down, we schedule the stats snapshot task
and destroy all the TAP connection instances as part of the
shutdown process. As the stats snapshot task needs to persist
TAP stats, it should be scheduled before purging those TAP
connections from memory.

Change-Id: If1e154d68d2b8fa9f9859b9eda7274dad743a07d
Reviewed-on: http://review.couchbase.org/24759
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7810: Add -Wunused-variable to compiler warnings list

Change-Id: I3352562a066581dd336f20e3a5b49f847eea60d4
Reviewed-on: http://review.couchbase.org/24797
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7797: Lower bgfetch sleep time for 60s to 1s

Change-Id: I767599bb3d64c085ad406ee75d5d88ebb48effc3
Reviewed-on: http://review.couchbase.org/24842
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-5522: add warmup_min_{items,memory}_threshhold parameters for runtime settings.

Change-Id: I1c3fd788280f10fc8b46beadb5a86e1aa643242b
Reviewed-on: http://review.couchbase.org/24796
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: xiaoqin ma <maxiaoqin2005@gmail.com>
Revert "MB-7797: Lower bgfetch sleep time for 60s to 1s"

This reverts commit 62717fadb0f59cf50396908845625a4142b4a44f.

Change-Id: Idc9c18e12bac55be552e4fd32baa5f839dc3b145
Reviewed-on: http://review.couchbase.org/24901
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-7597: adjust the changes for high CPU usage."

This reverts commit eee2e9564ef844cf8cc435911d9e80af0dece244.

Change-Id: Ib30942fbde5287f903e8e7a38877380d7e1511ea
Reviewed-on: http://review.couchbase.org/24902
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-7597: correct the logic of updating bgfetcher global variable"

This reverts commit f6b583f3760cc1e7df85b5bf3abbc2e016a270fc.

Change-Id: Ic959fae6a1bb6cf9a2a382342518510aa9efcc4b
Reviewed-on: http://review.couchbase.org/24903
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/2.0.1' into 202

Change-Id: Ic24e7263f72b84d227954d289c08dafe8466b51f
MB-7830: Zero out key_stats buffer in observe logic

This will prevent us from sending garbage in some of the fields
if the key is not found on the server.

Change-Id: I561e95b46baee8f0994fe17b4df3e6c74ec3da4c
Reviewed-on: http://review.couchbase.org/24898
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7884: Don't creating a string from a NIL pointer

When running SmartOS I got a coredump in strlen() called
from one of the constructors creating a string out of a null
pointer. According to my copy of the C++ specification the
constructor with the char* requires the pointer to be a
non-null value.

Change-Id: I7ec7de7f294a9be7a2d398b7942969e0aa30c904
Reviewed-on: http://review.couchbase.org/24709
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7892: Use memset instead of bzero

memset is a standard function, whereas bzero is nonstandard
and possibly not found on all platforms (without special includes)

Change-Id: I13f40ecb443ed3316b6fe0a0cda6e77eb96099eb
Reviewed-on: http://review.couchbase.org/25088
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7652: Check for buffer overflow when logging messages

Change-Id: I356959257ce6263586cfd79b952695a5dab25407
Reviewed-on: http://review.couchbase.org/25074
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7893: Add a "catch-all" clause for file io exceptions

On SmartOS the exception thrown isn't a const &, causing the
catch to fail (and dump core). This patch work around this
problem by adding a "catch all" clause at the end.

Change-Id: I98e1cb3d298a26e0aa2ee29c28bb6eef4381698c
Reviewed-on: http://review.couchbase.org/25087
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7836: Give more granularity for minutes in cbstats

 disk_vbstate_snapshot (472 total)
    1us - 2us       : (  0.21%)   1
    2ms - 4ms       : ( 96.82%) 456 ##############
    8ms - 16ms      : ( 97.46%)   3
    16ms - 32ms     : ( 97.67%)   1
    32ms - 65ms     : ( 98.73%)   5 #
    65ms - 131ms    : ( 99.36%)   3
    131ms - 262ms   : ( 99.58%)   1
    4s - 8s         : ( 99.79%)   1
    1m:07s - 2m:14s : (100.00%)   1
    Avg             : (  154ms)

Change-Id: Ic29583d21cd0479280ef01dafc1a27254c2c1dff
Reviewed-on: http://review.couchbase.org/25073
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>