Merge remote-tracking branch 'gerrit/2.0.1' into 2.0.2

Change-Id: I03ef5ebbae2231cc926f1b77f117e25420f2030c
Merge remote-tracking branch 'gerrit/2.0.2'

Change-Id: I2e8f70465f1efcc9b43201117af62b8b182a57e8
CBD-821 Extend an item's NRU value to two bits.

The current NRU scheme allocates only one bit per item in memory
cache and resets it to false whenever it is updated or referenced.
However, this scheme doesn't have the re-reference frequency of
each item and treats update and read operations with the same
weight in a cache management.

To support the easy and simple prediction of the re-reference
frequency, this change extends an item's NRU value to two bits,
so that it can have the re-reference hint for each item.

Basically, when a new item is inserted, its NRU value is set to
the intermediate value (i.e., two), and decremented as long as
the value is greater than zero whenever the item is referenced
later. The new item eviction algorithm (shown in another change)
uses an item's NRU value to collect the list of items with high
NRU values, which will be candidates for evictions.

Change-Id: Id863d68f1e0cfed6382a2ad53a2d7b089029df70
Reviewed-on: http://review.couchbase.org/24439
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
CBD-821 Adapt the item pager to use the extended NRU bits.

An item's NRU value is extended to two bits, which allows us
to have the range of 0 - 3 values. More frequently accessed
items will likely have zero NRU value as their NRU values are
decremented everytime they are referenced.

The item pager is adapted to leverage the extended NRU bits:

Phase 1:
1. Scan the next vbucket and collect items whose NRU score is 3
2. Evict those items with NRU score '3'
3. Check the memory usage and go to step 1 if the memory usage
   is still above the low watermark.

Once the phase 1 is completed, then the pager goes to the phase
2 if the memory usage is still above the high watermark.

Phase 2:
1. Scan the next vbucket and increment every item's
   NRU score by 1
2. If its NRU score is equal to '3', then generate a random
   number N whose range is from 0 to 1, and evict that item
   if N < P where P is the probability that is calculated based
   on the current memory usage and memory low watermark.
   Note that we also differentiate the probability P depending on
   the vbucket state (active, replica).
3. Check the memory usage and go to step 1 if the memory usage is
   still above the low watermark.

Change-Id: Idfcfda834e5fe89061cd49d5abaefba0621c6a42
Reviewed-on: http://review.couchbase.org/24526
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-7728: Add copyright headers to all files

This is something that our cpplint tool turns on and it makes
our product look a little bit more profesional. I have also
enabled the tool to begin checking for these headers so it will
start complaining if we add a new file without a copyright header.

Change-Id: Ie956fdf7027bf56d9665ee455acd3c109fcacc76
Reviewed-on: http://review.couchbase.org/24560
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7483: Remove duplicate stats holding bucket size

ep_max_data_size (removed)
ep_max_size

Change-Id: Ie0dd4f94b15917ad397c1902449cd6f193e73c89
Reviewed-on: http://review.couchbase.org/24638
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-7483: Remove duplicate stats holding bucket size"

This reverts commit 7e84fc9ce1488722e055a496382af302f655b417.

Change-Id: I889e1bb0411e0c991e061fa3eb7e9300e00e0230
Reviewed-on: http://review.couchbase.org/24669
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-7509: Skip merging checkpoint end items in chk queues

Change-Id: Idd73a04fc0ff8cce774ee956c006b879bfb1fec5
Reviewed-on: http://review.couchbase.org/24675
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7509 Removed item should be removed from keyIndex too.

Change-Id: I842be6f77606e99ca7b47a451fd5f03e4b95feb6
Reviewed-on: http://review.couchbase.org/24730
Reviewed-by: Jin Lim <jin@couchbase.com>
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: xiaoqin ma <maxiaoqin2005@gmail.com>
MB-7524 Schedule the stats snapshot before purging TAP connections

When the engine shuts down, we schedule the stats snapshot task
and destroy all the TAP connection instances as part of the
shutdown process. As the stats snapshot task needs to persist
TAP stats, it should be scheduled before purging those TAP
connections from memory.

Change-Id: If1e154d68d2b8fa9f9859b9eda7274dad743a07d
Reviewed-on: http://review.couchbase.org/24759
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-100: fix ep_engine compilation errors/warnings (master branch)

Change-Id: I296fd1937e33588d8b862b421a2ec0ad2d532f06
Reviewed-on: http://review.couchbase.org/24582
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7810: Add -Wunused-variable to compiler warnings list

Change-Id: I3352562a066581dd336f20e3a5b49f847eea60d4
Reviewed-on: http://review.couchbase.org/24797
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7797: Lower bgfetch sleep time for 60s to 1s

Change-Id: I767599bb3d64c085ad406ee75d5d88ebb48effc3
Reviewed-on: http://review.couchbase.org/24842
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-5522: add warmup_min_{items,memory}_threshhold parameters for runtime settings.

Change-Id: I1c3fd788280f10fc8b46beadb5a86e1aa643242b
Reviewed-on: http://review.couchbase.org/24796
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: xiaoqin ma <maxiaoqin2005@gmail.com>
Revert "MB-7797: Lower bgfetch sleep time for 60s to 1s"

This reverts commit 62717fadb0f59cf50396908845625a4142b4a44f.

Change-Id: Idc9c18e12bac55be552e4fd32baa5f839dc3b145
Reviewed-on: http://review.couchbase.org/24901
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-7597: adjust the changes for high CPU usage."

This reverts commit eee2e9564ef844cf8cc435911d9e80af0dece244.

Change-Id: Ib30942fbde5287f903e8e7a38877380d7e1511ea
Reviewed-on: http://review.couchbase.org/24902
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Revert "MB-7597: correct the logic of updating bgfetcher global variable"

This reverts commit f6b583f3760cc1e7df85b5bf3abbc2e016a270fc.

Change-Id: Ic959fae6a1bb6cf9a2a382342518510aa9efcc4b
Reviewed-on: http://review.couchbase.org/24903
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'gerrit/2.0.1' into 202

Change-Id: Ic24e7263f72b84d227954d289c08dafe8466b51f
MB-7830: Zero out key_stats buffer in observe logic

This will prevent us from sending garbage in some of the fields
if the key is not found on the server.

Change-Id: I561e95b46baee8f0994fe17b4df3e6c74ec3da4c
Reviewed-on: http://review.couchbase.org/24898
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-7884: Don't creating a string from a NIL pointer

When running SmartOS I got a coredump in strlen() called
from one of the constructors creating a string out of a null
pointer. According to my copy of the C++ specification the
constructor with the char* requires the pointer to be a
non-null value.

Change-Id: I7ec7de7f294a9be7a2d398b7942969e0aa30c904
Reviewed-on: http://review.couchbase.org/24709
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>