MB-12271: Set the default dcp producer noop interval to 20 seconds

This is a fix to support backwards compatibility between 3.0 and
3.0.1+ versions of Couchbase. The problem is that 3.0 has a noop
interval of 20 seconds by default and 3.0.1 has a default noop
interval of 200 seconds. In 3.0.1 the consumer explicitly sets
the noop interval of the producer so in 3.0.1+ clusters the 20
second default will be overriden by the consumer, but in the
scenario when a 3.0.1+ node connects to a 3.0 node the 20 second
default will remain.

Change-Id: I2e18e9ad68037f3a82abe5167f2bca89f381f318
Reviewed-on: http://review.couchbase.org/42136
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-12226: Handle ENOENT error during unlink

If the file is not found during an unlink, it shouldn't be added
to the pending file deletions queue.

Change-Id: Ief306277bbbc946ae18e39dd4819f811f12ea76c
Reviewed-on: http://review.couchbase.org/42173
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Sriram Ganesan <sriram@couchbase.com>
Hide the -a argument in cbstats

The -a argument is used by cbcollectinfo so we can't just
get rid of it.

Change-Id: I0f04c2d454c117271b44176df904e90a938965e9
Reviewed-on: http://review.couchbase.org/42297
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Merge remote-tracking branch 'couchbase/3.0.1'

Change-Id: Iea18be488592f879e6eaa5b8bc16cf9d33836a58
Remove unused functions from checkpoint manager and item pager

Change-Id: I09788ffd82e15559a63f5e6f7a8e1a98c28af497
Reviewed-on: http://review.couchbase.org/42408
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-12350 DCP info message contains misleading statistics.

Add a new parameter to backfill received which contains information to the source
of the backfill (memory vs disk). Increment stat counters based upon this
data as the current stat accounting was always counting disk.

Change-Id: I5ab0602540deedad536e78517850393594341b32
Reviewed-on: http://review.couchbase.org/42153
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
Update test_memory_limit to support memory_tracker

When the memory tracker is enabled, mem_used accurately reflects the
actual amount of memory used by the bucket. This test assumes that
mem_used is only the the actual document key & value size, and so when
the memory_tracker is enabled the mem_used figure (correctly)
increases significantly.

Update the testcase to work with much larger initial mem_used, and
calculate the value length based on a fraction of the available space,
not on a fixed overhead.

Change-Id: I95df3544d1569f3e7d1a4efbe18e1f2c9cd99830
Reviewed-on: http://review.couchbase.org/42237
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Update ep_testsuite mem_used checks to support memory tracker

A number of ep_testsuite tests check for mem_used to *exactly* match
it's value before the test began. This is fragile, particulary when
using the memory tracker as any number of new/delete calls may be
made _in addition_ to the expected stored documents, and hence it is
quite possible the final mem_used value isn't exactly the same.

Update the tests to either check for a relative change, for example
less than the high point, to handle such fluctuations; or remove the
check where the low-level mem_used isn't actually useful in the test.

Change-Id: Ief4d7add1117a4c4eb34d2faf86f7e781c8e5459
Reviewed-on: http://review.couchbase.org/42238
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Update test_item_pager to support memory tracker

test_item_pager has fixed values for how many documents to
create. This fails when the minimum bucket quota (max_size) must be
increased in-line with the actual amount of memory a bucket takes -
when the memory tracker is enabled.

Update the test to create as many document as possible before TEMP_OOM
is encountered. Also reduce the amount of memory used by temporary
test objects to leave more for actual stored documents.

Change-Id: Id539c7654f81f3cab3a08da6b96521ad388ead58
Reviewed-on: http://review.couchbase.org/42239
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Update test_flush_stats to remove mem_used checks

When accurate memory tracking is enabled, there is too many
allocations going on 'under the hood' to be able to check that
mem_used increases relative to a baseline when only two small keys are
added. Therefore just remove the mem_used checks from this testcase.

Change-Id: I841b8601234a88f1b0b2417b51178ede9091e706
Reviewed-on: http://review.couchbase.org/42240
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Don't check for exact mem_used in DGM tests

The D>M tests assume that mem_used will be *exactly* the same after
the test as before. When using accurate memory tracking this is
unlikely to be the case. Seeing as these checks aren't the main thrust
of these two tests, remove them.

Change-Id: Ieaed79ffc0daeb99721a9b994ba6dccce8bab175
Reviewed-on: http://review.couchbase.org/42463
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Add unit test for memory tracking

Memory tracking (by linking with tcmalloc/jemalloc) is enabled in
production for accurate accounting of per-bucket memory usage, and so
it should be enabled as part of unit tests if we are to have
representative memory usage numbers. Add a test to this effect.

Change-Id: I17fe915441c790019b833189756cab2c93a8b641
Reviewed-on: http://review.couchbase.org/42462
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10789: [1] BloomFilter + MurmurHash

Implementation of the filter and hash functions

Change-Id: I42f7fb9e4c54fe7dae18b953aa6962eb9abfd08f
Reviewed-on: http://review.couchbase.org/41857
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Reviewed-by: Sundararaman Sridharan <sundar@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10789: [2] Bloomfilter for every vbucket

Add bloomfilter object and support to the vbucket class.

Change-Id: Icfa664d502d5a2fa7523b17fc2d493fc0de13849
Reviewed-on: http://review.couchbase.org/41858
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10789: [3] Bloom filter use is configurable

Also add the residency threshold paramter which
will be used in the full eviction scenario only.

2 parameters dynamically configurable with cbepctl:
    - bfilter_enabled
    - bfilter_residency_threshold

Change-Id: I7348ae125365d0ec67918fe295fc76d41a24813a
Reviewed-on: http://review.couchbase.org/41859
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10789: [4] Bloomfilters on compaction

Create/resize bloom filters during compaction.
Time_purge_hook callback of compaction that
iterates over all keys will be responsible for
populating the bloomfilter with all the necessary
keys.

Change-Id: I8f30488599f426324bac28a973ae006b6d1454c0
Reviewed-on: http://review.couchbase.org/41860
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10789: [5] Bloomfilter for sets/gets/deletes/evicts

For VALUE-ONLY EVICTION:
- Add all deletes to bloomfilter
- Check bloomfilter before getMetaData

For FULL EVICTION:
- Add all deletes to bloomfilter
- Add evicted items to bloomfilter

Check bloomfilter, if item not in hashtable for:
    GetInternal, DelWithMeta, DelItem, Add,
    SetWithMeta, Set, GetLocked, Touch, etc.

Change-Id: I2766226228d18b448ad464525ce80e6803a31772
Reviewed-on: http://review.couchbase.org/41861
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10789: [6] Unit tests for bloomfilters

For Value-only and full eviction modes

Change-Id: I8916f8ded688d0e20fa57adcace73960e473a0eb
Reviewed-on: http://review.couchbase.org/41862
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-10789: [7] Disabling bloomfilters' use by default

Change-Id: Ib5aab5f45605276b6e0f7dca37e3fcf4ef3c87ce
Reviewed-on: http://review.couchbase.org/42638
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Fix build break on WIN32

New versions of MSVC have stdint.h

Change-Id: Ica591b67c9e09541648d3a47e6c8fc98afde2547
Reviewed-on: http://review.couchbase.org/42716
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>