CBD-80 Fix to sporadic unit test failures on Linux

Add test_setup func and wait_for_warmup_complete to ensure that
loading of ep_engine completes before running test case.
Bulk of test failures on Linux were due to incomplete warmup of
ep_engine prior to running test cases.

Change-Id: I2db65c23c3754743323cc2eb07e801a106409e67
Reviewed-on: http://review.couchbase.org/15920
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Added and subsequently removed items shouldn't affect size.

There are lots of places where we attempt to account for adding and
removing items, but adding a couple of simple hash_table_test tests
shows that it was leaking the amount of data believed to be stored in
a few cases.

Change-Id: I4501759cc2ab14567f02e2e0149dcb61a0c2a4ea
Reviewed-on: http://review.couchbase.org/15980
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Always adjust stats properly on hashtable changes.

The hash table itself adjusted the stats on the way in and on
deletions, but on reset, the caller would adjust the stats.
Sometimes.

This change adjusts all memory stats in the hashtable itself and
demonstrates correctness through unit tests.

Change-Id: I7ef7bc307f6781af9f637888d86900e06dde4fa6
Reviewed-on: http://review.couchbase.org/15981
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Refactor/document StoredValue::toItem

The bulk of this method was redundant and got in the way of some work
I was doing.

Change-Id: Ie649992ce0252b082f33a58d1fae249c58a8ee91
Reviewed-on: http://review.couchbase.org/15982
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Better test error reporting with checkeq

Instead of reporting an assertion has failed, we can report what we
expected and what we got in simple equality tests.

Change-Id: I969c18d7c4bc1f80b75fc7e73037438d662fb63f
Reviewed-on: http://review.couchbase.org/15983
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Include unistd.h for usleep(3)

Fixes compiler errors like this:

In file included from vbucket.hh:18:0,
                 from backfill.cc:3:
stored-value.hh: In destructor 'HashTable::~HashTable()':
stored-value.hh:973:23: error: 'usleep' was not declared in this scope

Change-Id: I7bc2c26a8e7e1fec928d51f36a78ba028a6486d7
Reviewed-on: http://review.couchbase.org/16024
Tested-by: Sergey Avseyev <sergey.avseyev@gmail.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5312 Call AM_PROG_AR to fix compilation with automake 1.12.

Change-Id: I6c91bb9f141389ab4110343f3d66ed6fa9f1c519
Reviewed-on: http://review.couchbase.org/16058
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-5278 Add correct assertions to TAP BG Fetch functions

Change-Id: I7bfa200a14f05f1821295a1981ebab1c0a0db2de
Reviewed-on: http://review.couchbase.org/15894
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16077
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-4418: tap_throttle_queue_cap in cbflushctl takes 'infinite' argument

The infinite argument will map to -1 for the tap_throttle_queue_cap. We
are adding this mapping because in order for cbflushctl to be able to
parse -1 it would require doing '-- -1' and this is confusing and not
intuitive for ordinary users. Those who are familiar with the -- will
however still be able to use it.

Change-Id: I44b534d330a7cb7c9b132113b2cb0564901cfcfc
Reviewed-on: http://review.couchbase.org/15895
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16078
Add all of the available flush parameter to help and alphabetize

Change-Id: Ifc228abd0278f11f437d5273c055267892c61614
Reviewed-on: http://review.couchbase.org/15896
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16079
MB-5278 Drop TAP BG fetched items if a connection is newly created

TAP BG fetch jobs can be still in the dispatcher queue when their
corresponding TAP connection is completely removed. If a TAP
connection is newly created for the same client, those TAP BG
fetched items should be dropped.

Change-Id: Ic6018d76c44789e0fa77a7bf17a810b8282bfc7e
Reviewed-on: http://review.couchbase.org/15910
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16080
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5297 Fix to the bug in removing vbucket from backfill vb list

Even if a given vbucket doesn't exist anymore, the TAP producer
should remove that vbucket from its backfill vb list so that it
can maintain the correct list of backfill vbuckets.

Change-Id: I82fe55a888b195327450a11349a01858c53a2948
Reviewed-on: http://review.couchbase.org/16013
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16081
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100 Move TAP backfill scheduling log messages to warning level

As TAP backfill log messages are quite critical to debug any
data inconsistency issues, we decide to promote them to the
warning level.

Change-Id: Id5deacbbb99743a94de7bb2a321ae94075a31594
Reviewed-on: http://review.couchbase.org/16014
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16082
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5297 Don't check backfill completion from backfill tasks.

As TAP backfill completion is checked regularly in the
corresponding TAP producer, we don't need to check a given TAP
producer's backfill completion from the backfill tasks
scheduled by the ReadOnly dispatcher.

Change-Id: I986feb1421730dd6c70b5dabc9f85af501f017ff
Reviewed-on: http://review.couchbase.org/16015
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16083
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100 Add more logs to TapProducer::setCursorToOpenCheckpoint

Change-Id: I1330d2576415e5f0893e8f71605c73f97dde7fff
Reviewed-on: http://review.couchbase.org/16016
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16084
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100 Promote TAP VB_SET / OPAQUE log messages to warning level

As TAP VB_SET and OPAQUE log messages are quite useful to debug
any rebalance or replication failures, we decide to promote them
to the warning level.

Change-Id: I7b7aa164df3373f823f02ea37e07bec9108910de
Reviewed-on: http://review.couchbase.org/16017
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16085
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100 Convert TAP OPAQUE code correctly in the log message

Change-Id: Ie7ff5600c9eaa30ca73a5f6a2ae445211989e4bb
Reviewed-on: http://review.couchbase.org/16020
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16086
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5297 Run TAP backfill task after sending all OPAQUE messages.

Change-Id: If806edab4840cc3d8f2bb37e62210d7994a06601
Reviewed-on: http://review.couchbase.org/16042
Reviewed-by: Jin Lim <jin@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16087
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Revert "Add an extra reference to the tap connection during..."

This reverts commit 6739fac1c7e0b7b0913a8d65fffce6f85256157f.

Change-Id: If1ed0e00ab3bdfbc428ca818abbb5ff5154de448
Reviewed-on: http://review.couchbase.org/15917
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16088
MB-5311 Remove potential deadlock from TAP connection map

Now that tap connections may run in all worker threads we may
have a deadlock if we try to release the reference with the
tap connection map locked. Delay the release call until it's
executed by the dispatcher.

Change-Id: Idb0293d69acde1b7f36a25e38de2dd7a9d88ac79
Reviewed-on: http://review.couchbase.org/15976
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Reviewed-on: http://review.couchbase.org/16089