Fix race condition in TaskQueue shutdown

There is a bug in the use of ExecutorThread.state when sleeping a
TaskQueue - TaskQueue::_doSleep() doesn't atomically transition the
state from RUNNING -> SLEEPING. This can cause a deadlock when
shutting down a ExecutorThread:

    Thread A:                           Thread B:
    --------------------------------    ------------------------------
    if (t.state == RUNNING) {  // true
                                        t.state = SHUTDOWN
        t.state = SLEEPING              cb_join_thread(Thread A)
                                        // wait forever
    ...
    if (t.state == SHUTDOWN) { // FALSE
      exit(0) // NEVER REACHED
    }

Fix by changing ExecutorThread.state to be an AtomicValue, and use
compare-and-exchange to move from RUNNING -> SLEEPING (and SLEEPING ->
RUNNING).

Change-Id: I9fab90a83978ae2aa6a0dcdd3b079a1c2f369402
Reviewed-on: http://review.couchbase.org/43134
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
fix compilation error in executorthread.cc

Atomic variables cannot be read directly without load()

Change-Id: If7a75a363f1096b0e6e5cfeb7bc7d4a07fb48433
Reviewed-on: http://review.couchbase.org/43144
Reviewed-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: abhinav dangeti <abhinav@couchbase.com>
MB-12067 [1/6]: Initial version of HashTable defragmenter

Create a new Defragmenter task, which visits all members of the
HashTable and defragments them. Currently the defragmentation
algorithm is very naive - it simply reallocates all Blob objects
encountered, regardless of how fragmented or not the pages they reside
on. This is obviously very wasteful (as we will move object in both
fragmented and not fragmented pages on each pass), but however is
sufficient to significantly reduce RSS of the memcached process, if
jemalloc is used as the memory allocator.

jemalloc is required as it prefers low addresses during re-use;
and so the act of allocating a new Blob object, copying the old to the
new and then free()ing the old Blob will result in the new objects
being packed with other objects in the same size class.

To obtain this behaviour, we need to disable jemalloc's thread cache
for the thread which is performning the defragmentation, for the
duration of that task, and hence this requires the use of new
functions from the allocator hooks API.

Change-Id: I8f0e8daaa5e5d7ea8fde58f1c8377bf30312c85a
Reviewed-on: http://review.couchbase.org/41494
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067 [2/6]: Expose run_defragmenter_task from ep_engine via epctl

Add support en/disabling defragmentation, and explicitly triggering
defragmentation via the epctl interface. The main use-case for this is
to allow unit-tests to explicitly trigger the defragmenter.

Change-Id: Ieaeadfbd4186213e35a1e89bc0202d39ff27ab96
Reviewed-on: http://review.couchbase.org/41495
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067 [3/6]: Unit test for degragmenter task

Basic smoketest for the defragmenter: Create a number of documents,
spanning at least two or more pages, then delete most (but not all) of
them - crucially ensure that one document from each page is still
present. This will result in the rest of that page being
'wasted'. Then verify that after defragmentation the actual memory
usage drops down to (close to) mem_used.

Change-Id: Ibfe8780fb02ad5eb504242b01e7583bc1e305184
Reviewed-on: http://review.couchbase.org/41496
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067 [4/6]: Add age field to Blob; use as part of defragmenter policy

When a Blob object is allocated record the current 'age' of it. The
age is incremented when the runs, allowing us to measure how many
defragmenter passes have occurred, and consequently can be used by the
defragmenter to select Blob objects which have been in existance (and
haven't changed size) for a long period, and hence are candiates for
defragmentation.

Change-Id: I6380e6601ae5ee86ca630031e4edea9711a88433
Reviewed-on: http://review.couchbase.org/41497
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067 [5/6]: pause/resume visitor support for epStore & HashTable

Add new visiting methods to HashTable and EventuallyPersistentStore to
allow a client to pause the visit at a particular position, and then
resume from approximately the same position at a later time.

The methods operate in a similar way to the existing visit() /
visitDepth() methods, with the addition of a start_pos parameter to
specify where to start at, and returning the position the visitation
got to.

Clients would typically call pauseResumeVisit() with a default
Position initially (to start from the beginning), and then when they
want to pause they return false from their visit() method.

Change-Id: I2542efc165e3200f0cc74d8120a16e2175f1341d
Reviewed-on: http://review.couchbase.org/41498
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067 [6/6]: Use pause/resume visitor for defragmenter task

Expand the defragmenter to make use of the pause/resume visitor
methods on epStore & HashTable. This allows it to run for a
constrained period of time, pausing (and recording how far it got) if
it reaches its time limit. It can then resume from where it left off
at the next defragmentation internatl.

Change-Id: I82520aa907bb386f9c55fc7e92ac5338bf862cdf
Reviewed-on: http://review.couchbase.org/41499
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067: Add additional defragmenter_num_visited stat

Records how many items have been visited (considered for
defragmentation). Useful to measure this against
defragmenter_num_moved.

Change-Id: I56f3509645349b0ef482afcb9b68c3049154143b
Reviewed-on: http://review.couchbase.org/43052
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067: Allow runtime adjustment of defrag params

Allow defragmenter_{interval,age_threshold,chunk_duration} to be
adjusted at runtime, to assist in tuning sensible defaults.

Change-Id: Ifc5a1cd845cebb844d13f7f4fc2264630ad7c0fc
Reviewed-on: http://review.couchbase.org/43053
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
MB-12067: Disable defragmenter by default

Disable the Defragmenter task by default, to allow profiling of just
jemalloc compared to tcmalloc, without the defragmenter being in play.

Change-Id: I87d5c15c217f0f53b8a264ccf17377d12f7c7e82
Reviewed-on: http://review.couchbase.org/43242
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Defragmenter: Use ExTask RCPTR

Use a ExTask (reference-counted pointer) for the defragmenterTask to
ensure that there isn't any races between the ExecutorPool using it
and shutting the task down.

Fixes some intermittent unit-test failures seen on OS X.

Change-Id: Icc7bbea8bbdd3fb26c8183106a2568dcb95275d5
Reviewed-on: http://review.couchbase.org/43243
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
test_memory_limit: Increase document size for TEMP_OOM test

When running under jemalloc we appear to have a lower mem_used for
this test, with the result that occasionally it will fail as the
second store() passes when it's expected to hit TEMP_OOM.

Fix this by increasing the size of the docuement we set.

Change-Id: I8b65f6e138b7ba1544957927d16ffd4d39867aea
Reviewed-on: http://review.couchbase.org/43246
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
MB-12647: Ensure CAS value will always be unique

In windows, the gethrtime() returns the same timestamp when 2 requests
are made in a very short interval, thus causing the same CAS to return.
This can race conditions with 2 consecutive requests, resulting in
data corruption.

Change-Id: I4f396de3f14129504ca406ebb8d4c7a9f3a89bd8
Reviewed-on: http://review.couchbase.org/43271
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix race in use of gmtime()

As identified by ThreadSanitizer:

    WARNING: ThreadSanitizer: data race (pid=17259)
      Write of size 8 at 0x7fec86e44de0 by main thread (mutexes: write M1161):
	#0 gmtime ??:0 (libtsan.so.0+0x000000025135)
	#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be)
	#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35)
	#3 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
	#4 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
	#5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000832d8)
	#6 wait_for_stat_change(engine_interface*, engine_interface_v1*, char const*, int, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:846 (ep_testsuite.so+0x0000000838d6)
	#7 test_setup /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:178 (ep_testsuite.so+0x00000001ee69)
	#8 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1050 (exe+0x000000005970)
	#9 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)

      Previous write of size 8 at 0x7fec86e44de0 by thread T7:
	#0 gmtime ??:0 (libtsan.so.0+0x000000025135)
	#1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:3369 (ep.so+0x00000010f4be)
	#2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4339 (ep.so+0x000000113c35)
	#3 EventuallyPersistentStore::snapshotStats() /repos/couchbase/server/source/ep-engine/src/ep.cc:1465 (ep.so+0x0000000e150e)
	#4 StatSnap::run() /repos/couchbase/server/source/ep-engine/src/tasks.cc:79 (ep.so+0x000000174db2)
	#5 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:110 (ep.so+0x00000014a0e7)
	#6 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x000000149930)
	#7 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
	#8 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)

Switch to gmtime_r which is thread-safe.

Change-Id: Id0773df65f4fc569c0a173b6185b1ef8bd91862d
Reviewed-on: http://review.couchbase.org/43312
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix race in TaskQueue.{ready,future,pending}Queue access

Fix race as identified by ThreadSanitizer:

    WARNING: ThreadSanitizer: data race (pid=4243)
      Read of size 8 at 0x7d04000fde60 by main thread (mutexes: write M1367):
	#0 std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >::operator++() /usr/include/c++/4.8/bits/stl_list.h:235 (ep.so+0x00000013a129)
	#1 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::__distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::input_iterator_tag) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:82 (ep.so+0x000000138b67)
	#2 std::iterator_traits<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >::difference_type std::distance<std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> > >(std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >, std::_List_const_iterator<SingleThreadedRCPtr<GlobalTask> >) /usr/include/c++/4.8/bits/stl_iterator_base_funcs.h:118 (ep.so+0x000000136b63)
	#3 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::size() const /usr/include/c++/4.8/bits/stl_list.h:874 (ep.so+0x000000135538)
	#4 ExecutorPool::doTaskQStat(EventuallyPersistentEngine*, void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:654 (ep.so+0x0000001331b6)
	#5 EventuallyPersistentEngine::doWorkloadStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4198 (ep.so+0x000000112ed3)
	#6 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4454 (ep.so+0x000000114cb4)
	#7 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
	#8 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
	#9 test_workload_stats /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:7094 (ep_testsuite.so+0x00000004e931)
	#10 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc)
	#11 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)

      Previous write of size 8 at 0x7d04000fde60 by thread T5 (mutexes: write M45):
	#0 RCValue::_rc_decref() const /repos/couchbase/server/source/ep-engine/src/atomic.h:293 (ep.so+0x000000096797)
	#1 __gnu_cxx::new_allocator<std::_List_node<SingleThreadedRCPtr<GlobalTask> > >::allocate(unsigned long, void const*) /usr/include/c++/4.8/ext/new_allocator.h:104 (ep.so+0x00000017b42b)
	#2 std::_List_base<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_get_node() /usr/include/c++/4.8/bits/stl_list.h:334 (ep.so+0x00000017b0dc)
	#3 std::_List_node<SingleThreadedRCPtr<GlobalTask> >* std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_create_node<SingleThreadedRCPtr<GlobalTask> const&>(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:502 (ep.so+0x00000017a25a)
	#4 void std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::_M_insert<SingleThreadedRCPtr<GlobalTask> const&>(std::_List_iterator<SingleThreadedRCPtr<GlobalTask> >, SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1561 (ep.so+0x000000178e7b)
	#5 std::list<SingleThreadedRCPtr<GlobalTask>, std::allocator<SingleThreadedRCPtr<GlobalTask> > >::push_back(SingleThreadedRCPtr<GlobalTask> const&) /usr/include/c++/4.8/bits/stl_list.h:1016 (ep.so+0x00000017817b)
	#6 TaskQueue::_fetchNextTask(ExecutorThread&, bool) /repos/couchbase/server/source/ep-engine/src/taskqueue.cc:126 (ep.so+0x000000176932)
	#7 TaskQueue::fetchNextTask(ExecutorThread&, bool) /repos/couchbase/server/source/ep-engine/src/taskqueue.cc:142 (ep.so+0x000000176acf)
	#8 ExecutorPool::_nextTask(ExecutorThread&, unsigned char) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:214 (ep.so+0x000000130707)
	#9 ExecutorPool::nextTask(ExecutorThread&, unsigned char) /repos/couchbase/server/source/ep-engine/src/executorpool.cc:229 (ep.so+0x0000001307a5)
	#10 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:78 (ep.so+0x000000149da0)
	#11 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a)
	#12 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
	#13 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)

Fix by adding ew helper methods which will return the size of the
various queues (while holding the TaskQueue's mutex).

Change-Id: If5e8d357e45803d78c4ba6ed1475e6e1a90e1c89
Reviewed-on: http://review.couchbase.org/43313
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix race in updating Vbucket.file{SpaceUsed,Size}

As identified by ThreadSantizer:

    WARNING: ThreadSanitizer: data race (pid=4009)
      Write of size 8 at 0x7d440000c5b0 by thread T6 (mutexes: write M55):
	#0 KVStatsCallback::callback(KVStatsCtx&) /repos/couchbase/server/source/ep-engine/src/ep.cc:933 (ep.so+0x0000000f0a22)
	#1 CouchKVStore::commit2couchstore(Callback<KVStatsCtx>*, unsigned long, unsigned long) /repos/couchbase/server/source/ep-engine/src/couch-kvstore/couch-kvstore.cc:1697 (ep.so+0x0000001aa8c6)
	#2 CouchKVStore::commit(Callback<KVStatsCtx>*, unsigned long, unsigned long) /repos/couchbase/server/source/ep-engine/src/couch-kvstore/couch-kvstore.cc:1040 (ep.so+0x0000001a6483)
	#3 EventuallyPersistentStore::flushVBucket(unsigned short) /repos/couchbase/server/source/ep-engine/src/ep.cc:2909 (ep.so+0x0000000e780b)
	#4 Flusher::flushVB() /repos/couchbase/server/source/ep-engine/src/flusher.cc:283 (ep.so+0x00000013f363)
	#5 Flusher::step(GlobalTask*) /repos/couchbase/server/source/ep-engine/src/flusher.cc:174 (ep.so+0x00000013e9c8)
	#6 FlusherTask::run() /repos/couchbase/server/source/ep-engine/src/tasks.cc:44 (ep.so+0x000000174a85)
	#7 ExecutorThread::run() /repos/couchbase/server/source/ep-engine/src/executorthread.cc:110 (ep.so+0x00000014a0c1)
	#8 launch_executor_thread /repos/couchbase/server/source/ep-engine/src/executorthread.cc:34 (ep.so+0x00000014990a)
	#9 platform_thread_wrap /repos/couchbase/server/source/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x000000002d8b)
	#10 __tsan_write_range ??:0 (libtsan.so.0+0x00000001b1c9)

      Previous read of size 8 at 0x7d440000c5b0 by main thread (mutexes: write M193510842443017784):
	#0 VBucketCountVisitor::visitBucket(RCPtr<VBucket>&) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2889 (ep.so+0x00000010c631)
	#1 VBucketCountAggregator::visitBucket(RCPtr<VBucket>&) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2926 (ep.so+0x000000121392)
	#2 EventuallyPersistentStore::visit(VBucketVisitor&) /repos/couchbase/server/source/ep-engine/src/ep.cc:3278 (ep.so+0x0000000e99d9)
	#3 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:2955 (ep.so+0x00000010cb46)
	#4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:4344 (ep.so+0x000000113c0f)
	#5 EvpGetStats /repos/couchbase/server/source/ep-engine/src/ep_engine.cc:217 (ep.so+0x000000102b14)
	#6 mock_get_stats /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:195 (exe+0x0000000026de)
	#7 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000832d8)
	#8 wait_for_flusher_to_settle(engine_interface*, engine_interface_v1*) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:900 (ep_testsuite.so+0x000000083cd4)
	#9 wait_for_persisted_value(engine_interface*, engine_interface_v1*, char const*, char const*, unsigned short) /repos/couchbase/server/source/ep-engine/tests/ep_test_apis.cc:917 (ep_testsuite.so+0x000000083de5)
	#10 test_io_stats /repos/couchbase/server/source/ep-engine/tests/ep_testsuite.cc:6279 (ep_testsuite.so+0x0000000468b3)
	#11 execute_test /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000059fc)
	#12 main /repos/couchbase/server/source/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x000000006606)

Change to AtomicValues to fix this.

Change-Id: Ie7f9a403f809e751ff1802cceb5d2a77a483a586
Reviewed-on: http://review.couchbase.org/43314
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix race in Flusher._state

Fix race identified by ThreadSanitizer:

    WARNING: ThreadSanitizer: data race (pid=10664)
      Read of size 4 at 0x7d4400009888 by main thread (mutexes: write M18321):
        #0 Flusher::stateName() const /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:119 (ep.so+0x0000001fc22e)
        #1 EventuallyPersistentEngine::doEngineStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3025 (ep.so+0x000000180649)
        #2 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4374 (ep.so+0x00000018d960)
        #3 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000174062)
        #4 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d)
        #5 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000dae46)
        #6 verify_curr_items(engine_interface*, engine_interface_v1*, int, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:834 (ep_testsuite.so+0x0000000e1d31)
        #7 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3589 (ep_testsuite.so+0x000000094c9a)
        #8 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83)
        #9 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84)

      Previous write of size 4 at 0x7d4400009888 by thread T5:
        #0 Flusher::transition_state(flusher_state) /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:114 (ep.so+0x0000001fb7df)
        #1 Flusher::step(GlobalTask*) /home/vagrant/couchbase-server/ep-engine/src/flusher.cc:167 (ep.so+0x0000001fc9d1)
        #2 FlusherTask::run() /home/vagrant/couchbase-server/ep-engine/src/tasks.cc:44 (ep.so+0x00000027870e)
        #3 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x0000002160ff)
        #4 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x00000021560a)
        #5 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4)

Change-Id: Iaeb60efdc1032de7ba344e04cce454cc1d876d40
Reviewed-on: http://review.couchbase.org/43315
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix race in DCP Stream.itemReady

Race identified by ThreadSanitizer:

      Read of size 1 at 0x7d480000b458 by main thread (mutexes: write M18384, write M898, write M18351):
        #0 ActiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:386 (ep.so+0x0000000ff1a9)
        #1 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-producer.cc:502 (ep.so+0x0000000efcd3)
        #2 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3725 (ep.so+0x0000001ba97d)
        #3 ConnStatBuilder std::for_each<std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder>(std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_algo.h:4417 (ep.so+0x0000001ba5b5)
        #4 void ConnMap::each_UNLOCKED<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:148 (ep.so+0x0000001ba519)
        #5 void ConnMap::each<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:140 (ep.so+0x0000001b2c4e)
        #6 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3978 (ep.so+0x00000018928e)
        #7 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4384 (ep.so+0x00000018dc73)
        #8 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000174062)
        #9 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d)
        #10 get_int_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:799 (ep_testsuite.so+0x0000000dae46)
        #11 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, int, bool) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3365 (ep_testsuite.so+0x0000000afc65)
        #12 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3596 (ep_testsuite.so+0x000000094ed0)
        #13 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83)
        #14 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84)

      Previous write of size 1 at 0x7d480000b458 by thread T10 (mutexes: write M18363, write M18357):
        #0 ActiveStream::markDiskSnapshot(unsigned long, unsigned long) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:202 (ep.so+0x0000000fd0c7)
        #1 DCPBackfill::create() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:138 (ep.so+0x0000000d057c)
        #2 DCPBackfill::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:95 (ep.so+0x0000000cfe3b)
        #3 BackfillManager::backfill() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:171 (ep.so+0x0000000c778d)
        #4 BackfillManagerTask::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:50 (ep.so+0x0000000c745a)
        #5 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x00000021646f)
        #6 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x00000021597a)
        #7 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4)

Change-Id: I02cdf26c697f559a0afde91b377f1462c482349c
Reviewed-on: http://review.couchbase.org/43316
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>
Fix race in ActiveStream.backfillItems

Race identified by ThreadSanitizer:

  Read of size 8 at 0x7d480000b518 by main thread (mutexes: write M18375, write M898, write M18336):
        #0 void STATWRITER_NAMESPACE::add_casted_stat<unsigned long>(char const*, unsigned long const&, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/statwriter.h:45 (ep.so+0x0000000aacf9)
        #1 ActiveStream::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:378 (ep.so+0x0000000fee1e)
        #2 DcpProducer::addStats(void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*) /home/vagrant/couchbase-server/ep-engine/src/dcp-producer.cc:502 (ep.so+0x0000000efcd3)
        #3 ConnStatBuilder::operator()(SingleThreadedRCPtr<ConnHandler>&) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3725 (ep.so+0x0000001ba92d)
        #4 ConnStatBuilder std::for_each<std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder>(std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, std::_List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_algo.h:4417 (ep.so+0x0000001ba565)
        #5 void ConnMap::each_UNLOCKED<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:148 (ep.so+0x0000001ba4c9)
        #6 void ConnMap::each<ConnStatBuilder>(ConnStatBuilder) /home/vagrant/couchbase-server/ep-engine/src/connmap.h:140 (ep.so+0x0000001b2bfe)
        #7 EventuallyPersistentEngine::doDcpStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:3978 (ep.so+0x00000018923e)
        #8 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:4384 (ep.so+0x00000018dc23)
        #9 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/vagrant/couchbase-server/ep-engine/src/ep_engine.cc:214 (ep.so+0x000000174012)
        #10 mock_get_stats /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:196 (exe+0x0000000a741d)
        #11 get_ull_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/vagrant/couchbase-server/ep-engine/tests/ep_test_apis.cc:817 (ep_testsuite.so+0x0000000e1aa6)
        #12 dcp_stream(engine_interface*, engine_interface_v1*, char const*, void const*, unsigned short, unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, int, int, int, int, int, bool) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3370 (ep_testsuite.so+0x0000000afd38)
        #13 test_dcp_producer_stream_req_disk(engine_interface*, engine_interface_v1*) /home/vagrant/couchbase-server/ep-engine/tests/ep_testsuite.cc:3596 (ep_testsuite.so+0x000000094ed0)
        #14 execute_test /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1055 (exe+0x0000000a3e83)
        #15 main /home/vagrant/couchbase-server/memcached/programs/engine_testapp/engine_testapp.c:1313 (exe+0x0000000a1d84)

      Previous write of size 8 at 0x7d480000b518 by thread T10 (mutexes: write M276545986342477096, write M18342):
        #0 ActiveStream::backfillReceived(Item*, backfill_source_t) /home/vagrant/couchbase-server/ep-engine/src/dcp-stream.cc:229 (ep.so+0x0000000fd6ab)
        #1 CacheCallback::callback(CacheLookup&) /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:58 (ep.so+0x0000000cf5cc)
        #2 CouchKVStore::recordDbDump(_db*, _docinfo*, void*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:1577 (ep.so+0x0000002f3fe7)
        #3 recordDbDumpC(_db*, _docinfo*, void*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:74 (ep.so+0x0000002f2b70)
        #4 lookup_callback(couchfile_lookup_request*, _sized_buf const*, _sized_buf const*) /home/vagrant/couchbase-server/couchstore/src/couch_db.cc:729 (libcouchstore.so+0x000000019e46)
        #5 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:101 (libcouchstore.so+0x0000000132d7)
        #6 btree_lookup_inner(couchfile_lookup_request*, unsigned long, int, int) /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:71 (libcouchstore.so+0x000000012bd3)
        #7 btree_lookup /home/vagrant/couchbase-server/couchstore/src/btree_read.cc:133 (libcouchstore.so+0x00000001236b)
        #8 couchstore_changes_since /home/vagrant/couchbase-server/couchstore/src/couch_db.cc:774 (libcouchstore.so+0x0000000196ca)
        #9 CouchKVStore::scan(ScanContext*) /home/vagrant/couchbase-server/ep-engine/src/couch-kvstore/couch-kvstore.cc:1227 (ep.so+0x0000002f2868)
        #10 DCPBackfill::scan() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:150 (ep.so+0x0000000d08ea)
        #11 DCPBackfill::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill.cc:97 (ep.so+0x0000000cfeb3)
        #12 BackfillManager::backfill() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:171 (ep.so+0x0000000c778d)
        #13 BackfillManagerTask::run() /home/vagrant/couchbase-server/ep-engine/src/dcp-backfill-manager.cc:50 (ep.so+0x0000000c745a)
        #14 ExecutorThread::run() /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:110 (ep.so+0x00000021641f)
        #15 launch_executor_thread(void*) /home/vagrant/couchbase-server/ep-engine/src/executorthread.cc:34 (ep.so+0x00000021592a)
        #16 platform_thread_wrap /home/vagrant/couchbase-server/platform/src/cb_pthreads.c:19 (libplatform.so.0.1.0+0x0000000033f4)

Change-Id: I330979be66ea5113f28846f664af70559379f1ae
Reviewed-on: http://review.couchbase.org/43317
Reviewed-by: Chiyoung Seo <chiyoung@couchbase.com>
Tested-by: Chiyoung Seo <chiyoung@couchbase.com>