MB-100 Add assertion checks to the memory tracker stats functions

Change-Id: Ia54b2e4f29089aaafbb8306d4c9b4112a66466e0
Reviewed-on: http://review.couchbase.org/17122
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5439 Store TAP engine_specific value while grabbing the lock

To avoid any possible race conditions, TAP instance should be
stored as the engine_specific value for a given connection while
the TAP conneciton map is locked.

Change-Id: I057a6cd3cbeb37e500ac33eb049a0a875c325285
Reviewed-on: http://review.couchbase.org/17123
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5444: Use abort instead of assert in allocation hooks

Assert calls memory allocating functions and causes recursive
hooks calls.

Change-Id: If0c6f640e4f4d654514c872e99653882d8cb56dd
Reviewed-on: http://review.couchbase.org/17124
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5431: We now account for create_instance overhead

Change-Id: If1f30745a420da11afd020823e3291989f7184e5
Reviewed-on: http://review.couchbase.org/17125
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5444 Fix to the abort condition in tracking memory usage.

Change-Id: Iaa1c933c30fca02aaaa123d601776df7f51e9bf0
Reviewed-on: http://review.couchbase.org/17126
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Added tap cursor names and checkpoint position in checkpoint stats

Example stats for vbucket 0 that has one TAP cursor with the name
'eq_tapq:rep_10.1.3.93':

vb_0:eq_tapq:rep_10.1.3.93:cursor_checkpoint_id:    1
vb_0:last_closed_checkpoint_id:                     0
vb_0:num_checkpoint_items:                          1
vb_0:num_checkpoints:                               1
vb_0:num_items_for_persistence:                     0
vb_0:num_tap_cursors:                               1
vb_0:open_checkpoint_id:                            1
vb_0:persisted_checkpoint_id:                       0
vb_0:state:                                         active

Change-Id: I20da6b00d2fd8343c25797cec0b9a51e4b083155
Reviewed-on: http://review.couchbase.org/17127
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5457 Remove duplicate memory threshold check on backfill.

Memory threshold check on backfill is actually performed right
before scanning a vbucket's database.

Change-Id: I73c80088052f8cbf8e1399cc80f55caeff29d29e
Reviewed-on: http://review.couchbase.org/17128
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5457 Set backfill memory threshold to 95% of bucket quota

More accurate memory accounting allows us to set the backfill
memory threshold to a higher value. In addition, having a
different memory threshold compared with the TAP throttle
memory threshold would help us avoid the operational deadlock.

Change-Id: I789c3c77612d2b71aa9d21c4aada2e35996d8861
Reviewed-on: http://review.couchbase.org/17129
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5471: Fixed memory leak in ep_engine.cc (tapThrottle)

Change-Id: Ifa1468eea190477e9507f189a4bee5ddcd65ff54
Reviewed-on: http://review.couchbase.org/17130
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5458: Correctly track engine creation

Change-Id: I8f044cf4ab32cc4c67877c6db5f81f99e977e5b4
Reviewed-on: http://review.couchbase.org/17131
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5458 Account the memory overhead from stats_snapshot task

Change-Id: Ia2acc4fecb8ceb0b9721ba6842042e7e8d50816e
Reviewed-on: http://review.couchbase.org/17132
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5458 Reset thread local value upon leaving from bucket instance

Due to the new memory accounting, the thread local value should
be reset to NULL when a worker thread leaves from the bucket
instance, in order to not account the upstream memory allocation
in the bucket instance.

Change-Id: I416f7dfb18ee82141642bb16cfcf5549173e5e91
Reviewed-on: http://review.couchbase.org/17133
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-100 Align white spaces in ObjectRegistry functions

Change-Id: Ief41dd6195a6c72834f1ee35c5097d42df6694df
Reviewed-on: http://review.couchbase.org/17134
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5458 Disable memory tracker if memory tracker stat is underflow

We observed that the memory accounting stats from the new memory
tracker suffer from underflow issues when a bucket is destroyed.
This would cause a node crash and rebalance failures in case of
multiple buckets. To avoid this issue, this change allows us to
switch to the old memory accounting stats for a bucket being
destroyed rather than borting the process.

As part of the future enhancement on memory accounting, we plan
to place memory usage barrier correctly across multiple
components.

Change-Id: I8e48bbaa9e67faa9629e91e03d8243e1b4f40742
Reviewed-on: http://review.couchbase.org/17135
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5305 use PRODUCT_VERSION instead of git_describe if it is set

Change-Id: I54df6117a16dc0923a3b3027fe2d64e7f72b4d48
Reviewed-on: http://review.couchbase.org/17136
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
MB-5510 Retry a disk fetch upon the schema changes in sqlite.

We use sqlite3_prepare_v2() to compile the SQL fetch statement,
which does not require the recompilation of the SQL statement
upon the schema changes, but simply need to retry it.

For more information, please refer to
http://www.sqlite.org/faq.html#q15
http://www.sqlite.org/c3ref/prepare.html

Change-Id: I5196722abf6e028eca03396276a1d8845c7873d1
Reviewed-on: http://review.couchbase.org/17137
Reviewed-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Tested-by: Chiyoung Seo <chiyoung.seo@gmail.com>
Merge branch 'branch-18'

* branch-18:
  MB-5510 Retry a disk fetch upon the schema changes in sqlite.
  MB-5305 use PRODUCT_VERSION instead of git_describe if it is set
  MB-5458 Disable memory tracker if memory tracker stat is underflow
  MB-100 Align white spaces in ObjectRegistry functions
  MB-5458 Reset thread local value upon leaving from bucket instance
  MB-5458 Account the memory overhead from stats_snapshot task
  MB-5458: Correctly track engine creation
  MB-5471: Fixed memory leak in ep_engine.cc (tapThrottle)
  MB-5457 Set backfill memory threshold to 95% of bucket quota
  MB-5457 Remove duplicate memory threshold check on backfill.
  Added tap cursor names and checkpoint position in checkpoint stats
  MB-5444 Fix to the abort condition in tracking memory usage.
  MB-5431: We now account for create_instance overhead
  MB-5444: Use abort instead of assert in allocation hooks
  MB-5439 Store TAP engine_specific value while grabbing the lock
  MB-100 Add assertion checks to the memory tracker stats functions

Conflicts:
	ep_engine.cc

Change-Id: I12e541300f10578e2a326a04f5bf10858ff6b547
Merge "Merge branch 'branch-18'"