SEG-11041: Membase has high bg_fetch with high no of conn with no client load

With the change in the eviction framework, get and getl return temporary
failure when the items are non-resident and eviction doesn't succeed in
freeing memory. The cases for getl in ascii mode and get/getl in binary
mode were not handled correctly under this condition.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
SEG-11119: Fix mbstats parsing for various stats

Fixed mbstats to take care of a couple of things:
- item_disk_age and item_memory_age now have the base unit set to
  seconds
- all_item_sizes is a merged histogram of resident_item_sizes and
  non_resident_item_sizes

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
Changes to enable disk based backfill

With the multiple KVStore design and the capabilities provided by WAL
mode, a bunch of changes were done to make disk based backfill possible:

- The hashtable walk that BackFillVisitor does to pick up non-resident
  items is now disabled for any vbucket on which disk backfill will be
  scheduled. Before beginning the disk walk, a cursor is placed on the
  open checkpoint and the items pending for flushing are picked up.
  After the walk, the flush list items are shipped across the tap,
  following which the checkpoints are shipped.
- Eviction checks are performed prior to starting the disk walk (which
  will cause it to snooze in case of OOM) and on every retrieved item
  when necessary.
- The disk backfill method is enabled only on WAL mode because this
  allows the usage of a separate dispatcher to keep the regular
  background fetches going. On TRUNCATE mode the flusher and the
  background fetches will be stalled due to which this has been disabled
  even if a single KVStore is on TRUNCATE for the vbucket in
  consideration.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>

Conflicts:

	backfill.cc
	ep.hh
Memory throttle for disk based backfill

Limited the number of items backfilledItems can hold at any point of
time. This is controlled by a configurable parameter bf_max_list_size
which is set to 1000 by default.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>

Conflicts:

	ep_engine.cc
Made bf_resident_threshold configurable via mbflushctl

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
Added stats for the separate dispatcher for backfill

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
SEG-11008: ep_total_cache_size accounting is incorrect with key is set with -ve expiry

The constructor for StoredValue was performing a wrong increment on
ep_total_cache_size. Since we follow totalSize() on every state change,
the constructor must follow the same.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
Handle TapConnection teardown cleanly during disk backfill.

When TapConnection is freed, abort the disk backfill dispatcher job instead of
continuing unnecessarily.

Signed-off-by: Vandana Shah <vashah@zynga.com>
In disk backfill, no need to check for memory if TapConnection is gone

Signed-off-by: Vandana Shah <vashah@zynga.com>
SEG-11235: Membase crash while replicating 250M keys

Deletes were not handled when flusher items are picked up before disk
based backfill. This fix uses queued_item in place of Item to record the
operation along with other item properties.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
SEG-11235: Membase crash while replicating 250M keys

This is based on Vandana's review. Since copyItemsFromFlushList is a
function available in EventuallyPersistentStore, it must populate all
possible fields into the queued_item list. Further, setting the flag
TAP_CONNECT_REQUEST_KEYS_ONLY defers the scheduling of disk based
backfill.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
Avoid hashtable lookup for an item that has been fetched from disk.

When doWalkTapQueue processes items in backfilledItems (these come from
the disk walk of disk backfill or tap-bg-fetch of backfill visitor), it
obtains the latest copy of the item from hashtable. This is not
necessary and only creates overhead due to locks taken on the hashtable
which might contend with frontend load. In any case, disk backfill
stocks up flusher items and both methods set checkpoints, ensuring no
mutations are lost. This fix removes this check when unnecessary.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
SEG-11263: Pause persistence on a per kvstore basis

This allows a particular kvstore to be started or stopped via
mbflushctl. The script takes in an extra parameter for the kvstore ID.
Not giving an ID performs the operation on all kvstores.

Changed pauseCounter logic to count each module exactly once using a
flag.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
SEG-11264: incorrect status for disk based backfill with multikv store

This commit sets up a way to be able to test the issue. Every kvstore
can be delayed by bf_disk_sleep_time seconds when enabled with
bf_disk_sleep_enable. With the dispatcher displaying the name of the
task under stats, it is possible to reproduce a scenario where a single
kvstore works slower than the rest. If backfill reports completion
before every kvstore is finished, it can be tracked.

Also added stats and documentation for all backfill related parameters.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
SEG-11264: incorrect status for disk based backfill with multikv store

When backfill is scheduled, it must account for the number of kvstores
that the task gets dispatched on so that the accounting is correct when
the decrement occurs during completion.

Also removed forced connection ops that were added to make disk backfill
work with tap_keepalive set to 0.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
SEG-11002: backfill replication triggers huge ep_overhead in master resulting in oom kill

Added a configurable parameter bf_num_keys_threshold which triggers a
disk backfill whenever the number of items in a vbucket happen to be
above this threshold. This will enable forcing disk based backfill when
there is a large number of keys with very small blob sizes.

Performed a few cleanups and added stats for both thresholds. Got rid of
minimum limit for bf_resident_threshold so that memory backfill can be
forcefully triggered by setting the right values.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>

Conflicts:

	backfill.cc
Use backfill dispatcher and kvstore for tap bg-fetches

To keep regular bg-fetches undisturbed while scheduling, tap bg-fetches
must be scheduled on the separate dispatcher for backfill for getting
the items from disk.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>

Conflicts:

	tapconnection.cc
	backfill.cc
SEG-11265: add stats for how is much remaining in flusher queue during disk based backfill

This increments the bf_flush_items_<kvId> stat when items are retrieved
from the flusher and decrements it when these are sent across the tap.

Signed-off-by: Abhishek Nanda <ananda@zynga.com>
Fixed tap producer to not call epstore->get to update sync registry

SEG-11271: running tap_example.py causes membase to crash in case of DGM
Removed unnecessary epstore->get call while updating sync registry for
replication from tap producer. It is okay to return the data read from disk
as the latest data will be sent across from checkpoint queues.

Signed-off-by: Vandana Shah <vashah@zynga.com>
Added tap_throttle_persist_threshold config to tune tap throttle threshold

This config allows tuning of tap throttling based on persistence queues.
Also changed the default to 2 million.

Signed-off-by: Vandana Shah <vashah@zynga.com>

Conflicts:

	management/mbflushctl