Compact vbuckets by their ID in ascending order

When updating an index, the updater opens all the vbucket
databases, reads all changes from each, starting with the
vbucket with lowest ID, and then closes each before reading
the changes from the next one.
Compacting the vbucket databases in order decreases the
amount of time pre-compaction database files are kept open.

Change-Id: Iff01a792b08bdbeb7afc9d2263d4e21db23a0ee3
Reviewed-on: http://review.couchbase.org/13735
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
don't try to force local ddoc replication on janitor pass.MB-4825

The original idea was to ping ddoc replication srv that some vbuckets
may have been created. This was when we replicated ddocs to each
vbucket database.

It was broken even from day 0. Because it only touches local ddoc
replication srv. Even if this node doesn't run bucket yet/anymore.

Good thing is that it's not needed anymore. Because we don't replicate
ddocs to vbucket databases. So ddoc_replication_srv doesn't need any
notifications from janitor.

Change-Id: I759e457c984f94384eecfc673a7a0921111de99f
Reviewed-on: http://review.couchbase.org/13752
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Merge branch 'branch-20dp4'

* branch-20dp4:
  don't try to force local ddoc replication on janitor pass.MB-4825

Change-Id: I2eb27ec54eacf80ae4ab0b731597ae5befb5b888
Added 7 stats to bucket analytics to cover couch disk usage

Document data size, view data size, total view and docs disk
size, view and docs fragmentation percentage and total size
of view + docs data.

Change-Id: I18708d895a8a20a27c2222a2fa3fde918cbe2851
Reviewed-on: http://review.couchbase.org/13672
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Move getenv_int() from to misc.erl

This function will also be used by xdcr manager. Hence moving it to a
common place.

Change-Id: Ic5726b8b56fb2d42923f186a886a93bed954ef40
Reviewed-on: http://review.couchbase.org/13662
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Set couch_replicator connnection timeout

The default timeout was too small and was causing the xdcr source to
retry requests too frequently. If the destination was already busy
servicing the original request, resending requests frequently would
make it worse.

The new value is 5 mins. Not sure if this is too conservative but if
it turns out to be so, it can be changed later or made an environment
variable.

Change-Id: I4b074ddd65f09afb8f43f936c8889620a06e0871
Reviewed-on: http://review.couchbase.org/13660
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
ns_server changes towards fixing xdcr

Change-Id: I55cfd8ae40812f9e9cb3fb06b3b93b7b2e7f3001
Reviewed-on: http://review.couchbase.org/12292
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Make get-meta returns the CAS

Return the CAS returned by ep-engine along with enoent for a non-
existent item.

Change-Id: I85e9d6eecbec38ce31cfedf2c2fabb5a78e85fdb
Reviewed-on: http://review.couchbase.org/13657
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Use set_with_meta instead of add_with_meta for new items

Even for an add through xdcr, ep-engine creates a temp item and
returns its CAS, which is supposed to be used while issuing the add
later. This is similar to issuing a set_with_meta command. Hence,
add_with_meta is redundant and can be removed.

Change-Id: Ic7339c74f7c2dd09f5b5e0516809e93085015567
Reviewed-on: http://review.couchbase.org/13658
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Add checkpointing support to xdcr

We leverage Couch's native replication checkpointing support which
works as follows: At the beginning of replication, the source and
destination are looked up to check whether a previous checkpointing
document exists. This is done to determine the sequence number at
which to start replication. While replication is going on,
checkpointing is done at regular intervals by a separate process.
Each time, a new checkpointing record (consisting of, among other
things, the last replicated sequence number and the session id) is
created and and appended to the history, and the document is sent for
an update to both the source and the destination.

Given this scheme of things, the following approach for xdcr
checkpointing seems to work best i.e., one involving the least changes
to the existing Couch replicator code (this may change in the future
if we decide to adopt a different approach).

We bypass ep-engine for opening and updating checkpoint documents. In
order to make this safe (since there's no coordination between Couch
and ep-engine), we store the documents in the master database instead
the individual vbucket databases. We also prepend the document id with
the vbucket id to avoid conflicts.

Opening a checkpoint doc is no different from opening a regular doc.
Updating a checkpoint doc requires us to wait for the in-memory items
(as of the time of receiving the doc) to be persisted to Couch before
updating the doc itself. This is a little conservative as the wait
times may end up being longer for nodes with slow disks. It is,
however, the simplest approach right now. Later, we can evaluate
whether we need another approach.

Change-Id: I8ba141cefc54e368071e1326ac0bb81e24c72d07
Reviewed-on: http://review.couchbase.org/13772
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Implementation of delete-with-meta in capi

Change-Id: I1ac7959e701a1bb94c4358383acea5166a4cd0d9
Reviewed-on: http://review.couchbase.org/13659
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Set timeout for set_with_meta

The gen_server call was using the default timeout of 5 secs which is
too small and was causing unnecessary timeouts. The new value is set
to ?TIMEOUT (30 secs).

Change-Id: I614a0ae70973f35f0307af6a2ea0885d1febe386
Reviewed-on: http://review.couchbase.org/13661
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Schedule vbucket replications in small batches

Previously, xdcr manager would trigger couch replications for all
active vbuckets simultaneously which was causing performance to
suffer. The new approach, instead, schedules replications for only a
small number of vbuckets (32 by default and configurable through an
environment variable called MAX_CONCURRENT_REPS_PER_DOC) at any given
time.

This change set only supports xdcr continuous mode replication. We
start by scheduling the specified number of vbuckets (in couch mode
non-continuous) and periodicaly monitor them. As one or more of them
are done, new vbuckets are scheduled. After we've run through all of
them in this manner, we repeat the process.

An alternate approach is likely better where we can assign a time
slot for each vbucket instead of letting it run through completion
before scheduling another. Implementing this approach would be
greatly simplified if the Couch replicator API supported pause and
resume functions. Until then, we can stick to the current approach.

Also, I found it harded to break this change set into smaller pieces.
If reviewing seems hard or if there's a better way to organize stuff,
please let me know.

Change-Id: Ifc905becb41296e68ee66dbdc73b912bc055d15a
Reviewed-on: http://review.couchbase.org/13663
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Use existing stats for disk usage and display data usage on bucket screen

Change-Id: I9dc6a113753ae3c0b2ab32c19ef5801d03529a55
Reviewed-on: http://review.couchbase.org/13836
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Only collect stats for membase buckets

Change-Id: Icb93b59b23d1d49302308826b9ddda34e1a5347c
Reviewed-on: http://review.couchbase.org/13837
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Account for diskUsed no longer reported for memcached buckets

Change-Id: Ia7f14f333230672640c4c8f1b54d2545a8ec2d40
Reviewed-on: http://review.couchbase.org/13838
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Connection pooling of ns_memcached instances

Currently there's only one ns_memcached data process per bucket.
However, with xdcr, there will likely be many concurrent replication
streams coming into a given node, where they'll end up getting
serialized due to only one ns_memcached process serving their
requests. This change set makes it so that there can be multiple
processes for serving data requests. Their number is configurable by
means of an environment variable.

Change-Id: Ie81fb223324f3ca789f54568d09853fffcb9eb2e
Don't use num_items_for_persistence during xdcr checkpointing

ep-engine flusher decrements num_items_for_persistence before
actually persisting an item, so relying on its value becoming zero
before updating the checkpoint document is not safe. Instead, we wait
for the current open checkpoint id to be persisted. If it gets
persisted before we timeout, the checkpoint document is updated.
Otherwise, we don't checkpoint.

Change-Id: I1e86be03a67d60f9a22a36d5ef052fae53b414e9
Explicitly trigger doc replication upon startup

Due to a race condition, it may so happen that after a new node is
added to a cluster, the design/replicator docs on existing nodes do
not replicated to the new node. This change set explicitly triggers
the replication of all existing docs by sending the
replicate_newnodes_docs message to all the existing nodes.

Change-Id: I3d3f47d3a2546334dfdb3b4052019e06b99c33f2