MB-11563 Update partition versions during updater checkpointing

The updater performs batch writes. After each short disk batch writes,
it will synchronize new header information and partition sequence numbers
upto which it has processed for each partitions. The view_group receives
the new header, seqnos and update the group header. We updated partition
versions for each partitions only after the complete updater instance run.

When new passive partitions are added to the current index as part of partition
migration from another node to current node, we have an way to notify newly
added passive partiton list to the currently running updater. This ensures that
updater doesn't need to be killed and restarted again with newly added passive
partitions, thereby wasting useful indexing time. When the updater receives
the new passive partitions list, it will start streaming changes from those
partitions by the updater-loader. When the updater checkpoints after it
received a new passive partition, it will be still sending a old partition
version list as part of new header. The view_group's updater checkpoint processor
calculates the missing partitions by checking which are the partitions which
are not present in checkpointed group's vbucket seqnos list. Since the seqnos
list contains the newly added passive partition, we will not be doing any
addition of that partition's entry in the partitions versions list. So the
view group's partitions version list has those passive partitions missing now.

At this instance if a higher priority happens, the view group terminates the
current updater instance (Remember that the correct partitions versions list
is only updated after complete updater run.). Hence nobody is going to fix
the partitions versions list and it can have missing passive partitions.

Eg.
Updater has partition verion list: [{1, v1}, {2, v2}]
Added new passive partition 5 and updater processed it.
Updater checkpointed new group,
seqnos list: [{1, 23}, {2, 23}, {5, 10}]
part versions list: [{1, v1}, {2, v2}]
View group kills updater due to higher priority task and now partition versions
list has missing entries.

Fixed updater to store partition versions for newly added partitons and send
upto date partitions versions list during checkpointing.

Change-Id: Ia47417e0d37429cc22dc41d826532b71ca116471
Reviewed-on: http://review.couchbase.org/38931
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
MB-11563 Add a unit test case for rollback after adding partitions

Change-Id: I2bea97b64f99a03d5d3f369c0d0ad30323227856
Reviewed-on: http://review.couchbase.org/38930
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11579 Use UPR_ADD_STREAM_FLAG_LATEST flag for upr stream request

When we request a stream for the first time and rollback happens due
to vbucket resets, there is race condition that could result in a
stream request with endseqno > vb_high_seqno of that vbucket. This
will cause stream to hang until more mutations are added to that vbucket.

Eg:
1. Requested stream for vb 228 for 0 - 6 and we received data
2. Requested stream for vb 229, but server told to rollback and we rolled
   back our entire index (Now, vb 228 sequence in index = 0, vbuuid =0 because
   of rollback)
3. To start index update, we obtained end_seqno from vbucket-seqno stats and
   say, seqno is 6
4. EP-engine resetted vbucket 228 in between
5. Index updater doesn't know about reset. It requested startseqno = 0,
   endseqno = 6

In order prevent stream from hanging, stream is requested with a special
flag that tells server to set endseqno. This flag needs to be set only
when we are requesting stream for a partition first time (ie, paritition
version is [{0, 0}]).

Change-Id: I292ed61fdbfc37b3c89edf32ec22527414e36187
Reviewed-on: http://review.couchbase.org/38954
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11461 Use a single upr message for gathering vbucket-seqno stats

The view engine has to get vbucket sequence numbers for all partitions
whenever a query is processed. The time taken for requesting the stats
and collecting partition sequence numbers affect the query latency
during stale=update_after and stale=false queries. Currently we are
requesting one message per partition to the EP-Engine. That means, for
1024 partitions, we will need to process 1024 upr messages.

Added support for sending single upr vbucket-seqno stats message and
parse interested partition seqnos. This should reduce the network
noise caused by stats requests and improve query latency.

Change-Id: I602e11923fd26980e1f08dab836b4dfa19a02eb3
Reviewed-on: http://review.couchbase.org/38691
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Add more details in upr stream timeout log

Added bucket, name, start_seqno and end_seqno information in
the stream timeout log message.

Change-Id: If2e2d84d8502b1f24c96669a3a74913836ec9ec4
Reviewed-on: http://review.couchbase.org/38733
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
MB-11598 Update partition versions list correctly after compaction

The compactor delta log files list currently does not have partition
versions information. Hence, if the compactor and updater is running in
parallel and additional passive partitions are added, it could end up
in a situation where additional passive partitions information is not
updated in the partition versions list.

Fixed compactor to include partition versions list and update
them accordingly.

Change-Id: I79b8c10f4db389f5762f62e6033c28902ef04474
Reviewed-on: http://review.couchbase.org/39015
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
MB-11506 Add a stat field for total sequence number difference

This stat field displays total sequence number difference
between KV store sequence number and View index sequence number
summed over all vbuckets. This gives an idea of how much indexing
is lagging behind w.r.t KV store.

Vbucket partitions that are indexible are considered and other
partitions are ignored

Change-Id: Ie810edcbf2a45617307c3d7762bb6ab492a55aaa
Reviewed-on: http://review.couchbase.org/38793
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
upr client: Fix a race condition in print_log callback

Stream end can happen independently at any time. Hence
we should not assume that stream information for a request
will be available when print log is requested.

Change-Id: I2bca4cf9b70377e74a6002155604eb46f8b060c9
Reviewed-on: http://review.couchbase.org/39091
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11505 upr client: Add time out log message for vbucket stats

Added a log message if any of the vbucket stats requests take
more than 2 seconds for receiving the response. Stats request
will also block wait until response is received or the upr client
crashes due to any internal error.

Change-Id: I76992f198b7b12e758aa50a8c43b4e37d215c29f
Reviewed-on: http://review.couchbase.org/39086
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
MB-11636: Don't delay sending of TCP packets

The UPR TCP connection haven't had the {nodelay, true} option set. Hence
the sending of packets was delayed.

Change-Id: I14851dfa37fba6f44809b8365d6d6c17c6b7dad1
Reviewed-on: http://review.couchbase.org/39111
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Dave Rigby <daver@couchbase.com>
Optimize for improving stale=false query latency

The stale=false query triggers an updater job if the current indexed
seqnos are less than current partition seqno if an updater is not
already running. Otherwise, if the query's wanted seqnos is less than
current indexed seqnos and an updater is already running, they are
added to a waiter list. All the query requests added into a waiter
list is only processed after completion of current updater instance.

When an updater starts processing mutations, it will process in batches
and synchronize headers to view group process frequently (Checkpointing).
But the reply to the waiting queries are only processed after the entire
updater instance run is complete. After each checkpointing if some of the
currently waiting query request's wanted_seqnos is less than current indexed
seqnos, we should be able to reply to the queries immediately.

Optimized checkpointing phase to reply query waiters if wanted_seqnos
criteria is met. This should improve query latency for stale=false queries.

Change-Id: I44d7067ebc9db7ce690679095e83554dd4244277
Reviewed-on: http://review.couchbase.org/39051
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
MB-11641 Update partition versions list after compaction delta is applied

When the view compactor process delta log records files, it should update
the new index header with new partitions versions list for that delta log.
Missing statement for updating partition versions was resulting in missing
entries for certain partitions in partition versions list if compactor
was triggered while new passive partitions are added.

Change-Id: I3aa093699af534d6fa2ffafb3e9224109f866e69
Reviewed-on: http://review.couchbase.org/39203
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Denser partition version logging

The logging of the partition versions when the group header was
updated was spread across several lines. That is neither useful
when skipping over the logs, nor when grepping in the logs. Hence
put the partition versions output into a single line.

Change-Id: I19f4872c4c063b54385d86ad1258110a8a6e04d2
Reviewed-on: http://review.couchbase.org/39206
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
MB-10921 Add a log message for couch_file delete

Added a log message when we delete a file. This will be helpful
to find which files are not getting cleanup after delete.

Change-Id: I9c79cfbd141edf05b1bb09918dd62d9d1e547dbf
Reviewed-on: http://review.couchbase.org/39233
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11672: Log every item that enters view engine updater through upr

Change-Id: I734918cdba29115c097a8e735cbe2a641ba21b8a