Adding flow control in couchdb upr client.

Change-Id: Id294aa24625d9b7e53ef4752090a1cb17c9475fe
Reviewed-on: http://review.couchbase.org/35600
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-10773 Handle dropping of connection gracefully

Currently if the connection to the UPR producer drops we end in an
error. This should be handled more gracefully.With this change, it
do retry and creates a new connection to the UPR producer.

Change-Id: Ie63c1581f025484420f9a50a8c7a0ad1c5304660
Reviewed-on: http://review.couchbase.org/36701
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Make sure startup errors are not lost

Change-Id: I1f75c88d73086fbff956bf678e236efa04862473
Reviewed-on: http://review.couchbase.org/36837
Tested-by: Sriram Melkote <siri@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-11371 Fix corruption in partition versions list

During view engine operations with replicas enabled, it
has been observed that parititon version data structure ends
up with duplicate entries for some partitions.

get_more_passive_partitions() helper function grabs all the
passive_partitions messages from its mailbox and merge into a
partitions list. The resulting partitions list may not be sorted
if multiple passive_partitions are processed. This partitions
list is used to construct partitions versions list and results
in unsorted partitions version list. Unsorted partition versions
list causes corruption in building another partition versions list
by merging the existing list. The partition version list
always needs to be sorted. Fixed get_more_passive_partitions()
method to return sorted partitions list.

Change-Id: I4d3794f3857cbded3017c5b842ed63361073eadb
Reviewed-on: http://review.couchbase.org/38139
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
Use the network read bytes for flow control

Currently message sizes are hardcoded.Now we have
changed that to use the network size length for the flow
control buffer ack.

Change-Id: I9c37a5068be943b31574464fb2edd3d2faf6a68d
Reviewed-on: http://review.couchbase.org/37954
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11044: Use data type supplied by UPR

With this change we now longer need to determine the data type of
a document within the view-engine (as implemented in
b9ae83f853671d0373bc6ba53c2a8f03a5eb717c [1]), but we can rely on the
the data type supplied by UPR.

[1]: https://github.com/couchbase/couchdb/commit/b9ae83f853671d0373bc6ba53c2a8f03a5eb717c

Change-Id: I7c2440bb5bc1429ab935673d012cc068a03a9d4f
Reviewed-on: http://review.couchbase.org/36703
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Revert "MB-10468 determine document type within view engine"

This reverts commit b9ae83f853671d0373bc6ba53c2a8f03a5eb717c.

The previous commit makes this one obsolete. The data type of a document
no longer needs to be determined within the view-engine, UPR provides that
information.

Change-Id: Idf2a5fc4ed6efa60b9d7ae196da71ded2b80dffd
Reviewed-on: http://review.couchbase.org/36704
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11290 Handle stale compactor notification in updater

During certain index group state changes we do not stop the existing
updator running process, but compactor might be stopped and restarted
after state changes are applied. This results in multiple compactor
notifications sent to the updater driver process. The updater driver
expects that only one compactor notification can happen at a time.
But that assumption is incorrect if compactor is stopped and restarted
while updater keeps running. When multiple compactor notification arrive
due to restarts of compactor processes, it needs to send ack to the most
recent process and ignore any further notifications.

Change-Id: Icfab19904057172b31f5c7d6294399bb596be6fb
Reviewed-on: http://review.couchbase.org/37904
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11416 Make control buffer size configurable for QE tests

Make buffer size configurable, so that QE can configure it to lower
value and verify that unacked_bytes are less then 20 % of buffer size.

Change-Id: I75fd8bbcc4333916bd6cfb09c76b3432553fddb8
Reviewed-on: http://review.couchbase.org/38227
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
MB-10273: Reduce the number of btree flushes

With commit d707d352a2cd03cfb9c8dd726f007ad32566d415 [1] the updater
was flushing to disk (i.e. updating the btrees) whenever there was
a snapshot boundary. This lead to a lot of i/o. This commit is
reverting this change and replaces it with some in memory operations.

Before changes are flushed to disk, a certain number of items (based
on the total size) is buffered in memory. This buffer must contain unique
items only, before flushing.

We now allow temporarily to have duplicated items in that buffer and
de-duplicate them right before they get inserted into the btree. This
saves i/o. A quick an dirty test showed that the flushes are reduced
by around 50%.

[1]: http://github.com/couchbase/couchdb/commit/d707d352a2cd03cfb9c8dd726f007ad32566d415

Change-Id: Ib8cf8463347cbccf3b249f47f68aa88c450993a9
Reviewed-on: http://review.couchbase.org/38308
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Fix to allow ep-engine change to work

Change-Id: I044fa606378d3cae23a20cc023baee103a44cb93