MB-11458 Fix parsing of noop message from server

The upr server sends noop request and the upr client should
reply as response message. Currently the message parser expects
noop message from server to be response type. This assumption is
wrong.

Renamed no_op to noop to make it consistent in the upr client
implementation.

Change-Id: Id55ec6f483e340f2a51c1c8effe9e10e10479c3a
Reviewed-on: http://review.couchbase.org/38421
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
MB-11458 upr client should crash if any fatal errors happen

upr client should not supress any erlang process crash that is happening
to the receive worker and try restarting receive worker. The receive
worker process should be restarted only if the upr connection is dropped.

Change-Id: I3815247c9981dfeb9af51deced0ac1dffda05257
Reviewed-on: http://review.couchbase.org/38470
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Move get_seqs() to couch_set_view_util module

Change-Id: I697769a9bd5d62c358a44e19d16d1eda0437fb09
Reviewed-on: http://review.couchbase.org/38401
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
MB-11387 Compute index seq nos before starting updater

The end sequence numbers upto which the updater has to index
should be precalculated before starting the updater. We update
the index one partition at a time. Currently the end sequence number
is computed just before reading from the partition upr stream.
By the time one parition stream has been read, the next partition
sequence would have advanced to a higher value and it accumulates
further for each partition. This results in stale=false queries
to wait for long time since one updater instance would take too
much time to notify all the waiters.

If a compactor process has been already started, it will have to
chase the updater and do all the work of applying lot of deltas.
This will also result in compactor never able to catchup and index
files will remain fragmented all the time.

Change-Id: Ib768fced2a7251d797812f3687159f2cc0e59663
Reviewed-on: http://review.couchbase.org/38402
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
Clear existing streams from upr client if updater crash

If updater crash due to some reason, the next updater run
should not be affected. If updater crashes while reading from
a upr stream, the upr stream request will be active within the
upr client. We need to remove the active stream if updater exits
with an error.

Change-Id: I36ee7bf3ef7673cab2547762db7e3db5ae18c355
Reviewed-on: http://review.couchbase.org/38480
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
MB-11472 Avoid btree update call when there is nothing to flush

The updater writer process data in batches of certain size.
If all kvs data is processed in last batch and current batch has
empty kvs, this is the final batch. We do not need to execute
btree update codepath for just updating the partition seqnos.

Since kvs list is empty and there is no active flush session,
id_btree sortfile will be nil and that case is currently unhandled
in the btree update code.

Change-Id: I9b18faf60c215560fb12c96909e83e02717c2216
Reviewed-on: http://review.couchbase.org/38489
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11296: don't treat empty value as valid json

Because it's not. It was previously considered valid json due to
"anything and not just object or array is permitted" code that wraps
doc value with square braces. Which obviously caused empty value to be
mis-detected as json.

Change-Id: Id15621cac360047a9353ba0814cb7bbe1c5832b0
Reviewed-on: http://review.couchbase.org/38628
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
MB-11490 Close open tmpfile fds before starting native view updater

All open sort tmpfile fds should be closed before starting the native
view updater. Observed few failures recently with error codes such as
COUCHSTORE_ERROR_OPEN_FILE, COUCHSTORE_ERROR_NO_SUCH_FILE and
COUCHSTORE_ERROR_READ while updater operating on sort tmpfiles.

Change-Id: Ia3140131b2968ab78971e3e7d4aaf899dfc40e97
Reviewed-on: http://review.couchbase.org/38630
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
MB-11462 Acknowledge bytes received for close requested streams

When a close_stream request is send to the upr server, by that
time upr server would have already placed some mutations in
its network buffer for that stream. The upr client has to read
those mutations from the socket and ignore them. But even though
the client's user is not interested in it, since we received those
bytes we have to acknowledge using flow control ack message.

Whenever a high priority task arrives, the view engine cancels
an updater job and sends close stream message for currently active
upr stream. Since we did not send ack message for closed stream's
received bytes, connection was getting stuck when upr flow control
window gets full.

Change-Id: Ieedc664c9716e5e38c00852c1c35056649e69e6c
Reviewed-on: http://review.couchbase.org/38636
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
MB-11462 Fix missing state assignment after flow control check

The state variable tracks how much user has read from the buffer.
Return value from check_and_send_buffer_ack() method has to be returned
as the new state for handle_info callback.

Change-Id: Id2f2989b1453c1edbdaa998e13e4f8ed3a4fad4f
Reviewed-on: http://review.couchbase.org/38639
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Ack for unacked bytes when upr client has no active stream

When upr client completes processing of last stream_end message
for the last active stream, send flow control ack for the remaining
unacked bytes. This will help tests to identify whether client has
processed all bytes sent.

Change-Id: Ic797dc0e19c5ec8e46d4c1d769e2d81359f13aff
Reviewed-on: http://review.couchbase.org/38643
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11462 Stop leftover upr streams when compactor restarts updater

If the compactor has finished and verifies that it has caught up
already and the currently running updater hasn't completed update of
current delta, it stops the current updater and switches to the compacted
index file. It has to clear active upr streams after stopping updater process.

Change-Id: I8609cb241748c2c5f04518995cdcbd2f806efc54
Reviewed-on: http://review.couchbase.org/38644
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11504 Remove stale delta logs from compactor delta log list

The index compactor can be started anytime without bothering if
an updater is already running or not. But updater always send its
log files to group coordinator for compactor's usage. But for a
compactor, it is only interested in delta files which are generated
after it is started. We should be able to discard any files created
before it is started. If a compactor is started while an updater is
running, it will ask for an explicit ack from updater. This ack can
be used to flush all old delta log files.

Stale files were being added to the compactor delta log list and
updater deletes those files from disk if a compactor is currently not
running. But log files remain in the compactor delta log list. When a
compactor is started next time, it tries to apply log files which are
not present on disk. This was leading failures in couch_view_file_merger.

This commit might also might slow down fragmentation by eliminating
stale delta log files being applied to the compacted index file.

Change-Id: Ic100a28337edcd3156cc011fc80b5225dae5e685
Reviewed-on: http://review.couchbase.org/38645
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Revert "MB-11472 Avoid btree update call when there is nothing to flush"

This reverts commit d7d99c5d2b42eac9b4268899076dd0b3f2e0d464.
It is identified that this commit causes data corruption (MB-11486)

Change-Id: I3337cd5f368853710c17bcd11ef1d85ad59daceb
Reviewed-on: http://review.couchbase.org/38647
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11486 Fix data corruption caused by commit d7d99

Fix for MB-11472 had a race condition that could cause data corruption.
The updater finish its operation when there is no id sortfile present
as well as kvs to be flushed is empty. But there could be a native updater
instance running in the background processing the last batch. We should
check and wait for currently running updater to finish before declaring
update operation as finished.

Change-Id: I404da8fbe77f1d28cbf6f5058e563712e0dee455
Reviewed-on: http://review.couchbase.org/38648
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11462 Fix flow control ack handling for remove stream message

When a user requests remove_stream for a vbucket through upr client,
the actual remove of internal stream queue structure is done after receiving
reply from the server for close_stream upr command. But the flow control ack
is perfomed before sending the close_stream message by calculating the
current mutations bytes present in the queue. By the time actual
server response arrive, some more mutations for that stream would have
queued if upr server had sent them during the window of time before it
received the close_stream request. The mutation queue gets removed when
the server response is processed. This race condition causes few mutations
being silently removed from the queue and unacknowledged.

Eg.

Time T1:
User requests close_stream and current stream queue has 10 items
Before sending close_stream message to upr server, upr client
sends an flow control ack for 10 items.
upr client sent close_stream request.

Time T2:
upr client received 5 mutations for the stream for which we requested
close stream. The server would have sent these mutations before it received
close_stream request.

Time T3:
upr client received close_stream response from server
Removed the mutation queue for the stream with 5 mutations in it.
Nobody acked for 5 mutations which were present in the queue.

Change-Id: I4f0be7827c073dfd6c82b1205458a9a72608329d
Reviewed-on: http://review.couchbase.org/38650
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-11259 Change bytes field in upr buffer ack message

The protocol message for buffer ack is modified to have
non-zero extlen.

Change-Id: Ia5a355b55f2e27ece494983e62ac3cd04a026889
Reviewed-on: http://review.couchbase.org/38736
Reviewed-by: Michael Wiederhold <mike@couchbase.com>
Tested-by: Michael Wiederhold <mike@couchbase.com>
MB-11563 Reset partition versions of recent partitions after rollback

When an index rollback is performed, we restore an old index header. In an
old index header, the partitions membership of the current index might be
different. Some recently added partitions may not be present in the index
snapshot pointed by the rollbacked header. Hence, we need to add entries
for newly added partitions which are not present in the partition lists
of old snapshot. We are currently updating seqnos list and cleanup bitmask.
But parititions version list also has to be updated. Otherwise, when the
updater tries to start an upr stream, it will not find the partition version
for a newly added partition and it would crash.

Change-Id: I4296cade0ff58929e4282e02ec6e154030f6205b
Reviewed-on: http://review.couchbase.org/38878
Tested-by: Nimish Gupta <nimish@couchbase.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
MB-11176 Detect vbucket resets by checking start_seqno > end_seqno

During special scenarios like upgrade from Couchbase 2.0 to Couchbase 3.0,
UPR server may reset vbuckets multiple times and hence end_seqno might
become lower than last indexed sequence number. The end_seqno is obtained
from vbucket-seqno stats and it gets resetted when a vbucket reset happens.
In such an event, view engine should optimistically check if a rollback is
required and should perform index rollback.

Change-Id: I6a702adb05f2207e2adcde20e7c418d00393c4cc
Reviewed-on: http://review.couchbase.org/38876
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Nimish Gupta <nimish@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
Fix a race condition in unit tests

The tests 20-debug-params.t and 14-duplicated-keys-per-doc.t are not
waiting for all running updater processes to be finished before
shutdown of the view_group. This results in crash if the background
updater is trying to read paritition db files while shutdown group
command is in the process of deleting db files.

Change-Id: Ide04ea61b2d05a77261fcb1a3e1be68d2ce997d4
Reviewed-on: http://review.couchbase.org/38789
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>