MB-9209: hibernate idle couch_file readers

During xdcr those processes are reading tons of binaries. And after
burst of activity tend to go idle for long time. It makes sense to
hibernate those folks in this state. Releasing unused
memory. Especially given that it's state is just two args which is
cheap to gc/hibernate.

Change-Id: I6793dd7fcd29457f711a871bc12f3b7e275b56ff
Reviewed-on: http://review.couchbase.org/30357
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Checkpoint progress for docs with _design prefix and too large bodies

If the last document in the sequence tree of a database file had its ID
with the "_design/" prefix, we would completley ignore it and skip
checkpointing. This was a problem if no other partition database files
had any updates and this document was associated with the last sequence
of the database - in this case each indexer run would process that sequence
number over and over, which would block forever stale=false queries - unless
a new document was added after to that database, or an existing one updated
or deleted. So just treat a document with a _design/ prefix in its ID as
any other document, as it shouldn't have any special meaning in regular
databases.

Similarly, if a document with a too large body is skipped, ensure that we
don't end in the same loop condition as above, by ensuring its sequence
number is checkpointed.

This was diagnosed in CBSE-893.

Change-Id: I7390db0329b00b137a0d9f1ff8e29971a58af143
Reviewed-on: http://review.couchbase.org/31341
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sarath Lakshman <sarathlakshman@slynux.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Merge remote-tracking branch 'origin/2.1.1'

* origin/2.1.1:
  Checkpoint progress for docs with _design prefix and too large bodies
  MB-9209: hibernate idle couch_file readers

Conflicts:
	src/couch_set_view/src/couch_set_view_updater.erl

Change-Id: Ida38d0c32d6c1b12d70e8ed9af8b0438b9c40a23
MB-9771 Integrate native incremental index updater

The main updater-writer process creates batches of incremental
changes to be applied to the view btrees and id btree. The list
of operations for each batch are written into a sorted file (one
for each btree in the index file). The viewgroup info along with
the sorted file paths are passed to an external native process.
The external process reads the operations from the sorted file and
applies to the btrees in the index file efficiently.

Change-Id: I122252a3803c29ac9351b8902e939c8c17c02825
Reviewed-on: http://review.couchbase.org/31362
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9771 Remove sorting step for incremental mapreduce view updates

For mapreduce views, native incremental index updater performs sorting
of btree ops before performing the update. Hence, sorting is not required
to be done from erlang side.

Change-Id: I433731c7bd9f6e1b75a5e120bfb1c98c39f7e7c9
Reviewed-on: http://review.couchbase.org/31550
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Remove an unwanted enclosing of a variable in list

Change-Id: I2f705c510c49aab6bbc06ed93e683f9d0324e23d
Reviewed-on: http://review.couchbase.org/31662
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Add helper method for gracefully stopping external programs

Change-Id: I781e295596bd908c913184a4074f34b1f31c8836
Reviewed-on: http://review.couchbase.org/32308
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9771 Use port stream option to read stdout response

For native incremental index updater, use stream method to
read lines from stdout and parse view group header.
Earlier {line, N} option was used to receive stdout response.
It has limited control for receiving binary raw data (header data)
since the port driver itself tries parse lines.

Change-Id: I4104d494a7e56031e4e60cb7599d8382c10d0f3a
Reviewed-on: http://review.couchbase.org/31551
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-7629 Integrate native view compactor

Invoke native view compactor program, couch_view_group_compactor
to perform btree compactions.

During a test run (on Mac/SSD) with 7M documents and 3 simple views,
time taken for view compaction before and after native compactor is
as follows:

Before:
86.739063s

After:
49.133651s

Change-Id: I8c7526a5f2c46423e370d0823d34d2311d0ce14b
Reviewed-on: http://review.couchbase.org/32309
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-7629 Fix view compactor stopper to stop native process

Compactor stopper has to send a stop message to the
native compactor process and it has to die gracefully.
It should be waiting for native process to die after
sending stop message.

Change-Id: I07c02bd5585538b51e05e9d59595f5b684d8cc3c
Reviewed-on: http://review.couchbase.org/32310
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Refactor native btree updater wrapper to make it reusable

Refactored native btree updater helper code to be
reusable by incremental updater as well as view compactor
delta log apply methods.

During a test run (on Mac/SSD) with 7M documents and 3 simple views,
the taken taken before and after integration of native sequential sorting
and btree updater is as follows:

Before:
Indexing time: 657.884 seconds
Sorting + Btree updates: 603.450 seconds

After:
Indexing time: 386.529 seconds
Sorting + Btree updates: 346.332 seconds

Change-Id: I23172de5fd5a09b8a9f94f37476599f13f41a71e
Reviewed-on: http://review.couchbase.org/32311
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-7629 Use native updater for applying compactor delta logs

Replaced erlang btree updater with native btree updater to
apply delta log files to the compacted btrees.

Change-Id: Ib2e09a450cfe0924484574b5e0680f0d6725912c
Reviewed-on: http://review.couchbase.org/32312
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9951: Trim rev-id to 16 bytes

With adding the datatype and flexible metadata to the the
metadata field [1], it is no longer fixed to 16 bytes, but
is at least 2 bytes bigger, one for the flex meta code and
one for the datatype field.

Hence trim the revision ID down to a maximum of 16 bytes.

[1] http://review.couchbase.org/31046

Change-Id: I25898dc4e70c1d7d071c228e2bcdd2c03e06ff6a
Reviewed-on: http://review.couchbase.org/32743
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-7629 Prevent sorting compactor delta log files

Send a flag to the native btree updater to
turn off sorting compactor delta log files.

Change-Id: I64293a045e73dba9b80c87bfb7321c65e00791af
Reviewed-on: http://review.couchbase.org/32313
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9968 Use shutdown_sync() for stopping view compactor

During a scenario when stop_compactor() is invoked by couch_view_group process,
it sends `stop` message to the native compactor driver process and block waits
for that process to exit. But if at the same time view compactor driver has
completed the native process and tries to send `compact_done` using
gen_server:call() method to the couch_view_group process, both of them can
be performing blocked call on each other and it becomes a circular dependency
entering into a deadlocks state.

The native compactor can die by itself whenever the erlang port is closed
by detecting the stdin close. Since the compactor does not operate on the
current active index file, when the compactor erlang driver process is
terminated we do not need to ensure that native process has been died.
It can detect the stdin pipe close and exit asynchronously.
Replaced shutdown_wait() method with erlang process force kill method,
shutdown_sync() to stop the running compactor.

This reverts commit d0d76be2cd926e647cc7c7f7d4c07341c9813187.

Change-Id: I34e616ee2127d8a271f4364884fb8337f91a03fa
Reviewed-on: http://review.couchbase.org/33034
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
MB-10090 Fix receive_group_header buffered stdout reader method

receive_group_header method simulates blocked read of bytes in
given length by operating on the port interface in stream mode.
In stream mode, data is received as messages asynchronously.
If data accumulator received by this method is less than the
given length, it has to wait for data to arrive through port
and append to the existing data to extend the buffered data.

The current buffer was getting replaced by newly arrived data
rather than appending it and hence it indefinitely waits for
enough bytes to arrive. The test needs to use 1024 vbuckets
to go through this code path.

Change-Id: I757ecda73ac29ffba983aed64362cdf9b2f9914c
Reviewed-on: http://review.couchbase.org/33073
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Ignore non-port related messages in native compactor driver

The view native compactor driver process needs to handle only
Port related messages. It can ignore receiving any messages
such as pause message sent by unit test.

Change-Id: Iafd4f15219933f37a28d618db79ea95bbad5b5e9
Reviewed-on: http://review.couchbase.org/33161
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-7629 Fix view compactor incremental progress stats

The native view compactor can now send intermediate
progress of how much kvs are written at every percentage
progress intervals.

Change-Id: I4c9ddbfee0b0531733552d54f2eb92a5fe44cf6b
Reviewed-on: http://review.couchbase.org/32314
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Fix updater stats task to stay until disk write is complete

The the updater progress stats is populated from
the docloader when documents are loaded into MapQueue.
Even after all documents are loaded to MapQueue, it takes
additional amount of time to sort records and and build/update
ondisk indexes. Even though loading is complete, we need to make
sure that the progress stats task should not dissapear until
the whole updater processing is complete.

Change-Id: Idf3d3304d488a0bf3851db1b2b172ea6e796f1e0
Reviewed-on: http://review.couchbase.org/33117
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Add support for stopping native index builder

Similar to the native compacter and native btree updater,
added ability to send a stop message to the index builder
and wait for native process to exit.

Change-Id: Ibda0ef728100266508407f6512f6f12dac2080d3
Reviewed-on: http://review.couchbase.org/32549
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>