MB-9209: hibernate idle couch_file readers

During xdcr those processes are reading tons of binaries. And after
burst of activity tend to go idle for long time. It makes sense to
hibernate those folks in this state. Releasing unused
memory. Especially given that it's state is just two args which is
cheap to gc/hibernate.

Change-Id: I6793dd7fcd29457f711a871bc12f3b7e275b56ff
Reviewed-on: http://review.couchbase.org/30357
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
MB-9730 Integrate native viewgroup cleanup support

An external native program `couch_view_group_cleanup`
is invoked to perform viewgroup cleanup. On successful cleanup,
viewgroup couch_file processes are refreshed to use new views
after cleanup.

A comparison of time taken for cleanup before and after this change
benchmarked by running `test/11-updates-cleanup-many-views.t` with
6400000 documents are as follows:

Before:
Removed 30000 values from the index in 4.468 seconds

After:
Removed 30000 values from the index in 2.364 seconds

Change-Id: I1ef010a5c1dad8366e90dbbba4bf93117a92377f
Reviewed-on: http://review.couchbase.org/30787
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
 MB-9730 Add support to stop cleaner native process

stop_cleaner() function needs to ensure that the
viewgroup cleanup native process has exited.
Added a mechanism to send exit message to native viewgroup
cleanup process and wait for process to exit and send exit status.

Change-Id: If187e2114a0608a8f704f6b43a3c36a4817035ab
Reviewed-on: http://review.couchbase.org/30988
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Remove an unused variable in rollback unit test

Remove an unused variable since it is violating the fact
that couch_file process pid can change in between updates since
viewgroup cleaner resets the couch file process for the viewgroup.

Change-Id: I8e3f9020e769c195970ae2984ea4bcaf2d5afee6
Reviewed-on: http://review.couchbase.org/31040
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Checkpoint progress for docs with _design prefix and too large bodies

If the last document in the sequence tree of a database file had its ID
with the "_design/" prefix, we would completley ignore it and skip
checkpointing. This was a problem if no other partition database files
had any updates and this document was associated with the last sequence
of the database - in this case each indexer run would process that sequence
number over and over, which would block forever stale=false queries - unless
a new document was added after to that database, or an existing one updated
or deleted. So just treat a document with a _design/ prefix in its ID as
any other document, as it shouldn't have any special meaning in regular
databases.

Similarly, if a document with a too large body is skipped, ensure that we
don't end in the same loop condition as above, by ensuring its sequence
number is checkpointed.

This was diagnosed in CBSE-893.

Change-Id: I7390db0329b00b137a0d9f1ff8e29971a58af143
Reviewed-on: http://review.couchbase.org/31341
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sarath Lakshman <sarathlakshman@slynux.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Merge remote-tracking branch 'origin/2.1.1'

* origin/2.1.1:
  Checkpoint progress for docs with _design prefix and too large bodies
  MB-9209: hibernate idle couch_file readers

Conflicts:
	src/couch_set_view/src/couch_set_view_updater.erl

Change-Id: Ida38d0c32d6c1b12d70e8ed9af8b0438b9c40a23
MB-9771 Integrate native incremental index updater

The main updater-writer process creates batches of incremental
changes to be applied to the view btrees and id btree. The list
of operations for each batch are written into a sorted file (one
for each btree in the index file). The viewgroup info along with
the sorted file paths are passed to an external native process.
The external process reads the operations from the sorted file and
applies to the btrees in the index file efficiently.

Change-Id: I122252a3803c29ac9351b8902e939c8c17c02825
MB-9771 Remove sorting step for incremental mapreduce view updates

For mapreduce views, native incremental index updater performs sorting
of btree ops before performing the update. Hence, sorting is not required
to be done from erlang side.

Change-Id: I433731c7bd9f6e1b75a5e120bfb1c98c39f7e7c9
MB-9771 Use port stream option to read stdout response

For native incremental index updater, use stream method to
read lines from stdout and parse view group header.
Earlier {line, N} option was used to receive stdout response.
It has limited control for receiving binary raw data (header data)
since the port driver itself tries parse lines.

Change-Id: I4104d494a7e56031e4e60cb7599d8382c10d0f3a
Remove an unwanted enclosing of a variable in list

Change-Id: I2f705c510c49aab6bbc06ed93e683f9d0324e23d
Add helper method for gracefully stopping external programs

Change-Id: I781e295596bd908c913184a4074f34b1f31c8836
MB-7629 Integrate native view compactor

Invoke native view compactor program to perform
btree compactions. For id_bree and mapreduce view btrees,
compaction is performed by couch_set_view_group_compactor.

Change-Id: I8c7526a5f2c46423e370d0823d34d2311d0ce14b
MB-7629 Fix view compactor stopper to stop native process

Compactor stopper has to send a stop message to the
native compactor process and it has to die gracefully.
It should be waiting for native process to die after
sending stop message.

Change-Id: I07c02bd5585538b51e05e9d59595f5b684d8cc3c
Refactor native btree updater wrapper to make it reusable

Refactored native btree updater helper code to be
reusable by incremental updater as well as view compactor
delta log apply methods.

Change-Id: I23172de5fd5a09b8a9f94f37476599f13f41a71e
MB-7629 Use native updater for applying compactor delta logs

Replaced erlang btree updater with native btree updater to
apply delta log files to the compacted btrees.

Change-Id: Ib2e09a450cfe0924484574b5e0680f0d6725912c
MB-7629 Prevent sorting compactor delta log files

Send a flag to the native btree updater to
turn off sorting compactor delta log files.

Change-Id: I64293a045e73dba9b80c87bfb7321c65e00791af
MB-7629 Fix view compactor incremental progress stats

The native view compactor can now send intermediate
progress of how much kvs are written at every percentage
progress intervals.

Change-Id: I4c9ddbfee0b0531733552d54f2eb92a5fe44cf6b