MB-9771 Integrate native incremental index updater

The main updater-writer process creates batches of incremental
changes to be applied to the view btrees and id btree. The list
of operations for each batch are written into a sorted file (one
for each btree in the index file). The viewgroup info along with
the sorted file paths are passed to an external native process.
The external process reads the operations from the sorted file and
applies to the btrees in the index file efficiently.

Change-Id: I122252a3803c29ac9351b8902e939c8c17c02825
Reviewed-on: http://review.couchbase.org/31362
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9771 Remove sorting step for incremental mapreduce view updates

For mapreduce views, native incremental index updater performs sorting
of btree ops before performing the update. Hence, sorting is not required
to be done from erlang side.

Change-Id: I433731c7bd9f6e1b75a5e120bfb1c98c39f7e7c9
Reviewed-on: http://review.couchbase.org/31550
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Remove an unwanted enclosing of a variable in list

Change-Id: I2f705c510c49aab6bbc06ed93e683f9d0324e23d
Reviewed-on: http://review.couchbase.org/31662
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9771 Use port stream option to read stdout response

For native incremental index updater, use stream method to
read lines from stdout and parse view group header.
Earlier {line, N} option was used to receive stdout response.
It has limited control for receiving binary raw data (header data)
since the port driver itself tries parse lines.

Change-Id: I4104d494a7e56031e4e60cb7599d8382c10d0f3a
Add helper method for gracefully stopping external programs

Change-Id: I781e295596bd908c913184a4074f34b1f31c8836
MB-7629 Integrate native view compactor

Invoke native view compactor program to perform
btree compactions. For id_bree and mapreduce view btrees,
compaction is performed by couch_set_view_group_compactor.

During a test run (on Mac/SSD) with 7M documents and 3 simple views,
time taken for view compaction before and after native compactor is
as follows:

Before:
86.739063s

After:
49.133651s

Change-Id: I8c7526a5f2c46423e370d0823d34d2311d0ce14b
MB-7629 Fix view compactor stopper to stop native process

Compactor stopper has to send a stop message to the
native compactor process and it has to die gracefully.
It should be waiting for native process to die after
sending stop message.

Change-Id: I07c02bd5585538b51e05e9d59595f5b684d8cc3c
Refactor native btree updater wrapper to make it reusable

Refactored native btree updater helper code to be
reusable by incremental updater as well as view compactor
delta log apply methods.

During a test run (on Mac/SSD) with 7M documents and 3 simple views,
the taken taken before and after integration of native sequential sorting
and btree updater is as follows:

Before:
Indexing time: 657.884 seconds
Sorting + Btree updates: 603.450 seconds

After:
Indexing time: 386.529 seconds
Sorting + Btree updates: 346.332 seconds

Change-Id: I23172de5fd5a09b8a9f94f37476599f13f41a71e
MB-7629 Use native updater for applying compactor delta logs

Replaced erlang btree updater with native btree updater to
apply delta log files to the compacted btrees.

Change-Id: Ib2e09a450cfe0924484574b5e0680f0d6725912c
MB-7629 Prevent sorting compactor delta log files

Send a flag to the native btree updater to
turn off sorting compactor delta log files.

Change-Id: I64293a045e73dba9b80c87bfb7321c65e00791af
MB-7629 Fix view compactor incremental progress stats

The native view compactor can now send intermediate
progress of how much kvs are written at every percentage
progress intervals.

Change-Id: I4c9ddbfee0b0531733552d54f2eb92a5fe44cf6b
Add support for stopping native index builder

Similar to the native compacter and native btree updater,
added ability to send a stop message to the index builder
and wait for native process to exit.

Change-Id: Ibda0ef728100266508407f6512f6f12dac2080d3
Remove sorting step for mapreduce initial index builder

Native btree index builder takes care of sorting before
the btree build. Hence sorting step is not required from
erlang side for mapreduce views.

Change-Id: Iafee54d0d65ca4c5cd90389d91e258792776989e