Add helper method for gracefully stopping external programs

Change-Id: I781e295596bd908c913184a4074f34b1f31c8836
Reviewed-on: http://review.couchbase.org/32308
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9771 Use port stream option to read stdout response

For native incremental index updater, use stream method to
read lines from stdout and parse view group header.
Earlier {line, N} option was used to receive stdout response.
It has limited control for receiving binary raw data (header data)
since the port driver itself tries parse lines.

Change-Id: I4104d494a7e56031e4e60cb7599d8382c10d0f3a
Reviewed-on: http://review.couchbase.org/31551
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-7629 Integrate native view compactor

Invoke native view compactor program, couch_view_group_compactor
to perform btree compactions.

During a test run (on Mac/SSD) with 7M documents and 3 simple views,
time taken for view compaction before and after native compactor is
as follows:

Before:
86.739063s

After:
49.133651s

Change-Id: I8c7526a5f2c46423e370d0823d34d2311d0ce14b
Reviewed-on: http://review.couchbase.org/32309
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-7629 Fix view compactor stopper to stop native process

Compactor stopper has to send a stop message to the
native compactor process and it has to die gracefully.
It should be waiting for native process to die after
sending stop message.

Change-Id: I07c02bd5585538b51e05e9d59595f5b684d8cc3c
Reviewed-on: http://review.couchbase.org/32310
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Refactor native btree updater wrapper to make it reusable

Refactored native btree updater helper code to be
reusable by incremental updater as well as view compactor
delta log apply methods.

During a test run (on Mac/SSD) with 7M documents and 3 simple views,
the taken taken before and after integration of native sequential sorting
and btree updater is as follows:

Before:
Indexing time: 657.884 seconds
Sorting + Btree updates: 603.450 seconds

After:
Indexing time: 386.529 seconds
Sorting + Btree updates: 346.332 seconds

Change-Id: I23172de5fd5a09b8a9f94f37476599f13f41a71e
Reviewed-on: http://review.couchbase.org/32311
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-7629 Use native updater for applying compactor delta logs

Replaced erlang btree updater with native btree updater to
apply delta log files to the compacted btrees.

Change-Id: Ib2e09a450cfe0924484574b5e0680f0d6725912c
Reviewed-on: http://review.couchbase.org/32312
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9951: Trim rev-id to 16 bytes

With adding the datatype and flexible metadata to the the
metadata field [1], it is no longer fixed to 16 bytes, but
is at least 2 bytes bigger, one for the flex meta code and
one for the datatype field.

Hence trim the revision ID down to a maximum of 16 bytes.

[1] http://review.couchbase.org/31046

Change-Id: I25898dc4e70c1d7d071c228e2bcdd2c03e06ff6a
Reviewed-on: http://review.couchbase.org/32743
Tested-by: abhinav dangeti <abhinav@couchbase.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Junyi Xie <junyi.couchbase@gmail.com>
MB-7629 Prevent sorting compactor delta log files

Send a flag to the native btree updater to
turn off sorting compactor delta log files.

Change-Id: I64293a045e73dba9b80c87bfb7321c65e00791af
Reviewed-on: http://review.couchbase.org/32313
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9968 Use shutdown_sync() for stopping view compactor

During a scenario when stop_compactor() is invoked by couch_view_group process,
it sends `stop` message to the native compactor driver process and block waits
for that process to exit. But if at the same time view compactor driver has
completed the native process and tries to send `compact_done` using
gen_server:call() method to the couch_view_group process, both of them can
be performing blocked call on each other and it becomes a circular dependency
entering into a deadlocks state.

The native compactor can die by itself whenever the erlang port is closed
by detecting the stdin close. Since the compactor does not operate on the
current active index file, when the compactor erlang driver process is
terminated we do not need to ensure that native process has been died.
It can detect the stdin pipe close and exit asynchronously.
Replaced shutdown_wait() method with erlang process force kill method,
shutdown_sync() to stop the running compactor.

This reverts commit d0d76be2cd926e647cc7c7f7d4c07341c9813187.

Change-Id: I34e616ee2127d8a271f4364884fb8337f91a03fa
MB-7629 Fix view compactor incremental progress stats

The native view compactor can now send intermediate
progress of how much kvs are written at every percentage
progress intervals.

Change-Id: I4c9ddbfee0b0531733552d54f2eb92a5fe44cf6b
Add support for stopping native index builder

Similar to the native compacter and native btree updater,
added ability to send a stop message to the index builder
and wait for native process to exit.

Change-Id: Ibda0ef728100266508407f6512f6f12dac2080d3
Remove sorting step for mapreduce initial index builder

Native btree index builder takes care of sorting before
the btree build. Hence sorting step is not required from
erlang side for mapreduce views.

Change-Id: Iafee54d0d65ca4c5cd90389d91e258792776989e