Better path handling in Python tests

This fixed issues where these would not run correctly on Windows.

Change-Id: I931f7009305262efa4b66d573de4ecffe93b9e5d
Reviewed-on: http://review.couchbase.org/32694
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sarath Lakshman <sarath@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Mute warnings that we should use _strdup

Otherwise the compiler emits tons of warnings that we
should be using _strdup instead of strdup.

Change-Id: I09c83f3e9c91293fabc46f5fc45ecd9d3914745f
Reviewed-on: http://review.couchbase.org/32723
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Use remove() instead of the unlink()

Change-Id: I5ccf22cc48a1477cf40450fc22cbd868d227481b
Reviewed-on: http://review.couchbase.org/32724
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
MB-7629 Add option to skip sorting in native btree updater

For compactor delta logs update, we do not want to
sort the log files since they are already sorted.

Change-Id: I298db564db2ebba677f0aa58924d97461348a0d5
Reviewed-on: http://review.couchbase.org/32315
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
V8 version is set by the topmost script

Change-Id: I86a3796168b7064986a43c6db721477bd0fc6ad3
Reviewed-on: http://review.couchbase.org/32864
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
MB-9999 Do not free btree roots if they haven't changed

When btree cleanup or update is performed, if there is no modification
required in the btree (Eg. if same kv pairs are added again) the root
remains the same. In such cases, it results in double freeing and
may cause segfault.

Added a check to prevent freeing if root node is still the same after
cleanup or update.

Change-Id: If8e5b1665f607559d939d402d88067677a1d965a
Reviewed-on: http://review.couchbase.org/32853
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: buildbot <build@couchbase.com>
MB-9999 Add stdin integer reader method

Using fscanf directly to read newline separated integers
from stdin have few pitfalls. For example, if we are trying
to read a newline separated integer from stdin using
fscanf(stdin, "%d\n", &x), when you send exactly "123\n"
to stdin, still fscanf will block until a non whitespace
character (mismatching isspace()) arrives or EOF is reached.

Since the erlang port does not close the other end of stdin,
it will never receive EOF. Hence, it will indefinitely block
expecting a non whitespace character. This case happens in
the native view group cleaner if there is no reducers for a
view.

Added a separate method for reading a line from stdin and
parse integer using sscanf.

Change-Id: Ie64cadf59b87c580d3251b9f67bb9ddff21b4a9a
Reviewed-on: http://review.couchbase.org/32854
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Add exit message handler for native index builder

Change-Id: If60d7eeef715d72caa89ccfbd3caac0daf28af59
MB-9968 Add support for native view components to exit on pipe close

The erlang side spawns view group native programs cleaner, updater
and compactor and sends messages via stdin. If the parent erlang
process gets killed, these processes needs to die by itself.
A read() call on stdin pipe returns zero if the other end has closed
the pipe. Added a check to identify stdin close and exit the process.

Change-Id: I34e616ee2127d8a271f4364884fb8337f91a03fa
MB-7629 Implement view compactor incremental progress stats

When view compactor makes progress, a stats callback
is used to update the current progress of inserted kvs.
For every percentage progress, the compactor program sends
inserted count stats through stdout.

Change-Id: I550a8ed077e6dda78753f4a944d0bff36a624540
Replace few string allocs and copy with strdup

Change-Id: I33cba00877556171693989e07af23ace7c0eba90
Add kvs sorting step in initial index builder

Change-Id: I7e79d7cd13fb53cb17db4f5fad83f6c1d5963aa0
Add feed callback and skip_writeback option in file sorter

For the file sorter, we write out intermediate sorted files
to disk and perform a k-way merge to produce final sorted
records file. The final records file is consumed by index
builder to build btree indexes. But, ideally we could skip
writing the final result of k-way merge to disk if we could
pipeline the results coming out of k-way merge directly into
index builder. A callback mechanism has been added to the file
sorter/merger to call a feed_record method during the final
phase of file merger. Also added a flag to skip writing final
sorted results to a file.

The consumer of file sorter can save a lot of disk IO
by avoiding unnecessary write-read of final sort results.

Change-Id: If9b15092bdc7242772981859344f4f88dec45998
Add tests for file sorter callback and skip_writeback option

Change-Id: I87246cf28fdb3b524d219d7f92c82ffb425c4326
Use file sorter callback to build initial index

Initial btree index was built by reading the records
from a sorted records file. File sorter now supports
a callback which gets invoked as soon as final sorted
records are available from final sort-merge phase.

Initial index builder is refactored to build the btree
by avoiding final write phase of sorter and it builds
up through the sorted result stream.

Change-Id: I67f46ee71f5c9afed6ca321c1da8dd265c82ce42
Refactor view record sorter to use system quicksort

The cpu timing benchmarks shows that list mergesort currently
used by the view record sorter is expensive compared to the
system quicksort. Refactored sorter to avoid building linked list
and use pointer array holders to make use of system qsort.

The sort time benchmark shows that there is a 2x improvement in
sorting time by using qsort instead of list mergesort.

Benchmark using an initial index build on tmpfs with 10M records and
3 views is as follows:

Before:
real   7m13.660s
user   7m13.344s
sys    0m5.796s

After:
real   5m35.510s
user   5m35.832s
sys    0m5.040s

Change-Id: I346b7e980135d895ed2e45677ba776b9d487428c
Add parallelism for file sorter chunk sorting

The file sorter reads 64M of records, sort it and
writes to disk upto 16 files and uses this intemediate
sort files for merge phase. This in-memory sort is
cpu bound (45 % time is spend in comparator).

Added ability to run two sorts jobs at a time
in parallel. The benchmark results of running an
initial index build with 10M documents and 3 views are
as follows (Mac/SSD):

Before:
real 8m14.328s
user 8m1.810s
sys  0m10.472s

After:
real 5m35.886s
user 8m37.032s
sys  0m11.566s

Change-Id: I13457ecb482c717533e92c99776a180854c98dc9
Optimize sorter view record overhead memory footprint

The view record structure used by sorter has an overhead
of 40 bytes per record which accounts for large part of
memory allocated for sorting. Optimized the structure to
have only 7 bytes per record overhead.

The heap profiling results show that for 64MB file of records,
memory footprint has reduced from 140 MB to 86 MB.

Change-Id: I479699c93c3bc221c8d051bf9af840e76045e3fc