MB-7629 Implement view compactor incremental progress stats

When view compactor makes progress, a stats callback
is used to update the current progress of inserted kvs.
For every percentage progress, the compactor program sends
inserted count stats through stdout.

Change-Id: I550a8ed077e6dda78753f4a944d0bff36a624540
Replace few string allocs and copy with strdup

Change-Id: I33cba00877556171693989e07af23ace7c0eba90
Add kvs sorting step in initial index builder

Change-Id: I7e79d7cd13fb53cb17db4f5fad83f6c1d5963aa0
Add feed callback and skip_writeback option in file sorter

For the file sorter, we write out intermediate sorted files
to disk and perform a k-way merge to produce final sorted
records file. The final records file is consumed by index
builder to build btree indexes. But, ideally we could skip
writing the final result of k-way merge to disk if we could
pipeline the results coming out of k-way merge directly into
index builder. A callback mechanism has been added to the file
sorter/merger to call a feed_record method during the final
phase of file merger. Also added a flag to skip writing final
sorted results to a file.

The consumer of file sorter can save a lot of disk IO
by avoiding unnecessary write-read of final sort results.

Change-Id: If9b15092bdc7242772981859344f4f88dec45998
Add tests for file sorter callback and skip_writeback option

Change-Id: I87246cf28fdb3b524d219d7f92c82ffb425c4326
Use file sorter callback to build initial index

Initial btree index was built by reading the records
from a sorted records file. File sorter now supports
a callback which gets invoked as soon as final sorted
records are available from final sort-merge phase.

Initial index builder is refactored to build the btree
by avoiding final write phase of sorter and it builds
up through the sorted result stream.

Change-Id: I67f46ee71f5c9afed6ca321c1da8dd265c82ce42
Refactor view record sorter to use system quicksort

The cpu timing benchmarks shows that list mergesort currently
used by the view record sorter is expensive compared to the
system quicksort. Refactored sorter to avoid building linked list
and use pointer array holders to make use of system qsort.

The sort time benchmark shows that there is a 2x improvement in
sorting time by using qsort instead of list mergesort.

Benchmark using an initial index build on tmpfs with 10M records and
3 views is as follows:

Before:
real   7m13.660s
user   7m13.344s
sys    0m5.796s

After:
real   5m35.510s
user   5m35.832s
sys    0m5.040s

Change-Id: I346b7e980135d895ed2e45677ba776b9d487428c
Add parallelism for file sorter chunk sorting

The file sorter reads 64M of records, sort it and
writes to disk upto 16 files and uses this intemediate
sort files for merge phase. This in-memory sort is
cpu bound (45 % time is spend in comparator).

Added ability to run two sorts jobs at a time
in parallel. The benchmark results of running an
initial index build with 10M documents and 3 views are
as follows (Mac/SSD):

Before:
real 8m14.328s
user 8m1.810s
sys  0m10.472s

After:
real 5m35.886s
user 8m37.032s
sys  0m11.566s

Change-Id: I13457ecb482c717533e92c99776a180854c98dc9
Optimize sorter view record overhead memory footprint

The view record structure used by sorter has an overhead
of 40 bytes per record which accounts for large part of
memory allocated for sorting. Optimized the structure to
have only 7 bytes per record overhead.

The heap profiling results show that for 64MB file of records,
memory footprint has reduced from 140 MB to 86 MB.

Change-Id: I479699c93c3bc221c8d051bf9af840e76045e3fc
Move native updater sort step to happen just before btree update

Currently record sorting for all btrees are performed one by one
and after that btree updates are performed in sequential order.
The disk buffer cache for sort files might get overwritten by
the time actual btree update happens and hence it has to pay
the cost of reading from disk again. By moving sort step just before
every btree update operation ensures better utilization of disk
buffer cache.

Change-Id: I5b7a720c4888e6b3af8ffc045ba3663fde892216