MB-7413 More efficient and faster incremental updates

At any time, insert larger batches of kv pairs into the
btrees. These batches are on disk sorted files, and while
they're applied to the btrees, the next batch is prepared
in parallel (maps, conversion to binaries, sorting, etc).
This reduces index update time, slows down fragmentation
and requires less disk space.

This has a very significant positive impact if the number
of changes to apply to an index are large, such as the case
when adding a group of new partitions (active, passive of
replica) to an index.

Small/medium scale performance test descriptions follows.

Test 1:

Initially 3.2M documents, design document with 4 views (both
from evperf lucky8 tests), index was built and later more
3.2M documents were added (new 512 partitions added to the
index as active).

Initial build took:                        11m17.911s
Index file size after initial build:       380Mb

** Before this change

Incremental update took:                   75m43.575s
Index file size after incremental update:  30.0Gb

** After this change

Incremental update took:                   18m36.762s
Index file size after incremental update:  4.0Gb

Test 2:

Initially 3.2M documents (from lucky8 evperf test), design
document with a single view (emits meta.id key and null value
for each document), index was built and later more 3.2M documents
were added (new 512 partitions added to the index as active).

Initial build took:                        6m23.199s
Index file size after initial build:       134Mb

** Before this change

Incremental update took:                   21m31.615s
Index file size after incremental update:  8.6Gb

** After this change

Incremental update took:                   7m13.394s
Index file size after incremental update:  1.8Gb

Change-Id: I66fde0adeb0f7a810c125bf7cb42cdd17dd03d9d
Reviewed-on: http://review.couchbase.org/23674
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
MB-7522 Optimize index state transitions (rebalance)

Optimize 2 types of common state transitions:

1) If a state transition consists only of moving partitions
between the passive and active states (or vice-versa), that
is, no new partitions are added nor existing partitions are
marked for cleanup, don't restart the updater (at the expense
of more complex logic to correct snapshots received from a
currently running updater).

2) If a state transition only adds new partitions to the
passive state (as ns_server's rebalance frequently calls),
don't restart the updater, and send the updater the new
list of passive partitions, so that it has a chance to
iterate over those partitions' content and index them in
the same run.

As long as the rebalancer in ns_server doesn't keep changing
its logic regarding state transitions all the time, this
provides a significant reduction of the indexing time. However
for at least some scenarios, there's still significant portion
of rebalance time not spent on indexing nor index compaction.

Change-Id: I2b7960fd73052ce476fbacee6c1aca4d01490d5a
Reviewed-on: http://review.couchbase.org/23871
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Merge remote-tracking branch 'origin/2.0.1'

Change-Id: I934648468334dc14f2f4c07d0ec3b8d08188f2ee