MB-9307: Size of couch_file:read_header_bin/2 is not used

Returning the size of the header is not needed anywhere.

Change-Id: I9f7f97d0a4e00fcb35635332da3be75dd83b3a01
Reviewed-on: http://review.couchbase.org/30859
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9307: Add indexer rollback

Add the functionality to rollback to a certain sequence
number of a certain partition. It rolls back to a previous
header, which means that other partitions might also be
rolled back.

"make check" is epected to fail and is fix with the next
commit.

Change-Id: I62cae905fd3e6e733310057e36a3f66c7200d315
Reviewed-on: http://review.couchbase.org/29732
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Don't replace the full header

After a rollback, don't use the full header information
from the on-disk header, but only the state of the
ID-btree and the views trees.

Also update the sequence numbers of the indexable and
unindexable partitions to match the ones of the on-disk
header it was rolled back to.

Change-Id: Iecf95d38f5111923cb6239adb0b7d824e625b5a3
Reviewed-on: http://review.couchbase.org/30016
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Abort compaction during rollback

Change-Id: I810a69d7625b59b4364e0786f6360bfb3c56c156
Reviewed-on: http://review.couchbase.org/30017
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Use timeout for rollback call

Change-Id: I1f515e66cfcfb49c7b8af539c51433a4d21d8ccb
Reviewed-on: http://review.couchbase.org/30018
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Find header in block size increments

To find a header, don't scan the file byte by byte, but
only on block boundaries where the headers are stored.

Change-Id: Ia6fd86bc0b707db723b86b4d76488ba0d2b78b51
Reviewed-on: http://review.couchbase.org/30019
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Take unindexable partitions into account

Partitions might be set to unindexable, but that doesn't mean
that we can't roll back to them and use them as a starting point
for future updates.

Change-Id: I401e4ea0de6f5c8f95c823cd67f780b159860413
Reviewed-on: http://review.couchbase.org/30020
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Don't rollback if partition didn't exist

Don't roll back if a partition doesn't exist in a
previous header (or has sequence number 0).

Change-Id: Idb99d557aa27339989d3e5c2b9c3a28c158e16c9
Reviewed-on: http://review.couchbase.org/30021
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Don't include new partitions due to rollback

When a rollback is performed, the new header might contain
partitions that are not part of the current index. Mark those
for cleanup and update the header accordingly.

Change-Id: If5ddc95a1ef8866418d0bffcc4eff4773ac227b0
Reviewed-on: http://review.couchbase.org/30023
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Rollback with multiple partitions

Instead of rolling back taking a single partition and sequence
number into account, it's now possible to rollback with supplying
multiple partitions and sequence numbers at the same time. It
is only rolled back if a header satisfies all given partition/
sequence number pairs.

Change-Id: I9492fd894a8b1fcc06d81aef93fe960322cfade6
Reviewed-on: http://review.couchbase.org/30058
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Add test for indexable/undindexable switch

Add a test for the rollback when a partition changes from
unindexable to indexable.

Change-Id: I99414a2d81c8c90da3281e59aa925e1e55856c52
Reviewed-on: http://review.couchbase.org/30860
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Volker Mische <volker.mische@gmail.com>
MB-9307: Better log message

Change-Id: Idc07ed0947b54d1224d8ed36e548d8f787aa4a3b
Reviewed-on: http://review.couchbase.org/30867
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: buildbot <build@couchbase.com>
CBD-485 fix descriptors leak in jump_to_another_version

Change-Id: Idef3c2d31cc12483b27015e0ea7bb6848d39f512
Reviewed-on: http://review.couchbase.org/31123
Tested-by: Artem Stemkovski <artem@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9307: Add typespecs to newly added functions

Change-Id: I7d48215ef6fb760236446f5bfb4006075ba092d9
Reviewed-on: http://review.couchbase.org/30958
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Sarath Lakshman <sarathlakshman@slynux.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Fix post 3.19 v8 support related wrong typedef

Change-Id: I275fcfdf502224725337221a6caf36026dbd701a
Reviewed-on: http://review.couchbase.org/31127
Reviewed-by: Trond Norbye <trond.norbye@gmail.com>
Tested-by: Trond Norbye <trond.norbye@gmail.com>
Reviewed-by: Matt Ingenthron <matt@couchbase.com>
MB-9730 Add support for couch_file to refresh EOF

The couch_file process and its child writer process keeps track of
EOF information. This EOF position is incremented when data is
appended to the file by the writer process. The reader also
makes use of stored EOF position for reading latest header from
the current file process.

When an external process opens the same file and appends data into
the file, the EOF information held by the couch_file process becomes
invalid. Hence, we need to update the EOF information held by the
couch_file process explicitly.

Added couch_file:refresh_eof(Pid) method to refresh the EOF position
by reopening the backing write fd for the file.

Change-Id: I427646e0a0707a2ccaca5af8a354552dc184dfc8
Reviewed-on: http://review.couchbase.org/30989
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-9730 Integrate native viewgroup cleanup support

An external native program `couch_view_group_cleanup`
is invoked to perform viewgroup cleanup. On successful cleanup,
viewgroup couch_file processes are refreshed to use new views
after cleanup.

A comparison of time taken for cleanup before and after this change
benchmarked by running `test/11-updates-cleanup-many-views.t` with
6400000 documents are as follows:

Before:
Removed 30000 values from the index in 4.468 seconds

After:
Removed 30000 values from the index in 2.364 seconds

Change-Id: I1ef010a5c1dad8366e90dbbba4bf93117a92377f
Reviewed-on: http://review.couchbase.org/30787
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
 MB-9730 Add support to stop cleaner native process

stop_cleaner() function needs to ensure that the
viewgroup cleanup native process has exited.
Added a mechanism to send exit message to native viewgroup
cleanup process and wait for process to exit and send exit status.

Change-Id: If187e2114a0608a8f704f6b43a3c36a4817035ab
Reviewed-on: http://review.couchbase.org/30988
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Remove an unused variable in rollback unit test

Remove an unused variable since it is violating the fact
that couch_file process pid can change in between updates since
viewgroup cleaner resets the couch file process for the viewgroup.

Change-Id: I8e3f9020e769c195970ae2984ea4bcaf2d5afee6
Reviewed-on: http://review.couchbase.org/31040
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Merge branch 'master' into upr

Change-Id: I09d6f70c3e3ec40a1b8397ce1558be83e40fccda