Add assertion to set view updater

Change-Id: I14e0913509cb55642a9315812b90cd65ee515449
Reviewed-on: http://review.couchbase.org/15517
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
MB-5211 Add couch_set_view.hrl to Makefile

Whenever couch_set_view.hrl was modified, if one of the
records which are used by any src/couchdb module is updated,
incremental builds (like the debug builds) wouldn't recompile
those erlang modules, which results in weird runtime badmatch
errors.

The solution here is to ensure those modules (such as the
view merger ones) get recompiled if couch_set_view.hrl is
updated.

Change-Id: I61b7e815d30daa1a00a5706f2c3d1f4c8b3c6c81
Reviewed-on: http://review.couchbase.org/15602
Tested-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-5189 Fix deadlock on view group shutdown

If the cleanup process is stopped and it dies with an error
reason, the view group's terminate callback will attempt to
shutdown the cleanup process, which is not alive anymore and
it will wait forever for an EXIT message.

Change-Id: I47212a5634d71608a1f0a314b1fedc931b5e309f
Reviewed-on: http://review.couchbase.org/15605
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-5234 Make stale parameter value case insensitive

Testrunner sometimes passes ?stale=False, which makes the
query parser throw an exception:

2012-05-04 16:39:59,164 - root - INFO - Params {'connection_timeout': 60000, 'full_set': 'true', 'stale': False}
2012-05-04 16:39:59,164 - root - INFO - index query url: http://192.168.1.75:9000/couchBase/default/_design/dev_test_compare_views_all_nodes/_view/dev_test_compare_views_all_nodes?connection_timeout=60000&full_set=true&stale=False
2012-05-04 16:39:59,199 - root - ERROR - http://192.168.1.75:9000/couchBase/default/_design/dev_test_compare_views_all_nodes/_view/dev_test_compare_views_all_nodes?connection_timeout=60000&full_set=true&stale=False error 400 reason: query_parse_error {"error":"query_parse_error","reason":"stale only available as stale=ok or as stale=update_after"}

Also updated the error message to list stale=false as a
possible stale value.

Change-Id: I92c59d0c18615c7e58b38f74714ff11e4cc7c3ee
Reviewed-on: http://review.couchbase.org/15654
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-5240 run msbuild from cmd.exe

The generated msbuild.bat doesn't have execution permission.

Change-Id: I885bda71888fde0cf1a5f5700cedfd3c6b3da6d8
Reviewed-on: http://review.couchbase.org/15661
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Steve Yen <steve.yen@gmail.com>
Tested-by: Bin Cui <bin.cui@gmail.com>
More logging and unmask errors during db delete

Making any issues during database and file deletion easier to debug.

Change-Id: Ia83f08bad07a9d17e1186347d5a7d7c34bd892da
Reviewed-on: http://review.couchbase.org/15676
Tested-by: Damien Katz <damien@couchbase.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Remove unused function clause

Change-Id: I6d7dd228e3035defd13b14b1533733b6f909e1ca
Reviewed-on: http://review.couchbase.org/15616
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-5241 Add missing replica cleanup operation

Currently if a replica partition transitions to the active
state, a process that transfer the partition's data from the
replica index to main index is started.
If before that process finishes someone asks to transition the
partition to the passive state, the proper actions happen on the
main index, but nothing is done on the replica index, leaving the
partition's data in the replica index. This doesn't seem to be
harmful however, it only consumes unnecessary disk space and IO
(due to automatic replica index updates).

Change-Id: If513a61247ff888da431d34bd9ac5c29cc7b1293
Reviewed-on: http://review.couchbase.org/15665
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-130 Remove unnecessary case expression

Change-Id: I1ba33c29d1da8fdb7f385f3d65c38a40503f89db
Reviewed-on: http://review.couchbase.org/15773
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-130 Better logging on db set exit / partition deleted

Change-Id: I6e7d577193cf176db0933dc1c2a1bf0add401dfd
Reviewed-on: http://review.couchbase.org/15775
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-130 Log information regarding replica group fix on startup

Change-Id: Ia1153350444b1e8e0d353dbdad8f84231e600a6a
Reviewed-on: http://review.couchbase.org/15776
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-137 Fix active/passive bitmasks when compactor finishes

When the compactor finishes, we set his group snapshot's replicas on
transfer field to the current set of replicas marked as on transfer in
the current group, but we don't update the active and passive bitmasks.
This is wrong, because if both groups' replicas on transfer are different,
it means some partitions had a transition from the passive state to the
active state, therefore the passive and active bitmasks must be updated in
the group snapshot received from the compactor.

Change-Id: Icae3afc9636705fc63acfb8d4ea7e53d0815daf7
Reviewed-on: http://review.couchbase.org/15856
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-104 Remove unnecessary header fields

The update_seqs and purge_seqs fields, which are long lists
(1024 entries) with elements that have the form
{PartitionId::integer(), Seq::integer()}, and were part of the
view states, were not really used. We need one of this lists only,
which applies to all views in the view group and already exist as
top level fields of the index header record.

This reduces memory copying of data structures when a view group
is passed between processes (query requests, compactor, updater,
cleaner) and results in smaller headers.

Change-Id: I117b0e4ec225c1fd064aa8ab106eed1fc2eadcf9
Reviewed-on: http://review.couchbase.org/15617
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-106 Lighter group snapshots for non-debug requests

Instead of sending a big fat group snapshot full of debug
information and unncessary fields for querying to any request,
use a much more lightweight version to send to non-debug mode
requests. This reduces term copying when sending the snapshots
to requesters and reduces contention on the main view group
process.

Change-Id: I09a5ac8f676911b61f5e6465ead45b1d1c92809f
Reviewed-on: http://review.couchbase.org/15735
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-104 Commit and fsync header in compactor process

After a group compaction finishes, the header was written and
the new file fsync'ed in the main view group process - this can
take a long period of time, which blocks the main view group
process, making view query requests wait for the write/fsync
to finish in order to get a group snapshot from the main view
group process and start folding a view.

The header is now written and the new file fsync'ed by the
compactor process. The main view group process will only write
a new header and fsync the new file if some important metadata
changed while the compaction was running - this happens for
the non-steady scenario, namely during (and after for a short
period) cluster rebalances and failovers. If this happens, the
main view group process is only writing and fsync'ing one header,
which is cheap - the compactor process alreadt fsync'ed all the
data and one obsolete header.

Change-Id: Ib004ea76b12ec1c3f3c891f13746f67b4f9b7d29
Reviewed-on: http://review.couchbase.org/15913
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-104 Checkpoint indexing progress in the updater

When building/updating an index, do the checkpoints (header writes)
through the updater process, and only send a new group snapshot
when a checkpoint is performed (every 5 seconds). This reduces
contention in the main view group process and avoids the overhead
of copying group snapshot terms after every batch is written to
the index.

Change-Id: I4af0b5c7be076bc41313e1593969bb4b0d5a59d2
Reviewed-on: http://review.couchbase.org/15914
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-130 Shutdown sync cleaner on terminate

On terminate just shutdown synchronously the cleaner.
There's no point in waiting for it to terminate more
gracefully by writing a new path from a leaf to the
root btree node, as in terminate we don't write a new
header after stopping the cleaner.

Change-Id: I8cdbee02fd0fcbc23d2155a1a9cdb772ee660187
Reviewed-on: http://review.couchbase.org/16021
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-130 Don't shutdown on cleanup/compactor error

If the cleanup or compactor processes exit with an
unexpected reason, don't shutdown the view group
process, just log an error message. Restarting the
view group process (after a subsequent request) is a
fairly expensive operation.

Change-Id: I9de136810797b21f38a84277ba527283ab4e72f8
Reviewed-on: http://review.couchbase.org/16022
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-5270 Ignore deletes of databases in pending cleanup

If a database is in the list of pending cleanup partitions,
ignore db deleted events for this database (just like if it
was not pending).

If the updater starts and its input group snapshot has a
pending transition, perform a full cleanup before reading
changes from the partitions to update the index. This is
needed to ensure it doesn't open databases that were silently
recreated and starts indexing data from them without removing
first all previous data originated from those databases.
This full cleanup step is interruptible (just like everywhere
else) and the partition cleanup list consists of the union of
the current cleanup list and the pending transition cleanup list.

Change-Id: Ibc8bc29fdfa3865c0b81a5d8906c230c1a435727
Reviewed-on: http://review.couchbase.org/15929
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Fix argument type in function call

The first argument for ddoc_not_found_msg/2 must be a
binary (database name) and not a #db record.

Change-Id: If45121369a72000eb9c02e88af2f3637ebb007ec
Reviewed-on: http://review.couchbase.org/16183
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>