CBD-279 Faster check if group is up to date

The function seqs_up_to_date/2 is slower then just
using plain operator erlang:'>'/2. It was in place
since the early days of the indexer to assert that
both list of seq numbers were sorted and each position
referenced the same partition ID. Existing tests will
now detect if the list of seq numbers doesn't match
the expected format.

Change-Id: If75ace2b759e670ea1b344856de59e3f64b9674b
Reviewed-on: http://review.couchbase.org/16366
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Fix timing issues in test/10-updates-cleanup.t

If the compaction finishes very quickly, the monitor's message
reason field will be 'noproc' and not 'normal'.

Change-Id: I482fe60636c4f61b99c1fc704a2d55406ec94fb0
Reviewed-on: http://review.couchbase.org/16462
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-279 Remove excessive and expensive logging

These log messages are not needed anymore. They were added
before DP4 to help development. All the logging information
regarding current group state is now only logged by the main
view group process when state changes are requested and applied
successfully.

This very verbose logging was expensive, slightly dropping
overall performance as verified with the evperf tests.

Change-Id: I0a54d6a9918779ef3e92f8d3ddc563337292d4e1
Reviewed-on: http://review.couchbase.org/16397
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: buildbot <build@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Set set view tests number of worker threads to 2

So that the build bot doesn't get too overloaded causing
many test failures due to timeouts.

Change-Id: I9e0ddb5279bfe4a07dbbab012ddfaadaf1d27d5c
Reviewed-on: http://review.couchbase.org/16476
Tested-by: buildbot <build@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Farshid Ghods <farshid@couchbase.com>
Tested-by: Farshid Ghods <farshid@couchbase.com>
CBD-294 Simplify/cleanup httpd layer for performance

Removed all the features not used in Couchbase Server
(vhosts, url rewriting, access log, etc).
These were increasing, very significantly, query response
times/latencies.

Tests repeated on Linux and Mac OS X, 2 and 4 nodes clusters,
with evperf and the erlang script view_query_perf.

Some sample results with view_query_perf below.

$ ./run --queries 25000 --workers 1 --query-url 'http://localhost:9500/default/_design/test2/_view/view1?limit=10' --output-times before
Spawning 1 workers, each will perform 25000 view queries
View query URL is:  http://localhost:9500/default/_design/test2/_view/view1?limit=10

Waiting for workers to finish...

All workers finished. Final statistics are:

    Average response time:   6.4329149200000275ms
    Highest response time:   168.963ms
    Lowest response time:    5.382ms
    Response time std dev:   1.4596464282289152ms
    # of errors:             0

Saving query response times to file before

$ ./run --queries 25000 --workers 1 --query-url 'http://localhost:9500/default/_design/test2/_view/view1?limit=10' --output-times after
Spawning 1 workers, each will perform 25000 view queries
View query URL is:  http://localhost:9500/default/_design/test2/_view/view1?limit=10

Waiting for workers to finish...

All workers finished. Final statistics are:

    Average response time:   5.704996960000008ms
    Highest response time:   161.67ms
    Lowest response time:    4.836ms
    Response time std dev:   1.463934100590158ms
    # of errors:             0

Saving query response times to file after

$ cat before | perl -MStatistics::Histogram -e '@data = <>; chomp @data; print get_histogram(\@data);'
Count: 25000
Range:  5.382 - 168.963; Mean:  6.433; Median:  6.310; Stddev:  1.460
Percentiles:  90th:  6.987; 95th:  7.342; 99th:  8.240
   5.382 -    7.861: 24472 #####################################################
   7.861 -   11.304:   486 #
  11.304 -   16.083:    28 |
  16.083 -   22.720:     3 |
  22.720 -   31.935:     7 |
  31.935 -   44.729:     1 |
  44.729 -   87.161:     1 |
  87.161 -  121.409:     1 |
 121.409 -  168.963:     1 |

$ cat after | perl -MStatistics::Histogram -e '@data = <>; chomp @data; print get_histogram(\@data);'
Count: 25000
Range:  4.836 - 161.670; Mean:  5.705; Median:  5.602; Stddev:  1.464
Percentiles:  90th:  6.191; 95th:  6.499; 99th:  7.333
   4.836 -    7.140: 24633 #####################################################
   7.140 -   10.354:   336 #
  10.354 -   14.837:    24 |
  14.837 -   29.811:     3 |
  29.811 -   41.976:     1 |
  41.976 -   58.945:     1 |
  58.945 -  161.670:     2 |

Change-Id: I7c029c334df5b2322837e1b213efabe4818b0361
Reviewed-on: http://review.couchbase.org/16460
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-294 Remove unnecessary view merge parameters

The rereduce function and its language don't need to
be configurable via the HTTP/REST API. The rereduce
function is always extracted from the design document
and its language is always JavaScrit - this is the use
case in Couchbase Server 2.0.

Change-Id: I32e85cce2dd6677d5d9b624c8677c72f180df57d
Reviewed-on: http://review.couchbase.org/16461
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-54 Ability to pause/unpause partition indexing

These new changes allow a caller to temporarily prevent
the indexer from processing data of a subset of the current
active and passive partitions.
Queries will still be able to access existing data that
originated from those partitions.

Change-Id: Ibdfd4a30ce3cdc12228f345b19a29fddfe5fd958
Reviewed-on: http://review.couchbase.org/15586
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-302 Automatic partition cleanup on database deletion

Instead of relying on an external component (ns_server, mccouch)
to tell indexes to cleanup partitions when their databases are
about to be deleted, let the indexers be notified of the database
deletes via the couchdb core database and automatically mark the
corresponding partitions for cleanup in the indexes.

Change-Id: Iefec826033abb2847b04aed6b98cf6c8baf5a65b
Reviewed-on: http://review.couchbase.org/16510
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Remove unnecessary code

When the updater's process that load changes starts, the
pending cleanup partition list can never be non-empty, so
this code was useless.

Change-Id: Ie81ec907f9abedb17f19bec9d04e69464d56604d
Reviewed-on: http://review.couchbase.org/16527
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-305 Ability to monitor index partition updates

It's now possible for a process to get notified when the
index has indexed all the content of a partition (vbucket).
This is needed for a better integration with ns_server.

Currently only active and passive partitions can be monitored.
There's no support for replica partition monitoring.

Change-Id: I0f560c43ceb5c530749d1dfabf827475535c603b
Reviewed-on: http://review.couchbase.org/16563
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-313 Log information about view compactor retries

Change-Id: Ia4574aa21b5033eb71ddaad3a64722b6575b57df
Reviewed-on: http://review.couchbase.org/16615
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-314 Skip barrier for updates started by view compactor

If the updater is spawned by the compactor, to apply a delta, it
waits to enter the barrier. This is not needed because the delta
is small and by not doing it, we're not blocking ns_server's
compaction daemon, allowing it to start and monitor other compactions
sooner.

Change-Id: I80a0d6f33f6d78b008b8bb30a85b0d923c5b5b71
Reviewed-on: http://review.couchbase.org/16619
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Revert "MB-100 Remove unnecessary code"

This reverts commit 230469d00fe4740bd21f2c25fc4588aaae1c2412.

The change was not correct because it would cause updaters
spawned by the compactor to crash (if the pending cleanup
partitions are deleted but not yet recreated) or to temporarily
index data from the recreated partitions (marked for pending
cleanup) just to later be deleted by an updater spawned by the
view group or by a cleanup process spawned by the view group.

Change-Id: Ifffdba338f4875c6e77f0b40928de618f13defb3
Reviewed-on: http://review.couchbase.org/16620
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-317 Fix detection of compact group staleness

Change-Id: I44be039fa2fff80a7e4b0206f5dcf3030b1245ff
Reviewed-on: http://review.couchbase.org/16641
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Make set view tests run faster

Without sacrificing reliability.

Change-Id: I9d51e29d892409f559296a7deaf8f19af97aaf60
Reviewed-on: http://review.couchbase.org/16666
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Set number of test jobs to 2

It's actually faster than 3.

Change-Id: I27a1f523f63af0bf2d1f3c641e28bb032f78ed3a
Reviewed-on: http://review.couchbase.org/16667
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Fix ocassional test failure

The child process sometimes finishes too fast, before
we get a chance to monitor it. In that case the monitor's
message has a reason 'noproc'.

Change-Id: I9e802c441505ce572ffa12aee2417329ab42eb60
Reviewed-on: http://review.couchbase.org/16690
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Cleanup exit message in mailbox (if any)

Change-Id: Ib3c9309165ab6bf4705127d58977ee272625c3fa
Reviewed-on: http://review.couchbase.org/16684
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-324 Simplify set view group partition state management

This change obsoletes MB-5270, CBD-320 and CBD-321. This
simpler approach removes the problems identified by those
3 issues.

Change-Id: I7778fd03b1f065699bfd5dce680df07bb25432e3
Reviewed-on: http://review.couchbase.org/16689
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-324 Add missing file flush call when cleanup finishes

Change-Id: I3571a6c9aef2eb211ea21c3969bc26352632d6b6
Reviewed-on: http://review.couchbase.org/16704
Tested-by: buildbot <build@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>