MB-6056 Fix distributed skip and limit

When used together, limit can not be propagated as it is without
summing the value of skip to it. We don't know in advance how
many rows are discarded from each node, so in worst case N (where
N is limit value) rows are discarded from a node, effectively
counting as a limit of 0 for that node.
Regression introduced in cf25a4b26c923645eb9acaee1b7b282f71af5c21.

Change-Id: Ia129cc3463d79e2b1e5f04ca8e5ec69320734db9
Reviewed-on: http://review.couchbase.org/19034
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Add a dateToArray() function to map context

This converts from a date object, or from a string like
this: "2012-07-30T23:58:22.193Z" or anything else the V8
date parser can handle.

The output is the familiar time segmented array for using in
keys, something like this:

[2011,9,27,1,46,40]

Usage of the whole thing is like this:

function(doc) {
	if (doc.date) {
		emit(dateToArray(doc.date), null)
	}
}

Change-Id: Ided5d8e0dbfb79424404c47c72b92bb1fc444d73
Reviewed-on: http://review.couchbase.org/19048
Reviewed-by: Chris Anderson <jchris@couchbase.com>
Tested-by: Chris Anderson <jchris@couchbase.com>
MB-100: include ddoc signature into get_group_data_size reply

Which is required for new ns_server per-ddoc stats gathering
implementation.

Change-Id: Id9679d1298f814635af35cfcecfde428e20a743b
Reviewed-on: http://review.couchbase.org/19111
Reviewed-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
More space efficient KeyDocId for view indexes

Move 16bit docid length prefix. It is unnecessary as we already know
how many remaining bytes are left after the emitted json key.

Change-Id: I4486e9c0d29435bd7f1285621058956d933dbc79
Reviewed-on: http://review.couchbase.org/19049
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>
MB-6071 Fix _all_docs collation with ?include_docs=true

Regression introduced by CBD-32, as row tuples have variable
sizes since then.

Change-Id: Id91f5ea3d6d854acbaf54a1545e131acaf3a98da
Reviewed-on: http://review.couchbase.org/19135
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 Don't log debug message so often

The compaction daemon in ns_server calls the old CouchDB index file
cleanup function very often, causing too much useless output in the
log when no files are deleted.

Change-Id: I77b4cac2d24e776a0d2a3e472c2ea794713a856b
Reviewed-on: http://review.couchbase.org/19140
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-426 Reject writes if keys/values/reductions are too large

Rather than accept the accept the writes, which will overflow the
size fields, and then find later hard to debug failures at read
time - which can be very similar to file corruption issues and
therefore difficult to distinguish.

Change-Id: Ia2f54c185ceff1b1ca7078aaabdd979bb2dea52c
Reviewed-on: http://review.couchbase.org/19014
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-426 Update ejson collator

This is the latest version, which is thread safe (ICU collators are not
guaranteed to be thread safe nor reentrant) and has several other
improvements.

This was reviewed previously and merged, but somehow not present anymore
(maybe there was some git/gerrit issue that caused the rollback?).
Fortunately this last and better version was still intact in Apache CouchDB.

This is used for old CouchDB views (development views) and part of this
will be re-used by the new raw JSON collator (CBD-426).

Change-Id: Ibc65dfb6c71ae2af2af6ad64acf0790bc5d72e14
Reviewed-on: http://review.couchbase.org/19016
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-426 Add raw json collator

This allows comparing JSON keys in their raw form, without the
need to decode them into Erlang terms.
Algorithm and original implementation by Jens Alfke (TouchDB).

Change-Id: I2b6f7222cf10c72a63b3ab08589ac52589e4df90
Reviewed-on: http://review.couchbase.org/19121
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-426 Use raw collator for index builds

For the same dataset (2M docs, 3 views), with the raw collator
index build time was 6m9.554s and before was 14m15.181s.
Without the new file format (CBD-426) it took 5m30.656s.

Change-Id: I87f441a35a24ad29ef67b620f597721e09b61eff
Reviewed-on: http://review.couchbase.org/19158
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-426 Use less space for temporary files

4 bytes length prefix for values is not needed because they are
the rest of the binary.

Change-Id: If320ad4337950c5385b10801555c3dc33d216ab2
Reviewed-on: http://review.couchbase.org/19159
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-426 More efficient query path for new index format

Restore the performance/efficiency of the query path to at
least match the one we had prior to introducing an Erlang
termless index file format.

Change-Id: I168cfac36784c209117e07d2ccdbc27ef710bd12
Reviewed-on: http://review.couchbase.org/19212
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-279 Remove unnecessary header/check

No need to make application/json content type required
for view merge requests between nodes. This was inherited
from CouchDB codebase and not needed.

Change-Id: I004d5cb891f1c60d79c834dcb2b60bb1aca7b441
Reviewed-on: http://review.couchbase.org/19251
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100 On shutdown, delete group's ets entry

If the ddoc is deleted, or updated, when the view group
shutdowns just delete its entry from the ets table.
Otherwise a too many ddoc deletes/updates could lead to
a memory leak.

Change-Id: I8a7e8bfd1dd3f15356f8fb951374340dff8b70f8
Reviewed-on: http://review.couchbase.org/19252
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-181 Allow passing user status to set view compactor.

Currently ns_server has two types of compactions: scheduled and
triggered manually. A progress of certain compaction is tracked using
the status announced via couch_task_status. Sometimes it's really
useful to discriminate two aforementioned cases. But currently it's
not possible since announced status is not controlled by
ns_server. This change allows to pass arbitrary initial status to
set view compactor. So that ns_server could pass some initial status
that would allow it to tell manual and scheduled compactions apart.

Change-Id: I3b7569ba73e74b55b31ff6301d67f0b9e3519e3e
Reviewed-on: http://review.couchbase.org/19269
Tested-by: Aliaksey Artamonau <aliaksiej.artamonau@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-423 Add couch_set_view:trigger_update/3

Change-Id: Ic732c72d013529d38d1abb93d3433e1039f6221a
Reviewed-on: http://review.couchbase.org/19295
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
CBD-279 More efficient updater/compactor task creation

Getting a list of design doc IDs with a specific signature is
now much cheaper, as it avoids reading all the design documents
from the master database (and opening the database), decompressing
and JSON decoding the document bodies, and finally compute their
signatures.

Change-Id: I9beaaafaf22fd87f7590742e8af5d7cc68987051
Reviewed-on: http://review.couchbase.org/19299
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-6126 Correctly split database name into components

If the bucket name had forward slashes in it (/), then all the
code that splitted a database name into bucket name (set name)
and vbucket Id (partition Id) didn't work correctly.

Change-Id: I6b266f8d16b30a97c9ddafa2ce1bb70ae5408875
Reviewed-on: http://review.couchbase.org/19301
Reviewed-by: Volker Mische <volker.mische@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100: ignore mark-indexable calls on non-unindexable partitions

Because that's more convenient for stateless capi_set_view_manager
inside ns_server.

Change-Id: I90c7799a7e6dc5a7555fd76895a5dbd72531d0e7
Reviewed-on: http://review.couchbase.org/19350
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
MB-100: don't add already added active/passive/cleanup partitions

So that 'unindexable' check is not falsely triggered.

capi_set_view_manager in ns_server is always passing all vbuckets it
has for all vbucket states. If any of those have already been added
and then marked as 'unindexed', then previous code would incorrectly
fail.

Change-Id: Ic2412c23895402915ae6f6ca42ce719ece523187
Reviewed-on: http://review.couchbase.org/19351
Tested-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>