Added no-op 'couch_viewgen' tool

This will become the view-file generator but does nothing yet.

Change-Id: I98032a7477087abf92168b6c6965cfa19c2323b8
Reviewed-on: http://review.couchbase.org/18712
Reviewed-by: Jens Alfke <jens@couchbase.com>
Tested-by: Jens Alfke <jens@couchbase.com>
Factored merge-sort code out of db_compact.c

Created new 'class' TreeWriter. Currently the only client is
db_compact.c, but the view generator will use it too.

Change-Id: I2893bd7419f3104f256fceea90f0ff5804ac7b57
Reviewed-on: http://review.couchbase.org/18713
Reviewed-by: Jens Alfke <jens@couchbase.com>
Tested-by: Jens Alfke <jens@couchbase.com>
Add JSON collator function

Change-Id: Ib32f9dfd3a11327c8f03f7e94d3fe341fb0ec353
Reviewed-on: http://review.couchbase.org/18714
Reviewed-by: Jens Alfke <jens@couchbase.com>
Tested-by: Jens Alfke <jens@couchbase.com>
TreeWriter takes custom key-comparison fn

Change-Id: If0f51557e6bb03fd23aed2add3e4a1101ec56822
Reviewed-on: http://review.couchbase.org/18715
Reviewed-by: Jens Alfke <jens@couchbase.com>
Tested-by: Jens Alfke <jens@couchbase.com>
Filled out viewgen.c to actually call the indexer.

Also changed the names of the tools in the Xcode project
to match the ones from the makefile,
e.g. dbdump --> couch_dbdump.

Change-Id: I80f7e0ef46ac7637e689ef0faf39c5d40e113303
Reviewed-on: http://review.couchbase.org/18716
Reviewed-by: Jens Alfke <jens@couchbase.com>
Tested-by: Jens Alfke <jens@couchbase.com>
Implemented reduce functions for view indexer.

Change-Id: Idc640d577bb3637453d216681018e0e94f96ad21
Reviewed-on: http://review.couchbase.org/18717
Reviewed-by: Jens Alfke <jens@couchbase.com>
Tested-by: Jens Alfke <jens@couchbase.com>
Update to new format of primary index keys

Change-Id: I346a96c25ef51d09539a4ebdbc84c1a1f5683ad9
Reviewed-on: http://review.couchbase.org/19122
Tested-by: Aaron Miller <apage43@ninjawhale.com>
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Breaking out tree_file 'superclass' of Db

This will allow us to use the B-tree functions to operate on files
that aren't database files, such as the view-index files.

Change-Id: I04e931813d72768508895bd955ed839f5c0bf748
Reviewed-on: http://review.couchbase.org/19123
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
Use big-endian ints in temporary key/value files

Now that this file format is being used by Couchbase Server to
prepare index data, not just ephemerally by the compactor, it seems
safer to use a standard cross-platform format for integers in it.

Change-Id: Ideafd94517ac4ee9121888c926434601d6ce78b8
Reviewed-on: http://review.couchbase.org/19164
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
MB-6118: Indexer now generates special index files

* Refactored a pseudo "base class" tree_file out of Db; basically
  everything but the roots.
* Changed the B-tree code, including TreeWriter, to operate on
  tree_files instead of Dbs.
* Added new API to create/write/close index files and put it in a
  new public header 'couch_index.h'.
* The index files have a new header format that contains a variable
  number of tree roots.

Change-Id: If79421eb0bfb56ab84f5c3d92ed75a6c9a5fd6b3
Reviewed-on: http://review.couchbase.org/19165
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
MB-6114: Added test for CouchStoreIndex

Change-Id: I84bad585ec5b840c05b5c00be1dda5c26399284c
Reviewed-on: http://review.couchbase.org/19274
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
MB-6113: Add standard JSON reduce functions to indexer

It can do _count, _sum and _stats. But only one at a time.

Change-Id: Ia5e05b9cb93f1c0b27463ae5f79ac27867d9dfd9
Reviewed-on: http://review.couchbase.org/19275
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
MB-6145: Add back-index support to indexer

The viewgen tool now has a "--back" flag that indicates the next
input file contains the KV pairs for a back-index.

Change-Id: I6cf09e674730bb3b5888c3b733c826d356d8a922
Reviewed-on: http://review.couchbase.org/19331
Tested-by: Aaron Miller <apage43@ninjawhale.com>
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
MB-6186: Fix bug in JSON collator

It could fall off the end of the input string while parsing a numeric key.

Change-Id: I17afa26456908998b1fb124afcd0f15e760e8faf
Reviewed-on: http://review.couchbase.org/19477
Reviewed-by: Filipe David Borba Manana <fdmanana@gmail.com>
Tested-by: Jens Alfke <jens@couchbase.com>
MB-6293: Better bit and packed-struct twiddling.

(Revised to obey C strict aliasing rules!)

Replaced the bit-access functions in bitfield.h with a series of
structs representing big-endian integers of sizes 1-6 bytes, and
a set of functions to read and write them.

This then allows us to define actual structs for the database data
structures; these are in node_types.h.

Then use those structs when reading and writing structured data,
which makes the code a lot clearer since it's accessing named fields
instead of magic offset/size constants.

Change-Id: I4addff9d215f6f6cfe67f86442dcf70c85a47d60
Reviewed-on: http://review.couchbase.org/19815
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
MB-6703: Incorrect warning in fatbuf_get if size=0

It's just a write to stderr and only in debug builds, but it's
showing up during some unit tests I'm writing so I want to quash it.

Change-Id: I0702dc5cc28622068896d03a1ac7ab3240b5d584
Reviewed-on: http://review.couchbase.org/21001
Tested-by: Aaron Miller <apage43@ninjawhale.com>
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
MB-101: Pedantic-warning fixes

Change-Id: I8f999888d4748fcdb5a8889f9b97c67c911d84d4
Reviewed-on: http://review.couchbase.org/21002
Tested-by: Aaron Miller <apage43@ninjawhale.com>
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
MB-6548: Added public couchstore_alloc_docinfo()

Change-Id: Iddee4a253020deca2b084ae476b34c7a8327ed38
Reviewed-on: http://review.couchbase.org/21003
Reviewed-by: Aaron Miller <apage43@ninjawhale.com>
Tested-by: Aaron Miller <apage43@ninjawhale.com>
MB-6928: import ctypes with fallback

Fallback steps:
1. Try to import ctypes from our library
2. If import fails - remove couchbase library from system path and
try again.
3. Exit if it still fails; otherwise return library to system path
and keep working.

Change-Id: I2b556597bb070b1426ca6824af6e9f58bf363953
Reviewed-on: http://review.couchbase.org/21750
Reviewed-by: Bin Cui <bin.cui@gmail.com>
Tested-by: Pavel Paulau <pavel.paulau@gmail.com>
MB-6945 Backport file format refactor from indexer

Pulling in changes from commit c7b1432 on the indexer branch.
Originally reviewed at http://review.couchbase.org/19815

These changes consolidate the file format definition and
make code dealing with the file format cleaner, and also
make it much easier to change the file format.

Change-Id: I4ee37e66f451fb77fcc3f6264a262c9bbb5417e6
Reviewed-on: http://review.couchbase.org/22103
Reviewed-by: Jens Alfke <jens@couchbase.com>
Reviewed-by: Damien Katz <damien@couchbase.com>
Tested-by: Damien Katz <damien@couchbase.com>