test: add utility function to sort a json list

So far we only need this one place, but it's a bit messy to inline
test: add notmuch_search_files_sanitize and use it

We do this in a lot of places, so make it a helper in the test-lib.
test: rearrange the test corpus into subfolders, fix tests

We will need this for improved folder search tests, but having some
folders should exercise our code paths better anyway.

Modify the relevant test accordingly to make it pass.

This reorganization triggers a bug in the test suite, namely that it
expects the output of --output=files to be in a certain order. So we
add the fix for that into the same commit.

This mainly involves sorting, although the case --duplicate=$n
requires more subtlety.
lib: refactor folder term update after filename removal

Abstract some blocks of code for reuse. No functional changes.
lib: add support for path: prefix searches

The path: prefix is a literal boolean prefix matching the paths,
relative from the maildir root, of the message files.

path:foo matches all message files in foo (but not in foo/new or
foo/cur).

path:foo/new matches all message files in foo/new.

path:"" matches all message files in the top level maildir.

path:foo/** matches all message files in foo and recursively in all
subdirectories of foo.

path:** matches all message files recursively, i.e. all messages.
test: make insert test use the path: prefix

This is a more strict test for the insert test.
lib: make folder: prefix literal

In xapian terms, convert folder: prefix from probabilistic to boolean
prefix, matching the paths, relative from the maildir root, of the
message files, ignoring the maildir new and cur leaf directories.

folder:foo matches all message files in foo, foo/new, and foo/cur.

folder:foo/new does *not* match message files in foo/new.

folder:"" matches all message files in the top level maildir and its
new and cur subdirectories.

This change constitutes a database change: bump the database version
and add database upgrade support for folder: terms. The upgrade also
adds path: terms.

Finally, fix the folder search test for literal folder: search, as
some of the folder: matching capabilities are lost in the
probabilistic to boolean prefix change.
test: add tests for the new boolean folder: and path: prefixes

Additional tests for the boolean folder: and path: prefixes using the
full corpus.
test: add database upgrade test from format version 1 to 2

Test the upgrade from probabilistic to boolean folder: terms, and
addition of path: terms.

The test depends on the pre-built test corpus and database tarball and
checksum file being in place. If it's not, the test is skipped. The
mechanism to fetch the test database will be added later.

At the time of writing, a working test database and checksum file is
available at

   http://notmuchmail.org/releases/test-databases/

It has been noted that some non-GNU environments make lack
sha256sum. We leave this portability issue for a followup patch.
devel: add script to generate test databases

Add script to generate notmuch test databases using specified versions
of notmuch. This is useful for generating material for database
upgrade tests.

This reuses the test infrastructure to have a sandbox environment for
notmuch new etc.
test: commit database checksum, ignore actual databases

The checksum file is used by the test infrastructure to verify the downloaded
test database is the one we had in mind.  Note that this test is
rather strict, and the the checksum file needs to be recommitted when
the database is regenerated.

add a pattern .gitignore to ignore the actual databases
test: add machinery to download and verify databases

Note that it is intentional that the checksum file is not
downloaded. The intent is to check those into git.
man: update man pages for folder: and path: search terms

Text from review by Austin Clements <amdragon@MIT.EDU>.
man: try to clarify the folder: and path: vs. --output=files confusion
perf-test: use command line arguments for directories

It seems that between version 1.26 and 1.27 of gnu tar, directories to
be extracted read with --files-from are no longer recursively extacted.
This patch puts them on the command line instead.
util: Fix two corner-cases in boolean term quoting function

Previously, make_boolean_term did not quote empty boolean terms or
boolean terms that started with '('.  These cases are incompatible
with Xapian: empty terms cannot be omitted, and boolean terms that
start with '(' trigger an alternate term quoting syntax.

Fix this by quoting empty terms and terms that contain '('.
doc: build man pages at build time; introduce HAVE_SPHINX, HAVE_RST2MAN

This helps avoid build artifacts (namely, nroff and gzipped-nroff man
pages) owned by root.

The variables allow choosing which generator to use for the man page.
These will be hooked to configure in a following commit.
doc: build man pages into hierarchy, fix help test.

It turns out there was a reason the old man pages were stored in a man
compatible hierarchy, namely so that we could run man on them before
installing.

Hardcode doc build location into test suite.  This isn't ideal, but
let's unbreak the test suite for now.
doc: configure detection of sphinx and rst2man

Because sphinx-build does not provide a convenient way of listing
which builders exist, and some people actually have pre 1.0 sphinx, we
try loading a relevant python module.

Currently the assumption is that no python in path -> no sphinx-build
in path.
doc: cosmetic fix for prerst2man.py

Fix a particular egregious combination of format and string
concatenation.