[IRIS-2011] find messages in INBOX

when searching all messages using criteria which don't hit the search
engine, like 'DRAFT'.  The code that generated the messages was using a
prefix like "user.foo.*" instead of "user.foo" and filtering based on
the length.  No excuse - I've fixed this bug in other code so many times.
[IRIS-2006] Escape more Sphinx metachars
cunit: run tests in human-friendly mode by default

and generate XML only when run with the magic environment variable that
tools/jenkins-build.sh uses.
cunit: update documentation.
Don't close fd 0 in XNSIPPETS

The indexing lock changes broke this; the data structure is shared
between the indexing code and the XSNIPPETS code but was only being
initialised properly in the indexing code.
Add ptrarray_tail()
sphinx: reimplement how queries are generated

to be more forgiving of small issues like boolean nodes with no
children.  This was easier to fix in the sphinx integration code than to
avoid doing in build_query().
sphinx: add a unit test for query building

the new get_internalised() method lets us peek into the guts of the
Sphinx extended mode query that is generated by the builder, so we can
actually test it.
sphinx: don't escape queries in verbose logs

it just makes it harder to read in the syslog.
Turn down the verbosity for XSNIPPETS
Fix error message in squatter.c

"error opening looking up mailbox... " what!?!
sphinx: log indexed keywords at verbose > 3

using the Sphinx CALL KEYWORDS syntax to report how the searchd broke
the search text into keywords for indexing purposes.  Useful for testing
the behaviour of Sphinx configuration items like charset_table,
ngram_chars, and ngram_len.
[IRIS-2007] index & search Chinese/Japanese/Korean

Note that this requires re-indexing any messages which contained such
characters, as they were previously completely ignored by the indexing
code.

The only other non-latin characters supported are Russian, but those
aren't stemmed yet.  So we really only support English, Chinese,
Japanese, and Korean.

Of course all of this is subject to verification by testers who actually
speak those languages.  I just copied some text from a Wiki.
[IRIS-2012] Actually fix the case Marian found

This fixes commit

1855529 sphinx: reimplement how queries are generated

to do what it said on the box.
sphinx: the colon after IOERROR in syslogs matters

because monitorcyrus looks for m/IOERROR:/ and won't notify us of those
errors otherwise.
lib: add nonblocking+shared mode to lock API

and improve the API somewhat while maintaining backwards
compatibility macros.  The new code is more flexible and
smaller.
Replace mailbox_open_iwlnb with _irlnb

This was causing problems in production.  The rolling squatter could not
open a mailbox index for indexing because an imapd had the mailbox
selected and thus held the mboxname lock in shared mode.  Squatter would
retry many times with the same result and eventually give up and evict
the mailbox from its queue.  It turns out that squatter was opening the
mailbox in nonblocking but exclusive mode, which would of course always
fail.  The solution is to open the mailbox in nonblocking but shared
mode.  As squatter was the only code to call the mailbox_open_iwlnb()
API we just replace it with mailbox_open_irlnb().
squatter: remove initial 32ms delay when rolling

The purpose of this delay was to give the lmtpd or imapd the chance to
drop the mboxname lock before we took it to index the mailbox.  This
turned out not to be a useful tactic because imapds can keep the
mboxname lock held for longer than squatter is willing to keep retrying.
As we are now taking the lock in shared mode for indexing, there's no
need for the initial 32ms delay before the first try.
squatter: -e option accepts UTF-8
Improve readability in mboxname_lock

Review comments from BronG.