index: strip dead search code
typos fix - code review
more
index: silence warnings
mailbox: add mailbox_yield_index API
search: use mailbox_yield_index
Add command_popen() and friends.

This makes the whole fork-and-pipe-and-exec dance a little bit simpler.

v2: merged in
fbec395 Actually add the command unit tests
sphinx: separate the two receiver classes

There are effectively two subclasses of search_text_receiver_t used for
the Sphinx integration code.  One is used when updating the index with
new messages, and the other in the XSNIPPETS command when generating
snippets for search matches.  Because both of them talked to searchd via
the MySQL protocol and both of them received message text, they were
implemented using the same struct.  However we want to reimplement the
way updating is done, while leaving the snippets code alone, so as a
first step we separate the two into different structs.
sphinx: actually use indexing_unlock()

but only to automate the close() of the fd, which is the only
way we were unlocking.
sphinxmgr: add GETCONF command

which returns the config path of a sphinx instance without actually
starting the searchd (or, currently, setting up the directory).
sphinx: use the fmindex program for indexing

instead of starting up a searchd and using an SQL INSERT statement,
we now generate a big lump of XML, run fmindex, and give it the XML
on its stdin.

Currently we still need to start a searchd to manipulate the "latest"
table, so we're not quite there yet.
sphinx: store latest uid in a twoskip DB

instead of in the "latest" table in Sphinx itself.  This means we can
now check the indexing status of any folder without starting a searchd,
and we can also index a message without starting a searchd.

v2: merged in
f1e52d9 sphinx: fix memleak
sphinxmgr: be more graceful shutting down searchd

Seen in production: if a searchd is shut down outside of sphinxmgr's
knowledge, then 10 min later when sphinxmgr goes to shut it down the
shutdown fails, and then keeps failing every 1 second thereafter,
spamming the syslog.
sphinx: feed fmindex a file name rather than stdin

This will help with the new feature of fmindex where it delegates the
indexing to searchd if searchd is running, but otherwise will do the
indexing in-process to avoid starting a searchd.
Don't update indexes from XCONVMULTISORT

as rolling indexing should soon be working again.
TEMP HACK: dont unlink failing XML
search: propagate an error from fmindex

all the way out to squatter's main loop, so that we can
localise failures and not spam the syslog forever.
sphinx: call {end,begin}_mailbox when yielding

One important side effect of the previous indexing batching code was
that the search engine was notified of the boundaries of batches, so
that it can actually flush whatever it might have pending when a batch
ends.  This effect was lost in the shuffle when this code was refactored
to yield the mailbox lock rather than go back through the mailbox close
and open path again.
charset: be more pedantic about UTF-8 decode

because now we could be going downstream to the libexpat parser being
run from Sphinx, and it's as pedantic as anything and the failure mode
is not pretty.
sphinx: be more pedantic about XML escaping

because now we could be going downstream to the libexpat parser being
run from Sphinx, and it's as pedantic as anything and the failure mode
is not pretty.