It will presumably generate an intermediate data representation which can be more easily fed to the database. That is, deeply nested trees of messages are flattened. If anyone is seriously interested in taking this project and running with it, I'll see about getting permission to release that code. The module which stores the output of the parser on disk in some quickly- retrievable format. And more to the point, the text searches have to take advantage of the tagging of the data, so that, for example, constraining a search to be in the subject and not the body actually makes the search go faster instead of slower.
Incremental updates are probably pretty important.
I doubt we could get away with a setup that required a nightly update. It seems clear that RDF would be the way go go here. All of the web search engines force the user to type in boolean expressions. Of course, that doesn't handle deeper boolean expressions, or textual searches. Maybe it's a dumb idea.