mongodb/mongo — pull request #729 by jedestep (2014-08-01 21:04:45+00:00)
Title: SERVER-14578 improve performance on $text queries with negated terms

Part of work on SERVER-14296. The current implementation for negated terms in $text queries (for example "foo -bar" to search for foo but not bar) fetches all documents matching positive terms and scans every word of every field for the negated terms. When the number of positive results is large (or when I/O is inexpensive), it is more efficient to do index scans over the negated terms and take a set difference of the results. In my current implementation the extra index scans will kick in if more than 1000 documents with a positive match were returned. Some quick benchmarking indicated that this value generally captured the best of both approaches, at least on hardware on which I/O was extremely slow.

An additional minor change to the $text stage is to create fewer stemmers. The libstemmer documentation recommends creating as fewer stemmers as possible, and the current implementation built one per field per document, even if the fields had a language for which a stemmer had already been built. Caching stemmers and storing them by language should decrease the amount of time taken to build and update the index, as well as to do fetch-based negation checking.