mozilla/fjord — pull request #308 by willkg (2014-06-16 17:32:03+00:00)
Title: [bug 1026019] Rework bigram generation

This reworks the tokenizing part of bigram generation so that it's no
longer (ab)using Elasticsearch to do the work. Now it does its own
tokenizing.

I also added some really rough tests to make sure the new tokenizing
wasn't really horrible. The tests run roughly the same before and after
the code changes. The new code handles the "add-on" case better--the old
code broke that into "add" and "on" and then ditched "on".

This radically improves reindexing. Cuts reindexing times in half or better.

r?