factorie/factorie — pull request #37 by alextp (2013-08-19 18:35:50+00:00)
Title: Phrase tokenizer

Two changes: adds a tokenizer that quickly chunks tokens into a predefined list of phrases and code which can build a list of phrases from bigram statistics.