factorie/factorie — pull request #89 by harshal-pandya (2013-10-27 19:07:11+00:00)
Title: doc annotator that can concatenate hyphenated tokens if they represent a single word

This doc annotator works on the output of the tokenizer and concatenates tokens if based on a user-provided dictionary or based on other tokens from the same document.
It modifies the tokens so any other attrs on the tokens may be lost if tokens are concatenated.