New document annotator that can concatenate hyphenated tokens if they represent a single word
now works in place. name changed to dehyphenator
dehyphenator is now dehyphenating tokenizer since it itself tokenizes the document instead of taking as input a pre-tokenized doc
Merge pull request #89 from harshal-pandya/master

doc annotator that can concatenate hyphenated tokens if they represent a single word