latin-language-toolkit/llt-tokenizer — pull request #7 by balmas (2014-01-15 19:39:00+00:00)
Title: fixes #6 handling xml entities

Hi guys - I'm not entirely sure about the treatment of ' (i.e. whether it should be punctuation or word) - this might be something we want to make configurable. I think the rest of the xml entities are punctuation though.