latin-language-toolkit/llt-tokenizer — pull request #15 by LFDM (2014-01-20 18:59:43+00:00)
Title: Spaces in xml attrs and chevron handling

Adds xml option to Tokenizer, which determines how chevrons are handled. If true every plain chevron is treated as xml tag, chevrons inside of a text need to be escaped.
Instances of Punctuation take the unescaped value as `@string`.

Fixes #13, fixes #10, fixes #6.