latin-language-toolkit/arethusa — pull request #513 by LFDM (2014-11-26 10:04:28+00:00)
Title: Minor: Terminator status of tokens

Even when we have tokens of multiple sentence in state, we want to know the last word of a sentence. The `TreebankRetriever` formerly checked for punctuation, which is unsafe (punctuation could be in the middle of a sentence - as abbreviation dot - or other sentences might come without any sentence closer whatsoever) - this is more robust now.

When `artificialTokens` are inserted behind this terminator, the program now shifts this `terminator` state around to the right place.