latin-language-toolkit/llt-tokenizer — pull request #27 by LFDM (2014-08-18 10:01:28+00:00)
Title: Consistent splitting

Closes #1

We do want to split things like sive, neve, neu, seu, μηδέ, οὐδέ, μήτε, οὔτε, εἰτε etc.

Quoting Francesco Mambrini's reasoning:

I think the line should be drawn here: if a word consists on two tokens that clearly perform two different syntactical functions (and in particular, if one of these has an important structural role, like COORD -- head of coordination) they must be split all the time (i.e. even when the second conjunction is not used as COORD). For example, οὐδέ must be split consistently in two tokens οὐ + δέ, because we need to annotate both parts, especially (but not only) when δέ is head of coordination. The same goes with κἄγω. Both functions are syntactically active in the sentence and annotators do have very hard time annotating when the word is not split. Compound-words, on the contrary, most often are not formed by two syntactically independent parts that perform a visible role in the structure of the sentence: thus, in "hic" there is no difference in the syntactic behavious of the two parts.
Thus, to take up your examples, I would say that, yes, "sive, seu, nisi" (together with "nec, neque" and other -que words) must be tokenized in two. These goes also for Greek: μηδέ, οὐδέ, μήτε, οὔτε, εἰτε and the krasis (but the list may be incomplete).

---------------------

Mind that `neve` can also be a very valid token on its own, but I guess we will encounter `ne -ve` more often - therefore we split and live with this error.