Repository: cltk/cltk

Issue:
UnigramLemmatizer.choose_tag returns '', not None, for certain words, short-circuiting BackoffGreekLemmatizer
**Describe the bug**

The [backoff Greek lemmatizer](https://docs.cltk.org/en/stable/greek.html#lemmatization-backoff-method) is supposed to use the following chain of sub-lemmatizers ([ref](https://github.com/cltk/cltk/blob/de3f4fb72ed557101538e215b0e8096ad7ccb811/cltk/lemmatize/greek/backoff.py#L56-L61)):
1. `DictLemmatizer`
2. `UnigramLemmatizer`
3. `RegexpLemmatizer`
4. `DictLemmatizer`
5. `IdentityLemmatizer`

The `SequentialBackoffLemmatizer` superclass tries each sub-lemmatizer in turn, moving to the next whenever one returns `None` ([ref](https://github.com/cltk/cltk/blob/98a64abf428643186d9b21de675597ecc456d28d/src/cltk/lemmatize/backoff.py#L104-L105)). But certain words cause `UnigramLemmatizer` to return `''`, not `None`, which causes `SequentialBackoffLemmatizer` to return `''` as the lemma, without trying `RegexpLemmatizer` and the other following sub-lemmatizers. One such word is `'διοτρεφές'`.

```
>>> from cltk.lemmatize.greek.backoff import BackoffGreekLemmatizer
>>> from cltk.corpus.utils.formatter import cltk_normalize
>>> word = cltk_normalize('διοτρεφές')
>>> lemmatizer = BackoffGreekLemmatizer()
>>> lemmatizer.lemmatize([word])
[('διοτρεφές', '')]
```

By walking the chain of sub-lemmatizers manually, we find that it is `UnigramLemmatizer` (`backoff4`) that returns an empty string. If it returned `None`, then backoff would continue and eventually find a lemma with `backoff2`.

```
>>> lemmatizer.backoff5.choose_tag([word], 0, None)
>>> lemmatizer.backoff4.choose_tag([word], 0, None)
''
>>> lemmatizer.backoff3.choose_tag([word], 0, None)
>>> lemmatizer.backoff2.choose_tag([word], 0, None)
'διοτρεφής'
>>> lemmatizer.backoff1.choose_tag([word], 0, None)
'διοτρεφές'
>>> lemmatizer.backoff4
<UnigramLemmatizer: CLTK Sentence Training Data>
```

I cannot find a place in the inheritance chain of `UnigramLemmatizer` (through [`UnigramTagger`](https://github.com/nltk/nltk/blob/3.5/nltk/tag/sequential.py#L326), [`NgramTagger`](https://github.com/nltk/nltk/blob/3.5/nltk/tag/sequential.py#L259), and [`ContextTagger`](https://github.com/nltk/nltk/blob/3.5/nltk/tag/sequential.py#L108)) that explicitly returns an empty string, so I suppose it must be happening somewhere in the model.

**To Reproduce**

Steps to reproduce the behavior:
1. Install Python version 3.7.3.
2. Install CLTK version 0.1.121 with greek_models_cltk at commit [a68b983734d34df16fd49661f11c4ea037ab173a](https://github.com/cltk/grc_models_cltk/tree/a68b983734d34df16fd49661f11c4ea037ab173a).
   ```
   python3 -m venv venv
   source venv/bin/activate
   pip3 install cltk
   python3 -c 'from cltk.corpus.utils.importer import CorpusImporter; CorpusImporter('greek').import_corpus('greek_models_cltk')'
   ```
3. In a script or REPL, run the following code:
   ```
   >>> from cltk.lemmatize.greek.backoff import BackoffGreekLemmatizer
   >>> from cltk.corpus.utils.formatter import cltk_normalize
   >>> word = cltk_normalize('διοτρεφές')
   >>> lemmatizer = BackoffGreekLemmatizer()
   >>> lemmatizer.lemmatize([word])
   ```
4. See error:
   ```
   [('διοτρεφές', '')]
   ```

**Expected behavior**

```
[('διοτρεφές', 'διοτρεφής')]
```

**Desktop (please complete the following information):**

Debian 10.7

**Additional context**

`UnigramLemmatizer` does not return `''` for all strings. For example, giving it a string of non-Greek text falls all the way through to `IdentityLemmatizer`, as expected.

```
>>> lemmatizer.lemmatize(['foo'])
[('foo', 'foo')]
```

This is how we are using the lemmatizer. We are inserting our own `DictLemmatizer` with corrections we have found in our corpus at the beginning of the backoff chain.

https://github.com/sasansom/sedes/blob/85ba9e2a2b5e9fbf52655368451b6057922582a6/src/lemma.py#L206-L216



Discussion:
> so I suppose it must be happening somewhere in the model.

I found 57 entries in `UnigramLemmatizer._context_to_tag` (about 0.3% of 19459 entries total) that map to an empty string.

So it's looking like this issue may belong instead in https://github.com/cltk/grc_models_cltk. Should I copy it there, or is here the best place?

```
>>> from cltk.lemmatize.greek.backoff import BackoffGreekLemmatizer
>>> lemmatizer = BackoffGreekLemmatizer()
>>> for k in sorted([k for (k, v) in lemmatizer.backoff4._context_to_tag.items() if not v]): print(k)
20
;̓
·.
Αἰγούσσαις
Βαλιαρεῖς
Βριάρεων
Λυκουργίδας
Μέλητον
Μεργάνην
Νηληϊάδη
Πλάκῳ
Φθίης
α
αἱματη
β
βοῶπις
γαμψώνυχες
γεμόν
διοτρεφές
εὐ
εὐρυσθενές
εὐχερεί
ησυδουπιάπιτα
θαλείῃ
κεκλή
μελις
ξενοτί
ξψπρια
παθοῦς
πλεόνεσσιν
πρευμενεῖς
σβυτοδόκοι
στασινυς
τέξῃ
φέροιτο
φρ
ἀν
ἀνθεμίζομαι
ἀπε
ἀπρός
ἀρ
ἀῤ
ἀώρ
ἄ
ἄρκον
Ἀλευαδῶν
Ἀράξεω
Ἀρριβίου
ἐκικλή
ἐπάμιδα
ἐπελπομέν
ἑπτήρη
ἕπε
ἵ
Ἱππακρίτας
ὁμοί
Ὑκκάρου
```
Hi, Stephen. Thanks for all the detail and yes this is the right place to post. @diyclassics wrote this code and he's been pretty busy lately, so not sure the timeline of getting a fix in.

> I cannot find a place in the inheritance chain of UnigramLemmatizer (through UnigramTagger, NgramTagger, and ContextTagger) that explicitly returns an empty string, so I suppose it must be happening somewhere in the model.

[UnigramLemmatizer](https://github.com/cltk/cltk/blob/36a9feba26e71d5d311e34c009128e001feb2f5c/src/cltk/lemmatize/backoff.py#L209) yeah I skimmed a bit … there is a lot of inheritance going on here! I have the feeling you're right about this coming from the model

> I found 57 entries in UnigramLemmatizer._context_to_tag (about 0.3% of 19459 entries total) that map to an empty string.

Ah, you seem to have found it. I commend your bravery! What do you recommend should be done? Is there a clean way to overwrite the contents of the model? I have never inspected their contents.

> We are inserting our own DictLemmatizer with corrections we have found in our corpus at the beginning of the backoff chain.

Generally speaking, this is just how I recommend end users massage the outputs of many of our tools. We know these statistical methods create known-bad results and fixing them at the tail end is easier and more consistent than trying to manipulate training data to *maybe* come to the right answer.

This is interesting. Sometime we'd be very interested to know more of your scansion work and how it differs from our approaches (have some stuff for the German languages and Latin).
I am not Stephen (that's @sasansom), but David Fifield, his collaborator on the Sedes project.

> What do you recommend should be done? Is there a clean way to overwrite the contents of the model?

I was worried that the lemmatization failures might be something systemic—a failure to handle certain suffixes, for example. But that seems not to be the case. As long as it's just a small number of random words, it's something we can work around.

The training data for `UnigramLemmatizer` come from the file [greek_lemmatized_sents.pickle](https://github.com/cltk/grc_models_cltk/blob/master/lemmata/backoff/greek_lemmatized_sents.pickle) ([ref](https://github.com/cltk/cltk/blob/de3f4fb72ed557101538e215b0e8096ad7ccb811/cltk/lemmatize/greek/backoff.py#L29)). It appears the be a list of known lemmatization pairs, with context:
```
>>> with open(os.path.expanduser("~/cltk_data/greek/model/greek_models_cltk/lemmata/backoff/greek_lemmatized_sents.pickle"), "rb") as f:
...     model = pickle.load(f)
>>> for entry in model[:4]: print(entry)
[('Κάδμου', 'Κάδμος'), ('πολῖται', 'πολίτης'), (',', ','), ('χρὴ', 'χρή'), ...]
[('εἰ', 'εἰ'), ('μὲν', 'μέν'), ('γὰρ', 'γάρ'), ('εὖ', 'εὖ'), ('πράξαιμεν', 'πράσσω'), ...]
[('εἰ', 'εἰ'), ('δ̓', 'δέ'), ('αὖθ̓', 'αὖτε'), (',', ','), ('ὃ', 'ὅς'), ('μὴ', 'μή'), ...]
[('ὑμᾶς', 'σύ'), ('δὲ', 'δέ'), ('χρὴ', 'χρή'), ('νῦν', 'νῦν'), (',', ','), ...]
```
I believe this list is then passed through [`ContextTagger._train`](https://github.com/nltk/nltk/blob/3.5/nltk/tag/sequential.py#L154), which counts frequencies and builds a map of words to their most frequent lemma.

Curiously, although there are 2 occurrences of `('διοτρεφές', '')` in the model, there are 3 of `('διοτρεφές', 'Διοτρεφής')` and 16 of `('διοτρεφές', 'διοτρεφής')`:
```
>>> collections.Counter((word, lemma) for line in model for word, lemma in line if word == "διοτρεφές")
Counter({('διοτρεφές', 'διοτρεφής'): 16, ('διοτρεφές', 'Διοτρεφής'): 3, ('διοτρεφές', ''): 2})
```
We would therefore expect the training process to arrive at `διοτρεφής` as the most frequent lemma of `διοτρεφές`. What I think is happening is the [pre-training subsetting step](https://github.com/cltk/cltk/blob/de3f4fb72ed557101538e215b0e8096ad7ccb811/cltk/lemmatize/greek/backoff.py#L40-L47) in `BackoffGreekLemmatizer` is just getting unlucky for this word and throwing away its most common lemmata.

So one way to deal with this, in the absence of more detailed information about where the greek_lemmatized_sents.pickle model comes from and how it came to contain empty strings, would be to have [`UnigramLemmatizer.__init__`](https://github.com/cltk/cltk/blob/98a64abf428643186d9b21de675597ecc456d28d/src/cltk/lemmatize/backoff.py#L214) filter out obviously bogus lemmata like `''` from its `train` argument.

Another way may be to remove the subsetting step from `BackoffGreekLemmatizer`, and hope that the numbers work out to settle on the correct lemmata. The subsetting, if I am reading it correctly, retains only 4000 word–lemma pairs, out of 549906 available. (It at first appears that the subsetting is randomized, however the randomization uses a [fixed seed](https://github.com/cltk/cltk/blob/de3f4fb72ed557101538e215b0e8096ad7ccb811/cltk/lemmatize/greek/backoff.py#L23), so it is actually deterministic.) But I would expect the removal of subsetting to have a large and unknown effect on the output lemmatization, as `UnigramLemmatizer` takes over a larger fraction of the input.
```
>>> sum(len(line) for line in model)
549906
```
Ah, David, my mistake, sorry. Really appreciate how carefully you traced this backwards.

> have UnigramLemmatizer.__init__ filter out obviously bogus lemmata like '' from its train argument

> Another way may be to remove the subsetting step from BackoffGreekLemmatizer

Am fine with either, depends on whether Patrick wants to retrain or not. He is looking at this now.


> So it's looking like this issue may belong instead in https://github.com/cltk/grc_models_cltk. Should I copy it there, or is here the best place?

Thank you for catching this—working on it now. You raise a number of good points that I am looking into.

First...
- I think you should copy over the issue of blank (i.e. '') lemmas to grc_models_cltk. I will update the token-lemma pair sentences; for now, I will just remove pairs with missing lemmas. (I will then turn some attention to putting together an updated list of sentences and releasing a new version of this lemmatizer.)

I will also make two updates to the lemmatizer code itself...
1. Look into adding a check for blank ('') lemmas to UnigramLemmatizer
2. Change the value in, as you refer to it, the subsetting step from 4000 (sentences, I believe—I will check) to 90% of the sents. (Also, when I have a chance to update the sentences, I can create an official train-test split in the models repo.)

Again, thank you for bringing these to our attention—looking forward to working with you all more on improving these tools/models.
@kylepjohnson Placeholder issue/comment—1. how should we version different tools/models within the language-specific model repos (like e.g. grc_models_cltk)?; 2. how should we coordinate versioning of the main repo with versioning of the corpora, models, etc.?
>  1. how should we version different tools/models within the language-specific model repos (like e.g. grc_models_cltk

Everything is in Git, so while not convenient, it does have an archive of all versions. 

> 2. how should we coordinate versioning of the main repo with versioning of the corpora, models, etc.?

You could also consider tagging a branch w/ its corresponding cltk version number. However this is not convenient for you or the users. ...

Question @diyclassics will your fix be solely of the model? or will this involve a code fix, too? If both, then if you merge the pull requests at the same (models repo and this repo) and also push to pypi -- well, this will be in synch enough for me.

Am I following your question, Patrick?
Thinking aloud about two related but separate things—1. I am updating only *part* of the models repo, but updating the version of the *whole* models repo (whether in a git commit number or tag, etc.); making sure this is the best way to handle incremental changes to the datasets, models, etc.; 2. when the main repo refers to one of the corpus/model repos (e.g. with the default Backoff lemmatizers, or the corpus readers, etc.), is there any indication of versioning for the user, e.g. when the tool is called? (This is what I meant by 'coordinate'; didn't know what word fit best.) Just trying to anticipate problems that may arise from fixing the model, the code, or both.
>  is there any indication of versioning for the user, e.g. w

I think you're asking whether we have *model versioning* to sync up w/ the *software versioning* of the core code. The answer is NO, only that we keep the models dir up-to-date w/ whatever the latest prod version is. Yes, this is definitely not ideal … there's a cost (measured in time) to do such model versioning that we haven't been willing to pay … absolutely, it's worth figuring out.

A cheap trick could be put a model version into the model's filename (`lat_lemmatizer_v0.0.127.pickle`) and then somehow this is hardcoded into the code. The burden falls from the users to us (nothing wrong with that, just not sure we have the bandwidth to deal with the upkeep and breakage).

Heh look at me, now I'm the one thinking aloud.
FYI https://github.com/cltk/cltk/issues/988#issuecomment-787486041
@whoopsedesy @diyclassics Is there an update on this? I don't want the ticket to linger, especially the error results from an implementation choice. But of course, we very much want the patch if this will affect other users in the future.

Resolving patch:
diff --git a/src/cltk/lemmatize/grc.py b/src/cltk/lemmatize/grc.py
--- a/src/cltk/lemmatize/grc.py
+++ b/src/cltk/lemmatize/grc.py
@@ -55,13 +55,13 @@ def __init__(
 
         def _randomize_data(train: List[list], seed: int):
             import random
-
             random.seed(seed)
             random.shuffle(train)
-            pos_train_sents = train[:4000]
+            train_size = int(.9 * len(train))
+            pos_train_sents = train[:train_size]
             lem_train_sents = [[(item[0], item[1]) for item in sent] for sent in train]
-            train_sents = lem_train_sents[:4000]
-            test_sents = lem_train_sents[4000:5000]
+            train_sents = lem_train_sents[:train_size]
+            test_sents = lem_train_sents[train_size:]
 
             return pos_train_sents, train_sents, test_sents
 
diff --git a/src/cltk/lemmatize/lat.py b/src/cltk/lemmatize/lat.py
--- a/src/cltk/lemmatize/lat.py
+++ b/src/cltk/lemmatize/lat.py
@@ -541,13 +541,13 @@ def __init__(
 
         def _randomize_data(train: List[list], seed: int):
             import random
-
             random.seed(seed)
             random.shuffle(train)
-            pos_train_sents = train[:4000]
+            train_size = int(.9 * len(train))
+            pos_train_sents = train[:train_size]
             lem_train_sents = [[(item[0], item[1]) for item in sent] for sent in train]
-            train_sents = lem_train_sents[:4000]
-            test_sents = lem_train_sents[4000:5000]
+            train_sents = lem_train_sents[:train_size]
+            test_sents = lem_train_sents[train_size:]
 
             return pos_train_sents, train_sents, test_sents
 


Test patch: