Repository: explosion/spaCy

Issue:
Lemmatizer exceptions: `lemma_strings` get sorted anyway
Lemmatization exceptions have been working inconsistently, even when they are added directly in `corpora/en/wordnet/dict/verb.exc`

The minimal test case of `coping` at https://github.com/explosion/spaCy/issues/389 reveals that, at 
https://github.com/explosion/spaCy/blob/master/spacy/lemmatizer.py#L94, the lemmatizer's list of  potential forms (where the thing in lemmatizer.exceptions is item [0]) is cast to a `set` (and so loses ordering); then in https://github.com/explosion/spaCy/blob/master/spacy/morphology.pyx#L149, that `set` gets sorted. So lemmatizer exceptions only work if they also come first alphabetically!

I've implemented the fix for this, but I need this issue in order to submit the PR!

One question, though, for @honnibal: this can be fixed either as I did it locally -- return the whole list, with exceptions and then anything that comes back from the lemmatizer's `rules` -- or skip the rules altogether if we used an exception. I think it's more useful downstream if we keep all lemma candidates, even if we're not using them in the default pipeline. But it also seems only destructive to do `sorted(set())` on them!


Discussion:
Thanks for bringing this up. I do think something is off in the lemmatization logic here.

However, if we want the exceptions list to have priority, shouldn't we just return if an exception is found?

Once we've generated multiple analyses in `lemmatize()`, I think it's actually reasonable to use lexical order as a decision criterion. I don't see a more principled way to do it, and at least lexical order is predictable?
I do think it's fine, and saves useless rule-application, if we just use the exception if it's present. The exception list also seems to have tuples as values.

I haven't looked at the ways the analyzer generate candidate lemmas, but I can imagine plenty of implementations where there's no principle to apply for choosing (I guess some rulebases could make the *last* rule to be applied the 'best' one). In the case of `coping`, `hating` and others, the real problem is that Wordnet has verbs entered for 'hat' and 'cop' which will always win.

Maybe an alternative is to prune the Wordnet-sourced exceptions for cases like `cop` and `hat`

Oops, sorry for hitting the `close` button

Resolving patch:
diff --git a/spacy/lemmatizer.py b/spacy/lemmatizer.py
--- a/spacy/lemmatizer.py
+++ b/spacy/lemmatizer.py
@@ -78,15 +78,16 @@ def lemmatize(string, index, exceptions, rules):
     #    forms.append(string)
     forms.extend(exceptions.get(string, []))
     oov_forms = []
-    for old, new in rules:
-        if string.endswith(old):
-            form = string[:len(string) - len(old)] + new
-            if not form:
-                pass
-            elif form in index or not form.isalpha():
-                forms.append(form)
-            else:
-                oov_forms.append(form)
+    if not forms:
+        for old, new in rules:
+            if string.endswith(old):
+                form = string[:len(string) - len(old)] + new
+                if not form:
+                    pass
+                elif form in index or not form.isalpha():
+                    forms.append(form)
+                else:
+                    oov_forms.append(form)
     if not forms:
         forms.extend(oov_forms)
     if not forms:


Test patch:
diff --git a/spacy/tests/regression/test_issue1387.py b/spacy/tests/regression/test_issue1387.py
new file mode 100644
--- /dev/null
+++ b/spacy/tests/regression/test_issue1387.py
@@ -0,0 +1,22 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from ...symbols import POS, VERB, VerbForm_part
+from ...vocab import Vocab
+from ...lemmatizer import Lemmatizer
+from ..util import get_doc
+
+import pytest
+
+def test_issue1387():
+    tag_map = {'VBG': {POS: VERB, VerbForm_part: True}}
+    index = {"verb": ("cope","cop")}
+    exc = {"verb": {"coping": ("cope",)}}
+    rules = {"verb": [["ing", ""]]}
+    lemmatizer = Lemmatizer(index, exc, rules)
+    vocab = Vocab(lemmatizer=lemmatizer, tag_map=tag_map)
+    doc = get_doc(vocab, ["coping"])
+    doc[0].tag_ = 'VBG'
+    assert doc[0].text == "coping"
+    assert doc[0].lemma_ == "cope"
+
diff --git a/spacy/tests/tagger/test_lemmatizer.py b/spacy/tests/tagger/test_lemmatizer.py
--- a/spacy/tests/tagger/test_lemmatizer.py
+++ b/spacy/tests/tagger/test_lemmatizer.py
@@ -47,3 +47,20 @@ def test_tagger_lemmatizer_lemma_assignment(EN):
     assert all(t.lemma_ == '' for t in doc)
     EN.tagger(doc)
     assert all(t.lemma_ != '' for t in doc)
+
+
+from ...symbols import POS, VERB, VerbForm_part
+from ...vocab import Vocab
+from ...lemmatizer import Lemmatizer
+from ..util import get_doc
+def test_tagger_lemmatizer_exceptions():
+    index = {"verb": ("cope","cop")}
+    exc = {"verb": {"coping": ("cope",)}}
+    rules = {"verb": [["ing", ""]]}
+    tag_map = {'VBG': {POS: VERB, VerbForm_part: True}}
+    lemmatizer = Lemmatizer(index, exc, rules)
+    vocab = Vocab(lemmatizer=lemmatizer, tag_map=tag_map)
+    doc = get_doc(vocab, ["coping"])
+    doc[0].tag_ = 'VBG'
+    assert doc[0].text == "coping"
+    assert doc[0].lemma_ == "cope"