Repository: explosion/spaCy

Issue:
💫 Lemmatizer should apply rules on OOV words
@juanmirocks points out in #327 that the lemmatizer fails on OOV words:

```python

>>> nlp.vocab.morphology.lemmatizer(u'endosomes', 'noun', morphology={'number': 'plur'})set([u'endosomes'])
>>> nlp.vocab.morphology.lemmatizer(u'chromosomes', 'noun', morphology={'number': 'plur'})
set([u'chromosome'])
```

Suggested patch to lemmatizer.py

```python

    oov_forms = []
    for old, new in rules:
        if string.endswith(old):
            form = string[:len(string) - len(old)] + new
            if form in index or not form.isalpha():
                forms.append(form)
            else:
                oov_forms.append(form)
    if not forms:
        forms.extend(oov_forms)
```


## Your Environment
<!-- Include details of your environment -->
* Operating System: 
* Python Version Used: 
* spaCy Version Used: 
* Environment Information: 



Discussion:
@honnibal thanks for looking into this -- Know I understand the problem better.

Looking forward to a fix as I think it will greatly improve the performance of my methods.

Kindly let me know if I can help with the patch.
@juanmirocks If you have time to add the patch to `lemmatizer.py` and make a pull request, that would be great 👍  

The only thing that's important is to make sure it's properly tested. In this case, I think it'd be fine to just add a [regression test](https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md#fixing-bugs) for this issue, using the above examples.
Thanks Ines.

I will gladly help and try. Is that patch code all what is necessary as per
decided for Hannibal?

I will dive in
On Mon, 30 Jan 2017 at 14:48, Ines Montani <notifications@github.com> wrote:

> @juanmirocks <https://github.com/juanmirocks> If you have time to add the
> patch to lemmatizer.py and make a pull request, that would be great 👍
>
> The only thing that's important is to make sure it's properly tested. In
> this case, I think it'd be fine to just add a regression test
> <https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md#fixing-bugs>
> for this issue, using the above examples.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/explosion/spaCy/issues/781#issuecomment-276066468>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAGQH0KVj1NvUEf7qdGmE9e2roqlMaOAks5rXeoZgaJpZM4LwgLo>
> .
>

Other examples that I will use for debugging and testing:

* "colocalizes" is lemmatized the same, "colocalizes"
  * > PP2A colocalizes with shugoshin at centromeres and is required for centromeric protection .

Resolving patch:
diff --git a/spacy/lemmatizer.py b/spacy/lemmatizer.py
--- a/spacy/lemmatizer.py
+++ b/spacy/lemmatizer.py
@@ -86,13 +86,16 @@ def lemmatize(string, index, exceptions, rules):
     #if string in index:
     #    forms.append(string)
     forms.extend(exceptions.get(string, []))
+    oov_forms = []
     for old, new in rules:
         if string.endswith(old):
             form = string[:len(string) - len(old)] + new
             if form in index or not form.isalpha():
                 forms.append(form)
+            else:
+                oov_forms.append(form)
     if not forms:
-        forms.append(string)
+        forms.extend(oov_forms)
     return set(forms)
 
 


Test patch:
diff --git a/spacy/tests/regression/test_issue781.py b/spacy/tests/regression/test_issue781.py
new file mode 100644
--- /dev/null
+++ b/spacy/tests/regression/test_issue781.py
@@ -0,0 +1,10 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import pytest
+
+
+# Note: "chromosomes" worked previous the bug fix
+@pytest.mark.parametrize('word,lemmas', [("chromosomes", ["chromosome"]), ("endosomes", ["endosome"]), ("colocalizes", ["colocalize", "colocaliz"])])
+def test_issue781(path, lemmatizer, word, lemmas):
+    assert lemmatizer(word, 'noun', morphology={'number': 'plur'}) == set(lemmas)