Repository: explosion/spaCy
Pull request: Spanish tokenizer exception and examples improvement
Fixes #5530  Adds to #3056 

## Enhancement to lang/es tokenizer_exceptions.py and added Example sentences to lang/es/examples.py

File: spacy/lang/es/tokenizer_exceptions.py

Currently the lines 7-10 contain

```
_exc = {
    "pal": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
    "pala": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
}

```
These normalizations are "lunfardo" (slang) used in certain regions, but they can create errors in tokenization due to the actual meaning of the words, which follow:

From dle.rae.es (Real Academia Española)

"pal"
Del fr. pal, y este del lat. palus 'palo'.
1. m. Heráld. Palo o partición y mueble del escudo.
2. m. Mar. Linguete grande, y en especial, el del cabrestante.

It is basically a big stick

"pala" means shovel

I propose to put these two exceptions in another variable that can be selectively used if needed or wanted in the following alternative normalization, that would be more accurate (including the spelling pause that is made during the pronunciation):

```
_slang_exc = {
    "pa'l": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
    "pa'la": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
    "pal": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
    "pala": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
}

```


## Checklist
- [x ] I have submitted the spaCy Contributor Agreement.
- [ x] I ran the tests, and all new and existing tests passed.
- [ x] My changes don't require a change to the documentation, or if they do, I've added all required information.


Problem statement:
Enhancement to Spanish tokenizer_exceptions.py
## Enhancement to lang/es tokenizer_exceptions.py

File: spacy/lang/es/tokenizer_exceptions.py

Currently the lines 7-10 contain

```
_exc = {
    "pal": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
    "pala": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
}

```
These normalizations are "lunfardo" (slang) used in certain regions, but they can create errors in tokenization due to the actual meaning of the words, which follow:

From dle.rae.es (Real Academia Española)

"pal"
Del fr. pal, y este del lat. palus 'palo'.
1. m. Heráld. Palo o partición y mueble del escudo.
2. m. Mar. Linguete grande, y en especial, el del cabrestante.

It is basically a big stick

"pala" means shovel

I propose to put these two exceptions in another variable that can be selectively used if needed or wanted
An alternative normalization, that would be more accurate (including the spelling pause that is made during the pronunciation) would be to replace for the following:

```
_slang_exc = {
    "pa'l": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
    "pa'la": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
}

```

## Your Environment
* **spaCy version:** 2.3.0.dev0
* **Platform:** Linux-5.4.0-29-generic-x86_64-with-glibc2.29
* **Python version:** 3.8.2

Hints from issue discussion:

Commit message: Spanish tokenizer exception additions. Added Spanish question examples

Patch to review:
diff --git a/spacy/lang/es/examples.py b/spacy/lang/es/examples.py
index 0e31b56af1e..7ab0a7dfe0c 100644
--- a/spacy/lang/es/examples.py
+++ b/spacy/lang/es/examples.py
@@ -18,5 +18,9 @@
     "El gato come pescado.",
     "Veo al hombre con el telescopio.",
     "La araña come moscas.",
-    "El pingüino incuba en su nido.",
+    "El pingüino incuba en su nido sobre el hielo.",
+    "¿Dónde estais?",
+    "¿Quién es el presidente Francés?",
+    "¿Dónde está encuentra la capital de Argentina?",
+    "¿Cuándo nació José de San Martín?",
 ]
diff --git a/spacy/lang/es/tokenizer_exceptions.py b/spacy/lang/es/tokenizer_exceptions.py
index 2c263108640..b7bb3eb71a7 100644
--- a/spacy/lang/es/tokenizer_exceptions.py
+++ b/spacy/lang/es/tokenizer_exceptions.py
@@ -4,15 +4,24 @@
 from ...symbols import ORTH, LEMMA, NORM, PRON_LEMMA
 
 
-_exc = {
+# Not used by default as they are only used in certains regional speech modes, left here for completeness
+_slang_exc = {
+    "pa'l": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
+    "pa'la": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
     "pal": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
     "pala": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
 }
 
+_exc = {}
+
 
 for exc_data in [
+    {ORTH: "n°", LEMMA: "número"},
+    {ORTH: "°C", LEMMA: "grados Celcius"},
     {ORTH: "aprox.", LEMMA: "aproximadamente"},
     {ORTH: "dna.", LEMMA: "docena"},
+    {ORTH: "dpto.", LEMMA: "departamento"},
+    {ORTH: "ej.", LEMMA: "ejemplo"},
     {ORTH: "esq.", LEMMA: "esquina"},
     {ORTH: "pág.", LEMMA: "página"},
     {ORTH: "p.ej.", LEMMA: "por ejemplo"},
@@ -20,6 +29,8 @@
     {ORTH: "Vd.", LEMMA: PRON_LEMMA, NORM: "usted"},
     {ORTH: "Uds.", LEMMA: PRON_LEMMA, NORM: "ustedes"},
     {ORTH: "Vds.", LEMMA: PRON_LEMMA, NORM: "ustedes"},
+    {ORTH: "vol.", NORM: "volúmen"},
+
 ]:
     _exc[exc_data[ORTH]] = [exc_data]
 
@@ -39,10 +50,14 @@
 for orth in [
     "a.C.",
     "a.J.C.",
+    "d.C.",
+    "d.J.C.",
     "apdo.",
     "Av.",
     "Avda.",
     "Cía.",
+    "Dr.",
+    "Dra.",
     "EE.UU.",
     "etc.",
     "fig.",
@@ -58,8 +73,10 @@
     "Prof.",
     "Profa.",
     "q.e.p.d.",
+    "Q.E.P.D."
     "S.A.",
     "S.L.",
+    "S.R.L."
     "s.s.s.",
     "Sr.",
     "Sra.",


Code review comments:
diff_hunk: @@ -4,22 +4,33 @@
 from ...symbols import ORTH, LEMMA, NORM, PRON_LEMMA
 
 
-_exc = {
+# Not used by default as they are only used in certains regional speech modes, left here for completeness
+_slang_exc = {
path: spacy/lang/es/tokenizer_exceptions.py
text: @user1:
I don't think we want additional lists of exceptions. I feel like this may blow up and we might end up maintaining various incompatible lists, and it could be slightly subjective what exactly should go in which list. We should have reasonable rules for the "proper"/official language constructs, and if people want to model slang or other traits of specific language, they can write their own exceptions.

@author:
I agree, I did it that way as I didn't know how you would want to handle the situation.
~~Should I modify the pull request, or is something you'll see from your side?~~ I modified the file already

diff_hunk: @@ -4,22 +4,33 @@
 from ...symbols import ORTH, LEMMA, NORM, PRON_LEMMA
 
 
-_exc = {
+# Not used by default as they are only used in certains regional speech modes, left here for completeness
+_slang_exc = {
+    "pa'l": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "l", LEMMA: "el", NORM: "el"}],
+    "pa'la": [{ORTH: "pa", LEMMA: "para"}, {ORTH: "la", LEMMA: "la", NORM: "la"}],
path: spacy/lang/es/tokenizer_exceptions.py
text: @user1:
You're right that "pala" shouldn't be in the normal list of exceptions. In fact, we removed it already 2 months ago, but apparently only on the `develop` branch. I probably should have done it on `master` instead. See here: https://github.com/explosion/spaCy/pull/5265/files (and the preceding discussion here: https://github.com/explosion/spaCy/issues/5255)

@user1:
Following your reasoning, I think we can also just remove "pal" from the exceptions list.

@author:
I'm sorry that I didn't check the dev branch. I do think we should remove "pal" too