Repository: tatuylonen/wiktextract

Issue:
[es] Issues with conjugation parsing
I'm using the Spanish Wiktionary extract from Kaikki (specifically https://kaikki.org/dictionary/downloads/es/es-extract.jsonl). I noticed some inconsistencies in the extracted verb conjugations.

**First**: There are some verbs where the form contains footnotes (e.g., a reference for vos) in the verb form. In the English extract these are provided as tags.
**Second**: There are "forms" where there is a text, e.g., at the bottom of `hablar` for the negative imperative form:

```
shape: (123, 5)
┌────────┬─────────────────────────────────┬─────────────┬─────────────────────┬────────────────────────┐
│ verb   ┆ form                            ┆ mode        ┆ tense               ┆ person                 │
│ ---    ┆ ---                             ┆ ---         ┆ ---                 ┆ ---                    │
│ str    ┆ str                             ┆ enum        ┆ str                 ┆ enum                   │
╞════════╪═════════════════════════════════╪═════════════╪═════════════════════╪════════════════════════╡
...
│ hablar ┆ hablásᵛᵒˢ                       ┆ indicative  ┆ present             ┆ second-person-singular │
│ hablar ┆ hablas^(tú)                     ┆ indicative  ┆ present             ┆ second-person-singular │
...
│ hablar ┆ hables^(tú)                     ┆ subjunctive ┆ present             ┆ second-person-singular │
│ hablar ┆ hablésᵛᵒˢ²                      ┆ subjunctive ┆ present             ┆ second-person-singular │
...
│ hablar ┆ hayas^(tú) hablado              ┆ subjunctive ┆ present perfect     ┆ second-person-singular │
│ hablar ┆ hayásᵛᵒˢ² hablado               ┆ subjunctive ┆ present perfect     ┆ second-person-singular │
...
│ hablar ┆ habláᵛᵒˢ                        ┆ imperative  ┆ affirmative         ┆ second-person-singular │
│ hablar ┆ habla^(tú)                      ┆ imperative  ┆ affirmative         ┆ second-person-singular │
...
│ hablar ┆ Se emplea el presente del modo… ┆ imperative  ┆ negative            ┆ second-person-singular │
└────────┴─────────────────────────────────┴─────────────┴─────────────────────┴────────────────────────┘
```

I can deal with both the first and second issue in my application, if you tell me that this won't change and is to be expected.

**Third**: For some verbs the conjugation tables aren't parsed correctly which leads to missing tags on each form. For `hablar` it works well. For `amar` for example it does not (even though `amar` is the _paradigm_ by which `hablar` is conjugated):

```
shape: (138, 5)
┌──────┬──────────────────┬─────────────┬──────────────────────────┬────────────┐
│ verb ┆ form             ┆ mode        ┆ tense                    ┆ person     │
│ ---  ┆ ---              ┆ ---         ┆ ---                      ┆ ---        │
│ str  ┆ str              ┆ enum        ┆ str                      ┆ enum       │
╞══════╪══════════════════╪═════════════╪══════════════════════════╪════════════╡
│ amar ┆ amando           ┆ impersonal  ┆ gerund                   ┆ impersonal │
│ amar ┆ habiendo amado   ┆ impersonal  ┆ gerund                   ┆ impersonal │
│ amar ┆ haber amado      ┆ impersonal  ┆ infinitive               ┆ impersonal │
│ amar ┆ amar             ┆ impersonal  ┆ infinitive               ┆ impersonal │
│ amar ┆ amado            ┆ impersonal  ┆ participle               ┆ impersonal │
│ amar ┆ amaré            ┆ indicative  ┆ future                   ┆ null       │
│ amar ┆ amarás           ┆ indicative  ┆ future                   ┆ null       │
│ amar ┆ amarás           ┆ indicative  ┆ future                   ┆ null       │
│ amar ┆ amará            ┆ indicative  ┆ future                   ┆ null       │
│ amar ┆ amaremos         ┆ indicative  ┆ future                   ┆ null       │
│ amar ┆ amaréis          ┆ indicative  ┆ future                   ┆ null       │
│ amar ┆ amarán           ┆ indicative  ┆ future                   ┆ null       │
│ amar ┆ habrá amado      ┆ indicative  ┆ future compound          ┆ null       │
│ amar ┆ habrás amado     ┆ indicative  ┆ future compound          ┆ null       │
│ amar ┆ habré amado      ┆ indicative  ┆ future compound          ┆ null       │
│ amar ┆ habrás amado     ┆ indicative  ┆ future compound          ┆ null       │
│ amar ┆ habremos amado   ┆ indicative  ┆ future compound          ┆ null       │
│ amar ┆ habréis amado    ┆ indicative  ┆ future compound          ┆ null       │
│ amar ┆ habrán amado     ┆ indicative  ┆ future compound          ┆ null       │
│ amar ┆ hubieron amado   ┆ indicative  ┆ past anterior            ┆ null       │
│ amar ┆ hubisteis amado  ┆ indicative  ┆ past anterior            ┆ null       │
│ amar ┆ hubimos amado    ┆ indicative  ┆ past anterior            ┆ null       │
│ amar ┆ hubo amado       ┆ indicative  ┆ past anterior            ┆ null       │
│ amar ┆ hubiste amado    ┆ indicative  ┆ past anterior            ┆ null       │
│ amar ┆ hube amado       ┆ indicative  ┆ past anterior            ┆ null       │
│ amar ┆ hubiste amado    ┆ indicative  ┆ past anterior            ┆ null       │
│ amar ┆ amabais          ┆ indicative  ┆ past imperfect           ┆ null       │
│ amar ┆ amaba            ┆ indicative  ┆ past imperfect           ┆ null       │
│ amar ┆ amabas           ┆ indicative  ┆ past imperfect           ┆ null       │
│ amar ┆ amabas           ┆ indicative  ┆ past imperfect           ┆ null       │
│ amar ┆ amaba            ┆ indicative  ┆ past imperfect           ┆ null       │
│ amar ┆ amaban           ┆ indicative  ┆ past imperfect           ┆ null       │
│ amar ┆ amábamos         ┆ indicative  ┆ past imperfect           ┆ null       │
│ amar ┆ habías amado     ┆ indicative  ┆ pluperfect               ┆ null       │
│ amar ┆ había amado      ┆ indicative  ┆ pluperfect               ┆ null       │
│ amar ┆ habías amado     ┆ indicative  ┆ pluperfect               ┆ null       │
│ amar ┆ había amado      ┆ indicative  ┆ pluperfect               ┆ null       │
│ amar ┆ habíamos amado   ┆ indicative  ┆ pluperfect               ┆ null       │
│ amar ┆ habíais amado    ┆ indicative  ┆ pluperfect               ┆ null       │
│ amar ┆ habían amado     ┆ indicative  ┆ pluperfect               ┆ null       │
│ amar ┆ aman             ┆ indicative  ┆ present                  ┆ null       │
│ amar ┆ amáis            ┆ indicative  ┆ present                  ┆ null       │
│ amar ┆ amamos           ┆ indicative  ┆ present                  ┆ null       │
│ amar ┆ ama              ┆ indicative  ┆ present                  ┆ null       │
│ amar ┆ amás             ┆ indicative  ┆ present                  ┆ null       │
│ amar ┆ amas             ┆ indicative  ┆ present                  ┆ null       │
│ amar ┆ amo              ┆ indicative  ┆ present                  ┆ null       │
│ amar ┆ amaron           ┆ indicative  ┆ present perfect          ┆ null       │
│ amar ┆ amé              ┆ indicative  ┆ present perfect          ┆ null       │
│ amar ┆ amaste           ┆ indicative  ┆ present perfect          ┆ null       │
│ amar ┆ amaste           ┆ indicative  ┆ present perfect          ┆ null       │
│ amar ┆ amó              ┆ indicative  ┆ present perfect          ┆ null       │
│ amar ┆ amamos           ┆ indicative  ┆ present perfect          ┆ null       │
│ amar ┆ amasteis         ┆ indicative  ┆ present perfect          ┆ null       │
│ amar ┆ has amado        ┆ indicative  ┆ present perfect compound ┆ null       │
│ amar ┆ he amado         ┆ indicative  ┆ present perfect compound ┆ null       │
│ amar ┆ has amado        ┆ indicative  ┆ present perfect compound ┆ null       │
│ amar ┆ han amado        ┆ indicative  ┆ present perfect compound ┆ null       │
│ amar ┆ habéis amado     ┆ indicative  ┆ present perfect compound ┆ null       │
│ amar ┆ hemos amado      ┆ indicative  ┆ present perfect compound ┆ null       │
│ amar ┆ ha amado         ┆ indicative  ┆ present perfect compound ┆ null       │
│ amar ┆ amarías          ┆ conditional ┆ conditional              ┆ null       │
│ amar ┆ amarían          ┆ conditional ┆ conditional              ┆ null       │
│ amar ┆ amaríais         ┆ conditional ┆ conditional              ┆ null       │
│ amar ┆ amaríamos        ┆ conditional ┆ conditional              ┆ null       │
│ amar ┆ amaría           ┆ conditional ┆ conditional              ┆ null       │
│ amar ┆ amarías          ┆ conditional ┆ conditional              ┆ null       │
│ amar ┆ amaría           ┆ conditional ┆ conditional              ┆ null       │
│ amar ┆ habríamos amado  ┆ conditional ┆ conditional compound     ┆ null       │
│ amar ┆ habrían amado    ┆ conditional ┆ conditional compound     ┆ null       │
│ amar ┆ habría amado     ┆ conditional ┆ conditional compound     ┆ null       │
│ amar ┆ habríais amado   ┆ conditional ┆ conditional compound     ┆ null       │
│ amar ┆ habrías amado    ┆ conditional ┆ conditional compound     ┆ null       │
│ amar ┆ habrías amado    ┆ conditional ┆ conditional compound     ┆ null       │
│ amar ┆ habría amado     ┆ conditional ┆ conditional compound     ┆ null       │
│ amar ┆ amare            ┆ subjunctive ┆ future                   ┆ null       │
│ amar ┆ amareis          ┆ subjunctive ┆ future                   ┆ null       │
│ amar ┆ amáremos         ┆ subjunctive ┆ future                   ┆ null       │
│ amar ┆ amares           ┆ subjunctive ┆ future                   ┆ null       │
│ amar ┆ amares           ┆ subjunctive ┆ future                   ┆ null       │
│ amar ┆ amare            ┆ subjunctive ┆ future                   ┆ null       │
│ amar ┆ amaren           ┆ subjunctive ┆ future                   ┆ null       │
│ amar ┆ hubieren amado   ┆ subjunctive ┆ future compound          ┆ null       │
│ amar ┆ hubiere amado    ┆ subjunctive ┆ future compound          ┆ null       │
│ amar ┆ hubieres amado   ┆ subjunctive ┆ future compound          ┆ null       │
│ amar ┆ hubieres amado   ┆ subjunctive ┆ future compound          ┆ null       │
│ amar ┆ hubiere amado    ┆ subjunctive ┆ future compound          ┆ null       │
│ amar ┆ hubiereis amado  ┆ subjunctive ┆ future compound          ┆ null       │
│ amar ┆ hubiéremos amado ┆ subjunctive ┆ future compound          ┆ null       │
│ amar ┆ amase            ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amaran           ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amara            ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amases           ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amases           ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amaras           ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amaras           ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amara            ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amaseis          ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amase            ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amasen           ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amáramos         ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amásemos         ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ amarais          ┆ subjunctive ┆ past imperfect           ┆ null       │
│ amar ┆ hubieses amado   ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubieras amado   ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubieses amado   ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubierais amado  ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubieseis amado  ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubieran amado   ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubiésemos amado ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubieras amado   ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubiéramos amado ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubiese amado    ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubiera amado    ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubiese amado    ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubiera amado    ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ hubiesen amado   ┆ subjunctive ┆ pluperfect               ┆ null       │
│ amar ┆ ames             ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ amen             ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ ames             ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ ame              ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ améis            ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ amemos           ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ ame              ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ amés             ┆ subjunctive ┆ present                  ┆ null       │
│ amar ┆ hayamos amado    ┆ subjunctive ┆ present perfect          ┆ null       │
│ amar ┆ haya amado       ┆ subjunctive ┆ present perfect          ┆ null       │
│ amar ┆ hayáis amado     ┆ subjunctive ┆ present perfect          ┆ null       │
│ amar ┆ haya amado       ┆ subjunctive ┆ present perfect          ┆ null       │
│ amar ┆ hayas amado      ┆ subjunctive ┆ present perfect          ┆ null       │
│ amar ┆ hayas amado      ┆ subjunctive ┆ present perfect          ┆ null       │
│ amar ┆ hayan amado      ┆ subjunctive ┆ present perfect          ┆ null       │
│ amar ┆ ama              ┆ imperative  ┆ present                  ┆ null       │
│ amar ┆ amá              ┆ imperative  ┆ present                  ┆ null       │
│ amar ┆ ame              ┆ imperative  ┆ present                  ┆ null       │
│ amar ┆ amemos           ┆ imperative  ┆ present                  ┆ null       │
│ amar ┆ amad             ┆ imperative  ┆ present                  ┆ null       │
│ amar ┆ amen             ┆ imperative  ┆ present                  ┆ null       │
└──────┴──────────────────┴─────────────┴──────────────────────────┴────────────┘
```

`amar` does not include any person information (it should contain tags like `first-person`, `second-person`, `singular`, `plural` etc). It also does not contain the `compound` tag in all instances as one would expect going from the result from `hablar`. There for example we get the `compound` tag on `impersonal` forms, e.g.,:

```
┌────────┬─────────────────────────────────┬─────────────┬─────────────────────┬────────────────────────┐
│ verb   ┆ form                            ┆ mode        ┆ tense               ┆ person                 │
│ ---    ┆ ---                             ┆ ---         ┆ ---                 ┆ ---                    │
│ str    ┆ str                             ┆ enum        ┆ str                 ┆ enum                   │
╞════════╪═════════════════════════════════╪═════════════╪═════════════════════╪════════════════════════╡
│ hablar ┆ hablando                        ┆ impersonal  ┆ gerund              ┆ impersonal             │
│ hablar ┆ habiendo hablado                ┆ impersonal  ┆ gerund compound     ┆ impersonal             │
│ hablar ┆ hablar                          ┆ impersonal  ┆ infinitive          ┆ impersonal             │
│ hablar ┆ haber hablado                   ┆ impersonal  ┆ infinitive compound ┆ impersonal             │
...
│ hablar ┆ hablo                           ┆ indicative  ┆ present             ┆ first-person-singular  │
│ hablar ┆ hablásᵛᵒˢ                       ┆ indicative  ┆ present             ┆ second-person-singular │
│ hablar ┆ hablas^(tú)                     ┆ indicative  ┆ present             ┆ second-person-singular │
│ hablar ┆ habla                           ┆ indicative  ┆ present             ┆ third-person-singular  │
│ hablar ┆ hablamos                        ┆ indicative  ┆ present             ┆ first-person-plural    │
│ hablar ┆ habláis                         ┆ indicative  ┆ present             ┆ second-person-plural   │
│ hablar ┆ hablan                          ┆ indicative  ┆ present             ┆ third-person-plural    │
...
```

**Fourth**: There are some forms marked as archaic in the Spanish Wiktionary but these forms are not tagged as such in the extract. Is that something we could include?

Hints from issue comments:
@xxyzz: 1.  ~I'll try to add the bottom footnote as raw tag, if it's too much trouble I'll remove the footnote.~ Footnote is already removed in the new template code.
2.  Spanish Wiktionary recently updated their inflection table templates, and page "hablar" was updated after 20250501, page "amar" was updated before 20250501, the new template doesn't seem to have this problem.
3. the new "es.v" table template doesn't have "primera", "segunda", "tercera" table header, so no "first-person" tag. But these "yo", "tú", "vos" table headers are extracted to the "raw_tags" list. The "compound" tags will be added if the header has "compuesto", the first table doesn't have this header but some rows in the following tables have this tag.
4. please provide some examples

In conclusion, the page "hablar" data is extracted from a template they don't use anymore and you won't see this kind of data in the next dump file.@MLNW: 1. Great, thanks!
2. So once all verb templates are updated this won't happen again. Do you have any experience how long this kind of change takes to propagate to all affected pages? In this case it seems to affect all verbs which should be >17000. I can imagine that takes quite a while if there is no automation used.
3. Are there plans to use `yo`, `tu` to again provide tags like `first-person`, `second-person` or should I handle that conversion in my application code?
4. All (most?) verb conjugation tables have some rows that are marked with a `†` and the legend says `† arcaico`. It's always the same rows though since those verb forms are no longer user. E.g., for [amar](https://es.wiktionary.org/wiki/amar):
  ![Image](https://github.com/user-attachments/assets/3f0a6f47-a154-4c84-ab8e-2c3a82df43c2)@xxyzz: - all obsoleted table templates were replaced before 20250520
- I think all the "yo", "tú", "vos" row headers can't be translated to "*-person" tags
- I think I could add the footnote to "raw_tags" or create a new "note" field, will try this tomorrow@kristian-clausal: >     * I think all the "yo", "tú", "vos" row headers can't be translated to "*-person" tags

Technically, in these cases we *should* do it with alternative tag sets that duplicate the form. So there's one form for one meaning of "yo", another for another meaning of "yo", etc. I've been avoiding the issue a bit because... it's kind of hard to get right. That's why the tag system for EN has alternative tag sets (represent as space-separated single strings that are later chopped up) inside a list.
@xxyzz: IMO better leave the original pronoun table headers and let users decide what classification they prefer. [Spanish pronouns](https://en.wikipedia.org/wiki/Spanish_pronouns) seem pretty complex, I'm not sure how to convert them to tags without causing confusion or losing data detail(users won't know what algorithm is used to convert the headers when compare to Wiktionary and we can't explain the conversion rules just like the current en code situation).@MLNW: The Wiki page you linked has a table that shows the classification quite well:

![Image](https://github.com/user-attachments/assets/7a7598ea-9faf-459c-b85f-77f4949b202e)

There are some wrinkles here described in the two footnotes. But if the `raw_tags` are left as they are users can still retrieve that information from the `raw_tags`.@xxyzz: Table headers are translated to tags and the original headers are still in "raw_tags".

Solution patch:
diff --git a/src/wiktextract/extractor/es/conjugation.py b/src/wiktextract/extractor/es/conjugation.py
index b723df35..86a475ae 100644
--- a/src/wiktextract/extractor/es/conjugation.py
+++ b/src/wiktextract/extractor/es/conjugation.py
@@ -36,6 +36,31 @@ class SpanHeader:
     span: int
 
 
+# https://en.wikipedia.org/wiki/Spanish_pronouns
+PRONOUN_TAGS = {
+    "yo": ["first-person", "singular"],
+    "que yo": ["first-person", "singular"],
+    "tú": ["second-person", "singular"],
+    "que tú": ["second-person", "singular"],
+    "(tú)": ["second-person", "singular"],
+    "vos": ["second-person", "singular", "vos-form"],
+    "que vos": ["second-person", "singular", "vos-form"],
+    "(vos)": ["second-person", "singular", "vos-form"],
+    "él, ella, usted": ["third-person", "singular"],
+    "que él, que ella, que usted": ["third-person", "singular"],
+    "(usted)": ["third-person", "singular"],
+    "nosotros": ["first-person", "plural"],
+    "que nosotros": ["first-person", "plural"],
+    "(nosotros)": ["first-person", "plural"],
+    "vosotros": ["second-person", "plural"],
+    "que vosotros": ["second-person", "plural"],
+    "(vosotros)": ["second-person", "plural"],
+    "ustedes, ellos": ["third-person", "plural"],
+    "que ustedes, que ellos": ["third-person", "plural"],
+    "(ustedes)": ["third-person", "plural"],
+}
+
+
 def process_es_v_template(
     wxr: WiktextractContext, template_node: TemplateNode
 ) -> tuple[list[Form], list[str]]:
@@ -63,6 +88,7 @@ def process_es_v_template(
             col_headers.clear()  # new table
 
         col_index = 0
+        is_archaic_row = False
         for cell in row.find_child(
             NodeKind.TABLE_HEADER_CELL | NodeKind.TABLE_CELL
         ):
@@ -81,6 +107,7 @@ def process_es_v_template(
                     )
                     col_index += colspan
                 else:
+                    is_archaic_row = cell_text.endswith("^†")
                     row_header = cell_text.removesuffix("^†").strip()
             else:
                 cell_nodes = []
@@ -100,8 +127,13 @@ def process_es_v_template(
                             and col_index < col_head.index + col_head.span
                         ):
                             form.raw_tags.append(col_head.text)
+                            form.tags.extend(
+                                PRONOUN_TAGS.get(col_head.text, [])
+                            )
                     if row_header != "":
                         form.raw_tags.append(row_header)
+                    if is_archaic_row:
+                        form.tags.append("archaic")
                     if form.form not in ["", "―"]:
                         translate_raw_tags(form)
                         forms.append(form)


Test patch:
diff --git a/tests/test_es_conj.py b/tests/test_es_conj.py
index e8036c4e..774cd8b0 100644
--- a/tests/test_es_conj.py
+++ b/tests/test_es_conj.py
@@ -73,12 +73,24 @@ etymology text
                 {"form": "haber dado", "tags": ["impersonal", "infinitive"]},
                 {
                     "form": "diera",
-                    "tags": ["subjunctive", "past", "imperfect"],
+                    "tags": [
+                        "first-person",
+                        "singular",
+                        "subjunctive",
+                        "past",
+                        "imperfect",
+                    ],
                     "raw_tags": ["que yo"],
                 },
                 {
                     "form": "diese",
-                    "tags": ["subjunctive", "past", "imperfect"],
+                    "tags": [
+                        "first-person",
+                        "singular",
+                        "subjunctive",
+                        "past",
+                        "imperfect",
+                    ],
                     "raw_tags": ["que yo"],
                 },
             ],