ERROR:root:mwparserfromhell
State: closed | Author: Shiro-LK (NONE)
Labels: name: dataset bug

Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`



Comments: It looks like it comes from `mwparserfromhell`.

Would it be possible to get the bad `section` that causes this issue ? The `section` string is from `datasets/wikipedia.py:L548` ? You could just add a `try` statement and print the section if the line `section_text.append(section.strip_code().strip())` crashes.

It will help us know if we have to fix it on our side or if it is a `mwparserfromhell` issue.

Hi, 

Thank you for you answer.
I have try to print the bad section using `try` and `except`, but it is a bit weird as the error seems to appear 3 times for instance, but the two first error does not print anything (as if the function did not go in the `except` part).
For the third one, I got that (I haven't display the entire text) :

> error : ==== Parque nacional Cajas ====
> {{AP|Parque nacional Cajas}}
> [[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
> El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
>  [[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
> leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
> Para acceder al parque desde la costa, la vía Molleturo-Cuenca es también la mejor opción.

How can I display the link instead of the text ? I suppose it will help you more 

The error appears several times as Apache Beam retries to process examples up to 4 times irc.

I just tried to run this text into `mwparserfromhell` but it worked without the issue.

I used this code (from the `wikipedia.py` script):
```python
import mwparserfromhell as parser
import re
import six

raw_content = r"""==== Parque nacional Cajas ====
{{AP|Parque nacional Cajas}}
[[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
[[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
"""

wikicode = parser.parse(raw_content)

# Filters for references, tables, and file/image links.
re_rm_wikilink = re.compile("^(?:File|Image|Media):", flags=re.IGNORECASE | re.UNICODE)

def rm_wikilink(obj):
    return bool(re_rm_wikilink.match(six.text_type(obj.title)))

def rm_tag(obj):
    return six.text_type(obj.tag) in {"ref", "table"}

def rm_template(obj):
    return obj.name.lower() in {"reflist", "notelist", "notelist-ua", "notelist-lr", "notelist-ur", "notelist-lg"}

def try_remove_obj(obj, section):
    try:
        section.remove(obj)
    except ValueError:
        # For unknown reasons, objects are sometimes not found.
        pass

section_text = []
for section in wikicode.get_sections(flat=True, include_lead=True, include_headings=True):
    for obj in section.ifilter_wikilinks(matches=rm_wikilink, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_templates(matches=rm_template, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_tags(matches=rm_tag, recursive=True):
        try_remove_obj(obj, section)

    section_text.append(section.strip_code().strip())
```

Not sure why we're having this issue. Maybe could you get also the file that's causing that ?

thanks for your answer.
How can I know which file is causing the issue ? 
I am trying to load the spanish wikipedia data. 

Because of the way Apache Beam works we indeed don't have access to the file name at this point in the code.
We'll have to use some tricks I think :p 

You can append `filepath` to `title` in `wikipedia.py:L512` for example. [[EDIT: it's L494 my bad]]
Then just do `try:...except:` on the  call of `_parse_and_clean_wikicode` L500 I guess.

Thanks for diving into this ! I tried it myself but I run out of memory on my laptop
As soon as we have the name of the file it should be easier to find what's wrong.

Thanks for your help.

I tried to print the "title" of the document inside the` except (mwparserfromhell.parser.ParserError) as e`,the title displayed was : "Campeonato Mundial de futsal de la AMF 2015". (Wikipedia ES) Is it what you were looking for ?

Thanks a lot @Shiro-LK !

I was able to reproduce the issue. It comes from [this table on wikipedia](https://es.wikipedia.org/wiki/Campeonato_Mundial_de_futsal_de_la_AMF_2015#Clasificados) that can't be parsed.

The file in which the problem occurs comes from the wikipedia dumps, and it can be downloaded [here](https://dumps.wikimedia.org/eswiki/20200501/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2)

Parsing the file this way raises the parsing issue:

```python
import mwparserfromhell as parser
from tqdm.auto import tqdm
import bz2
import six
import logging
import codecs
import xml.etree.cElementTree as etree

filepath = "path/to/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2"

def _extract_content(filepath):
    """Extracts article content from a single WikiMedia XML file."""
    logging.info("generating examples from = %s", filepath)
    with open(filepath, "rb") as f:
        f = bz2.BZ2File(filename=f)
        if six.PY3:
            # Workaround due to:
            # https://github.com/tensorflow/tensorflow/issues/33563
            utf_f = codecs.getreader("utf-8")(f)
        else:
            utf_f = f
        # To clear root, to free-up more memory than just `elem.clear()`.
        context = etree.iterparse(utf_f, events=("end",))
        context = iter(context)
        unused_event, root = next(context)
        for unused_event, elem in tqdm(context, total=949087):
            if not elem.tag.endswith("page"):
                continue
            namespace = elem.tag[:-4]
            title = elem.find("./{0}title".format(namespace)).text
            ns = elem.find("./{0}ns".format(namespace)).text
            id_ = elem.find("./{0}id".format(namespace)).text
            # Filter pages that are not in the "main" namespace.
            if ns != "0":
                root.clear()
                continue
            raw_content = elem.find("./{0}revision/{0}text".format(namespace)).text
            root.clear()

            if "Campeonato Mundial de futsal de la AMF 2015" in title:
                yield (id_, title, raw_content)

for id_, title, raw_content in _extract_content(filepath):
    wikicode = parser.parse(raw_content)
```

The copied the raw content that can't be parsed [here](https://pastebin.com/raw/ZbmevLyH).

The minimal code to reproduce is:
```python
import mwparserfromhell as parser
import requests

raw_content = requests.get("https://pastebin.com/raw/ZbmevLyH").content.decode("utf-8")
wikicode = parser.parse(raw_content)

```

I will create an issue on mwparserfromhell's repo to see if we can fix that


This going to be fixed in the next `mwparserfromhell` release :)

Fixed in `mwparserfromhell` version 0.6.