devopsmarc/my-issues-dataset huggingface dataset:
[Bug] FileLock dependency incompatible with filesystem 
 I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that. 
 I'm wondering if this can be revisited. In some managed environments the same person using HF cannot change the file-system mount flags, (and the organization may be unwilling to change these flags due to other concerns) but can ensure that there won't be concurrent writes, for example because HF is offline and the models/datasets were downloaded earlier. 

The real fix would be to FileLock itself, which does not seem very active and seems to not deal with failed system flock calls , which would be one way to fix this, as they mention in the issue below also raised by @jarednielsen 

https://github.com/tox-dev/py-filelock/issues/67
[Bug] FileLock dependency incompatible with filesystem 
 I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that. 
 > I'm wondering if this can be revisited. In some managed environments the same person using HF cannot change the file-system mount flags, (and the organization may be unwilling to change these flags due to other concerns) but can ensure that there won't be concurrent writes, for example because HF is offline and the models/datasets were downloaded earlier.

I am one of those users.  Is there a work around for this?
[Bug] FileLock dependency incompatible with filesystem 
 I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that. 
 The machines I use have a shared FS which has the filelock problem as well as a local one that does not.  Using some env vars (HF_HOME, which controls both models and datasets, and HF_DATASETS_OFFLINE) for both transformers and datasets library one can influence where these downloads happen, and whether the locks get taken. I think some of the relevant documentation is here https://huggingface.co/docs/transformers/installation#cache-setup.  I do end up  using different settings when I download the models and when I use them, and have to rsync the models to the local file system using a separate script.
[Bug] FileLock dependency incompatible with filesystem 
 I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that. 
 Thanks @orm011 .  These filesystems are such a pain.  I'll dig around, looks like setting `cache_dir` to a non-lustre filesystem works for `transformers`  but not `datasets`.
[Bug] FileLock dependency incompatible with filesystem 
 I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that. 
 Note I `export HF_HOME=` in the shell prior to running python (I do not use the `cache_dir` argument, I think I ran into similar issues with it,   nor `HF_DATASETS_CACHE` , though maybe that works, or maybe you can set it in python prior to importing the library ), and I change no other variables. Then `datasets.load_dataset()` works without any additional flags, and they  go into `HF_HOME/datasets/` and the models go into `HF_HOME/transformers/`   (and the lock files are all there as well).
Fork dataset 
 We have a multi-task learning model training I'm trying to convert to using the Arrow-based nlp dataset. 

We're currently training a custom TensorFlow model but the nlp paradigm should be a bridge for us to be able to use the wealth of pre-trained models in Transformers.

Our preprocessing flow parses raw text and json with Entity and Relations annotations and creates 2 datasets for training a NER and Relations prediction heads.

Is there some good way to "fork" dataset-

EG

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 -> DatasetREL

or 

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 + DatasetNER -> DatasetREL

 
 To be able to generate the Arrow dataset you need to either use our csv or json utilities `load_dataset("json", data_files=my_json_files)` OR write your own custom dataset script (you can find some inspiration from the [squad](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py) script for example). Custom dataset scripts can be called locally with `nlp.load_dataset(path_to_my_script_directory)`.

This should help you get what you call "Dataset1".

Then using some dataset transforms like `.map` for example you can get to "DatasetNER" and "DatasetREL".
Fork dataset 
 We have a multi-task learning model training I'm trying to convert to using the Arrow-based nlp dataset. 

We're currently training a custom TensorFlow model but the nlp paradigm should be a bridge for us to be able to use the wealth of pre-trained models in Transformers.

Our preprocessing flow parses raw text and json with Entity and Relations annotations and creates 2 datasets for training a NER and Relations prediction heads.

Is there some good way to "fork" dataset-

EG

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 -> DatasetREL

or 

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 + DatasetNER -> DatasetREL

 
 Thanks for the helpful advice, @lhoestq  -- I wasn't quite able to get the json recipe working - 

```
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.py in __init__(self, source)
     60 
     61     def __init__(self, source):
---> 62         self._open(source)
     63 
     64 
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Tried reading schema message, was null or length 0
```

But I'm going to give the generator_dataset_builder a try.

1 more quick question -- can .map be used to output different length mappings -- could I skip one, or yield 2, can you map_batch
Fork dataset 
 We have a multi-task learning model training I'm trying to convert to using the Arrow-based nlp dataset. 

We're currently training a custom TensorFlow model but the nlp paradigm should be a bridge for us to be able to use the wealth of pre-trained models in Transformers.

Our preprocessing flow parses raw text and json with Entity and Relations annotations and creates 2 datasets for training a NER and Relations prediction heads.

Is there some good way to "fork" dataset-

EG

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 -> DatasetREL

or 

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 + DatasetNER -> DatasetREL

 
 I'll close this -- It's still unclear how to go about troubleshooting the json example as I mentioned above. If I decide it's worth the trouble, I'll create another issue, or wait for a better support for using nlp for making custom data-loaders.
Large dataset in Squad2-format 
 At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ? 
 I'm pretty sure you can get some inspiration from the squad_v2 script. It looks like the dataset is quite big so it will take some time for the users to generate it, but it should be reasonable.

Also you are saying that you are still making the dataset grow in size right ?
It's probably good practice to let the users do their training/evaluations with the exact same version of the dataset.
We allow for each dataset to specify a version (ex: 1.0.0) and increment this number every time there are new samples in the dataset for example. Does it look like a good solution for you ? Or would you rather have one final version with the full dataset ?
Large dataset in Squad2-format 
 At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ? 
 It would also be good if there is any possibility for versioning, I think this way is much better than the dynamic way.
If you mean that part to put the tiles into one is the generation it would take up to 15-20 minutes on home computer hardware.
Are there any compression or optimization algorithms while generating the dataset ?
Otherwise the hardware limit is around 32 GB ram at the moment.
If everything works well we will add some more gigabytes of data in future what would make it pretty memory costly.
Large dataset in Squad2-format 
 At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ? 
 15-20 minutes is fine !
Also there's no RAM limitations as we save to disk every 1000 elements while generating the dataset by default.
After generation, the dataset is ready to use with (again) no RAM limitations as we do memory-mapping.
Large dataset in Squad2-format 
 At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ? 
 Wow, that sounds pretty cool.
Actually I have the problem of running out of memory while tokenization on our local machine.
That wouldn't happen again, would it ?
Large dataset in Squad2-format 
 At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ? 
 You can do the tokenization step using `my_tokenized_dataset = my_dataset.map(my_tokenize_function)` that writes the tokenized texts on disk as well. And then `my_tokenized_dataset` will be a memory-mapped dataset too, so you should be fine :)
Large dataset in Squad2-format 
 At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ? 
 In your training loop, loading the tokenized texts is going to be fast and pretty much negligible compared to a forward pass. You shouldn't expect any slow down.
Error when calculating glue score 
 I was trying glue score along with other metrics here. But glue gives me this error;

```
import nlp
glue_metric = nlp.load_metric('glue',name="cola")

glue_score = glue_metric.compute(predictions, references)
```

```
---------------------------------------------------------------------------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-b9210a524504> in <module>()
----> 1 glue_score = glue_metric.compute(predictions, references)

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    191         """
    192         if predictions is not None:
--> 193             self.add_batch(predictions=predictions, references=references)
    194         self.finalize(timeout=timeout)
    195 

/usr/local/lib/python3.6/dist-packages/nlp/metric.py in add_batch(self, predictions, references, **kwargs)
    207         if self.writer is None:
    208             self._init_writer()
--> 209         self.writer.write_batch(batch)
    210 
    211     def add(self, prediction=None, reference=None, **kwargs):

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    155         if self.pa_writer is None:
    156             self._build_writer(pa_table=pa.Table.from_pydict(batch_examples))
--> 157         pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
    158         if writer_batch_size is None:
    159             writer_batch_size = self.writer_batch_size

/usr/local/lib/python3.6/dist-packages/pyarrow/types.pxi in __iter__()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

TypeError: an integer is required (got type str)
```
I'm not sure whether I'm doing this wrong or whether it's an issue. I would like to know a workaround. Thank you. 
 The glue metric for cola is a metric for classification. It expects label ids as integers as inputs.
Error when calculating glue score 
 I was trying glue score along with other metrics here. But glue gives me this error;

```
import nlp
glue_metric = nlp.load_metric('glue',name="cola")

glue_score = glue_metric.compute(predictions, references)
```

```
---------------------------------------------------------------------------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-b9210a524504> in <module>()
----> 1 glue_score = glue_metric.compute(predictions, references)

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    191         """
    192         if predictions is not None:
--> 193             self.add_batch(predictions=predictions, references=references)
    194         self.finalize(timeout=timeout)
    195 

/usr/local/lib/python3.6/dist-packages/nlp/metric.py in add_batch(self, predictions, references, **kwargs)
    207         if self.writer is None:
    208             self._init_writer()
--> 209         self.writer.write_batch(batch)
    210 
    211     def add(self, prediction=None, reference=None, **kwargs):

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    155         if self.pa_writer is None:
    156             self._build_writer(pa_table=pa.Table.from_pydict(batch_examples))
--> 157         pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
    158         if writer_batch_size is None:
    159             writer_batch_size = self.writer_batch_size

/usr/local/lib/python3.6/dist-packages/pyarrow/types.pxi in __iter__()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.6/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

TypeError: an integer is required (got type str)
```
I'm not sure whether I'm doing this wrong or whether it's an issue. I would like to know a workaround. Thank you. 
 MRPC is also a binary classification task, so its metric is a binary classification metric.

To evaluate if pairs of sentences are semantically equivalent, maybe you could take a look at models that compute if one sentence entails the other or not (typically the kinds of model that could work well on the MRPC task).
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 It looks like it comes from `mwparserfromhell`.

Would it be possible to get the bad `section` that causes this issue ? The `section` string is from `datasets/wikipedia.py:L548` ? You could just add a `try` statement and print the section if the line `section_text.append(section.strip_code().strip())` crashes.

It will help us know if we have to fix it on our side or if it is a `mwparserfromhell` issue.
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 Hi, 

Thank you for you answer.
I have try to print the bad section using `try` and `except`, but it is a bit weird as the error seems to appear 3 times for instance, but the two first error does not print anything (as if the function did not go in the `except` part).
For the third one, I got that (I haven't display the entire text) :

> error : ==== Parque nacional Cajas ====
> {{AP|Parque nacional Cajas}}
> [[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
> El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
>  [[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
> leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
> Para acceder al parque desde la costa, la vía Molleturo-Cuenca es también la mejor opción.

How can I display the link instead of the text ? I suppose it will help you more
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 The error appears several times as Apache Beam retries to process examples up to 4 times irc.

I just tried to run this text into `mwparserfromhell` but it worked without the issue.

I used this code (from the `wikipedia.py` script):
```python
import mwparserfromhell as parser
import re
import six

raw_content = r"""==== Parque nacional Cajas ====
{{AP|Parque nacional Cajas}}
[[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
[[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
"""

wikicode = parser.parse(raw_content)

# Filters for references, tables, and file/image links.
re_rm_wikilink = re.compile("^(?:File|Image|Media):", flags=re.IGNORECASE | re.UNICODE)

def rm_wikilink(obj):
    return bool(re_rm_wikilink.match(six.text_type(obj.title)))

def rm_tag(obj):
    return six.text_type(obj.tag) in {"ref", "table"}

def rm_template(obj):
    return obj.name.lower() in {"reflist", "notelist", "notelist-ua", "notelist-lr", "notelist-ur", "notelist-lg"}

def try_remove_obj(obj, section):
    try:
        section.remove(obj)
    except ValueError:
        # For unknown reasons, objects are sometimes not found.
        pass

section_text = []
for section in wikicode.get_sections(flat=True, include_lead=True, include_headings=True):
    for obj in section.ifilter_wikilinks(matches=rm_wikilink, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_templates(matches=rm_template, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_tags(matches=rm_tag, recursive=True):
        try_remove_obj(obj, section)

    section_text.append(section.strip_code().strip())
```
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 Not sure why we're having this issue. Maybe could you get also the file that's causing that ?
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 thanks for your answer.
How can I know which file is causing the issue ? 
I am trying to load the spanish wikipedia data.
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 Because of the way Apache Beam works we indeed don't have access to the file name at this point in the code.
We'll have to use some tricks I think :p 

You can append `filepath` to `title` in `wikipedia.py:L512` for example. [[EDIT: it's L494 my bad]]
Then just do `try:...except:` on the  call of `_parse_and_clean_wikicode` L500 I guess.

Thanks for diving into this ! I tried it myself but I run out of memory on my laptop
As soon as we have the name of the file it should be easier to find what's wrong.
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 Thanks for your help.

I tried to print the "title" of the document inside the` except (mwparserfromhell.parser.ParserError) as e`,the title displayed was : "Campeonato Mundial de futsal de la AMF 2015". (Wikipedia ES) Is it what you were looking for ?
ERROR:root:mwparserfromhell 
 Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

 
 Thanks a lot @Shiro-LK !

I was able to reproduce the issue. It comes from [this table on wikipedia](https://es.wikipedia.org/wiki/Campeonato_Mundial_de_futsal_de_la_AMF_2015#Clasificados) that can't be parsed.

The file in which the problem occurs comes from the wikipedia dumps, and it can be downloaded [here](https://dumps.wikimedia.org/eswiki/20200501/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2)

Parsing the file this way raises the parsing issue:

```python
import mwparserfromhell as parser
from tqdm.auto import tqdm
import bz2
import six
import logging
import codecs
import xml.etree.cElementTree as etree

filepath = "path/to/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2"

def _extract_content(filepath):
    """Extracts article content from a single WikiMedia XML file."""
    logging.info("generating examples from = %s", filepath)
    with open(filepath, "rb") as f:
        f = bz2.BZ2File(filename=f)
        if six.PY3:
            # Workaround due to:
            # https://github.com/tensorflow/tensorflow/issues/33563
            utf_f = codecs.getreader("utf-8")(f)
        else:
            utf_f = f
        # To clear root, to free-up more memory than just `elem.clear()`.
        context = etree.iterparse(utf_f, events=("end",))
        context = iter(context)
        unused_event, root = next(context)
        for unused_event, elem in tqdm(context, total=949087):
            if not elem.tag.endswith("page"):
                continue
            namespace = elem.tag[:-4]
            title = elem.find("./{0}title".format(namespace)).text
            ns = elem.find("./{0}ns".format(namespace)).text
            id_ = elem.find("./{0}id".format(namespace)).text
            # Filter pages that are not in the "main" namespace.
            if ns != "0":
                root.clear()
                continue
            raw_content = elem.find("./{0}revision/{0}text".format(namespace)).text
            root.clear()

            if "Campeonato Mundial de futsal de la AMF 2015" in title:
                yield (id_, title, raw_content)

for id_, title, raw_content in _extract_content(filepath):
    wikicode = parser.parse(raw_content)
```

The copied the raw content that can't be parsed [here](https://pastebin.com/raw/ZbmevLyH).

The minimal code to reproduce is:
```python
import mwparserfromhell as parser
import requests

raw_content = requests.get("https://pastebin.com/raw/ZbmevLyH").content.decode("utf-8")
wikicode = parser.parse(raw_content)

```

I will create an issue on mwparserfromhell's repo to see if we can fix that
Blog Authorship Corpus, Non Matching Splits Sizes Error, nlp viewer 
 Selecting `blog_authorship_corpus` in the nlp viewer throws the following error: 

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 172, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 132, in get
    builder_instance.download_and_prepare()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
```
@srush @lhoestq  
 This dataset seems to have a decoding problem that results in inconsistencies in the number of generated examples.
See #215.
That's why we end up with a `NonMatchingSplitsSizesError `.
Nested sequences with dicts 
 Am pretty much finished [adding a dataset](https://github.com/ghomasHudson/nlp/blob/DocRED/datasets/docred/docred.py) for [DocRED](https://github.com/thunlp/DocRED), but am getting an error when trying to add a nested `nlp.features.sequence(nlp.features.sequence({key:value,...}))`. 

The original data is in this format:
```python
{
  'title': "Title of wiki page",
  'vertexSet': [
                  [
                    { 'name': "mention_name", 
                      'sent_id': "mention in which sentence", 
                      'pos': ["postion of mention in a sentence"], 
                      'type': "NER_type"},
                    {another mention}
                  ], 
                  [another entity]
                ]
    ...
}
```
So to represent this I've attempted to write:
```
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": nlp.features.Sequence(nlp.features.Sequence({
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    })),
    ...
    }),
...
```
This is giving me the error:
```
pyarrow.lib.ArrowTypeError: Could not convert [{'pos': [[0,2], [2,4], [3,5]], "type": ["ORG", "ORG", "ORG"], "name": ["Lark Force", "Lark Force", "Lark Force", "sent_id": [0, 3, 4]}..... with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```
Do we expect the pyarrow stuff to break when doing this deeper nesting? I've checked that it still works when you do `nlp.features.Sequence(nlp.features.Sequence(nlp.Value("string"))` or `nlp.features.Sequence({key:value,...})` just not nested sequences with a dict.

If it's not possible, I can always convert it to a shallower structure. I'd rather not change the DocRED authors' structure if I don't have to though. 
 Oh yes, this is a backward compatibility feature with tensorflow_dataset in which a `Sequence` or `dict` is converted in a `dict` of `lists`, unfortunately it is not very intuitive, see here: https://github.com/huggingface/nlp/blob/master/src/nlp/features.py#L409

To avoid this behavior, you can just define the list in the feature with a simple list or a tuple (which is also simpler to write).
In your case, the features could be as follow:
``` python
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": [[{
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    }]],
    ...
    }),
...
```
Adding a dataset with multiple subtasks 
 I intent to add the datasets of the MT Quality Estimation shared tasks to `nlp`. However, they have different subtasks -- such as word-level, sentence-level and document-level quality estimation, each of which having different language pairs, and some of the data reused in different subtasks.

For example, in [QE 2019,](http://www.statmt.org/wmt19/qe-task.html) we had the same English-Russian and English-German data for word-level and sentence-level QE. 

I suppose these datasets could have both their word and sentence-level labels inside `nlp.Features`; but what about other subtasks? Should they be considered a different dataset altogether?

I read the discussion on #217 but the case of QE seems a lot simpler. 
 For one dataset you can have different configurations that each have their own `nlp.Features`.
We imagine having one configuration per subtask for example.
They are loaded with `nlp.load_dataset("my_dataset", "my_config")`.

For example the `glue` dataset has many configurations. It is a bit different from your case though because each configuration is a dataset by itself (sst2, mnli).
Another example is `wikipedia` that has one configuration per language.
[Question] Best way to batch a large dataset? 
 I'm training on large datasets such as Wikipedia and BookCorpus. Following the instructions in [the tutorial notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb), I see the following recommended for TensorFlow:

```python
train_tf_dataset = train_tf_dataset.filter(remove_none_values, load_from_cache_file=False)
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_tf_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]} 
labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
### Question about this last line ###
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
```

This code works for something like WikiText-2. However, scaling up to WikiText-103, the last line takes 5-10 minutes to run. I assume it is because tf.data.Dataset.from_tensor_slices() is pulling everything into memory, not lazily loading. This approach won't scale up to datasets 25x larger such as Wikipedia.

So I tried manual batching using `dataset.select()`:

```python
idxs = np.random.randint(len(dataset), size=bsz)
batch = dataset.select(idxs).map(lambda example: {"input_ids": tokenizer(example["text"])})
tf_batch = tf.constant(batch["ids"], dtype=tf.int64)
```

This appears to create a new Apache Arrow dataset with every batch I grab, and then tries to cache it. The runtime of `dataset.select([0, 1])` appears to be much worse than `dataset[:2]`. So using `select()` doesn't seem to be performant enough for a training loop.

Is there a performant scalable way to lazily load batches of nlp Datasets? 
 Update: I think I've found a solution.

```python
output_types = {"input_ids": tf.int64, "token_type_ids": tf.int64, "attention_mask": tf.int64}
def train_dataset_gen():
    for i in range(len(train_dataset)):
        yield train_dataset[i]
tf_dataset = tf.data.Dataset.from_generator(train_dataset_gen, output_types=output_types)
```

loads WikiText-2 in 20 ms, and WikiText-103 in 20 ms. It appears to be lazily loading via indexing train_dataset.