TypeError: Receiver() takes no arguments
I am trying to load a wikipedia data set

```
import nlp
from nlp import load_dataset

dataset = load_dataset("wikipedia", "20200501.en", split="train", cache_dir=data_path, beam_runner='DirectRunner')
#dataset = load_dataset('wikipedia', '20200501.sv', cache_dir=data_path, beam_runner='DirectRunner')
```

This fails in the apache beam runner.

```
Traceback (most recent call last):
  File "D:/ML/wikiembedding/gpt2_sv.py", line 36, in <module>
    dataset = load_dataset("wikipedia", "20200501.en", split="train", cache_dir=my_cache_dir, beam_runner='DirectRunner')
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\load.py", line 548, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\builder.py", line 462, in download_and_prepare
    self._download_and_prepare(
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\nlp\builder.py", line 969, in _download_and_prepare
    pipeline_results = pipeline.run()
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\pipeline.py", line 534, in run
    return self.runner.run_pipeline(self, self._options)
....
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\runners\worker\bundle_processor.py", line 218, in process_encoded
    self.output(decoded_value)
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\apache_beam\runners\worker\operations.py", line 332, in output
    cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File "C:\Users\seto\AppData\Local\Programs\Python\Python38\lib\site-packages\Cython\Shadow.py", line 167, in cast
    return type(*args)
TypeError: Receiver() takes no arguments

```

This is run on a Windows 10 machine with python 3.8. I get the same error loading the swedish wikipedia dump.
Thanks for trying
Let us know if you find clues of what caused this issue, or if you find a fix


Errors when I use

Errors when I use
I tried the following example code from https://huggingface.co/deepset/roberta-base-squad2 and got errors
I am using **transformers 3.0.2** code .

from transformers.pipelines import pipeline
from transformers.modeling_auto import AutoModelForQuestionAnswering
from transformers.tokenization_auto import AutoTokenizer

model_name = "deepset/roberta-base-squad2"

nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
QA_input = {
    'question': 'Why is model conversion important?',
    'context': 'The option to convert models between FARM and transformers gives freedom to the user and let people easily switch between frameworks.'
}
res = nlp(QA_input)

The errors are :

res = nlp(QA_input)
  File ".local/lib/python3.6/site-packages/transformers/pipelines.py", line 1316, in __call__
    for s, e, score in zip(starts, ends, scores)
  File ".local/lib/python3.6/site-packages/transformers/pipelines.py", line 1316, in <listcomp>
    for s, e, score in zip(starts, ends, scores)
KeyError: 0

Looks like an issue with 3.0.2 transformers version. Works fine when I use "master" version of transformers.


Caching doesn't work for map (non-deterministic)
The caching functionality doesn't work reliably when tokenizing a dataset. Here's a small example to reproduce it.

```python
import nlp
import transformers

def main():
    ds = nlp.load_dataset("reddit", split="train[:500]")

    tokenizer = transformers.AutoTokenizer.from_pretrained("gpt2")

    def convert_to_features(example_batch):
        input_str = example_batch["body"]
        encodings = tokenizer(input_str, add_special_tokens=True, truncation=True)
        return encodings

    ds = ds.map(convert_to_features, batched=True)

if __name__ == "__main__":
    main()
```

Roughly 3/10 times, this example recomputes the tokenization.

Is this expected behaviour?
Thanks for reporting !

To store the cache file, we compute a hash of the function given in `.map`, using our own hashing function.
The hash doesn't seem to stay the same over sessions for the tokenizer.
Apparently this is because of the regex at `tokenizer.pat` is not well supported by our hashing function.

I'm working on a fix


Caching doesn't work for map (non-deterministic)
The caching functionality doesn't work reliably when tokenizing a dataset. Here's a small example to reproduce it.

```python
import nlp
import transformers

def main():
    ds = nlp.load_dataset("reddit", split="train[:500]")

    tokenizer = transformers.AutoTokenizer.from_pretrained("gpt2")

    def convert_to_features(example_batch):
        input_str = example_batch["body"]
        encodings = tokenizer(input_str, add_special_tokens=True, truncation=True)
        return encodings

    ds = ds.map(convert_to_features, batched=True)

if __name__ == "__main__":
    main()
```

Roughly 3/10 times, this example recomputes the tokenization.

Is this expected behaviour?
Thanks everyone. Works great now.


nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```

In 0.4.0, the assertion in `concatenate_datasets ` is on the features, and not the schema.
Could you try to update `nlp` ?

Also, since 0.4.0, you can use `dset_wikipedia.cast_(dset_books.features)` to avoid the schema cast hack.


nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```

Or maybe the assertion comes from elsewhere ?


nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```

I'm using the master branch. The assertion failure comes from the underlying `pa.concat_tables()`, which is in the pyarrow package. That method does check schemas.

Since `features.type` does not contain information about nullable vs non-nullable features, the `cast_()` method won't resolve the schema mismatch. There is information in a schema which is not stored in features.


nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```

I'm doing a refactor of type inference in #363 . Both text fields should match after that


nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```

By default nullable will be set to True


nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```

It should be good now. I was able to run

```python
>>> from nlp import concatenate_datasets, load_dataset
>>>
>>> bookcorpus = load_dataset("bookcorpus", split="train")
>>> wiki = load_dataset("wikipedia", "20200501.en", split="train")
>>> wiki.remove_columns_("title")  # only keep the text
>>>
>>> assert bookcorpus.features.type == wiki.features.type
>>> bert_dataset = concatenate_datasets([bookcorpus, wiki])
```


nlp.Features does not distinguish between nullable and non-nullable types in PyArrow schema
Here's the code I'm trying to run:

```python
dset_wikipedia = nlp.load_dataset("wikipedia", "20200501.en", split="train", cache_dir=args.cache_dir)
dset_wikipedia.drop(columns=["title"])
dset_wikipedia.features.pop("title")
dset_books = nlp.load_dataset("bookcorpus", split="train", cache_dir=args.cache_dir)
dset = nlp.concatenate_datasets([dset_wikipedia, dset_books])
```

This fails because they have different schemas, despite having identical features.

```python
assert dset_wikipedia.features == dset_books.features # True
assert dset_wikipedia._data.schema == dset_books._data.schema # False
```

The Wikipedia dataset has 'text: string', while the BookCorpus dataset has 'text: string not null'. Currently I hack together a working schema match with the following line, but it would be better if this was handled in Features themselves.

```python
dset_wikipedia._data = dset_wikipedia.data.cast(dset_books._data.schema)
```

Thanks!


No 0.4.0 release on GitHub

No 0.4.0 release on GitHub
0.4.0 was released on PyPi, but not on GitHub. This means [the documentation](https://huggingface.co/nlp/) is still displaying from 0.3.0, and that there's no tag to easily clone the 0.4.0 version of the repo.
I did the release on github, and updated the doc :)
Sorry for the delay


No 0.4.0 release on GitHub

No 0.4.0 release on GitHub
0.4.0 was released on PyPi, but not on GitHub. This means [the documentation](https://huggingface.co/nlp/) is still displaying from 0.3.0, and that there's no tag to easily clone the 0.4.0 version of the repo.
Thanks!


ug

ug

whoops


ug

ug

please delete this


issues with downloading datasets for wmt16 and wmt19
I  have encountered multiple issues while trying to:
```
import nlp
dataset = nlp.load_dataset('wmt16', 'ru-en')
metric = nlp.load_metric('wmt16')
```
1. I had to do `pip install -e ".[dev]" ` on master, currently released nlp didn't work (sorry, didn't save the error) - I went back to the released version and now it worked. So it must have been some outdated dependencies that  `pip install -e ".[dev]" ` fixed.

2. it was downloading at 60kbs - almost 5 hours to get the dataset. It was downloading all pairs and not just the one I asked for.

I tried the same code with `wmt19` in parallel and it took a few secs to download and it only fetched data for the requested pair. (but it failed too, see below)

3. my machine has crushed and when I retried I got:

```
Traceback (most recent call last):
  File "./download.py", line 9, in <module>
    dataset = nlp.load_dataset('wmt16', 'ru-en')
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/builder.py", line 449, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/stas/anaconda3/envs/main/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/builder.py", line 422, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/stas/anaconda3/envs/main/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/stas/.cache/huggingface/datasets/wmt16/ru-en/1.0.0/4d8269cdd971ed26984a9c0e4a158e0c7afc8135fac8fb8ee43ceecf38fd422d.incomplete'
```
it can't handle resumes. but neither allows a new start. Had to delete it manually.

4. and finally when it downloaded the dataset, it then failed to fetch the metrics:
```
Traceback (most recent call last):
  File "./download.py", line 15, in <module>
    metric = nlp.load_metric('wmt16')
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/load.py", line 442, in load_metric
    module_path, hash = prepare_module(path, download_config=download_config, dataset=False)
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/load.py", line 258, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/metrics/wmt16/wmt16.py
```

5. If I run the same code with `wmt19`, it fails too:

```
ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-ru.tar.gz
```
I found `UNv1.0.en-ru.tar.gz` here: https://conferences.unite.un.org/uncorpus/en/downloadoverview, so it can be reconstructed with:
```
wget -c https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-ru.tar.gz.00
wget -c https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-ru.tar.gz.01
wget -c https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-ru.tar.gz.02
cat UNv1.0.en-ru.tar.gz.0* > UNv1.0.en-ru.tar.gz
```
it has other languages as well, in case https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/ is gone


issues with downloading datasets for wmt16 and wmt19
I  have encountered multiple issues while trying to:
```
import nlp
dataset = nlp.load_dataset('wmt16', 'ru-en')
metric = nlp.load_metric('wmt16')
```
1. I had to do `pip install -e ".[dev]" ` on master, currently released nlp didn't work (sorry, didn't save the error) - I went back to the released version and now it worked. So it must have been some outdated dependencies that  `pip install -e ".[dev]" ` fixed.

2. it was downloading at 60kbs - almost 5 hours to get the dataset. It was downloading all pairs and not just the one I asked for.

I tried the same code with `wmt19` in parallel and it took a few secs to download and it only fetched data for the requested pair. (but it failed too, see below)

3. my machine has crushed and when I retried I got:

```
Traceback (most recent call last):
  File "./download.py", line 9, in <module>
    dataset = nlp.load_dataset('wmt16', 'ru-en')
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/builder.py", line 449, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/stas/anaconda3/envs/main/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/builder.py", line 422, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/stas/anaconda3/envs/main/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/stas/.cache/huggingface/datasets/wmt16/ru-en/1.0.0/4d8269cdd971ed26984a9c0e4a158e0c7afc8135fac8fb8ee43ceecf38fd422d.incomplete'
```
it can't handle resumes. but neither allows a new start. Had to delete it manually.

4. and finally when it downloaded the dataset, it then failed to fetch the metrics:
```
Traceback (most recent call last):
  File "./download.py", line 15, in <module>
    metric = nlp.load_metric('wmt16')
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/load.py", line 442, in load_metric
    module_path, hash = prepare_module(path, download_config=download_config, dataset=False)
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/load.py", line 258, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/mnt/nvme1/code/huggingface/nlp-master/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/metrics/wmt16/wmt16.py
```

5. If I run the same code with `wmt19`, it fails too:

```
ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-ru.tar.gz
```
Further, `nlp.load_dataset('wmt19', 'ru-en')` has only the `train` and `val` datasets. `test` is missing.

Fixed locally for summarization needs, by running:
```
pip install sacrebleu
sacrebleu -t wmt19 -l ru-en --echo src > test.source
sacrebleu -t wmt19 -l ru-en --echo ref > test.target
```
h/t @sshleifer


Bookcorpus data contains pretokenized text
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
Yes indeed it looks like some `'` and spaces are missing (for example in `dont` or `didnt`).
Do you know if there exist some copies without this issue ?
How would you fix this issue on the current data exactly ? I can see that the data is raw text (not tokenized) so I'm not sure I understand how you would do it. Could you provide more details ?


Bookcorpus data contains pretokenized text
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
I'm afraid that I don't know how to obtain the original BookCorpus data. I believe this version came from an anonymous Google Drive link posted in another issue.

Going through the raw text in this version, it's apparent that NLTK's TreebankWordTokenizer was applied on it (I gave some examples in my original post), followed by:
`' '.join(tokens)`
You can retrieve the tokenization by splitting on whitespace. You can then "detokenize" it with TreebankWordDetokenizer class of NLTK (though, as I suggested, use the fixed version in my repo). This will bring the text closer to its original form, but some steps of TreebankWordTokenizer are destructive, so it wouldn't be one-to-one. Something along the lines of the following should work:
```
treebank_detokenizer = nltk.tokenize.treebank.TreebankWordDetokenizer()
db = nlp.load_dataset('bookcorpus', split=nlp.Split.TRAIN)
db = db.map(lambda x: treebank_detokenizer.detokenize(x['text'].split()))
```

Regarding other issues beyond the above, I'm afraid that I can't help with that.


Bookcorpus data contains pretokenized text
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
Ok I get it, that would be very cool indeed

What kinds of patterns the detokenizer can't retrieve ?


Bookcorpus data contains pretokenized text
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
The TreebankTokenizer makes some assumptions about whitespace, parentheses, quotation marks, etc. For instance, while tokenizing the following text:
```
Dwayne "The Rock" Johnson
```
will result in:
```
Dwayne `` The Rock '' Johnson
```
where the left and right quotation marks are turned into distinct symbols. Upon reconstruction, we can attach the left part to its token on the right, and respectively for the right part. However, the following texts would be tokenized exactly the same:
```
Dwayne " The Rock " Johnson
Dwayne " The Rock" Johnson
Dwayne     " The Rock" Johnson
...
```
In the above examples, the detokenizer would correct these inputs into the canonical text
```
Dwayne "The Rock" Johnson
```
However, there are cases where there the solution cannot easily be inferred (at least without a true LM - this tokenizer is just a bunch of regexes). For instance, in cases where you have a fragment that contains the end of quote, but not its beginning, plus an accidental space:
```
... and it sounds fantastic, " he said.
```
In the above case, the tokenizer would assume that the quotes refer to the next token, and so upon detokenization it will result in the following mistake:
```
... and it sounds fantastic, "he said.
```

While these are all odd edge cases (the basic assumptions do make sense), in noisy data they can occur, which is why I mentioned that the detokenizer cannot restore the original perfectly.


Bookcorpus data contains pretokenized text
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
To confirm, since this is preprocessed, this was not the exact version of the Book Corpus used to actually train the models described here (particularly Distilbert)?  https://huggingface.co/datasets/bookcorpus

Or does this preprocessing exactly match that of the papers?


Bookcorpus data contains pretokenized text
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
I believe these are just artifacts of this particular source. It might be better to crawl it again, or use another preprocessed source, as found here: https://github.com/soskek/bookcorpus


Bookcorpus data contains pretokenized text
It seem that the bookcoprus data downloaded through the library was pretokenized with NLTK's Treebank tokenizer, which changes the text in incompatible ways to how, for instance, BERT's wordpiece tokenizer works. For example, "didn't" becomes "did" + "n't", and double quotes are changed to `` and '' for start and end quotes, respectively.

On my own projects, I just run the data through NLTK's TreebankWordDetokenizer to reverse the tokenization (as best as possible). I think it would be beneficial to apply this transformation directly on your remote cached copy of the dataset. If you choose to do so, I would also suggest to use my fork of NLTK that fixes several bugs in their detokenizer (I've opened a pull-request, but they've yet to respond): https://github.com/nltk/nltk/pull/2575
Yes actually the BookCorpus on hugginface is based on [this](https://github.com/soskek/bookcorpus/issues/24#issuecomment-643933352). And I kind of regret naming it as "BookCorpus" instead of something like "BookCorpusLike".

But there is a good news ! @shawwn has replicated BookCorpus in his way, and also provided a link to download the plain text files. see [here](https://github.com/soskek/bookcorpus/issues/27). There is chance we can have a "OpenBookCorpus" !


rotten tomatoes movie review dataset taken down
In an interesting twist of events, the individual who created the movie review seems to have left Cornell, and their webpage has been removed, along with the movie review dataset (http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz). It's not downloadable anymore.
found a mirror: https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz


rotten tomatoes movie review dataset taken down
In an interesting twist of events, the individual who created the movie review seems to have left Cornell, and their webpage has been removed, along with the movie review dataset (http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz). It's not downloadable anymore.
fixed in #484


rotten tomatoes movie review dataset taken down
In an interesting twist of events, the individual who created the movie review seems to have left Cornell, and their webpage has been removed, along with the movie review dataset (http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz). It's not downloadable anymore.
Closing this one. Thanks again @jxmorris12 for taking care of this :)


Bugs : dataset.map() is frozen on ELI5

Bugs : dataset.map() is frozen on ELI5
Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?
This comes from an overflow in pyarrow's array.
It is stuck inside the loop that reduces the batch size to avoid the overflow.
I'll take a look


Bugs : dataset.map() is frozen on ELI5

Bugs : dataset.map() is frozen on ELI5
Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?
I created a PR to fix the issue.
It was due to an overflow check that handled badly an empty list.

You can try the changes by using
```
!pip install git+https://github.com/huggingface/nlp.git@fix-bad-type-in-overflow-check
```

Also I noticed that the first 1000 examples have an empty list in the `title_urls` field. The feature type inference in `.map` will consider it `null` because of that, and it will crash when it encounter the next example with a `title_urls` that is not empty.

Therefore to fix that, what you can do for now is increase the writer batch size so that the feature inference will take into account at least one example with a non-empty `title_urls`:

```python
# default batch size is 1_000 and it's not enough for feature type inference because of empty lists
valid_dataset = valid_dataset.map(make_input_target, writer_batch_size=3_000)
```

I was able to run the frozen cell with these changes.


Bugs : dataset.map() is frozen on ELI5

Bugs : dataset.map() is frozen on ELI5
Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?
@lhoestq Perfect and thank you very much!!
Close the issue.


Bugs : dataset.map() is frozen on ELI5

Bugs : dataset.map() is frozen on ELI5
Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?
@lhoestq mapping the function `make_input_target` was passed by your fixing.

However, there is another error in the final step of `valid_dataset.map(convert_to_features, batched=True)`

`ArrowInvalid: Could not convert Thepiratebay.vg with type str: converting to null type`
(The [same colab notebook above with new error message](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing#scrollTo=5sRrJ3_C8rLt))

Do you have some ideas? (I am really sorry I could not debug it by myself since I never used `pyarrow` before)
Note that `train_dataset.map(convert_to_features, batched=True)` can be run successfully even though train_dataset is 27x bigger than `valid_dataset` so I believe the problem lies in some field of `valid_dataset` again .


Bugs : dataset.map() is frozen on ELI5

Bugs : dataset.map() is frozen on ELI5
Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?
I got this issue too and fixed it by specifying `writer_batch_size=3_000` in `.map`.
This is because Arrow didn't expect `Thepiratebay.vg` in `title_urls `, as all previous examples have empty lists in `title_urls `


Bugs : dataset.map() is frozen on ELI5

Bugs : dataset.map() is frozen on ELI5
Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?
I am clear now . Thank so much again Quentin!


Export TFRecord to GCP bucket

Export TFRecord to GCP bucket
Previously, I was writing TFRecords manually to GCP bucket with : `with tf.io.TFRecordWriter('gs://my_bucket/x.tfrecord')`

Since `0.4.0` is out with the `export()` function, I tried it. But it seems TFRecords cannot be directly written to GCP bucket.

`dataset.export('local.tfrecord')` works fine,
but `dataset.export('gs://my_bucket/x.tfrecord')` does not work.

There is no error message, I just can't find the file on my bucket...

---

Looking at the code, `nlp` is using `tf.data.experimental.TFRecordWriter`, while I was using `tf.io.TFRecordWriter`.

**What's the difference between those 2 ? How can I write TFRecords files directly to GCP bucket ?**

@jarednielsen @lhoestq
Nevermind, I restarted my python session and it worked fine...

---

I had an authentification error, and I authenticated from another terminal. After that, no more error but it was not working. Restarting the sessions makes it work :)


Overview.ipynb throws exceptions with nlp 0.4.0
with nlp 0.4.0, the TensorFlow example in Overview.ipynb throws the following exceptions:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-48907f2ad433> in <module>
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

<ipython-input-5-48907f2ad433> in <dictcomp>(.0)
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

AttributeError: 'numpy.ndarray' object has no attribute 'to_tensor'
Thanks for reporting this issue

There was a bug where numpy arrays would get returned instead of tensorflow tensors.
This is fixed on master.

I tried to re-run the colab and encountered this error instead:

```
AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'to_tensor'
```

This is because the dataset returns a Tensor and not a RaggedTensor.
But I think we should always return a RaggedTensor unless the length of the sequence is fixed (it that case they can be stack into a Tensor).


Overview.ipynb throws exceptions with nlp 0.4.0
with nlp 0.4.0, the TensorFlow example in Overview.ipynb throws the following exceptions:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-48907f2ad433> in <module>
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

<ipython-input-5-48907f2ad433> in <dictcomp>(.0)
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

AttributeError: 'numpy.ndarray' object has no attribute 'to_tensor'
Hi, I got another error (on Colab):

```python
# You can read a few attributes of the datasets before loading them (they are python dataclasses)
from dataclasses import asdict

for key, value in asdict(datasets[6]).items():
    print('👉 ' + key + ': ' + str(value))

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-b8ace6c227a2> in <module>()
      2 from dataclasses import asdict
      3
----> 4 for key, value in asdict(datasets[6]).items():
      5     print('👉 ' + key + ': ' + str(value))

/usr/local/lib/python3.6/dist-packages/dataclasses.py in asdict(obj, dict_factory)
   1008     """
   1009     if not _is_dataclass_instance(obj):
-> 1010         raise TypeError("asdict() should be called on dataclass instances")
   1011     return _asdict_inner(obj, dict_factory)
   1012

TypeError: asdict() should be called on dataclass instances
```


Overview.ipynb throws exceptions with nlp 0.4.0
with nlp 0.4.0, the TensorFlow example in Overview.ipynb throws the following exceptions:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-48907f2ad433> in <module>
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

<ipython-input-5-48907f2ad433> in <dictcomp>(.0)
----> 1 features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
      2 labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
      3 labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
      4 tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)

AttributeError: 'numpy.ndarray' object has no attribute 'to_tensor'
Indeed we'll update the cola with the new release coming up this week.


test_load_real_dataset when config has BUILDER_CONFIGS that matter
It a dataset has custom `BUILDER_CONFIGS` with non-keyword arguments (or keyword arguments with non default values), the config is not loaded during the test and causes an error.
I think the problem is that `test_load_real_dataset` calls `load_dataset` with `data_dir=temp_data_dir` ([here](https://github.com/huggingface/nlp/blob/master/tests/test_dataset_common.py#L200)). This causes [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/builder.py#L201) to always be false because `config_kwargs` is not `None`. [This line](https://github.com/huggingface/nlp/blob/master/src/nlp/builder.py#L222) will be run instead, which doesn't use `BUILDER_CONFIGS`.

For an example, you can try running the test for lince:
` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lince`
which yields
> E           TypeError: __init__() missing 3 required positional arguments: 'colnames', 'classes', and 'label_column'
The `data_dir` parameter has been removed. Now the error is `ValueError: Config name is missing`

As mentioned in #470 I think we can have one test with the first config of BUILDER_CONFIGS, and another test that runs all of the configs in BUILDER_CONFIGS


test_load_real_dataset when config has BUILDER_CONFIGS that matter
It a dataset has custom `BUILDER_CONFIGS` with non-keyword arguments (or keyword arguments with non default values), the config is not loaded during the test and causes an error.
I think the problem is that `test_load_real_dataset` calls `load_dataset` with `data_dir=temp_data_dir` ([here](https://github.com/huggingface/nlp/blob/master/tests/test_dataset_common.py#L200)). This causes [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/builder.py#L201) to always be false because `config_kwargs` is not `None`. [This line](https://github.com/huggingface/nlp/blob/master/src/nlp/builder.py#L222) will be run instead, which doesn't use `BUILDER_CONFIGS`.

For an example, you can try running the test for lince:
` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lince`
which yields
> E           TypeError: __init__() missing 3 required positional arguments: 'colnames', 'classes', and 'label_column'
This was fixed in #527

Closing this one, but feel free to re-open if you have other questions


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

Hi ! Did you try to set the output format to pytorch ? (or tensorflow if you're using tensorflow)
It can be done with `dataset.set_format("torch", columns=columns)`  (or "tensorflow").

Note that for pytorch, string columns can't be converted to `torch.Tensor`, so you have to specify in `columns=` the list of columns you want to keep (`input_ids` for example)


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

Hello . Yes, I did set the output format as below for the two columns

  `train_dataset.set_format('torch',columns=['Text','Label'])`


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I think you're having this issue because you try to format strings as pytorch tensors, which is not possible.
Indeed by having "Text" in `columns=['Text','Label']`, you try to convert the text values to pytorch tensors.

Instead I recommend you to first tokenize your dataset using a tokenizer from transformers. For example

```python
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

train_dataset.map(lambda x: tokenizer(x["Text"]), batched=True)
train_dataset.set_format("torch", column=["input_ids"])
```

Another way to fix your issue would be to not set the format to pytorch, and leave the dataset as it is by default. In that case, the strings are returned normally when you get examples from your dataloader. It means that you would have to tokenize the examples in the training loop (or using a data collator) though.

Let me know if you have other questions


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-145-ca218223c9fc> in <module>()
----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
      2 val_f1 = f1_score_func(predictions, true_val)
      3 tqdm.write(f'Validation loss: {val_loss}')
      4 tqdm.write(f'F1 Score (Weighted): {val_f1}')

6 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in <genexpr>(.0)
    160
    161     def __getitem__(self, index):
--> 162         return tuple(tensor[index] for tensor in self.tensors)
    163
    164     def __len__(self):

TypeError: new(): invalid data type 'str'


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

> Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
> I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error
>
> TypeError Traceback (most recent call last)
> in ()
> ----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
> 2 val_f1 = f1_score_func(predictions, true_val)
> 3 tqdm.write(f'Validation loss: {val_loss}')
> 4 tqdm.write(f'F1 Score (Weighted): {val_f1}')
>
> 6 frames
> /usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in (.0)
> 160
> 161 def **getitem**(self, index):
> --> 162 return tuple(tensor[index] for tensor in self.tensors)
> 163
> 164 def **len**(self):
>
> TypeError: new(): invalid data type 'str'

I got the same error and fix it .
you can check your input where there may be string contained.
such as
```
a = [1,2,3,4,'<unk>']
torch.tensor(a)
```


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I didn't know tokenizers could return strings in the token ids. Which tokenizer are you using to get this @Doragd ?


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

> I didn't know tokenizers could return strings in the token ids. Which tokenizer are you using to get this @Doragd ?

i'm sorry that i met this issue in another place (not in huggingface repo).


invalid data type 'str' at _convert_outputs in arrow_dataset.py
I trying to build multi label text classifier model using Transformers lib.

I'm using Transformers NLP to load the data set, while calling trainer.train() method. It throws the following error

File "C:\***\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

I'm using pyarrow 1.0.0.  And I have simple custom data set with Text and Integer Label.
Ex: Data
 Text ,     Label  #Column Header
 I'm facing an Network issue, 1
 I forgot my password, 2

Error StackTrace:

File "C:\**\transformers\trainer.py", line 492, in train
    for step, inputs in enumerate(epoch_iterator):
  File "C:\**\tqdm\std.py", line 1104, in __iter__
    for obj in iterable:
  File "C:\**\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\**\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\**\nlp\arrow_dataset.py", line 414, in __getitem__
    output_all_columns=self._output_all_columns,
  File "C:\**\nlp\arrow_dataset.py", line 403, in _getitem
    outputs, format_type=format_type, format_columns=format_columns, output_all_columns=output_all_columns
  File "C:\**\nlp\arrow_dataset.py", line 343, in _convert_outputs
    v = command(v)
TypeError: new(): invalid data type 'str'

@akhilkapil do you have strings in your dataset ? When you set the dataset format to "pytorch" you should exclude columns with strings as pytorch can't make tensors out of strings


UnicodeDecodeError while loading PAN-X task of XTREME dataset
Hi 🤗  team!

## Description of the problem
I'm running into a `UnicodeDecodeError` while trying to load the PAN-X subset the XTREME dataset:

```
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-1d61f439b843> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    528     ignore_verifications = ignore_verifications or save_infos
    529     # Download/copy dataset processing script
--> 530     module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
    531
    532     # Get dataset builder class from the processing script

/usr/local/lib/python3.6/dist-packages/nlp/load.py in prepare_module(path, download_config, dataset, force_local_path, **download_kwargs)
    265
    266     # Download external imports if needed
--> 267     imports = get_imports(local_path)
    268     local_imports = []
    269     library_imports = []

/usr/local/lib/python3.6/dist-packages/nlp/load.py in get_imports(file_path)
    156     lines = []
    157     with open(file_path, mode="r") as f:
--> 158         lines.extend(f.readlines())
    159
    160     logger.info("Checking %s for additional imports.", file_path)

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)
```

## Steps to reproduce
Install from nlp's master branch
```python
pip install git+https://github.com/huggingface/nlp.git
```
then run
```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
```

## OS / platform details

- `nlp` version: latest from master
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

## Proposed solution
Either change [line 762](https://github.com/huggingface/nlp/blob/7ada00b1d62f94eee22a7df38c6b01e3f27194b7/datasets/xtreme/xtreme.py#L762) in `xtreme.py` to include UTF-8 encoding:

```
# old
with open(filepath) as f
# new
with open(filepath, encoding='utf-8') as f
```

or raise a warning that suggests setting the locale explicitly, e.g.
```python
import locale
locale.setlocale(locale.LC_ALL, 'C.UTF-8')
```
I have a preference for the first solution. Let me know if you agree and I'll be happy to implement the simple fix!
Indeed. Solution 1 is the simplest.

This is actually a recurring problem.
I think we should scan all the datasets with regexpr to fix the use of `open()` without encodings.
And probably add a test in the CI to forbid using this in the future.


UnicodeDecodeError while loading PAN-X task of XTREME dataset
Hi 🤗  team!

## Description of the problem
I'm running into a `UnicodeDecodeError` while trying to load the PAN-X subset the XTREME dataset:

```
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-1d61f439b843> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    528     ignore_verifications = ignore_verifications or save_infos
    529     # Download/copy dataset processing script
--> 530     module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
    531
    532     # Get dataset builder class from the processing script

/usr/local/lib/python3.6/dist-packages/nlp/load.py in prepare_module(path, download_config, dataset, force_local_path, **download_kwargs)
    265
    266     # Download external imports if needed
--> 267     imports = get_imports(local_path)
    268     local_imports = []
    269     library_imports = []

/usr/local/lib/python3.6/dist-packages/nlp/load.py in get_imports(file_path)
    156     lines = []
    157     with open(file_path, mode="r") as f:
--> 158         lines.extend(f.readlines())
    159
    160     logger.info("Checking %s for additional imports.", file_path)

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)
```

## Steps to reproduce
Install from nlp's master branch
```python
pip install git+https://github.com/huggingface/nlp.git
```
then run
```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
```

## OS / platform details

- `nlp` version: latest from master
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

## Proposed solution
Either change [line 762](https://github.com/huggingface/nlp/blob/7ada00b1d62f94eee22a7df38c6b01e3f27194b7/datasets/xtreme/xtreme.py#L762) in `xtreme.py` to include UTF-8 encoding:

```
# old
with open(filepath) as f
# new
with open(filepath, encoding='utf-8') as f
```

or raise a warning that suggests setting the locale explicitly, e.g.
```python
import locale
locale.setlocale(locale.LC_ALL, 'C.UTF-8')
```
I have a preference for the first solution. Let me know if you agree and I'll be happy to implement the simple fix!
I'm happy to tackle the broader problem - will open a PR when it's ready!


UnicodeDecodeError while loading PAN-X task of XTREME dataset
Hi 🤗  team!

## Description of the problem
I'm running into a `UnicodeDecodeError` while trying to load the PAN-X subset the XTREME dataset:

```
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-1d61f439b843> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    528     ignore_verifications = ignore_verifications or save_infos
    529     # Download/copy dataset processing script
--> 530     module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
    531
    532     # Get dataset builder class from the processing script

/usr/local/lib/python3.6/dist-packages/nlp/load.py in prepare_module(path, download_config, dataset, force_local_path, **download_kwargs)
    265
    266     # Download external imports if needed
--> 267     imports = get_imports(local_path)
    268     local_imports = []
    269     library_imports = []

/usr/local/lib/python3.6/dist-packages/nlp/load.py in get_imports(file_path)
    156     lines = []
    157     with open(file_path, mode="r") as f:
--> 158         lines.extend(f.readlines())
    159
    160     logger.info("Checking %s for additional imports.", file_path)

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)
```

## Steps to reproduce
Install from nlp's master branch
```python
pip install git+https://github.com/huggingface/nlp.git
```
then run
```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
```

## OS / platform details

- `nlp` version: latest from master
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

## Proposed solution
Either change [line 762](https://github.com/huggingface/nlp/blob/7ada00b1d62f94eee22a7df38c6b01e3f27194b7/datasets/xtreme/xtreme.py#L762) in `xtreme.py` to include UTF-8 encoding:

```
# old
with open(filepath) as f
# new
with open(filepath, encoding='utf-8') as f
```

or raise a warning that suggests setting the locale explicitly, e.g.
```python
import locale
locale.setlocale(locale.LC_ALL, 'C.UTF-8')
```
I have a preference for the first solution. Let me know if you agree and I'll be happy to implement the simple fix!
That would be awesome!


UnicodeDecodeError while loading PAN-X task of XTREME dataset
Hi 🤗  team!

## Description of the problem
I'm running into a `UnicodeDecodeError` while trying to load the PAN-X subset the XTREME dataset:

```
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-1d61f439b843> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    528     ignore_verifications = ignore_verifications or save_infos
    529     # Download/copy dataset processing script
--> 530     module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
    531
    532     # Get dataset builder class from the processing script

/usr/local/lib/python3.6/dist-packages/nlp/load.py in prepare_module(path, download_config, dataset, force_local_path, **download_kwargs)
    265
    266     # Download external imports if needed
--> 267     imports = get_imports(local_path)
    268     local_imports = []
    269     library_imports = []

/usr/local/lib/python3.6/dist-packages/nlp/load.py in get_imports(file_path)
    156     lines = []
    157     with open(file_path, mode="r") as f:
--> 158         lines.extend(f.readlines())
    159
    160     logger.info("Checking %s for additional imports.", file_path)

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)
```

## Steps to reproduce
Install from nlp's master branch
```python
pip install git+https://github.com/huggingface/nlp.git
```
then run
```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
```

## OS / platform details

- `nlp` version: latest from master
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

## Proposed solution
Either change [line 762](https://github.com/huggingface/nlp/blob/7ada00b1d62f94eee22a7df38c6b01e3f27194b7/datasets/xtreme/xtreme.py#L762) in `xtreme.py` to include UTF-8 encoding:

```
# old
with open(filepath) as f
# new
with open(filepath, encoding='utf-8') as f
```

or raise a warning that suggests setting the locale explicitly, e.g.
```python
import locale
locale.setlocale(locale.LC_ALL, 'C.UTF-8')
```
I have a preference for the first solution. Let me know if you agree and I'll be happy to implement the simple fix!
I've created a simple function that seems to do the trick:

```python
def apply_encoding_on_file_open(filepath: str):
    """Apply UTF-8 encoding for all instances where a non-binary file is opened."""

    with open(filepath, 'r', encoding='utf-8') as input_file:
        regexp = re.compile(r"""
                            (?!.*\b(?:encoding|rb|wb|wb+|ab|ab+)\b)
                            (open)
                            \((.*)\)
                            """)
        input_text = input_file.read()
        match = regexp.search(input_text)

        if match:
            print('Found match!', match.group())
            # append utf-8 encoding to matching groups in-place
            output = regexp.sub(lambda m: m.group()[:-1]+', encoding="utf-8")', input_text)
            with open(filepath, 'w', encoding='utf-8') as output_file:
                output_file.write(output)
        else:
            print("No match found!")
```

The regexp does a negative lookahead to avoid matching on cases where the encoding is already specified or when binary files are involved.

From an implementation perspective:

* Would it make sense to include this function in `nlp-cli` so that we can run something like
```
nlp-cli fix_encoding path/to/folder
```
and the command recursively fixes all files in the target?
* What is the desired behaviour in the CI test? Here we could either have a simple script that we run as a `job` in the CI and raises an error if a missing encoding is detected. Alternatively we could incorporate this behaviour into the CLI and run that in the CI.

Please let me know what you prefer among the alternatives.


UnicodeDecodeError while loading PAN-X task of XTREME dataset
Hi 🤗  team!

## Description of the problem
I'm running into a `UnicodeDecodeError` while trying to load the PAN-X subset the XTREME dataset:

```
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-1d61f439b843> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    528     ignore_verifications = ignore_verifications or save_infos
    529     # Download/copy dataset processing script
--> 530     module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
    531
    532     # Get dataset builder class from the processing script

/usr/local/lib/python3.6/dist-packages/nlp/load.py in prepare_module(path, download_config, dataset, force_local_path, **download_kwargs)
    265
    266     # Download external imports if needed
--> 267     imports = get_imports(local_path)
    268     local_imports = []
    269     library_imports = []

/usr/local/lib/python3.6/dist-packages/nlp/load.py in get_imports(file_path)
    156     lines = []
    157     with open(file_path, mode="r") as f:
--> 158         lines.extend(f.readlines())
    159
    160     logger.info("Checking %s for additional imports.", file_path)

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)
```

## Steps to reproduce
Install from nlp's master branch
```python
pip install git+https://github.com/huggingface/nlp.git
```
then run
```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
```

## OS / platform details

- `nlp` version: latest from master
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

## Proposed solution
Either change [line 762](https://github.com/huggingface/nlp/blob/7ada00b1d62f94eee22a7df38c6b01e3f27194b7/datasets/xtreme/xtreme.py#L762) in `xtreme.py` to include UTF-8 encoding:

```
# old
with open(filepath) as f
# new
with open(filepath, encoding='utf-8') as f
```

or raise a warning that suggests setting the locale explicitly, e.g.
```python
import locale
locale.setlocale(locale.LC_ALL, 'C.UTF-8')
```
I have a preference for the first solution. Let me know if you agree and I'll be happy to implement the simple fix!
I realised I was overthinking the problem, so decided to just run the regexp over the codebase and make the PR. In other words, we can ignore my comments about using the CLI 😸


DEFAULT_TOKENIZER import error in sacrebleu
Latest Version 0.3.0

When loading the metric "sacrebleu" there is an import error due to the wrong path
![image](https://user-images.githubusercontent.com/5303103/88633063-2c5e5f00-d0bd-11ea-8ca8-4704dc975433.png)

This issue was resolved by #447


Keep loading old file even I specify a new file in load_dataset
I used load a file called 'a.csv' by
```
dataset = load_dataset('csv', data_file='./a.csv')
```
And after a while, I tried to load another csv called 'b.csv'
```
dataset = load_dataset('csv', data_file='./b.csv')
```
However, the new dataset seems to remain the old 'a.csv' and not loading new csv file.

Even worse, after I load a.csv, the load_dataset function keeps loading the 'a.csv' afterward.

Is this a cache problem?

Same here !


Keep loading old file even I specify a new file in load_dataset
I used load a file called 'a.csv' by
```
dataset = load_dataset('csv', data_file='./a.csv')
```
And after a while, I tried to load another csv called 'b.csv'
```
dataset = load_dataset('csv', data_file='./b.csv')
```
However, the new dataset seems to remain the old 'a.csv' and not loading new csv file.

Even worse, after I load a.csv, the load_dataset function keeps loading the 'a.csv' afterward.

Is this a cache problem?

This is the only fix I could come up with without touching the repo's code.
```python
from nlp.builder import FORCE_REDOWNLOAD
dataset = load_dataset('csv', data_file='./a.csv', download_mode=FORCE_REDOWNLOAD, version='0.0.1')
```
You'll have to change the version each time you want to load a different csv file.
If you're willing to add a ```print```, you can go to ```nlp.load``` and add ```print(builder_instance.cache_dir)``` right before the ```return ds``` in the ```load_dataset``` method. It'll print the cache folder, and you'll just have to erase it (and then you won't need the change here above).


Cannot unpickle saved .pt dataset with torch.save()/load()
Saving a formatted torch dataset to file using `torch.save()`. Loading the same file fails during unpickling:

```python
>>> import torch
>>> import nlp

>>> squad = nlp.load_dataset("squad.py", split="train")
>>> squad
Dataset(features: {'source_text': Value(dtype='string', id=None), 'target_text': Value(dtype='string', id=None)}, num_rows: 87599)
>>> squad = squad.map(create_features, batched=True)
>>> squad.set_format(type="torch", columns=["source_ids", "target_ids", "attention_mask"])
>>> torch.save(squad, "squad.pt")

>>> squad_pt = torch.load("squad.pt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/torch/serialization.py", line 773, in _legacy_load
    result = unpickler.load()
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/splits.py", line 493, in __setitem__
    raise ValueError("Cannot add elem. Use .add() instead.")
ValueError: Cannot add elem. Use .add() instead.
```
where `create_features` is a function that tokenizes the data using `batch_encode_plus` and returns a Dict with `input_ids`, `target_ids` and `attention_mask`.
```python
def create_features(batch):
    source_text_encoding = tokenizer.batch_encode_plus(
        batch["source_text"],
        max_length=max_source_length,
        pad_to_max_length=True,
        truncation=True)

    target_text_encoding = tokenizer.batch_encode_plus(
        batch["target_text"],
        max_length=max_target_length,
        pad_to_max_length=True,
        truncation=True)

    features = {
        "source_ids": source_text_encoding["input_ids"],
        "target_ids": target_text_encoding["input_ids"],
        "attention_mask": source_text_encoding["attention_mask"]
    }

    return features
```

I found a similar issue in [issue 5267 in the huggingface/transformers repo](https://github.com/huggingface/transformers/issues/5267) which was solved by downgrading to `nlp==0.2.0`. That did not solve this problem, however.
This seems to be fixed in a non-released version.

Installing nlp from source
```
git clone https://github.com/huggingface/nlp
cd nlp
pip install .
```
solves the issue.


Issues: Adding a FAISS or Elastic Search index to a Dataset
It seems the DPRContextEncoder, DPRContextEncoderTokenizer cited[ in this documentation](https://huggingface.co/nlp/faiss_and_ea.html) is not implemented ? It didnot work with the standard nlp installation . Also, I couldn't find or use it with the latest nlp install from github in Colab.  Is  there any dependency on the latest PyArrow 1.0.0 ? Is it yet to be made generally available ?
`DPRContextEncoder` and `DPRContextEncoderTokenizer` will be available in the next release of `transformers`.

Right now you can experiment with it by installing `transformers` from the master branch.
You can also check the docs of DPR [here](https://huggingface.co/transformers/master/model_doc/dpr.html).

Moreover all the indexing features will also be available in the next release of `nlp`.


Issues: Adding a FAISS or Elastic Search index to a Dataset
It seems the DPRContextEncoder, DPRContextEncoderTokenizer cited[ in this documentation](https://huggingface.co/nlp/faiss_and_ea.html) is not implemented ? It didnot work with the standard nlp installation . Also, I couldn't find or use it with the latest nlp install from github in Colab.  Is  there any dependency on the latest PyArrow 1.0.0 ? Is it yet to be made generally available ?
@lhoestq Thanks for the info


Issues: Adding a FAISS or Elastic Search index to a Dataset
It seems the DPRContextEncoder, DPRContextEncoderTokenizer cited[ in this documentation](https://huggingface.co/nlp/faiss_and_ea.html) is not implemented ? It didnot work with the standard nlp installation . Also, I couldn't find or use it with the latest nlp install from github in Colab.  Is  there any dependency on the latest PyArrow 1.0.0 ? Is it yet to be made generally available ?
@lhoestq  I tried installing transformer from the master branch. Python imports for DPR again didnt' work.  Anyways, Looking forward to trying it in the next release of nlp


Issues: Adding a FAISS or Elastic Search index to a Dataset
It seems the DPRContextEncoder, DPRContextEncoderTokenizer cited[ in this documentation](https://huggingface.co/nlp/faiss_and_ea.html) is not implemented ? It didnot work with the standard nlp installation . Also, I couldn't find or use it with the latest nlp install from github in Colab.  Is  there any dependency on the latest PyArrow 1.0.0 ? Is it yet to be made generally available ?
@nsankar have you tried with the latest version of the library?


Issues: Adding a FAISS or Elastic Search index to a Dataset
It seems the DPRContextEncoder, DPRContextEncoderTokenizer cited[ in this documentation](https://huggingface.co/nlp/faiss_and_ea.html) is not implemented ? It didnot work with the standard nlp installation . Also, I couldn't find or use it with the latest nlp install from github in Colab.  Is  there any dependency on the latest PyArrow 1.0.0 ? Is it yet to be made generally available ?
@yjernite it worked. Thanks


New Datasets: IWSLT15+, ITTB

New Datasets: IWSLT15+, ITTB
**Links:**
[iwslt](https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/datasets/iwslt.html)
Don't know if that link is up to date.

[ittb](http://www.cfilt.iitb.ac.in/iitb_parallel/)
**Motivation**: replicate mbart finetuning results (table below)
![image](https://user-images.githubusercontent.com/6045025/88490093-0c1c8c00-cf67-11ea-960d-8dcaad2aa8eb.png)

For future readers, we already have the following language pairs in the wmt namespaces:

```
wmt14: ['cs-en', 'de-en', 'fr-en', 'hi-en', 'ru-en']
wmt15: ['cs-en', 'de-en', 'fi-en', 'fr-en', 'ru-en']
wmt16: ['cs-en', 'de-en', 'fi-en', 'ro-en', 'ru-en', 'tr-en']
wmt17: ['cs-en', 'de-en', 'fi-en', 'lv-en', 'ru-en', 'tr-en', 'zh-en']
wmt18: ['cs-en', 'de-en', 'et-en', 'fi-en', 'kk-en', 'ru-en', 'tr-en', 'zh-en']
wmt19: ['cs-en', 'de-en', 'fi-en', 'gu-en', 'kk-en', 'lt-en', 'ru-en', 'zh-en', 'fr-de']
```
Thanks Sam, we now have a very detailed tutorial and template on how to add a new dataset to the library. It typically take 1-2 hours to add one. Do you want to give it a try ?
The tutorial on writing a new dataset loading script is here: https://huggingface.co/nlp/add_dataset.html
And the part on how to share a new dataset is here: https://huggingface.co/nlp/share_dataset.html


New Datasets: IWSLT15+, ITTB

New Datasets: IWSLT15+, ITTB
**Links:**
[iwslt](https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/datasets/iwslt.html)
Don't know if that link is up to date.

[ittb](http://www.cfilt.iitb.ac.in/iitb_parallel/)
**Motivation**: replicate mbart finetuning results (table below)
![image](https://user-images.githubusercontent.com/6045025/88490093-0c1c8c00-cf67-11ea-960d-8dcaad2aa8eb.png)

For future readers, we already have the following language pairs in the wmt namespaces:

```
wmt14: ['cs-en', 'de-en', 'fr-en', 'hi-en', 'ru-en']
wmt15: ['cs-en', 'de-en', 'fi-en', 'fr-en', 'ru-en']
wmt16: ['cs-en', 'de-en', 'fi-en', 'ro-en', 'ru-en', 'tr-en']
wmt17: ['cs-en', 'de-en', 'fi-en', 'lv-en', 'ru-en', 'tr-en', 'zh-en']
wmt18: ['cs-en', 'de-en', 'et-en', 'fi-en', 'kk-en', 'ru-en', 'tr-en', 'zh-en']
wmt19: ['cs-en', 'de-en', 'fi-en', 'gu-en', 'kk-en', 'lt-en', 'ru-en', 'zh-en', 'fr-de']
```
Hi @sshleifer, I'm trying to add IWSLT using the link you provided but the download urls are not working. Only `[en, de]` pair is working. For others language pairs it throws a `404` error.


Google Colab - load_dataset - PyArrow exception
With latest PyArrow 1.0.0 installed, I get the following exception   . Restarting colab has the same issue

ImportWarning: To use `nlp`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition. If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.

The error goes only when I install version 0.16.0
i.e.  !pip install pyarrow==0.16.0
Indeed, we’ll make a new PyPi release next week to solve this. Cc @lhoestq


Google Colab - load_dataset - PyArrow exception
With latest PyArrow 1.0.0 installed, I get the following exception   . Restarting colab has the same issue

ImportWarning: To use `nlp`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition. If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.

The error goes only when I install version 0.16.0
i.e.  !pip install pyarrow==0.16.0
+1! this is the reason our tests are failing at [TextAttack](https://github.com/QData/TextAttack)

(Though it's worth noting if we fixed the version number of pyarrow to 0.16.0 that would fix our problem too. But in this case we'll just wait for you all to update)


Google Colab - load_dataset - PyArrow exception
With latest PyArrow 1.0.0 installed, I get the following exception   . Restarting colab has the same issue

ImportWarning: To use `nlp`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition. If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.

The error goes only when I install version 0.16.0
i.e.  !pip install pyarrow==0.16.0
Came to raise this issue, great to see other already have and it's being fixed so soon!

As an aside, since no one wrote this already, it seems like the version check only looks at the second part of the version number making sure it is >16, but pyarrow newest version is 1.0.0 so the second past is 0!


Google Colab - load_dataset - PyArrow exception
With latest PyArrow 1.0.0 installed, I get the following exception   . Restarting colab has the same issue

ImportWarning: To use `nlp`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition. If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.

The error goes only when I install version 0.16.0
i.e.  !pip install pyarrow==0.16.0
> Indeed, we’ll make a new PyPi release next week to solve this. Cc @lhoestq

Yes definitely


Google Colab - load_dataset - PyArrow exception
With latest PyArrow 1.0.0 installed, I get the following exception   . Restarting colab has the same issue

ImportWarning: To use `nlp`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition. If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.

The error goes only when I install version 0.16.0
i.e.  !pip install pyarrow==0.16.0
please fix this on pypi! @lhoestq