devopsmarc/my-issues-dataset huggingface dataset:
How to augment data ? 
 Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
``` 
 Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?
How to augment data ? 
 Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
``` 
 Some samples in the dataset are too long, I want to divide them in several samples.
How to augment data ? 
 Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
``` 
 Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.
How to augment data ? 
 Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
``` 
 It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...
How to augment data ? 
 Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
``` 
 Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq
How to augment data ? 
 Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
``` 
 As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.
🐛 [Metrics] ROUGE is non-deterministic 
 If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ? 
 > Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)
🐛 [Metrics] ROUGE is non-deterministic 
 If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ? 
 > If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
> 
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
> 
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
> 
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
> 
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_aggregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.
🐛 [Metrics] ROUGE is non-deterministic 
 If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ? 
 Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```
[Feature request] Add dataset.ragged_map() function for many-to-many transformations 
 `dataset.map()` enables one-to-one transformations. Input one example and output one example. This is helpful for tokenizing and cleaning individual lines.
`dataset.filter()` enables one-to-(one-or-none) transformations. Input one example and output either zero/one example. This is helpful for removing portions from the dataset.
However, some dataset transformations are many-to-many. Consider constructing BERT training examples from a dataset of sentences, where you map `["a", "b", "c"] -> ["a[SEP]b", "a[SEP]c", "b[SEP]c", "c[SEP]b", ...]`

I propose a more general `ragged_map()` method that takes in a batch of examples of length `N` and return a batch of examples `M`. This is different from the `map(batched=True)` method, which takes examples of length `N` and returns a batch of length `N`, processing individual examples in parallel. I don't have a clear vision of how this would be implemented efficiently and lazily, but would love to hear the community's feedback on this.

My specific use case is creating an end-to-end ELECTRA data pipeline. I would like to take the raw WikiText data and generate training examples from this using the `ragged_map()` method, then export to TFRecords and train quickly. This would be a reproducible pipeline with no bash scripts. Currently I'm relying on scripts like https://github.com/google-research/electra/blob/master/build_pretraining_dataset.py, which are less general.

 
 Actually `map(batched=True)` can already change the size of the dataset.
It can accept examples of length `N` and returns a batch of length `M` (can be null or greater than `N`).

I'll make that explicit in the doc that I'm currently writing.
[Feature request] Add dataset.ragged_map() function for many-to-many transformations 
 `dataset.map()` enables one-to-one transformations. Input one example and output one example. This is helpful for tokenizing and cleaning individual lines.
`dataset.filter()` enables one-to-(one-or-none) transformations. Input one example and output either zero/one example. This is helpful for removing portions from the dataset.
However, some dataset transformations are many-to-many. Consider constructing BERT training examples from a dataset of sentences, where you map `["a", "b", "c"] -> ["a[SEP]b", "a[SEP]c", "b[SEP]c", "c[SEP]b", ...]`

I propose a more general `ragged_map()` method that takes in a batch of examples of length `N` and return a batch of examples `M`. This is different from the `map(batched=True)` method, which takes examples of length `N` and returns a batch of length `N`, processing individual examples in parallel. I don't have a clear vision of how this would be implemented efficiently and lazily, but would love to hear the community's feedback on this.

My specific use case is creating an end-to-end ELECTRA data pipeline. I would like to take the raw WikiText data and generate training examples from this using the `ragged_map()` method, then export to TFRecords and train quickly. This would be a reproducible pipeline with no bash scripts. Currently I'm relying on scripts like https://github.com/google-research/electra/blob/master/build_pretraining_dataset.py, which are less general.

 
 You're two steps ahead of me :) In my testing, it also works if `M` < `N`.

A batched map of different length seems to work if you directly overwrite all of the original keys, but fails if any of the original keys are preserved.

For example,
```python
# Create a dummy dataset
dset = load_dataset("wikitext", "wikitext-2-raw-v1")["test"]
dset = dset.map(lambda ex: {"length": len(ex["text"]), "foo": 1})

# Do an allreduce on each batch, overwriting both keys
dset.map(lambda batch: {"length": [sum(batch["length"])], "foo": [1]})
# Dataset(schema: {'length': 'int64', 'foo': 'int64'}, num_rows: 5)

# Now attempt an allreduce without touching the `foo` key
dset.map(lambda batch: {"length": [sum(batch["length"])]})
# This fails with the error message below
```

```bash
  File "/path/to/nlp/src/nlp/arrow_dataset.py", line 728, in map
    arrow_schema = pa.Table.from_pydict(test_output).schema
  File "pyarrow/io.pxi", line 1532, in pyarrow.lib.Codec.detect
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named foo expected length 1 but got length 2
```

Adding the `remove_columns=["length", "foo"]` argument to `map()` solves the issue. Leaving the above error for future visitors. Perfect, thank you!
can't load SNLI dataset 
 `nlp` seems to load `snli` from some URL based on nlp.stanford.edu. This subdomain is frequently down -- including right now, when I'd like to load `snli` in a Colab notebook, but can't.

Is there a plan to move these datasets to huggingface servers for a more stable solution?

Btw, here's the stack trace:

```
File "/content/nlp/src/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/nlp/src/nlp/builder.py", line 466, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/content/nlp/src/nlp/datasets/snli/e417f6f2e16254938d977a17ed32f3998f5b23e4fcab0f6eb1d28784f23ea60d/snli.py", line 76, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 217, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in download
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/py_utils.py", line 190, in map_nested
    return function(data_struct)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://nlp.stanford.edu/projects/snli/snli_1.0.zip
``` 
 I just added the processed files of `snli` on our google storage, so that when you do `load_dataset` it can download the processed files from there :)

We are thinking about having available those processed files for more datasets in the future, because sometimes files aren't available (like for `snli`), or the download speed is too slow, or sometimes the files take time to be processed.
[Dataset requests] New datasets for Text Classification 
 We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- [x] TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
  - #386
- [x] Yelp-5
  - #1315
- [x] Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- [x] SST (Stanford Sentiment Treebank) **[include in glue]**
  - #1934
- [ ] Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- [x] Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
  - #791
  - #1389
- [x] 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
  - #410 
- [x] Sogou News dataset **[done]**
  - #450
- [x] Reuters news. The Reuters-21578 dataset [165] **[done]**
  - #471 
- [x] DBpedia. The DBpedia dataset [170]
  - #1116
- [ ] Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- [ ] EUR-Lex. The EUR-Lex dataset
- [x] WOS. The Web Of Science (WOS) dataset **[done]**
  - #424
- [ ] PubMed. PubMed [173]
- [x] TREC-QA: TREC-6 + TREC-50
  - See above: TREC-6 dataset
- [x] Quora. The Quora dataset [180]
  - #366

All these datasets are cited in https://arxiv.org/abs/2004.03705 
 - `nlp` has MR! It's called `rotten_tomatoes`
- SST is part of GLUE, or is that just SST-2?
- `nlp` also has `ag_news`, a popular news classification dataset

I'd also like to see:
- the Yahoo Answers topic classification dataset
- the Kaggle Fake News classification dataset
[Dataset requests] New datasets for Text Classification 
 We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- [x] TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
  - #386
- [x] Yelp-5
  - #1315
- [x] Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- [x] SST (Stanford Sentiment Treebank) **[include in glue]**
  - #1934
- [ ] Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- [x] Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
  - #791
  - #1389
- [x] 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
  - #410 
- [x] Sogou News dataset **[done]**
  - #450
- [x] Reuters news. The Reuters-21578 dataset [165] **[done]**
  - #471 
- [x] DBpedia. The DBpedia dataset [170]
  - #1116
- [ ] Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- [ ] EUR-Lex. The EUR-Lex dataset
- [x] WOS. The Web Of Science (WOS) dataset **[done]**
  - #424
- [ ] PubMed. PubMed [173]
- [x] TREC-QA: TREC-6 + TREC-50
  - See above: TREC-6 dataset
- [x] Quora. The Quora dataset [180]
  - #366

All these datasets are cited in https://arxiv.org/abs/2004.03705 
 Thanks @jxmorris12 for pointing this out. 

In glue we only have SST-2 maybe we can add separately SST-1.
[Dataset requests] New datasets for Text Classification 
 We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- [x] TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
  - #386
- [x] Yelp-5
  - #1315
- [x] Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- [x] SST (Stanford Sentiment Treebank) **[include in glue]**
  - #1934
- [ ] Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- [x] Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
  - #791
  - #1389
- [x] 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
  - #410 
- [x] Sogou News dataset **[done]**
  - #450
- [x] Reuters news. The Reuters-21578 dataset [165] **[done]**
  - #471 
- [x] DBpedia. The DBpedia dataset [170]
  - #1116
- [ ] Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- [ ] EUR-Lex. The EUR-Lex dataset
- [x] WOS. The Web Of Science (WOS) dataset **[done]**
  - #424
- [ ] PubMed. PubMed [173]
- [x] TREC-QA: TREC-6 + TREC-50
  - See above: TREC-6 dataset
- [x] Quora. The Quora dataset [180]
  - #366

All these datasets are cited in https://arxiv.org/abs/2004.03705 
 This is the homepage for the Amazon dataset: https://www.kaggle.com/datafiniti/consumer-reviews-of-amazon-products

Is there an easy way to download kaggle datasets programmatically? If so, I can add this one!
[Dataset requests] New datasets for Text Classification 
 We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- [x] TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
  - #386
- [x] Yelp-5
  - #1315
- [x] Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- [x] SST (Stanford Sentiment Treebank) **[include in glue]**
  - #1934
- [ ] Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- [x] Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
  - #791
  - #1389
- [x] 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
  - #410 
- [x] Sogou News dataset **[done]**
  - #450
- [x] Reuters news. The Reuters-21578 dataset [165] **[done]**
  - #471 
- [x] DBpedia. The DBpedia dataset [170]
  - #1116
- [ ] Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- [ ] EUR-Lex. The EUR-Lex dataset
- [x] WOS. The Web Of Science (WOS) dataset **[done]**
  - #424
- [ ] PubMed. PubMed [173]
- [x] TREC-QA: TREC-6 + TREC-50
  - See above: TREC-6 dataset
- [x] Quora. The Quora dataset [180]
  - #366

All these datasets are cited in https://arxiv.org/abs/2004.03705 
 Hi @jxmorris12 for now I think our `dl_manager` does not download from Kaggle.
@thomwolf , @lhoestq
[Dataset requests] New datasets for Text Classification 
 We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- [x] TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
  - #386
- [x] Yelp-5
  - #1315
- [x] Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- [x] SST (Stanford Sentiment Treebank) **[include in glue]**
  - #1934
- [ ] Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- [x] Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
  - #791
  - #1389
- [x] 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
  - #410 
- [x] Sogou News dataset **[done]**
  - #450
- [x] Reuters news. The Reuters-21578 dataset [165] **[done]**
  - #471 
- [x] DBpedia. The DBpedia dataset [170]
  - #1116
- [ ] Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- [ ] EUR-Lex. The EUR-Lex dataset
- [x] WOS. The Web Of Science (WOS) dataset **[done]**
  - #424
- [ ] PubMed. PubMed [173]
- [x] TREC-QA: TREC-6 + TREC-50
  - See above: TREC-6 dataset
- [x] Quora. The Quora dataset [180]
  - #366

All these datasets are cited in https://arxiv.org/abs/2004.03705 
 Great list. Any idea if Amazon Reviews has been added?

- ~40 GB of text (sadly no emoji)
- popular MLM pre-training dataset before bigger datasets like WebText https://arxiv.org/abs/1808.01371
- turns out that binarizing the 1-5 star rating leads to great Pos/Neg/Neutral dataset, T5 paper claims to get very high accuracy (98%!) on this with small amount of finetuning https://arxiv.org/abs/2004.14546

Apologies if it's been included (great to see where) and if not, it's one of the better medium/large NLP dataset for semi-supervised learning, albeit a bit out of date. 

Thanks!! 

cc @sshleifer
[Dataset requests] New datasets for Text Classification 
 We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- [x] TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
  - #386
- [x] Yelp-5
  - #1315
- [x] Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- [x] SST (Stanford Sentiment Treebank) **[include in glue]**
  - #1934
- [ ] Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- [x] Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
  - #791
  - #1389
- [x] 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
  - #410 
- [x] Sogou News dataset **[done]**
  - #450
- [x] Reuters news. The Reuters-21578 dataset [165] **[done]**
  - #471 
- [x] DBpedia. The DBpedia dataset [170]
  - #1116
- [ ] Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- [ ] EUR-Lex. The EUR-Lex dataset
- [x] WOS. The Web Of Science (WOS) dataset **[done]**
  - #424
- [ ] PubMed. PubMed [173]
- [x] TREC-QA: TREC-6 + TREC-50
  - See above: TREC-6 dataset
- [x] Quora. The Quora dataset [180]
  - #366

All these datasets are cited in https://arxiv.org/abs/2004.03705 
 On the Amazon Reviews dataset, the original UCSD website has noted these are now updated to include product reviews through 2018 -- actually quite recent compared to many other datasets. Almost certainly the largest NLP dataset out there with labels!
https://jmcauley.ucsd.edu/data/amazon/ 

Any chance someone has time to onboard this dataset in a HF way?

cc @sshleifer
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
See issues #242 and #307
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 It should be in `xtreme.py:L755`:
```python
        if self.config.name == "tydiqa" or self.config.name.startswith("MLQA") or self.config.name == "SQuAD":
            with open(filepath) as f:
                data = json.load(f)
```

Could you try to add the encoding parameter:
```python
open(filepath, encoding='utf-8')
```
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 Hello @jerryIsHere :) Did it work ?
If so we may change the dataset script to force the utf-8 encoding
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 @lhoestq sorry for being that late, I found 4 copy of xtreme.py. I did the changes as what has been told to all of them.
The problem is not solved
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 Could you provide a better error message so that we can make sure it comes from the opening of the `tydiqa`'s json files ?
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 @lhoestq 
The error message is same as before:
Exception has occurred: UnicodeDecodeError
'cp950' codec can't decode byte 0xe2 in position 111: illegal multibyte sequence
  File "D:\python\test\test.py", line 3, in <module>
    dataset = load_dataset('xtreme', 'tydiqa')

![image](https://user-images.githubusercontent.com/50871412/87748794-7c216880-c829-11ea-94f0-7caeacb4d865.png)

I said that I found 4 copy of xtreme.py and add the 「, encoding='utf-8'」 parameter to the open() function
these python script was found under this directory
C:\Users\USER\AppData\Local\Programs\Python\Python37\Lib\site-packages\nlp\datasets\xtreme
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 Hi there !
I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
I added ```encoding='UTF-8'``` to both lines that have ```open``` in ```imdb.py```  (108 and 114) and it worked for me.
Thank you !
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 > Hi there !
> I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
> I added `encoding='UTF-8'` to both lines that have `open` in `imdb.py` (108 and 114) and it worked for me.
> Thank you !

Hello !
Glad you managed to fix this issue on your side.
Do you mind opening a PR for IMDB ?
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 > This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
> Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
> See issues #242 and #307

Sorry for not responding for about a month.
I have just found that it is necessary to change / add the environment variable as what was told in #242.
Everything works after I add the new environment variable and restart my PC.

I think the encoding issue for windows isn't limited to the open() function call specific to few dataset, but actually in the entire library, depends on the machine / os you use.
'cp950' codec error from load_dataset('xtreme', 'tydiqa') 
 ![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
 
 Since #481 we shouldn't have other issues with encodings as they need to be set to "utf-8" be default.

Closing this one, but feel free to re-open if you gave other questions
Supporting documents in ELI5 
 I was attempting to use the ELI5 dataset, when I realized that huggingface does not provide the supporting documents (the source documents from the common crawl). Without the supporting documents, this makes the dataset about as useful for my project as a block of cheese, or some other more apt metaphor.  According to facebook, the entire document collection is quite large. However, it would still be helpful to at least include a subset of the supporting documents i.e., having some data is better than having a block of cheese, in my case at least.

If you choose not to include them, it would be helpful to have documentation mentioning this specifically. It is especially confusing because the hf nlp ELI5 dataset has the key `'document'` but there are no documents to be found :( 
 Hi @saverymax ! For licensing reasons, the original team was unable to release pre-processed CommonCrawl documents. Instead, they provided a script to re-create them from a CommonCrawl dump, but it unfortunately requires access to a medium-large size cluster:
https://github.com/facebookresearch/ELI5#downloading-support-documents-from-the-commoncrawl

In order to make the task accessible to people who may not have access to this kind of infrastructure, we suggest to use Wikipedia as a knowledge source rather than the full CommonCrawl. The following blog post shows how you can create Wikipedia support documents and get a performance that is on par with a system that uses CommonCrawl pages.
https://yjernite.github.io/lfqa.html#task_description

Hope that helps, using ElasticSearch to index Wiki40b and create the documents should take about 4 hours. Let us know if you have any trouble with the blog post though!
Supporting documents in ELI5 
 I was attempting to use the ELI5 dataset, when I realized that huggingface does not provide the supporting documents (the source documents from the common crawl). Without the supporting documents, this makes the dataset about as useful for my project as a block of cheese, or some other more apt metaphor.  According to facebook, the entire document collection is quite large. However, it would still be helpful to at least include a subset of the supporting documents i.e., having some data is better than having a block of cheese, in my case at least.

If you choose not to include them, it would be helpful to have documentation mentioning this specifically. It is especially confusing because the hf nlp ELI5 dataset has the key `'document'` but there are no documents to be found :( 
 Hi, thanks for the quick response. The blog post is quite an interesting working example, thanks for sharing it.
Two follow-up points/questions about my original question:

1. Yes, I read that the facebook team could not share the CommonCrawl b/c of licensing reasons. They state "No, we are not allowed to host processed Reddit or CommonCrawl data," which indicates they could also not share the Reddit data for licensing reasons. But it seems that HuggingFace is able to share the Reddit data, so why not a subset of CommonCrawl?

2. Thanks for the suggestion about ElasticSearch and Wiki40b. This is good to know about performance. I definitely could do the indexing and querying myself. What I like about the ELI5 dataset though, at least what is suggested by the paper, is that to create the dataset they had already selected the top 100 web sources and made a single support document from those. Though it doesn't appear to be too sophisticated an approach, having a single support document pre-computed (without having to run the facebook code or a replacement with another dataset) is super useful for my work, especially since I'm not working on developing the latest and greatest retrieval model. Of course, I don't expect HF NLP datasets to be perfectly tailored to my use-case. I know there is overhead to any project, I'm just illustrating a use-case of ELI5 which is not possible with the data provided as-is. If it's for licensing reasons, that is perfectly acceptable a reason, and I appreciate your response.
Loading CNN/Daily Mail dataset produces `nlp.utils.info_utils.NonMatchingSplitsSizesError` 
 ```
>>> import nlp
>>> nlp.load_dataset('cnn_dailymail', '3.0.0')
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/load.py", line 520, in load_dataset
    builder_instance.download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 431, in download_and_prepare
    self._download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=49424491, num_examples=11490, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='test', num_bytes=48931393, num_examples=11379, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='train', num_bytes=1249178681, num_examples=287113, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='train', num_bytes=1240618482, num_examples=285161, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='validation', num_bytes=57149241, num_examples=13368, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='validation', num_bytes=56637485, num_examples=13255, dataset_name='cnn_dailymail')}]
``` 
 I couldn't reproduce on my side.
It looks like you were not able to generate all the examples, and you have the problem for each split train-test-validation.
Could you try to enable logging, try again and send the logs ?
```python
import logging
logging.basicConfig(level=logging.INFO)
```
Loading CNN/Daily Mail dataset produces `nlp.utils.info_utils.NonMatchingSplitsSizesError` 
 ```
>>> import nlp
>>> nlp.load_dataset('cnn_dailymail', '3.0.0')
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/load.py", line 520, in load_dataset
    builder_instance.download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 431, in download_and_prepare
    self._download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=49424491, num_examples=11490, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='test', num_bytes=48931393, num_examples=11379, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='train', num_bytes=1249178681, num_examples=287113, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='train', num_bytes=1240618482, num_examples=285161, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='validation', num_bytes=57149241, num_examples=13368, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='validation', num_bytes=56637485, num_examples=13255, dataset_name='cnn_dailymail')}]
``` 
 In general if some examples are missing after processing (hence causing the `NonMatchingSplitsSizesError `), it is often due to either
1) corrupted cached files
2) decoding errors

I just checked the dataset script for code that could lead to decoding errors but I couldn't find any. Before we try to dive more into the processing of the dataset, could you try to clear your cache ? Just to make sure that it isn't 1)
Loading CNN/Daily Mail dataset produces `nlp.utils.info_utils.NonMatchingSplitsSizesError` 
 ```
>>> import nlp
>>> nlp.load_dataset('cnn_dailymail', '3.0.0')
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.26 GiB, total: 1.81 GiB) to /u/jm8wx/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0...

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/load.py", line 520, in load_dataset
    builder_instance.download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 431, in download_and_prepare
    self._download_and_prepare(
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/p/qdata/jm8wx/datasets/nlp/src/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=49424491, num_examples=11490, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='test', num_bytes=48931393, num_examples=11379, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='train', num_bytes=1249178681, num_examples=287113, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='train', num_bytes=1240618482, num_examples=285161, dataset_name='cnn_dailymail')}, {'expected': SplitInfo(name='validation', num_bytes=57149241, num_examples=13368, dataset_name='cnn_dailymail'), 'recorded': SplitInfo(name='validation', num_bytes=56637485, num_examples=13255, dataset_name='cnn_dailymail')}]
``` 
 Yes thanks for the support! I cleared out my cache folder and everything works fine now
[Bug] FileLock dependency incompatible with filesystem 
 I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that. 
 Environment is Ubuntu 18.04, Python 3.7.5, nlp==0.3.0, filelock=3.0.12.

The external volume is Amazon FSx for Lustre, and it by default creates files with limited permissions. My working theory is that FileLock creates a lockfile that isn't writable, and thus there's no way to acquire it by removing the .lock file. But Python is able to create new files and write to them outside of the FileLock package.

When I attempt to use FileLock within a Docker container by writing to `/root/.cache/hello.txt`, it succeeds. So there's some permissions issue. But it's not a Docker configuration issue; I've replicated it without Docker.
```bash
echo "hello world" >> hello.txt
ls -l

-rw-rw-r-- 1 ubuntu ubuntu 10 Jun 30 19:52 hello.txt
```
[Bug] FileLock dependency incompatible with filesystem 
 I'm downloading a dataset successfully with
`load_dataset("wikitext", "wikitext-2-raw-v1")`

But when I attempt to cache it on an external volume, it hangs indefinitely:
`load_dataset("wikitext", "wikitext-2-raw-v1", cache_dir="/fsx") # /fsx is an external volume mount`

The filesystem when hanging looks like this:
```bash
/fsx
----downloads
       ----94be...73.lock
----wikitext
       ----wikitext-2-raw
             ----wikitext-2-raw-1.0.0.incomplete
```

It appears that on this filesystem, the FileLock object is forever stuck in its "acquire" stage. I have verified that the issue lies specifically with the `filelock` dependency:
```python
open("/fsx/hello.txt").write("hello") # succeeds

from filelock import FileLock
with FileLock("/fsx/hello.lock"):
    open("/fsx/hello.txt").write("hello") # hangs indefinitely
```

Has anyone else run into this issue? I'd raise it directly on the FileLock repo, but that project appears abandoned with the last update over a year ago. Or if there's a solution that would remove the FileLock dependency from the project, I would appreciate that. 
 Looks like the `flock` syscall does not work on Lustre filesystems by default: https://github.com/benediktschmitt/py-filelock/issues/67.

I added the `-o flock` option when mounting the filesystem, as [described here](https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started-step2.html), which fixed the issue.