SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: [Dataset] `NonMatchingChecksumError` due to an update in the LinCE benchmark data

Comment:
Hi @thomwolf 

Thanks for the details! I just created a PR with the updated `dataset_infos.json` file (#550).
Issue: [Dataset] RACE dataset Checksums error

Comment:
`NonMatchingChecksumError` means that the checksum of the downloaded file is not the expected one.
Either the file you downloaded was corrupted along the way, or the host updated the file.
Could you try to clear your cache and run `load_dataset` again ? If the error is still there, it means that there was an update in the data, and we may have to update the expected checksum value.
Issue: [Dataset] RACE dataset Checksums error

Comment:
I just cleared the cache an run it again. The error persists ):

```
 nlp (master) $ rm -rf  /Users/abarbosa/.cache/huggingface/
 nlp (master) $ python
Python 3.8.5 (default, Aug  5 2020, 03:39:04)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nlp
>>> dataset = nlp.load_dataset("race")
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.39k/4.39k [00:00<00:00, 661kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.81k/1.81k [00:00<00:00, 644kB/s]
Using custom data configuration default
Downloading and preparing dataset race/default (download: 84.52 MiB, generated: 132.61 MiB, post-processed: Unknown size, total: 217.13 MiB) to /Users/abarbosa/.cache/huggingface/datasets/race/default/0.1.0/5461327f1a83549ca0d845a3159c806d2baf4f8d0d8f7d657157ce7cdf3899c2...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.4M/25.4M [01:03<00:00, 401kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/abarbosa/Documents/nlp/src/nlp/load.py", line 550, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/abarbosa/Documents/nlp/src/nlp/builder.py", line 471, in download_and_prepare
    self._download_and_prepare(
  File "/Users/abarbosa/Documents/nlp/src/nlp/builder.py", line 530, in _download_and_prepare
    verify_checksums(
  File "/Users/abarbosa/Documents/nlp/src/nlp/utils/info_utils.py", line 38, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
nlp.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://www.cs.cmu.edu/~glai1/data/race/RACE.tar.gz']
>>>
```
Issue: [Dataset] RACE dataset Checksums error

Comment:
Dealing with the same issue please update the checksum on nlp library end. The data seems to have changed on their end.
Issue: [Dataset] RACE dataset Checksums error

Comment:
We have a discussion on this datasets here: https://github.com/huggingface/nlp/pull/540

Feel free to participate if you have some opinion on the scope of data which should be included in this dataset.
Issue: [Dataset] RACE dataset Checksums error

Comment:
At least for me, the file that was downloaded from CMU isn't the complete dataset, but a small subset of it (~25MB vs ~85MB). I've previously downloaded the dataset directly, so for my personal needs I could just swap out the corrupted file with the correct one. Perhaps you could host it like you do for the Wikipedia and BookCorpus datasets.
Issue: [Dataset] RACE dataset Checksums error

Comment:
> At least for me, the file that was downloaded from CMU isn't the complete dataset, but a small subset of it (~25MB vs ~85MB). I've previously downloaded the dataset directly, so for my personal needs I could just swap out the corrupted file with the correct one. Perhaps you could host it like you do for the Wikipedia and BookCorpus datasets.

Could you upload this please?
Issue: [Dataset] RACE dataset Checksums error

Comment:
> > At least for me, the file that was downloaded from CMU isn't the complete dataset, but a small subset of it (~25MB vs ~85MB). I've previously downloaded the dataset directly, so for my personal needs I could just swap out the corrupted file with the correct one. Perhaps you could host it like you do for the Wikipedia and BookCorpus datasets.
> 
> Could you upload this please?

Not sure if I can upload it according to their license ("You agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purpose, any portion of the contexts and any portion of derived data.").
Issue: `list_datasets()` is broken.

Comment:
Thanks for reporting !
This has been fixed in #475 and the fix will be available in the next release
Issue: `list_datasets()` is broken.

Comment:
What you can do instead to get the list of the datasets is call

```python
print([dataset.id for dataset in nlp.list_datasets()])
```
Issue: File exists error when used with TPU

Comment:
Could you try to run `dataset = load_dataset("text", data_files=file_path, split="train")` once before calling the script ?

It looks like several processes try to create the dataset in arrow format at the same time. If the dataset is already created it should be fine
Issue: File exists error when used with TPU

Comment:
Thanks! I tested on 328MB text data on `n1-standard-8 (8 vCPUs, 30 GB memory)`. The main script ran without any issue, but it seems to require a huge space in the drive.

As suggested, I ran the following script before running the pre-training command with `xla_spawn.py`.

```python
from nlp import load_dataset

file_path="your_file_name"
load_dataset("text", data_files=file_path, split="train")
```
This will create `text-train.arrow` under the default cache directory. Then, I run the script with `xla_spawn.py`. It will load data from the cached file. My understanding is that there's no other way but to do this two-step process with the current version (0.4) of `nlp`.

During another caching process that happens in the main script:

```
08/26/2020 09:19:51 - INFO - nlp.utils.info_utils -   All the checksums matched successfully for post processing resources
08/26/2020 09:19:53 - INFO - nlp.arrow_dataset -   Caching processed dataset at /home/*****/.cache/huggingface/datasets/text/default-b0932b2bdbb63283/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d/cache-f90f341e5308a7469
8d872bcc88f9c0e.arrow
```

`nlp` generates a temporary file per core, each of which is three times larger than the original text data. If each process is actually writing on the disk, you will need a huge amount of space in your drive. (Maybe I'm missing something.)

```
-rw-r--r-- 1 ***** *****  674 Aug 26 09:19 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 09:19 LICENSE
-rw-r--r-- 1 ***** ***** 332M Aug 26 09:10 text-train.arrow
-rw------- 1 ***** ***** 940M Aug 26 09:31 tmp0k43sazw
-rw------- 1 ***** ***** 940M Aug 26 09:31 tmp7sxs9mj5
-rw------- 1 ***** ***** 939M Aug 26 09:31 tmpbbiqw2vp
-rw------- 1 ***** ***** 937M Aug 26 09:31 tmpjxb5ptyu
-rw------- 1 ***** ***** 933M Aug 26 09:31 tmpk3hkdh0e
-rw------- 1 ***** ***** 944M Aug 26 09:31 tmpnoalwftz
-rw------- 1 ***** ***** 931M Aug 26 09:31 tmpuxdr_dz3
-rw------- 1 ***** ***** 945M Aug 26 09:31 tmpxjyuy6dk
```
After the caching process, they seem to be merged into one file.

```
-rw------- 1  ***** ***** 989M Aug 26 09:32 cache-f90f341e5308a74698d872bcc88f9c0e.arrow
-rw-r--r-- 1  ***** *****  674 Aug 26 09:19 dataset_info.json
-rw-r--r-- 1  ***** *****    0 Aug 26 09:19 LICENSE
-rw-r--r-- 1  ***** ***** 332M Aug 26 09:10 text-train.arrow
```
Issue: File exists error when used with TPU

Comment:
Again it looks like every process tries to tokenize the full dataset at the same time.
If you do the tokenization before calling `xla_spawn.py` once, then each process will then use the tokenized cached file `cache-f90f341e5308a74698d872bcc88f9c0e.arrow` and not recompute it.

Not sure if there's a better way to do that cc @julien-c @thomwolf
Issue: File exists error when used with TPU

Comment:
I wrote a separate script just for preparing a cached file, including tokenization. Each process did use the tokenized cached file.

Currently I'm testing the pipeline on 24GB text data. It took about 1.5 hour to create a cached file on `n1-highmem-16 (16 vCPUs, 104 GB memory)`. I assume loading this cached file in the main script with `xla_spawn.py` won't be an issue (even if there are 8 processes).

```
total 98G
drwxr-xr-x 2 ***** ***** 4.0K Aug 26 13:38 .
drwxr-xr-x 3 ***** ***** 4.0K Aug 26 12:24 ..
-rw------- 1 ***** *****  74G Aug 26 13:38 cache-a7aa04134ba7b1aff5d9710f14a4e334.arrow
-rw-r--r-- 1 ***** *****  681 Aug 26 12:24 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 12:24 LICENSE
-rw-r--r-- 1 ***** *****  25G Aug 26 12:24 text-train.arrow
```
Issue: File exists error when used with TPU

Comment:
Sorry, I thought it was working, but actually the second call doesn't use the cached file that was generated separately, and it will generate another cache-****.arrorw file with a different name. If I run the training script again (with `xla_spawn.py`), it will use the second cached file, which was generated by the training script itself in the previous run.

```
drwxr-xr-x 2 ***** ***** 4.0K Aug 26 15:35 .
drwxr-xr-x 3 ***** ***** 4.0K Aug 26 15:29 ..
-rw------- 1 ***** *****  99M Aug 26 15:35 cache-0d77dfce704493dbe63f071eed6a5431.arrow
-rw------- 1 ***** *****  99M Aug 26 15:29 cache-69633651476e943b93c89ace715f9487.arrow
-rw-r--r-- 1 ***** *****  670 Aug 26 15:33 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 15:33 LICENSE
-rw-r--r-- 1 ***** *****  33M Aug 26 15:29 text-train.arrow
```
Issue: File exists error when used with TPU

Comment:
So if I understand correctly it means that the cached file generated by your separated script is different by the one used by the training script ?
Issue: File exists error when used with TPU

Comment:
Yes.

1. `cache-69633651476e943b93c89ace715f9487.arrow` was generated with a separate script. 
2. I ran the entire script with `xla_spawn.py`.
3. `cache-69633651476e943b93c89ace715f9487.arrow` is not used.
4. `cache-0d77dfce704493dbe63f071eed6a5431.arrow` is created.
5. training starts...

Now, if I kill the process at step 5, and do the step 2 again, it will use `cache-0d77dfce704493dbe63f071eed6a5431.arrow` (cached file created at step 4) without any issue.

I used the following to generate the first cached file.
```python
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=args.block_size), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
```
Issue: File exists error when used with TPU

Comment:
Thanks for all the details.
The two cached files are supposed to be the same. I suspect that the caching has a problem with the tokenizer.
Which tokenizer did you use ?
Issue: File exists error when used with TPU

Comment:
I trained a byte-level BPE tokenizer on my data with `tokenziers` library following this [example](https://github.com/huggingface/tokenizers/blob/master/bindings/python/examples/train_bytelevel_bpe.py).

And I put these model files in a directory named `"model_name"`. I also put config.json, which is the original RoBERTa config file.

```bash
%ls  model_name
config.json     merges.txt      vocab.json
```

[This](https://github.com/huggingface/transformers/blob/4bd7be9a4268221d2a0000c7e8033aaeb365c03b/examples/language-modeling/run_language_modeling.py#L196) is the line where `run_language_modeling.py` loads the tokenier.

```python
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
```

I use `"model_name"` for `model_args.tokenizer_name`. I don't specify `model_args.cache_dir`. It is 'None' by default.
Issue: File exists error when used with TPU

Comment:
In my separated script for caching, I'm using `use_fast=True` when initializing a tokenizer.

```python
tokenizer = AutoTokenizer.from_pretrained(args.config_name, use_fast=True)
```
I wasn't using that option in the main script. That could be the reason...
Issue: File exists error when used with TPU

Comment:
Yea it could definitely explain why you have two different cache files.
Let me know if using the same tokenizers on both sides fixes the issue
Issue: File exists error when used with TPU

Comment:
It still creates a new file even if I remove `use_fast=True`... 

Here's the script used to create a cached file.
```python 
#!/usr/bin/env python3

import argparse

from transformers import AutoTokenizer

from nlp import load_dataset


def main():
    parser = argparse.ArgumentParser(description='description')
    parser.add_argument('--config_name', type=str, help='Pretrained config name or path if not the same as model_name')
    parser.add_argument('--data_file', type=str, help='The input data file (a text file).')
    parser.add_argument('--block_size', type=int, default=-1, help='The training dataset will be truncated in block of this size for training')
    args = parser.parse_args()

    tokenizer = AutoTokenizer.from_pretrained(args.config_name)

    dataset = load_dataset("text", data_files=args.data_file, split="train")
    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                truncation=True, max_length=args.block_size), batched=True)
    dataset.set_format(type='torch', columns=['input_ids'])


if __name__ == "__main__":
    main()
```

Here's how the data is loaded in the modified `run_language_modeling.py`. [[original function](https://github.com/huggingface/transformers/blob/971d1802d009d9996b36a34a34477cee849ef39f/examples/language-modeling/run_language_modeling.py#L128-L135)]

```python
def get_dataset(args: DataTrainingArguments, tokenizer: PreTrainedTokenizer, evaluate=False):
    file_path = args.eval_data_file if evaluate else args.train_data_file
    split = "validation" if evaluate else "train"
    if args.line_by_line:
        # return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
        dataset = load_dataset("text", data_files=file_path, split="train")
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                              truncation=True, max_length=args.block_size), batched=True)
        dataset.set_format(type='torch', columns=['input_ids'])
        return dataset

    else:
        return TextDataset(
            tokenizer=tokenizer, file_path=file_path, block_size=args.block_size, overwrite_cache=args.overwrite_cache
        )
```

Probably I don't need this part in the main script,

```python
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                              truncation=True, max_length=args.block_size), batched=True)
        dataset.set_format(type='torch', columns=['input_ids'])
```
and simply do this?
```python
dataset = load_dataset("text", data_files=file_path, split="train")
return dataset
```
Issue: File exists error when used with TPU

Comment:
You need this part in the main script or it will use the dataset that is not tokenized
Issue: File exists error when used with TPU

Comment:
I can see that the tokenizer in `run_language_modeling.py` is not instantiated the same way as in your separated script.
Indeed we can see L196:
```python
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
```
Could you try to make it so they are instantiated the exact same way please ?
Issue: File exists error when used with TPU

Comment:
I updated my separated script, but it's creating a cached file again. If I don't use the `model_args.cache_dir`, both will get `None`, so they should be the same.

```python
#!/usr/bin/env python3
import argparse

from transformers import AutoTokenizer
from nlp import load_dataset

def main():
    parser = argparse.ArgumentParser(description='description')
    parser.add_argument('--tokenizer_name', type=str, help='Pretrained tokenizer name or path if not the same as model_name')
    parser.add_argument('--data_file', type=str, help='The input data file (a text file).')
    parser.add_argument('--cache_dir', type=str, default=None, help='Where do you want to store the pretrained models downloaded from s3')
    parser.add_argument('--block_size', type=int, default=-1, help='The training dataset will be truncated in block of this size for training')

    model_args = parser.parse_args()

    tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)

    dataset = load_dataset("text", data_files=model_args.data_file, split="train")
    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                truncation=True, max_length=model_args.block_size), batched=True)
    dataset.set_format(type='torch', columns=['input_ids'])

if __name__ == "__main__":
    main()
```

Is there a way to specify the cache file to load, and skip the re-computation?
Issue: File exists error when used with TPU

Comment:
Could you also check that the `args.block_size` used in the lambda function is the same as well ?
Issue: File exists error when used with TPU

Comment:
I ended up specifying the `cache_file_name` argument when I call `map` function.

```python
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True, truncation=True, max_length=args.block_size),
                      batched=True,
                      cache_file_name=cache_file_name)
```

Note:
- `text` dataset in `nlp` does not strip `"\n"`.  If you want the same output as in [`LineByLineTextDataset`](https://github.com/huggingface/transformers/blob/afc4ece462ad83a090af620ff4da099a0272e171/src/transformers/data/datasets/language_modeling.py#L88-L111), you would need to create your own dataset class where you replace `line` to `line.strip()` [here](https://github.com/huggingface/nlp/blob/master/datasets/text/text.py#L35).
Issue: wmt download speed example

Comment:
Thanks for creating the issue :)
The download link for wmt-en-de raw looks like a mirror. We should use that instead of the current url.
Is this mirror official ?

Also it looks like for `ro-en` it tried to download other languages. If we manage to only download the one that is asked it'd be cool

Also cc @patrickvonplaten
Issue: wmt download speed example

Comment:
Shall we host the files ourselves or it is fine to use this mirror in your opinion ?
Issue: wmt download speed example

Comment:
Should we add an argument in `load_dataset` to override some URL with a custom URL (e.g. mirror) or a local path?

This could also be used to provide local files instead of the original files as requested by some users (e.g. when you made a dataset with the same format than SQuAD and what to use it instead of the official dataset files).
Issue: wmt download speed example

Comment:
@lhoestq I think we should host it ourselves. I'll put the subset of wmt (without preprocessed files) that we need on s3 and post a link over the weekend.
Issue: wmt download speed example

Comment:
Is there a solution yet? The download speed is still too slow. 60-70kbps download for wmt16 and around 100kbps for wmt19. @sshleifer
Issue: [BUG] Metrics throwing new error on master since 0.4.0

Comment:
Update - maybe this is only failing on bleu because I was not tokenizing inputs to the metric
Issue: [BUG] Metrics throwing new error on master since 0.4.0

Comment:
Closing - seems to be just forgetting to tokenize. And found the helpful discussion in huggingface/evaluate#105
Issue: add MLDoc dataset

Comment:
This request is still an open issue waiting to be addressed by any community member, @GuillemGSubies.
Issue: dataset.shuffle(keep_in_memory=True) is never allowed

Comment:
This seems to be fixed in #513 for the filter function, replacing `cache_file_name` with `indices_cache_file_name` in the assert. Although not for the `map()` function @thomwolf
Issue: dataset.shuffle(keep_in_memory=True) is never allowed

Comment:
Maybe I'm a bit tired but I fail to see the issue here.

Since `cache_file_name` is `None` by default, if you set `keep_in_memory` to `True`, the assert should pass, no?
Issue: dataset.shuffle(keep_in_memory=True) is never allowed

Comment:
I failed to realise that this only applies to `shuffle()`. Whenever `keep_in_memory` is set to True, this is passed on to the `select()` function. However, if `cache_file_name` is None, it will be defined in the `shuffle()` function before it is passed on to `select()`. 

Thus, `select()` is called with `keep_in_memory=True` and a not None value for `cache_file_name`. 
This is essentially fixed in #513 

Easily reproducible:
```python
>>> import nlp
>>> data = nlp.load_dataset("cosmos_qa", split="train")
Using custom data configuration default
>>> data.shuffle(keep_in_memory=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 1398, in shuffle
    verbose=verbose,
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 1178, in select
    ), "Please use either `keep_in_memory` or `cache_file_name` but not both."
AssertionError: Please use either `keep_in_memory` or `cache_file_name` but not both.
>>>data.select([0], keep_in_memory=True)
# No error
```
Issue: dataset.shuffle(keep_in_memory=True) is never allowed

Comment:
Oh yes ok got it thanks. Should be fixed if we are happy with #513 indeed.
Issue: dataset.shuffle(keep_in_memory=True) is never allowed

Comment:
My bad. This is actually not fixed in #513. Sorry about that...
The new `indices_cache_file_name` is set to a non-None value in the new `shuffle()` as well. 

The buffer and caching mechanisms used in the `select()` function are too intricate for me to understand why the check is there at all. I've removed it in my local build and it seems to be working fine for my project, without really considering other implications of the change.
Issue: dataset.shuffle(keep_in_memory=True) is never allowed

Comment:
Ok I'll investigate and add a series of tests on the `keep_in_memory=True` settings which is under-tested atm
Issue: dataset.shuffle(keep_in_memory=True) is never allowed

Comment:
These are the steps needed to fix this issue:
1. add the following check to `Dataset.shuffle`:
```python
if keep_in_memory and indices_cache_file_name is not None:
    raise ValueError("Please use either `keep_in_memory` or `indices_cache_file_name` but not both.")
```
2. set `indices_cache_file_name` to `None` if `keep_in_memory` is True in the call to `select`
3. add a test with `shuffle(keep_in_memory=True)`
Issue: dataset.shuffle() and select() resets format. Intended?

Comment:
Hi @vegarab yes feel free to open a discussion here.

This design choice was not very much thought about.

Since `dataset.select()` (like all the method without a trailing underscore) is non-destructive and returns a new dataset it has most of its properties initialized from scratch (except the table and infos).

Thinking about it I don't see a strong reason against transmitting the format from the parent dataset to its newly created child. It's probably what's expected by the user in most cases. What do you think @lhoestq?

By the way, I've been working today on a refactoring of all the samples re-ordering/selection methods (`select`, `sort`, `shuffle`, `shard`, `train_test_split`). The idea is to speed them up by a lot (like, really a lot) by working as much as possible with an indices mapping table instead of doing a deep copy of the full dataset as we've been doing currently. You can give it a look and try it here: https://github.com/huggingface/nlp/pull/513
Feedbacks are very much welcome