SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Possible caching bug

Comment:
Hi, does this bug be fixed? when I load JSON files, I get the same errors by the command 
`!python3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/`

change the dateset to load json by refering to https://huggingface.co/docs/datasets/loading.html
`dataset = datasets.load_dataset('json', data_files=args.dataset)`

Errors:
`Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/json/default-c1e124ad488911b8/0.0.0/45636811569ec4a6630521c18235dfbbab83b7ab572e3393c5ba68ccabe98264...
`
Issue: Possible caching bug

Comment:
```ds = load_dataset("csv", data_files={'train': 'train.csv', 'test': 'test.csv'})```

Gives the output
```Using custom data configuration default-5c8ae7c208631aca```

and the code hangs there.
Issue: Possible caching bug

Comment:
> `ds = load_dataset("csv", data_files={'train': 'train.csv', 'test': 'test.csv'})`
> 
> Gives the output `Using custom data configuration default-5c8ae7c208631aca`
> 
> and the code hangs there.

Have you solved it? I met this problem too!
Issue: Possible caching bug

Comment:
Can you Ctrl+C to kill the process and share the stacktrace here ? It should show at which location in the code it was hanging
Issue: Possible caching bug

Comment:
I had the same issue and solved it by downgrading the datasets version from 2.7.0 -> 2.6.1
pip install -q datasets==2.6.1
Issue: Possible caching bug

Comment:
> I had the same issue and solved it by downgrading the datasets version from 2.7.0 -> 2.6.1 pip install -q datasets==2.6.1

Thanks, it works for me
Issue: "Checksums didn't match for dataset source files" error while loading openwebtext dataset

Comment:
Hi try, to provide more information please.

Example code in a colab to reproduce the error, details on what you are trying to do and what you were expected and details on your environment (OS, PyPi packages version).
Issue: "Checksums didn't match for dataset source files" error while loading openwebtext dataset

Comment:
> Hi try, to provide more information please.
> 
> Example code in a colab to reproduce the error, details on what you are trying to do and what you were expected and details on your environment (OS, PyPi packages version).

I have update the description, sorry for the incomplete issue by mistake.
Issue: "Checksums didn't match for dataset source files" error while loading openwebtext dataset

Comment:
Hi, I have manually downloaded the compressed dataset `openwebtext.tar.xz' and use the following command to preprocess the examples:
```
>>> dataset = load_dataset('/home/admin/workspace/datasets/datasets-master/datasets-master/datasets/openwebtext', data_dir='/home/admin/workspace/datasets')
Using custom data configuration default
Downloading and preparing dataset openwebtext/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02...
Dataset openwebtext downloaded and prepared to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02. Subsequent calls will reuse this data.
>>> len(dataset['train'])
74571
>>>
```
The size of the pre-processed example file is only 354MB, however the processed bookcorpus dataset is 4.6g. Are there any problems?
Issue: "Checksums didn't match for dataset source files" error while loading openwebtext dataset

Comment:
NonMatchingChecksumError: Checksums didn't match for dataset source files:

i got this issue when i try to work on my own datasets kindly tell me, from where i can get checksums of train and dev file in my github repo
Issue: "Checksums didn't match for dataset source files" error while loading openwebtext dataset

Comment:
Hi,  I got the similar issue for xnli dataset while working on colab with python3.7.  

`nlp.load_dataset(path = 'xnli')`

The above command resulted in following issue : 
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']
```

Any idea how to fix this  ?
Issue: "Checksums didn't match for dataset source files" error while loading openwebtext dataset

Comment:
Says fixed but I'm still getting it. 

command:

 dataset = load_dataset("ted_talks_iwslt", language_pair=("en", "es"), year="2014",download_mode="force_redownload")

got:

Using custom data configuration en_es_2014-35a2d3350a0f9823
Downloading and preparing dataset ted_talks_iwslt/en_es_2014 (download: 2.15 KiB, generated: Unknown size, post-processed: Unknown size, total: 2.15 KiB) to /home/ken/.cache/huggingface/datasets/ted_talks_iwslt/en_es_2014-35a2d3350a0f9823/1.1.0/43935b3fe470c753a023642e1f54b068c590847f9928bd3f2ec99f15702ad6a6...
Downloading:
2.21k/? [00:00<00:00, 141kB/s]

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?id=1Cz1Un9p8Xn9IpEMMrg2kXSDt0dnjxc4z&export=download']
Issue: need to redirect /nlp to /datasets and remove outdated info

Comment:
Should be fixed now: 

![image](https://user-images.githubusercontent.com/35882/95917301-040b0600-0d78-11eb-9655-c4ac0e788089.png)

Not sure I understand what you mean by the second part?
Issue: need to redirect /nlp to /datasets and remove outdated info

Comment:
Thank you!

> Not sure I understand what you mean by the second part?

Compare the 2:
* https://huggingface.co/datasets/wikihow
* https://huggingface.co/nlp/viewer/?dataset=wikihow&config=all
Can you see the difference? 2nd has formatting, 1st doesn't.
Issue: need to redirect /nlp to /datasets and remove outdated info

Comment:
For context, those are two different pages (not an old vs new one), one is from the dataset viewer (you can browse data inside the datasets) while the other is just a basic reference page displayed some metadata about the dataset.

For the second one, we'll move to markdown parsing soon, so it'll be formatted better.
Issue: Adding pseudo-labels to datasets

Comment:
Nice ! :)
It's indeed the first time we have such contributions so we'll have to figure out the appropriate way to integrate them.
Could you add details on what they could be used for ?
Issue: Adding pseudo-labels to datasets

Comment:
A new configuration for those datasets should do the job then.
Note that until now datasets like xsum only had one configuration. It means that users didn't have to specify the configuration name when loading the dataset. If we add new configs, users that update the lib will have to update their code to specify the default/standard configuration name (not the one with pseudo labels).
Issue: Adding pseudo-labels to datasets

Comment:
Oh yes why not. I'm more in favor of this actually since pseudo labels are things that users (not dataset authors in general) can compute by themselves and share with the community
Issue: Adding pseudo-labels to datasets

Comment:
![image](https://user-images.githubusercontent.com/6045025/96045248-b528a380-0e3f-11eb-9124-bd55afa031bb.png)

I assume I should (for example) rename the xsum dir, change the URL, and put the modified dir somewhere in S3?
Issue: Adding pseudo-labels to datasets

Comment:
You can use the `datasets-cli` to upload the folder with your version of xsum with the pseudo labels.

```
datasets-cli upload_dataset path/to/xsum
```
Issue: feat(dl_manager): add support for ftp downloads

Comment:
We only support http by default for downloading.
If you really need to use ftp, then feel free to use a library that allows to download through ftp in your dataset script (I see that you've started working on #722 , that's awesome !). The users will get a message to install the extra library when they load the dataset.

To make the download_manager work with a custom downloader, you can call `download_manager.download_custom` instead of `download_manager.download_and_extract`. The expected arguments are the following:
```
url_or_urls: url or `list`/`dict` of urls to download and extract. Each
        url is a `str`.
custom_download: Callable with signature (src_url: str, dst_path: str) -> Any
        as for example `tf.io.gfile.copy`, that lets you download from google storage
```
Issue: feat(dl_manager): add support for ftp downloads

Comment:
Also maybe it coud be interesting to have a direct support of ftp inside the `datasets` library. Do you know any good libraries that we might consider adding as a (optional ?) dependency ?
Issue: feat(dl_manager): add support for ftp downloads

Comment:
Downloading an `ftp` file is as simple as:
```python
import urllib 
urllib.urlretrieve('ftp://server/path/to/file', 'file')
```

I believe this should be supported by the library, as its not using any dependency and is trivial amount of code.
Issue: feat(dl_manager): add support for ftp downloads

Comment:
I know its unorthodox, but I added `ftp` download support to `file_utils` in the same PR https://github.com/huggingface/datasets/pull/722
So its possible to understand the interaction of the download component with the ftp download ability
Issue: feat(dl_manager): add support for ftp downloads

Comment:
@hoanganhpham1006 yes.
See pull request https://github.com/huggingface/datasets/pull/722 , it has a loader for this dataset, mostly ready.
There's one issue that delays it being merged - https://github.com/huggingface/datasets/issues/741 - regarding memory consumption.
Issue: feat(dl_manager): add support for ftp downloads

Comment:
The problem which I have now is that this dataset seems does not allow to download? Can you share it with me pls
Issue: feat(dl_manager): add support for ftp downloads

Comment:
The dataset loader is not yet ready, because of that issue.
If you want to just download the dataset the old-fashioned way, just go to: https://www-i6.informatik.rwth-aachen.de/ftp/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz (the ftp link is now broken, and its available over https)
Issue: OSError: Cannot find data file when not using the dummy dataset in RAG

Comment:
An update on my end. This seems like a transient issue. Reran the script from scratch overnight with no errors.
Issue: How to use similarity settings  other then "BM25" in Elasticsearch index ?

Comment:
Datasets does not use elasticsearch API to define custom similarity. If you want to use a custom similarity, the best would be to run a curl request directly to your elasticsearch instance (see sample hereafter, directly from ES documentation), then you should be able to use `my_similarity` in your configuration passed to datasets

```
curl -X PUT "localhost:9200/index?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index": {
      "similarity": {
        "my_similarity": {
          "type": "DFR",
          "basic_model": "g",
          "after_effect": "l",
          "normalization": "h2",
          "normalization.h2.c": "3.0"
        }
      }
    }
  }
}
'

```
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
Facing a similar issue here. My model using SQuAD dataset takes about 1h to process with in memory data and more than 2h with datasets directly.
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
Thanks for the tip @thomwolf ! I did not see that flag in the docs. I'll try with that.
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
We should add it indeed and also maybe a specific section with all the tips for maximal speed. What do you think @lhoestq @SBrandeis @yjernite ?
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
By default the datasets loaded with `load_dataset` live on disk.
It's possible to load them in memory by using some transforms like `.map(..., keep_in_memory=True)`.

Small correction to @thomwolf 's comment above: currently we don't have the `keep_in_memory` parameter for `load_dataset` AFAIK but it would be nice to add it indeed :)
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
Great! Thanks a lot.

I did a test using `map(..., keep_in_memory=True)` and also a test using in-memory only data.

```python
features = dataset.map(tokenize, batched=True, remove_columns=dataset['train'].column_names)
features.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask'])

features_in_memory = dataset.map(tokenize, batched=True, keep_in_memory=True, remove_columns=dataset['train'].column_names)
features_in_memory.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask'])

in_memory = [features['train'][i] for i in range(len(features['train']))]
```

For using the features without any tweak, I got **1min17s** for copying the entire DataLoader to CUDA:

```
%%time

for i, batch in enumerate(DataLoader(features['train'], batch_size=16, num_workers=4)):
    batch['input_ids'].to(device)
```

For using the features mapped with `keep_in_memory=True`, I also got **1min17s** for copying the entire DataLoader to CUDA:

```
%%time

for i, batch in enumerate(DataLoader(features_in_memory['train'], batch_size=16, num_workers=4)):
    batch['input_ids'].to(device)
```

And for the case using every element in memory, converted from the original dataset, I got **12.5s**:

```
%%time

for i, batch in enumerate(DataLoader(in_memory, batch_size=16, num_workers=4)):
    batch['input_ids'].to(device)
```

Taking a closer look in my SQuAD code, using a profiler, I see a lot of calls to `posix read` api. It seems that it is really reliying on disk, which results in a very high train time.
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
I am having the same issue here. When loading from memory I can get the GPU up to 70% util but when loading after mapping I can only get 40%.

In disk:
```
book_corpus = load_dataset('bookcorpus', 'plain_text', cache_dir='/home/ad/Desktop/bookcorpus', split='train[:20%]')
book_corpus = book_corpus.map(encode, batched=True, num_proc=20, load_from_cache_file=True, batch_size=2500)
book_corpus.set_format(type='torch', columns=['text', "input_ids", "attention_mask", "token_type_ids"])

training_args = TrainingArguments(
    output_dir="./mobile_bert_big",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=16,
    save_steps=50,
    save_total_limit=2,
    logging_first_step=True,
    warmup_steps=100,
    logging_steps=50,
    eval_steps=100,
    no_cuda=False,
    gradient_accumulation_steps=16,
    fp16=True)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=book_corpus,
    tokenizer=tokenizer)
```

In disk I can only get 0,17 it/s:
`[ 13/28907 01:03 < 46:03:27, 0.17 it/s, Epoch 0.00/1] `

If I load it with torch.utils.data.Dataset()
```
class BCorpusDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings

    def __getitem__(self, idx):
        item = [torch.tensor(val[idx]) for key, val in self.encodings.items()][0]
        return item

    def __len__(self):
        length = [len(val) for key, val in self.encodings.items()][0]
        return length

**book_corpus = book_corpus.select([i for i in range(16*2000)])** # filtering to not have 20% of BC in memory...
book_corpus = book_corpus(book_corpus)
```
I can get:
` [ 5/62 00:09 < 03:03, 0.31 it/s, Epoch 0.06/1]`

But obviously I can not get BookCorpus in memory xD

EDIT: it is something weird. If i load in disk 1% of bookcorpus:
```
book_corpus = load_dataset('bookcorpus', 'plain_text', cache_dir='/home/ad/Desktop/bookcorpus', split='train[:1%]')
```

I can get 0.28 it/s, (the same that in memory) but if I load 20% of bookcorpus:
```
book_corpus = load_dataset('bookcorpus', 'plain_text', cache_dir='/home/ad/Desktop/bookcorpus', split='train[:20%]')
```
I get again 0.17 it/s. 

I am missing something? I think it is something related to size, and not disk or in-memory.
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
There is a way to increase the batches read from memory? or multiprocessed it? I think that one of two or it is reading with just 1 core o it is reading very small chunks from disk and left my GPU at 0 between batches
Issue: Datasets performance slow? - 6.4x slower than in memory dataset

Comment:
My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.
Issue: Requirements should specify pyarrow<1

Comment:
@punitaojha, certainly. Feel free to work on this. Let me know if you need any help or clarity.
Issue: Requirements should specify pyarrow<1

Comment:
Hello @mathcass 
1. I did fork the repository and clone the same on my local system. 

2. Then learnt about how we can publish our package on pypi.org. Also, found some instructions on same in setup.py documentation.

3. Then I Perplexity document link that you shared above. I created a colab link from there keep both tensorflow and pytorch means a mixed option and tried to run it in colab but I encountered no errors at a point where you mentioned. Can you help me to figure out the issue. 

4.Here is the link of the colab file with my saved responses. 
https://colab.research.google.com/drive/1hfYz8Ira39FnREbxgwa_goZWpOojp2NH?usp=sharing
Issue: Requirements should specify pyarrow<1

Comment:
Thanks for looking at this @punitaojha and thanks for sharing the notebook. 

I just tried to reproduce this on my own (based on the environment where I had this issue) and I can't reproduce it somehow. If I run into this again, I'll include some steps to reproduce it. I'll close this as invalid. 

Thanks again.
Issue: Requirements should specify pyarrow<1

Comment:
I am sorry for hijacking this closed issue, but I believe I was able to reproduce this very issue. Strangely enough, it also turned out that running `pip install "pyarrow<1" --upgrade` did indeed fix the issue (PyArrow was installed in version `0.14.1` in my case).

Please see the Colab below:

https://colab.research.google.com/drive/15QQS3xWjlKW2aK0J74eEcRFuhXUddUST

Thanks!
Issue: TypeError: '<' not supported between instances of 'NamedSplit' and 'NamedSplit'

Comment:
Hi !
Thanks for reporting :) 
Indeed this is an issue on the `datasets` side.
I'm creating a PR
Issue: XNLI dataset is not loading 

Comment:
also i tried below code to solve checksum error 
`datasets-cli test ./datasets/xnli --save_infos --all_configs`

and it shows 

```
2020-10-02 07:06:16.588760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 268, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 474, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/./datasets/xnli/xnli.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 279, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 474, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/xnli/xnli.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/datasets-cli", line 36, in <module>
    service.run()
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 76, in run
    module_path, hash = prepare_module(path)
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 283, in prepare_module
    combined_path, github_file_path, file_path
FileNotFoundError: Couldn't find file locally at ./datasets/xnli/xnli.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/./datasets/xnli/xnli.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/xnli/xnli.py
```
Issue: XNLI dataset is not loading 

Comment:
Hi !
Yes the download url changed.
It's updated on the master branch. I'm doing a release today to fix that :)
Issue: XNLI dataset: NonMatchingChecksumError

Comment:
Thanks for reporting.
The data file must have been updated by the host.
I'll update the checksum with the new one.
Issue: XNLI dataset: NonMatchingChecksumError

Comment:
I'll do a release in the next few days to make the fix available for everyone.
In the meantime you can load `xnli` with
```
xnli = load_dataset('xnli', script_version="master")
```
This will use the latest version of the xnli script (available on master branch), instead of the old one.
Issue: `ArrowInvalid` occurs while running `Dataset.map()` function

Comment:
Hi !

This is because `encode` expects one single text as input (str), or one tokenized text (List[str]).
I believe that you actually wanted to use `encode_batch` which expects a batch of texts.
However this method is only available for our "fast" tokenizers (ex: BertTokenizerFast).
BertJapanese is not one of them unfortunately and I don't think it will be added for now (see https://github.com/huggingface/transformers/pull/7141)...
cc @thomwolf for confirmation.

Therefore what I'd suggest for now is disable batching and process one text at a time using `encode`.
Note that you can make it faster by using multiprocessing:

```python
num_proc = None  # Specify here the number of processes if you want to use multiprocessing. ex: num_proc = 4
encoded = train_ds.map(
    lambda example: {'tokens': t.encode(example['title'], max_length=1000)}, num_proc=num_proc
)
```
Issue: `ArrowInvalid` occurs while running `Dataset.map()` function

Comment:
Thank you very much for the kind and precise suggestion!
I'm looking forward to seeing BertJapaneseTokenizer built into the "fast" tokenizers.

I tried `map` with multiprocessing as follows, and it worked!

```python
# There was a Pickle problem if I use `lambda` for multiprocessing
def encode(examples):
    return {'tokens': t.encode(examples['title'], max_length=1000)}

num_proc = 8
encoded = train_ds.map(encode, num_proc=num_proc)
```

Thank you again!
Issue: Dataset browser url is still https://huggingface.co/nlp/viewer/

Comment:
Yes! might do it with @srush one of these days. Hopefully it won't break too many links (we can always redirect from old url to new)
Issue: The download instructions for c4 datasets are not contained in the error message

Comment:
Also not that C4 is a dataset that needs an Apache Beam runtime to be generated.
For example Dataflow, Spark, Flink etc.

Usually we generate the dataset on our side once and for all, but we haven't done it for C4 yet.
More info about beam datasets [here](https://huggingface.co/docs/datasets/beam_dataset.html)

Let me know if you have any questions
Issue: train_test_split returns empty dataset item

Comment:
Can you reproduce this example in a Colab so we can investigate? (or give more information on your software/hardware config)