SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Error loading C4 realnewslike dataset

Comment:
@bhavitvyamalik @lhoestq , just tried the above and got:
>>> a=datasets.load_dataset('c4','en.realnewslike')
Downloading: 3.29kB [00:00, 1.66MB/s]                                                                                                                                                                                                                                          
Downloading: 2.40MB [00:00, 12.6MB/s]                                                                                                                                                                                                                                          
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 819, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/load.py", line 701, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 1049, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 268, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/dshirron/.local/lib/python3.8/site-packages/datasets/builder.py", line 360, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig en.realnewslike not found. Available: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']
>>> 

datasets version is 1.11.0
Issue: Error loading C4 realnewslike dataset

Comment:
I think I had an older version of datasets installed and that's why I commented the old configurations in my last comment, my bad! I re-checked and updated it to latest version (`datasets==1.11.0`) and it's showing `available configs: ['en', 'realnewslike', 'en.noblocklist', 'en.noclean']`. 

I tried `raw_datasets = load_dataset('c4', 'realnewslike')` and the download started. Make sure you don't have any old copy of this dataset and you download it fresh using the latest version of datasets. Sorry for the mix up!
Issue: Error loading C4 realnewslike dataset

Comment:
It works. I probably had some issue with the cache. after cleaning it im able to download the dataset. Thanks
Issue: Unexpected type after `concatenate_datasets`

Comment:
Hi @JulesBelveze, thanks for your question.

Note that 🤗  `datasets` internally store their data in Apache Arrow format.

However, when accessing dataset columns, by default they are returned as native Python objects (lists in this case).

If you would like their columns to be returned in a more suitable format for your use case (torch arrays), you can use the method `set_format()`:
```python
concat_dataset.set_format(type="torch")
```

You have detailed information in our docs:
- [Using a Dataset with PyTorch/Tensorflow](https://huggingface.co/docs/datasets/torch_tensorflow.html)
- [Dataset.set_format()](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.set_format)
Issue: Second concatenation of datasets produces errors

Comment:
Hi @Aktsvigun! We are planning to address this issue before our next release, in a couple of weeks at most. 😅 

In the meantime, if you would like to contribute, feel free to open a Pull Request. You are welcome. Here you can find more information: [How to contribute to Datasets?](CONTRIBUTING.md)
Issue: Raise a proper exception when trying to stream a dataset that requires to manually download files

Comment:
Hi @severo, thanks for reporting.

As discussed, datasets requiring manual download should be:
- programmatically identifiable
- properly handled with more clear error message when trying to load them with streaming

In relation with programmatically identifiability, note that for datasets requiring manual download, their builder have a property `manual_download_instructions` which is not None:
```python
# Dataset requiring manual download:
builder.manual_download_instructions is not None
```
Issue: Cannot load `few-nerd` dataset

Comment:
Hi @Mehrad0711,

I'm afraid there is no "canonical" Hugging Face dataset named "few-nerd".

There are 2 kinds of datasets hosted at the Hugging Face Hub:
- canonical datasets (their identifier contains no slash "/"): we, the Hugging Face team, supervise their implementation and we make sure they work correctly by means of our test suite
- community datasets (their identifier contains a slash "/", where before the slash it is the username or the organization name): those datasets are uploaded to the Hub by the community, and we, the Hugging Face team, do not supervise them; it is the responsibility of the user/organization implementing them properly if they want them to be used by other users.

In this specific case, there is no "canonical" dataset named "few-nerd". On the other hand, there are two "community" datasets named "few-nerd":
- ["nbroad/few-nerd"](https://huggingface.co/datasets/nbroad/few-nerd)
- ["dfki-nlp/few-nerd"](https://huggingface.co/datasets/dfki-nlp/few-nerd)

If they were properly implemented, you should be able to load them this way:
```python
# "nbroad/few-nerd" community dataset
ds = load_dataset("nbroad/few-nerd", "supervised")

# "dfki-nlp/few-nerd" community dataset
ds = load_dataset("dfki-nlp/few-nerd", "supervised")
```

However, they are not correctly implemented and both of them give errors:
- "nbroad/few-nerd":
  ```
  TypeError: expected str, bytes or os.PathLike object, not dict
  ```
- "dfki-nlp/few-nerd":
  ```
  ConnectionError: Couldn't reach https://cloud.tsinghua.edu.cn/f/09265750ae6340429827/?dl=1
  ```

You could try to contact their users/organizations to inform them about their bugs and ask them if they are planning to fix them. Alternatively you could try to implement your own script for this dataset.
Issue: Cannot load `few-nerd` dataset

Comment:
Hello, @Mehrad0711; Hi, @albertvillanova  !
I am the maintainer of the `dfki/few-nerd" dataset script, sorry for the very late reply and hope this message finds you well!
We should use
```
dataset = load_dataset("dfki-nlp/few-nerd", name="supervised")
```
instead of not specifying the "name" argument, where name is from `["supervised", "inter", "intra"]`. Otherwise the method just treats "supervised" as `split`, which we reserve after specifying the name, since for each name, there are three splits: train, dev and test.

Also we use Tsinghua server source to download data files since it is the official source referred in the paper where the dataset is released (even though it is cc-by-sa-4.0 licensed, means we can copy the data anywhere after mentioning the license
). Sometimes the server just runs down due to high pressure, kinda weird (we encountered the same server problem serveral times a month when we conducted experiments on Few-NERD XD). I tried the script just now and it works perfectly!
```
>> dataset
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 131767
    })
    validation: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 18824
    })
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 37648
    })
})
>>> dataset["train"]
Dataset({
    features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
    num_rows: 131767
})
>>> dataset["train"][0]
{'id': '0', 'tokens': ['Paul', 'International', 'airport', '.'], 'ner_tags': [0, 0, 0, 0], 'fine_ner_tags': [0, 0, 0, 0]}
```
Anyways if you cannot stand the pain with the server and its slow download speed, you can also download the `dfki/few-nerd.py` script from HF and change the `_URLs` to your personal drive (after you once successfully download the data and upload to your cloud drive), and then load the .py script locally.

Hope this reply can still be any help. If you still have problems with it, feel free to ask here and I am glad to help!
Best wishes.
Issue: Cannot load `few-nerd` dataset

Comment:
Hi @chen-yuxuan, thanks for your answer.

Just a few comments:

- Please, note that as we use `datasets.load_dataset` implementation, we can pass the configuration name as the second positional argument (no need to pass explicitly `name=`) and it downloads the 3 splits:
```python
  In [4]: ds = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5k/11.5k [00:00<00:00, 2.85MB/s]
Downloading and preparing dataset few_nerd/supervised to .cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e40882b71f037a4a1f232025899170fbe8113cd2f4a26dddd2add7222a077255...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6M/14.6M [01:16<00:00, 190kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.9M/11.9M [01:14<00:00, 160kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.0M/12.0M [01:04<00:00, 186kB/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:58<00:00, 79.45s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.11it/s]
```

- On the other hand, please note that your script does not work on Windows machines, because you call `open()` without passing the encoding parameter:
```
~\.cache\huggingface\modules\datasets_modules\datasets\dfki-nlp___few-nerd\e40882b71f037a4a1f232025899170fbe8113cd2f4a26dddd2add7222a077255\few-nerd.py in <genexpr>(.0)
    276         assert filepath[-4:] == ".txt"
    277
--> 278         num_lines = sum(1 for _ in open(filepath))
    279         id = 0
    280

.venv\lib\encodings\cp1252.py in decode(self, input, final)
     21 class IncrementalDecoder(codecs.IncrementalDecoder):
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24
     25 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 5238: character maps to <undefined>
```

If you would like your script to be usable on Windows machines, you should pass `encoding="utf-8"` to every `open()` function:
- line 278: `num_lines = sum(1 for _ in open(filepath, encoding="utf-8"))`
- line 281: `with open(filepath, "r", encoding="utf-8")`
Issue: Cannot load `few-nerd` dataset

Comment:
Thank you @albertvillanova for your detailed feedback!

> no need to pass explicitly `name=`

Good catch! I thought `split` stands before `name` in the argument list... but now it is all clear to me, sounds cool! Thanks for the explanation.

Anyways in our old code it still looks bit confusing if we only want one split but the function downloads all, so to allow efficient downloading, I optimized the code a bit so that only the specified split data is downloaded. now we get
```
>>> x = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading and preparing dataset few_nerd/supervised to /home/user/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/8e7ab598946cd5b395dcec6ea239123c8dff5b58b8e1c03b0c595b540248a885...
Downloading: 100%|███████████████████████████████████████████████████████████████████| 14.6M/14.6M [01:01<00:00, 238kB/s]
100%|██████████████████████████████████████████████████████████████████████| 3359329/3359329 [00:12<00:00, 275462.84it/s]
100%|████████████████████████████████████████████████████████████████████████| 482037/482037 [00:01<00:00, 278633.64it/s]
100%|████████████████████████████████████████████████████████████████████████| 958765/958765 [00:03<00:00, 267472.83it/s]
Dataset few_nerd downloaded and prepared to /home/user/.cache/huggingface/datasets/few_nerd/supervised/0.0.0/8e7ab598946cd5b395dcec6ea239123c8dff5b58b8e1c03b0c595b540248a885. Subsequent calls will reuse this data.
```
where only one progress bar indicates downloading, and the three others just indicate pre-processing for the train, dev, test set.

For the encoding issue, I have made corresponding changes for the two lines you pointed out. However, I have no windows machine at hand, I would really appreciate it if you could help test on your end.

All the updates are uploaded to HF under `dfki-nlp` account where I am working for. 
Thank you again for your kind help!
Issue: Cannot load `few-nerd` dataset

Comment:
Hi @chen-yuxuan,

I have tested on Windows and now it works perfectly, after the fixing of the encoding issue:
```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("dfki-nlp/few-nerd", "supervised")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5k/11.5k [00:00<?, ?B/s]
Downloading and preparing dataset few_nerd/supervised to C:\Users\username\.cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e1ceeaee82073fea12206e4461c7cfcd67e68c8f3ebeca179bddcacee00c4511...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3359329/3359329 [00:25<00:00, 129427.23it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 482037/482037 [00:03<00:00, 134513.66it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 958765/958765 [00:06<00:00, 143152.35it/s]
Dataset few_nerd downloaded and prepared to C:\Users\username\.cache\huggingface\datasets\dfki-nlp___few_nerd\supervised\0.0.0\e1ceeaee82073fea12206e4461c7cfcd67e68c8f3ebeca179bddcacee00c4511. Subsequent calls will reuse this data.765 [00:06<00:00, 139045.03it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 174.71it/s]

In [3]: ds
Out[3]:
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 131767
    })
    validation: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 18824
    })
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags', 'fine_ner_tags'],
        num_rows: 37648
    })
})
```
Issue: Dataset JSON is incorrect

Comment:
As discussed, the metadata JSON files must be regenerated because the keys were nor properly generated and they will not be read by the builder:
> Indeed there is some problem/bug while reading the datasets_info.json file: there is a mismatch with the config.name keys in the file...
In the meanwhile, in order to be able to use the datasets_info.json file content, you can create the builder without passing the name :
```
In [25]: builder = datasets.load_dataset_builder("journalists_questions")
In [26]: builder.info.splits
Out[26]: {'train': SplitInfo(name='train', num_bytes=342296, num_examples=10077, dataset_name='journalists_questions')}
```

After regenerating the metadata JSON file for this dataset, I get the right key:
```
{"plain_text": {"description": "The journalists_questions corpus (
```
Issue: Improve detection of streamable file types

Comment:
maybe we should rather attempt to download a `Range` from the server and see if it works?
Issue: SacreBLEU update

Comment:
Hi @devrimcavusoglu, 
I tried your code with latest version of `datasets`and `sacrebleu==1.5.1` and it's running fine after changing one small thing:
```
sacrebleu = datasets.load_metric('sacrebleu')
predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
references = [["It is a guide to action that ensures that the military will forever heed Party commands"]]  # double brackets here should do the work
results = sacrebleu.compute(predictions=predictions, references=references)
print(results)
output: {'score': 41.180376356915765, 'counts': [11, 8, 6, 4], 'totals': [18, 17, 16, 15], 'precisions': [61.111111111111114, 47.05882352941177, 37.5, 26.666666666666668], 'bp': 1.0, 'sys_len': 18, 'ref_len': 16}
```
Issue: SacreBLEU update

Comment:
@bhavitvyamalik hmm. I forgot double brackets, but still didn't work when used it with double brackets. It may be an isseu with platform (using win-10 currently), or versions. What is your platform and your version info for datasets, python, and sacrebleu ?
Issue: SacreBLEU update

Comment:
You can check that here, I've reproduced your code in [Google colab](https://colab.research.google.com/drive/1X90fHRgMLKczOVgVk7NDEw_ciZFDjaCM?usp=sharing). Looks like there was some issue in `sacrebleu` which was fixed later from what I've found [here](https://github.com/pytorch/fairseq/issues/2049#issuecomment-622367967). Upgrading `sacrebleu` to latest version should work.
Issue: SacreBLEU update

Comment:
It seems that next release of `sacrebleu` (v2.0.0) will break our `datasets` implementation to compute it. See my Google Colab: https://colab.research.google.com/drive/1SKmvvjQi6k_3OHsX5NPkZdiaJIfXyv9X?usp=sharing

I'm reopening this Issue and making a Pull Request to fix it.
Issue: SacreBLEU update

Comment:
> It seems that next release of `sacrebleu` (v2.0.0) will break our `datasets` implementation to compute it. See my Google Colab: https://colab.research.google.com/drive/1SKmvvjQi6k_3OHsX5NPkZdiaJIfXyv9X?usp=sharing
> 
> I'm reopening this Issue and making a Pull Request to fix it.

How did you solve him
Issue: Add Microsoft Building Footprints dataset

Comment:
Motivation: this can be a useful dataset for researchers working on climate change adaptation, urban studies, geography, etc. I'll see if I can figure out how to add it!
Issue: Update CommonVoice with new release

Comment:
Does anybody know if there is a bundled link, which would allow direct data download instead of manual? 
Something similar to: `https://voice-prod-bundler-ee1969a6ce8178826482b88e843c335139bd3fb4.s3.amazonaws.com/cv-corpus-6.1-2020-12-11/ab.tar.gz` ? cc @patil-suraj
Issue: Concurrent use of same dataset (already downloaded)

Comment:
Launching simultaneous job relying on the same datasets try some writing issue. I guess it is unexpected since I only need to load some already downloaded file.
Issue: Concurrent use of same dataset (already downloaded)

Comment:
If i have two jobs that use the same dataset. I got :


  File "compute_measures.py", line 181, in <module>
    train_loader, val_loader, test_loader = get_dataloader(args)
  File "/gpfsdswork/projects/rech/toto/intRAOcular/dataset_utils.py", line 69, in get_dataloader
    dataset_train = load_dataset('paws', "labeled_final", split='train', download_mode="reuse_cache_if_exists")
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 582, in download_and_prepare
    self._save_info()
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 690, in _save_info
    self.info.write_to_directory(self._cache_dir)
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/info.py", line 195, in write_to_directory
    with open(os.path.join(dataset_info_dir, config.LICENSE_FILENAME), "wb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/gpfswork/rech/toto/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/LICENSE'
Issue: Concurrent use of same dataset (already downloaded)

Comment:
You can probably have a solution much faster than me (first time I use the library). But I suspect some write function are used when loading the dataset from cache.
Issue: Concurrent use of same dataset (already downloaded)

Comment:
I have the same issue:
```
Traceback (most recent call last):
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 1040, in _prepare_split
    with ArrowWriter(features=self.info.features, path=fpath) as writer:
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/arrow_writer.py", line 192, in __init__
    self.stream = pa.OSFile(self._path, "wb")
  File "pyarrow/io.pxi", line 829, in pyarrow.lib.OSFile.__cinit__
  File "pyarrow/io.pxi", line 844, in pyarrow.lib.OSFile._open_writable
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Failed to open local file '/dccstor/tslm-gen/.cache/csv/default-387f1f95c084d4df/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0.incomplete/csv-validation.arrow'. Detail: [errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/dccstor/tslm/elron/tslm-gen/train.py", line 510, in <module>
    main()
  File "/dccstor/tslm/elron/tslm-gen/train.py", line 246, in main
    datasets = prepare_dataset(dataset_args, logger)
  File "/dccstor/tslm/elron/tslm-gen/data.py", line 157, in prepare_dataset
    datasets = load_dataset(extension, data_files=data_files, split=dataset_split, cache_dir=dataset_args.dataset_cache_dir, na_filter=False, download_mode=dataset_args.dataset_generate_mode)
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/load.py", line 742, in load_dataset
    builder_instance.download_and_prepare(
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 574, in download_and_prepare
    self._download_and_prepare(
  File "/dccstor/tslm/envs/anaconda3/envs/trf-a100/lib/python3.9/site-packages/datasets/builder.py", line 654, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file. 
Original error:
[Errno 2] Failed to open local file '/dccstor/tslm-gen/.cache/csv/default-387f1f95c084d4df/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0.incomplete/csv-validation.arrow'. Detail: [errno 2] No such file or directory
```