ahmedfarazsyk/datasets-github-issues huggingface dataset:
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
Author: MinionAttack
State: closed
Labels: name: dataset-viewer

## Dataset viewer issue for '*oscar-corpus/OSCAR-2109*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/oscar-corpus/OSCAR-2109)*

*The dataset library cannot download any language from the oscar-corpus/OSCAR-2109 dataset. By entering the URL in your browser I can access the file.*

```
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

Am I the one who added this dataset ? No

Using the older version of [OSCAR](https://huggingface.co/datasets/oscar) I don't have any issues downloading languages with the dataset library.

Comments: Hi ! Thanks for reporting :)
I think this is because the dataset is behind an access page. We can fix the dataset viewer

If you also have this error when you use the `datasets` library in python, you should probably pass `use_auth_token=True` to the `load_dataset()` function to use your account to access the dataset.

Ah ok, I didn't realise about the login page. I'll try `use_auth_token=True` and see if that solves it.

Regards!

Hi, 

Using `use_auth_token=True` and downloading the credentials with `huggingface-cli login` (stored in .huggingface/token) solved the issue.

Should I leave the issue open until you fix the Dataset viewer issue?

Cool ! Yes let's keep this issue open until the viewer is fixed - I'll close it when this is fixed. Thanks

The error I get when trying to load OSCAR 21.09 is this
```
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

The URL I get in the browser is this
```
https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
```

Maybe URL is the issue? (resolve vs blob)

> The error I get when trying to load OSCAR 21.09 is this
> 
> ```
> ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
> ```
> 
> The URL I get in the browser is this
> 
> ```
> https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
> ```
> 
> Maybe URL is the issue? (resolve vs blob)

You need to download your login credentials. See `huggingface-cli login` documentation and when loading the dataset use `use_auth_token=True`:
`
load_dataset(corpus, language, split=None, use_auth_token=True, cache_dir=cache_folder)`

Fixed.

<img width="1542" alt="Capture d’écran 2022-04-12 à 13 57 24" src="https://user-images.githubusercontent.com/1676121/162957585-af96d19c-f86c-47fe-80c4-2b071083cee4.png">
Respect row ordering when concatenating datasets along axis=1
Author: mariosasko
State: closed
Labels: name: bug

Currently, there is a bug when concatenating datasets along `axis=1` if more than one dataset has the `_indices` attribute defined. In that scenario, all indices mappings except the first one get ignored.

A minimal reproducible example:
```python
>>> from datasets import Dataset, concatenate_datasets
>>> a = Dataset.from_dict({"a": [30, 20, 10]})
>>> b = Dataset.from_dict({"b": [2, 1, 3]})
>>> d = concatenate_datasets([a.sort("a"), b.sort("b")], axis=1)
>>> print(d[:3])              # expected: {'a': [10, 20, 30], 'b': [1, 2, 3]}
{'a': [10, 20, 30], 'b': [3, 1, 2]}
```

I've noticed the bug while working on #3195.
Make iter_archive work with ZIP files
Author: lhoestq
State: open
Labels: name: enhancement

Currently users can use `dl_manager.iter_archive` in their dataset script to iterate over all the files of a TAR archive.
It would be nice if it could work with ZIP files too !

Comments: Hello, is this issue open for any contributor ? can I work on it ?



Hi ! Sure this is open for any contributor. If you're interested feel free to self-assign this issue to you by commenting `#self-assign`. Then if you have any question or if I can help, feel free to ping me.

To begin with, feel free to take a look at both implementations of `iter_archive` for local downloads and for data streaming:

In the `DownloadManager` for local dowloads:
https://github.com/huggingface/datasets/blob/dfa334bd8dc6cbc854b170379c7d2cb7e3d3fe4f/src/datasets/utils/download_manager.py#L218-L242

In the `StreamingDownloadManager` to stream the content of the archive directly from the remote file:
https://github.com/huggingface/datasets/blob/dfa334bd8dc6cbc854b170379c7d2cb7e3d3fe4f/src/datasets/utils/streaming_download_manager.py#L502-L526

Notice the call to `xopen` that opens and streams a file given either an URL or a local path :)

Okay thank you for the information. I will work on this :) 

#self-assign
Dataset viewer issue for 'liweili/c4_200m'
Author: liliwei25
State: closed
Labels: name: dataset-viewer

## Dataset viewer issue for '*liweili/c4_200m*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/liweili/c4_200m)*

*Server Error*
```
Status code:   404
Exception:     Status404Error
Message:       Not found. Maybe the cache is missing, or maybe the ressource does not exist.
```

Am I the one who added this dataset ? Yes


Comments: Hi ! I think the issue comes from this [line](https://huggingface.co/datasets/liweili/c4_200m/blob/main/c4_200m.py#L87):
```python
path = filepath + "/*.tsv*"
```

You can fix this by doing this instead:
```python
path = os.path.join(filepath, "/*.tsv*")
```

Here is why:

Locally you can append `"/*.tsv*"` to your local path, however it doesn't work in streaming mode, and the dataset viewer does use the streaming mode.
In streaming mode, the download and extract part is done lazily. It means that instead of using local paths, it's still passing around URLs and [chained URLs](https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining)

Therefore in streaming mode, `filepath` is not a local path, but instead is equal to
```python
zip://::https://huggingface.co/datasets/liweili/c4_200m/resolve/main/data.zip
```
The `zip://` part means that we navigate inside the remote ZIP file.

You must use `os.path.join` to navigate inside it and get your TSV files:
```python
>>> os.path.join(filepath, "/*.tsv*")
zip://*.tsv*::https://huggingface.co/datasets/liweili/c4_200m/resolve/main/data.zip
```

`datasets` extends `os.path.join`, `glob.glob`, etc. in your dataset scripts to work with remote files.

hi @lhoestq ! thanks for the tip! i've updated the line of code but it's still not working. am i doing something else wrong? thank you!

Hi ! Your dataset code is all good now :)
```python
In [1]: from datasets import load_dataset

In [2]: d = load_dataset("liweili/c4_200m", streaming=True)
Downloading: 100%|█████████████████████████████████████████████| 2.79k/2.79k [00:00<00:00, 4.83MB/s]
Using custom data configuration default

In [3]: next(iter(d["train"]))
Out[3]: 
{'input': 'Bitcoin is for $7,094 this morning, which CoinDesk says.',
 'output': 'Bitcoin goes for $7,094 this morning, according to CoinDesk.'}
```
Though the viewer doesn't seem to be updated, I'll take a look at what's wrong

thank you @lhoestq! 😄 

It's working

<img width="1424" alt="Capture d’écran 2021-12-21 à 11 24 29" src="https://user-images.githubusercontent.com/1676121/146914238-24bf87c0-c68d-4699-8d6c-fa3065656d1d.png">
Checksum error for kilt_task_wow
Author: slyviacassell
State: closed
Labels: name: bug

## Describe the bug
Checksum failed when downloads kilt_tasks_wow. See error output for details.

## Steps to reproduce the bug
```python
import datasets
datasets.load_datasets('kilt_tasks','wow')
```

## Expected results
Download successful

## Actual results
```
Downloading and preparing dataset kilt_tasks/wow (download: 72.07 MiB, generated: 61.82 MiB, post-processed: Unknown size, total: 133.89 MiB) to /root/.cache/huggingface/datasets/kilt_tasks/wow/1.0.0/57dc8b2431e76637e0c6ef79689ca4af61ed3a330e2e0cd62c8971465a35db3a...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 5121.25it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1527.42it/s]
Traceback (most recent call last):
  File "kilt_wow.py", line 30, in <module>
    main()
  File "kilt_wow.py", line 27, in main
    train, dev, test = dataset.generate_k_shot_data(k=32, seed=seed, path="../data/")
  File "/workspace/projects/CrossFit/tasks/fewshot_gym_dataset.py", line 79, in generate_k_shot_data
    dataset = self.load_dataset()
  File "kilt_wow.py", line 21, in load_dataset
    return datasets.load_dataset('kilt_tasks','wow')
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 679, in _download_and_prepare
    verify_checksums(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://dl.fbaipublicfiles.com/KILT/wow-train-kilt.jsonl', 'http://dl.fbaipublicfiles.com/KILT/wow-dev-kilt.jsonl']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 4.0.1


Comments: Using `dataset = load_dataset("kilt_tasks", "wow", ignore_verifications=True)` may fix it, but I do not think it is a elegant solution.

Hi @slyviacassell, thanks for reporting.

Yes, there is an issue with the checksum verification. I'm fixing it.

And as you pointed out, in the meantime, you can circumvent the problem by passing `ignore_verifications=True`.
FET DATA
Author: FStell01
State: closed
Labels: name: dataset request

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Scifi_TV_Shows: Having trouble getting viewer to find appropriate files
Author: lara-martin
State: closed
Labels: name: dataset-viewer

## Dataset viewer issue for '*Science Fiction TV Show Plots Corpus  (Scifi_TV_Shows)*'

**Link:** [link](https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows)

I tried adding both a script (https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/blob/main/Scifi_TV_Shows.py) and some dummy examples (https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/tree/main/dummy), but the viewer still has a 404 error ("Not found. Maybe the cache is missing, or maybe the ressource does not exist."). I'm not sure what to try next. Thanks in advance!

Am I the one who added this dataset? Yes


Comments: Hi ! I think this is because `iter_archive` doesn't support ZIP files yet. See https://github.com/huggingface/datasets/issues/3272

You can navigate into the archive this way instead:
```python
# in split_generators
data_dir = dl_manager.download_and_extract(url)
train_filepath = os.path.join(data_dir, "all-sci-fi-data-train.txt")
return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={
                "filepath": train_filepath,
            },
        ),
...
])

# in generate_examples
with open(filepath, encoding="utf-8") as f:
    ...
```

It's working: https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/viewer/Scifi_TV_Shows/test

<img width="1494" alt="Capture d’écran 2021-12-21 à 11 23 51" src="https://user-images.githubusercontent.com/1676121/146914068-f4b7225f-42c5-471d-9c73-2adac722162f.png">
Use f-strings for string formatting 
Author: mariosasko
State: closed
Labels: name: good first issue

f-strings offer better readability/performance than `str.format` and `%`, so we should use them in all places in our codebase unless there is good reason to keep the older syntax.

> **NOTE FOR CONTRIBUTORS**: To avoid large PRs and possible merge conflicts, do 1-3 modules per PR. Also, feel free to ignore the files located under `datasets/*`.

Comments: Hi, I would be glad to help with this. Is there anyone else working on it?

Hi, I would be glad to work on this too.

#self-assign

Hi @Carlosbogo,

would you be interested in replacing the `.format` and `%` syntax with f-strings in the modules in the `datasets` directory since @Mehdi2402 has opened a PR that does that for all the other directories?

Oh I see. I will be glad to help with the `datasets` directory then.
SciELO dataset ConnectionError
Author: WojciechKusa
State: closed
Labels: name: bug

## Describe the bug
I get `ConnectionError` when I am trying to load the SciELO dataset. 


When I try the URL with `requests` I get:
```
>>> requests.head("https://ndownloader.figstatic.com/files/14019287")
<Response [302]>
```
And as far as I understand redirections in `datasets` are not supported for downloads. 
https://github.com/huggingface/datasets/blob/807341d0db0728073ab605c812c67f927d148f38/datasets/scielo/scielo.py#L45 



## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("scielo", "en-es")
```

## Expected results
Download SciELO dataset and load Dataset object


## Actual results

```
Downloading and preparing dataset scielo/en-es (download: 21.90 MiB, generated: 68.45 MiB, post-processed: Unknown size, total: 90.35 MiB) to /Users/test/.cache/huggingface/datasets/scielo/en-es/1.0.0/7e05d55a20257efeb9925ff5de65bd4884fc6ddb6d765f1ea3e8860449d90e0e...
Traceback (most recent call last):
  File "scielo.py", line 3, in <module>
    dataset = load_dataset("scielo", "en-es")
  File "../lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "../lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "../lib/python3.8/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/test/.cache/huggingface/modules/datasets_modules/datasets/scielo/7e05d55a20257efeb9925ff5de65bd4884fc6ddb6d765f1ea3e8860449d90e0e/scielo.py", line 77, in _split_generators
    data_dir = dl_manager.download_and_extract(_URLS[self.config.name])
  File "../lib/python3.8/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "../lib/python3.8/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "../lib/python3.8/site-packages/datasets/utils/py_utils.py", line 206, in map_nested
    return function(data_struct)
  File "../lib/python3.8/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "../lib/python3.8/site-packages/datasets/utils/file_utils.py", line 295, in cached_path
    output_path = get_from_cache(
  File "../lib/python3.8/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://ndownloader.figstatic.com/files/14019287

```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.12
- PyArrow version: 6.0.0
`GeneratorBasedBuilder` does not support `None` values
Author: pavel-lexyr
State: closed
Labels: name: bug

## Describe the bug
`GeneratorBasedBuilder` does not support `None` values.

## Steps to reproduce the bug
See [this repository](https://github.com/pavel-lexyr/huggingface-datasets-bug-reproduction) for minimal reproduction.

## Expected results
Dataset is initialized with a `None` value in the `value` column.

## Actual results
```
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    datasets.load_dataset("./bad-data")
  File ".../datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File ".../datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File ".../datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File ".../datasets/builder.py", line 1103, in _prepare_split
    example = self.info.features.encode_example(record)
  File ".../datasets/features/features.py", line 1033, in encode_example
    return encode_nested_example(self, example)
  File ".../datasets/features/features.py", line 808, in encode_nested_example
    return {
  File ".../datasets/features/features.py", line 809, in <dictcomp>
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File ".../datasets/features/features.py", line 855, in encode_nested_example
    return schema.encode_example(obj)
  File ".../datasets/features/features.py", line 299, in encode_example
    return float(value)
TypeError: float() argument must be a string or a number, not 'NoneType'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 6.0.0

Comments: Hi,

thanks for reporting and providing a minimal reproducible example. 

This line of the PR I've linked in our discussion on the Forum will add support for `None` values:
https://github.com/huggingface/datasets/blob/a53de01842aac65c66a49b2439e18fa93ff73ceb/src/datasets/features/features.py#L835

I expect that PR to be merged soon.
Adding ANERcorp-CAMeLLab dataset
Author: vitalyshalumov
State: open
Labels: name: dataset request


Comments: Adding ANERcorp dataset

## Adding a Dataset
- **Name:** *ANERcorp-CAMeLLab*
- **Description:** *Since its creation in 2008, the ANERcorp dataset (Benajiba & Rosso, 2008) has been a standard reference used by Arabic named entity recognition researchers around the world. However, over time, this dataset was copied over from user to user, modified slightly here and there, and split in many different configurations that made it hard to compare fairly across papers and systems.

In 2020, a group of researchers from CAMeL Lab (Habash, Alhafni and Oudah), and Mind Lab (Antoun and Baly) met with the creator of the corpus, Yassine Benajiba, to consult with him and collectively agree on an exact split, and accepted minor corrections from the original dataset. Bashar Alhafni from CAMeL Lab working with Nizar Habash implemented the decisions provided in this release.*

- **Paper:** *(a) Benajiba, Yassine, Paolo Rosso, and José Miguel Benedí Ruiz. "Anersys: An Arabic named entity recognition system based on maximum entropy." In International Conference on Intelligent Text Processing and Computational Linguistics, pp. 143-153. Springer, Berlin, Heidelberg, 2007.

(b)Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. "CAMeL Tools: An Open Source Python Toolkit, for Arabic Natural Language Processing." In Proceedings of the Conference on Language Resources and Evaluation (LREC 2020), Marseille, 2020.*
- **Data:** *https://camel.abudhabi.nyu.edu/anercorp/*
- **Motivation:** This is the standard dataset for evaluating NER performance in Arabic*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Couldn't reach data file for disaster_response_messages
Author: pandya6988
State: closed
Labels: name: dataset bug

## Describe the bug
Following command gives an ConnectionError.

## Steps to reproduce the bug
```python
disaster = load_dataset('disaster_response_messages')
```

## Error
```
ConnectionError: Couldn't reach https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv
```
## Expected results
It should load dataset without an error

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Google Colab
- Python version: 3.7
- PyArrow version: 


Comments: It looks like the dataset isn't available anymore on appen.com

The CSV files appear to still be available at https://www.kaggle.com/landlord/multilingual-disaster-response-messages though. It says that the data are under the CC0 license so I guess we can host the dataset elsewhere instead ?
Reuters21578 Couldn't reach 
Author: TingNLP
State: closed
Labels: name: dataset bug

``## Adding a Dataset
- **Name:** *Reuters21578*
- **Description:** *ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz*
- **Data:** *https://huggingface.co/datasets/reuters21578*

`from datasets import load_dataset`
`dataset = load_dataset("reuters21578", 'ModLewis')`

ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz

And I try to request the link as follow:
`import requests`
`requests.head('https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz')`

SSLError: HTTPSConnectionPool(host='kdd.ics.uci.edu', port=443): Max retries exceeded with url: /databases/reuters21578/reuters21578.tar.gz (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

This problem likes #575
What should I do ?


Comments: Hi ! The URL works fine on my side today, could you try again ?

thank you @lhoestq 
it works
wikitext description wrong
Author: hongyuanmei
State: closed
Labels: name: bug

## Describe the bug
Descriptions of the wikitext datasests are wrong. 

## Steps to reproduce the bug
Please see: https://github.com/huggingface/datasets/blob/f6dcafce996f39b6a4bbe3a9833287346f4a4b68/datasets/wikitext/wikitext.py#L50

## Expected results
The descriptions for raw-v1 and v1 should be switched. 

Comments: Hi @hongyuanmei, thanks for reporting.

I'm fixing it.

Duplicate of:
- #795