Minimartzz/github-issues huggingface dataset:
vivos (Vietnamese speech corpus) dataset not accessible
State: closed
Labels: description: A bug in a dataset script provided in the library
name: dataset bug

## Describe the bug
VIVOS data is not accessible anymore, neither of these links work (at least from France):
* https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (data)
* https://ailab.hcmus.edu.vn/vivos (dataset page) 

Therefore `load_dataset` doesn't work.

## Steps to reproduce the bug
```python
ds = load_dataset("vivos")
```

## Expected results
dataset loaded

## Actual results
```
ConnectionError: Couldn't reach https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='ailab.hcmus.edu.vn', port=443): Max retries exceeded with url: /assets/vivos.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d8a27d190>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))")))
```

Will try to contact the authors, as we wanted to use Vivos as an example in documentation on how to create scripts for audio datasets (https://github.com/huggingface/datasets/pull/4872), because it's small and straightforward and uses tar archives. 

Comments:
If you need an example of a small audio datasets, I just created few hours ago a speech dataset with only 300MB of compressed audio files https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia. It works also with streaming (@albertvillanova helped me adding this functionality) :-)

@cahya-wirawan omg this is awesome!! thank you! 

We have contacted the authors to ask them.
Dataset Viewer issue for ubuntu_dialogs_corpus
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
The dataset maintainers (https://huggingface.co/datasets/ubuntu_dialogs_corpus) decided to forbid the dataset from being downloaded automatically (https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download), and the dataset viewer respects this.
We will try to improve the error display though. Thanks for reporting.
Dataset Viewer issue for bigscience-biomedical/biosses
State: closed

### Link

https://huggingface.co/datasets/bigscience-biomedical/biosses

### Description

I've just been working on adding the dataset loader script to this dataset and working with the relative imports. I'm not sure how to interpret the error below (show where the dataset preview used to be) . 
```
Status code:   400
Exception:     ModuleNotFoundError
Message:       No module named 'datasets_modules.datasets.bigscience-biomedical--biosses.ddbd5893bf6c2f4db06f407665eaeac619520ba41f69d94ead28f7cc5b674056.bigbiohub'
```

### Owner

Yes

Comments:
Possibly not related to the dataset viewer in itself. cc @huggingface/datasets.

In particular, I think that the import of bigbiohub is not working here: https://huggingface.co/datasets/bigscience-biomedical/biosses/blob/main/biosses.py#L29 (requires a relative path?)

```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names('bigscience-biomedical/biosses')
Downloading builder script: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.00k/8.00k [00:00<00:00, 7.47MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 289, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1247, in dataset_module_factory
    raise e1 from None
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1220, in dataset_module_factory
    return HubDatasetModuleFactoryWithScript(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 931, in get_module
    local_imports = _download_additional_modules(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 215, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use bigscience-biomedical/biosses, you need to install the following dependency: bigbiohub.
Please install it using 'pip install bigbiohub' for instance'
```

Opened a PR here to (hopefully) fix the dataset script: https://huggingface.co/datasets/bigscience-biomedical/biosses/discussions/1/files

thanks for taking a look @severo . agree this isn't related to dataset viewer (sorry just clicked on the auto issue creator). also thanks @lhoestq , I see the format to use for relative imports. was a bit confused b/c it seems to be working here 

https://huggingface.co/datasets/bigscience-biomedical/scitail/blob/main/scitail.py#L31

I'll try this PR a see what happens. 

closing as I think the issue is relative imports and attempting to read json files directly in the repo (thanks again @lhoestq )
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- coqa
- hyperpartisan_news_detection
- opinosis
- scientific_papers
- scifact
- search_qa
- wiki_qa
- wiki_split
- wikisql

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add cc-by-nc-2.0 to list of licenses
State: closed

This PR adds the `cc-by-nc-2.0` to the list of licenses because it is required by `scifact` dataset: https://github.com/allenai/scifact/blob/master/LICENSE.md

Comments:
_The documentation is not available anymore as the PR was closed or merged._

this list needs to be kept in sync with the ones in moon-landing and hub-docs :)

@julien-c don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Also note that the licenses we are adding were all already present in our previous `licenses.json` file: are we regenerating it, step by step? Why don't we use a file with ALL the licenses we previously had in the list?

Licenses added:
- #4887
- #4930 

Previous `licenses.json` file:
- https://github.com/huggingface/datasets/blob/b7612754928e0fd43b9e3c3becb906ec280ff5d4/src/datasets/utils/resources/licenses.json
- removed in this commit: https://github.com/huggingface/datasets/pull/4613/commits/9f7725412dac1089b3e057f9e3fcf39cc222bc26

Let me know what you think and I can take care of this.

> Let me know what you think and I can take care of this.

What I think is that we shouldn't add licenses that are just used in a couple of datasets, and just use `license_details` for this.

> don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Yes, in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now? 


Feel free to delete the license list in `datasets` @albertvillanova ;)

Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)
Fixes a typo in loading documentation
State: closed

As show in the [documentation page](https://huggingface.co/docs/datasets/loading) here the `"tr"in` should be `"train`.

![image](https://user-images.githubusercontent.com/7144772/188390445-e1f04d54-e3e3-4762-8686-63ecbe4087e5.png)


Comments:
Concatenate_datasets loads everything into RAM
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
When loading the datasets seperately and saving them on disk, I want to concatenate them. But `concatenate_datasets` is filling up my RAM and the process gets killed. Is there a way to prevent this from happening or is this intended behaviour? Thanks in advance

## Steps to reproduce the bug
```python
gcs = gcsfs.GCSFileSystem(project='project')
datasets = [load_from_disk(f'path/to/slice/of/data/{i}', fs=gcs, keep_in_memory=False) for i in range(10)]

dataset = concatenate_datasets(datasets)
```

## Expected results
A concatenated dataset which is stored on my disk.

## Actual results
Concatenated dataset gets loaded into RAM and overflows it which gets the process killed.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 8.0.1
- Pandas version: 1.4.3

Comments:
decode mp3 with librosa if torchaudio is > 0.12 as a temporary workaround 
State: closed

`torchaudio>0.12` fails with decoding mp3 files if `ffmpeg<4`. currently we ask users to downgrade torchaudio, but sometimes it's not possible as torchaudio version is binded to torch version. as a temporary workaround we can decode mp3 with librosa (though it 60 times slower, at least it works)

another option would be to ask users to install the required version of `ffmpeg`, but is non-trivial on colab: it's not in apt packages in ubuntu 18 and `conda` is not preinstalled (with `conda` it would be easily installable)

- [x] decode with torchaudio anyway if the version of ffmpeg is correct? it's 60 times faster
- [x] tests 
- [x] DO NOT FORGET to get back all the tests

see https://github.com/huggingface/datasets/issues/4776 and https://github.com/huggingface/datasets/issues/3663#issuecomment-1225797165 (there is a Colab notebook to reproduce the error)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks ! Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

@lhoestq 

> Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

I'm not sure here, because from the one hand, if `torchaudio` works - it works 60 times faster then `librosa`.
But from the other hand, we will get inconsistent behavior (=different results of decoding) for users of `torchaudio>=0.12`. 
I'd better go for using `librosa` only to avoid inconsistency then. wdyt?

It seems a bit too constraining to not allow users who have a working torchaudio 0.12 setup to not use it. 

If the issue is about avoiding silent errors if the decoding changes, maybe we can log which back-end is used ? It can even be a warning with performance suggestions ("you're using librosa but torchaudio 0.xx is recommended").

Note that users can still have a requirements.txt or whatever in their projects if they really want full reproducibility (and it's the bare minimum imo)

There are multiple possible back-ends so it's maybe not reasonable to only allow one back-end, especially since each back-end has installation constrains and there's no "best" back-end.

Woohoo all green ! Feel free to merge if it's all good for you :)
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- eraser_multi_rc
- hotpot_qa
- metooma
- movie_rationales
- qanta
- quora
- quoref
- race
- ted_hrlr
- ted_talks_iwslt

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Unable to load local tsv files through load_dataset method
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Unable to load local tsv files through load_dataset method.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
data_files = {
    'train': 'train.tsv',
    'test': 'test.tsv'
}
raw_datasets = load_dataset('tsv', data_files=data_files)

## Expected results
I am pretty sure the data files exist in the current directory. The above code should load them as Datasets, but threw exceptions.

## Actual results
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-9-24207899c1af>](https://localhost:8080/#) in <module>
----> 1 raw_datasets = load_dataset('tsv', data_files='train.tsv')

2 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1244                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1245                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1246                     ) from None
   1247                 raise e1 from None
   1248     else:

FileNotFoundError: Couldn't find a dataset script at /content/tsv/tsv.py or any data file in the same directory. Couldn't find 'tsv' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/tsv/tsv.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments:
Hi @DataNoob0723,

Under the hood, we use `pandas` to load CSV/TSV files. Therefore, you should use "csv" and pass `sep="\t"`, as explained in our docs: https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/loading_methods#from-files
```python
ds = load_dataset('csv', sep="\t", data_files=data_files)
```
Dataset Viewer issue for pysentimiento/spanish-targeted-sentiment-headlines
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/pysentimiento/spanish-targeted-sentiment-headlines

### Description

After moving the dataset from my user (`finiteautomata`) to the `pysentimiento` organization, the dataset viewer says that it doesn't exist.

### Owner

_No response_

Comments:
Thanks for reporting, it's fixed now (I refreshed it manually). It's a known issue; we hope it will be fixed permanently in a few days.

<img width="1508" alt="Capture d’écran 2022-09-05 à 18 31 22" src="https://user-images.githubusercontent.com/1676121/188489762-0ed86a7e-dfb3-46e8-a125-43b815a2c6f4.png">


Thanks @severo!
Keys mismatch: make error message more informative
State: closed
Labels: description: New feature or request
name: enhancement

description: Good for newcomers
name: good first issue

**Is your feature request related to a problem? Please describe.**
When loading a dataset from disk with a defect in its `dataset_info.json` describing its features (I don’t know when/why/how this happens but it deserves its own issue), you will get an error message like:
`ValueError: Keys mismatch: between {'bar': Value(dtype='int64', id=None)} and {'foo': Value(dtype='int64', id=None)}`

Which is fine when you have only a few features like in the example but it gets very hard to read when you have a lot of features in your dataset.

**Describe the solution you'd like**
The error message should give the difference between the features (what keys are in A but missing in B and vice-versa). It should also tell which keys are inferred from `dataset.arrow` and which come from `dataset_info.json`.

Willing to help :)


Comments:
Good idea ! I think this can be improved in `Features.reorder_fields_as()` indeed at

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/features.py#L1739-L1740

Is it something you would be interested in contributing ?

Is this open to work on? I'd love to take on this as my first issue.

Hi @daspartho I’ve opened a PR #4919 
I don’t think there’s much left to do

ok : )
[Tests] Ensure `datasets` supports renamed repositories
State: open
Labels: description: Issues a bit more difficult than "Good First" issues
name: good second issue

On https://hf.co/datasets you can rename a dataset (or sometimes move it to another user/org). The website handles redirections correctly and AFAIK `datasets` does as well.

However it would be nice to have an integration test to make sure we don't break support for renamed datasets.

To implement this we can use the /api/repos/move endpoint on hub-ci to rename/move a repo (it is documented at https://huggingface.co/docs/hub/api)

Comments:
You could also switch to using `huggingface_hub` more directly, where such a guarantee is already tested =)

cc @Wauplin 

Hi! @lhoestq, I'd love to work on this issue as part of my ongoing contributions to `datasets`.

I'm planning to implement an integration test that ensures `load_dataset()` works even after a dataset repo is renamed or moved. I can work locally and interact with the `/api/repos/move` endpoint, using either `huggingface_hub` or `requests`, and will also run the test suite to validate behavior end-to-end.

Before I begin, I have two quick questions:
1. Is there a recommended test dataset or sandbox repo that I should use for safely renaming/moving during tests?
2. Do I need special permissions or tokens to use `hub-ci` or to move repos between users/orgs?

Once confirmed, I’ll proceed with writing and validating the test locally and open a PR soon after. Thank you!
Update GLUE evaluation metadata
State: closed

This PR updates the evaluation metadata for GLUE to:

* Include defaults for all configs except `ax` (which only has a `test` split with no known labels)
* Fix the default split from `test` to `validation` since `test` splits in GLUE have no labels (they're private)
* Fix the `task_id` for some existing defaults

cc @sashavor @douwekiela 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- asnq
- clue
- common_gen
- cosmos_qa
- guardian_authorship
- hindi_discourse
- py_ast
- x_stance

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896

Comments:
_The documentation is not available anymore as the PR was closed or merged._
None Type error for swda datasets
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10


Comments:
Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)
[LibriSpeech] Fix dev split local_extracted_archive for 'all' config
State: closed

We define the keys for the `_DL_URLS` of the dev split as `dev.clean` and `dev.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L60-L61

These keys get forwarded to the `dl_manager` and thus the `local_extracted_archive`.

However, when calling `SplitGenerator` for the dev sets, we query the `local_extracted_archive` keys `validation.clean` and `validation.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L212
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L219

The consequence of this is that the `local_extracted_archive` arg passed to `_generate_examples` is always `None`, as the keys `validation.clean` and `validation.other` do not exists in the `local_extracted_archive`.

When defining the `audio_file` in `_generate_examples`, since `local_extracted_archive` is always `None`, we always omit the `local_extracted_archive` path from the `audio_file` path, **even** if in non-streaming mode:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L259-L263

Thus, `audio_file` will only ever be the streaming path (`audio_file`, not `os.path.join(local_extracted_archive, audio_file)`).

This PR fixes the `.get()` keys for the `local_extracted_archive` for the dev splits.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This PR fixes a bug introduced in:
- #4184
Name the default config `default`
State: closed
Labels: description: New feature or request
name: enhancement

description: Further information is requested
name: question

Currently, if a dataset has no configuration, a default configuration is created from the dataset name.

For example, for a dataset loaded from the hub repository, such as https://huggingface.co/datasets/user/dataset (repo id is `user/dataset`), the default configuration will be `user--dataset`.

It might be easier to handle to set it to `default`, or another reserved word.

Comments:
Addressed in #5331.
Re-add code and und language tags
State: closed

This PR fixes the removal of 2 language tags done by:
- #4882

The tags are:
- "code": this is not a IANA tag but needed
- "und": this is one of the special scoped tags removed by 0d53202b9abce6fd0358cb00d06fcfd904b875af
  - used in "mc4" and "udhr" datasets

Comments:
_The documentation is not available anymore as the PR was closed or merged._