SurAyush/github_issues huggingface dataset:
Issue: Keys mismatch: make error message more informative
State: closed
Author: PaulLerner

**Is your feature request related to a problem? Please describe.**
When loading a dataset from disk with a defect in its `dataset_info.json` describing its features (I don’t know when/why/how this happens but it deserves its own issue), you will get an error message like:
`ValueError: Keys mismatch: between {'bar': Value(dtype='int64', id=None)} and {'foo': Value(dtype='int64', id=None)}`

Which is fine when you have only a few features like in the example but it gets very hard to read when you have a lot of features in your dataset.

**Describe the solution you'd like**
The error message should give the difference between the features (what keys are in A but missing in B and vice-versa). It should also tell which keys are inferred from `dataset.arrow` and which come from `dataset_info.json`.

Willing to help :)


Comments:
Good idea ! I think this can be improved in `Features.reorder_fields_as()` indeed at

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/features.py#L1739-L1740

Is it something you would be interested in contributing ?

Is this open to work on? I'd love to take on this as my first issue.

Hi @daspartho I’ve opened a PR #4919 
I don’t think there’s much left to do

ok : )
Issue: [Tests] Ensure `datasets` supports renamed repositories
State: open
Author: lhoestq

On https://hf.co/datasets you can rename a dataset (or sometimes move it to another user/org). The website handles redirections correctly and AFAIK `datasets` does as well.

However it would be nice to have an integration test to make sure we don't break support for renamed datasets.

To implement this we can use the /api/repos/move endpoint on hub-ci to rename/move a repo (it is documented at https://huggingface.co/docs/hub/api)

Comments:
You could also switch to using `huggingface_hub` more directly, where such a guarantee is already tested =)

cc @Wauplin
Issue: Update GLUE evaluation metadata
State: closed
Author: lewtun

This PR updates the evaluation metadata for GLUE to:

* Include defaults for all configs except `ax` (which only has a `test` split with no known labels)
* Fix the default split from `test` to `validation` since `test` splits in GLUE have no labels (they're private)
* Fix the `task_id` for some existing defaults

cc @sashavor @douwekiela 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Fix missing tags in dataset cards
State: closed
Author: albertvillanova

Fix missing tags in dataset cards:
- asnq
- clue
- common_gen
- cosmos_qa
- guardian_authorship
- hindi_discourse
- py_ast
- x_stance

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: None Type error for swda datasets
State: closed
Author: hannan72

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10


Comments:
Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)
Issue: [LibriSpeech] Fix dev split local_extracted_archive for 'all' config
State: closed
Author: sanchit-gandhi

We define the keys for the `_DL_URLS` of the dev split as `dev.clean` and `dev.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L60-L61

These keys get forwarded to the `dl_manager` and thus the `local_extracted_archive`.

However, when calling `SplitGenerator` for the dev sets, we query the `local_extracted_archive` keys `validation.clean` and `validation.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L212
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L219

The consequence of this is that the `local_extracted_archive` arg passed to `_generate_examples` is always `None`, as the keys `validation.clean` and `validation.other` do not exists in the `local_extracted_archive`.

When defining the `audio_file` in `_generate_examples`, since `local_extracted_archive` is always `None`, we always omit the `local_extracted_archive` path from the `audio_file` path, **even** if in non-streaming mode:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L259-L263

Thus, `audio_file` will only ever be the streaming path (`audio_file`, not `os.path.join(local_extracted_archive, audio_file)`).

This PR fixes the `.get()` keys for the `local_extracted_archive` for the dev splits.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This PR fixes a bug introduced in:
- #4184
Issue: Fix CI reporting
State: closed
Author: albertvillanova

Fix CI so that it reports defaults (failed and error) besides the custom (xfailed and xpassed) in the test summary.

This PR fixes a regression introduced by:
- #4845

This introduced the reporting of xfailed and xpassed, but wrongly removed the reporting of the defaults failed and error.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Name the default config `default`
State: closed
Author: severo

Currently, if a dataset has no configuration, a default configuration is created from the dataset name.

For example, for a dataset loaded from the hub repository, such as https://huggingface.co/datasets/user/dataset (repo id is `user/dataset`), the default configuration will be `user--dataset`.

It might be easier to handle to set it to `default`, or another reserved word.

Comments:
Addressed in #5331.
Issue: Re-add code and und language tags
State: closed
Author: albertvillanova

This PR fixes the removal of 2 language tags done by:
- #4882

The tags are:
- "code": this is not a IANA tag but needed
- "und": this is one of the special scoped tags removed by 0d53202b9abce6fd0358cb00d06fcfd904b875af
  - used in "mc4" and "udhr" datasets

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Dataset Viewer issue for timit_asr
State: closed
Author: InayatUllah932

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
Yes, the dataset viewer is based on `datasets`, and the following does not work:

```
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('timit_asr')
Downloading builder script: 7.48kB [00:00, 6.69MB/s]
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 354, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/timit_asr/43f9448dd5db58e95ee48a277f466481b151f112ea53e27f8173784da9254fb2/timit_asr.py", line 117, in _split_generators
    data_dir = os.path.abspath(os.path.expanduser(dl_manager.manual_dir))
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/posixpath.py", line 231, in expanduser
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

cc @huggingface/datasets 

Due to license restriction, this dataset needs manual downloading of the original data.

This information is in the dataset card: https://huggingface.co/datasets/timit_asr
> The dataset needs to be downloaded manually from https://catalog.ldc.upenn.edu/LDC93S1

Maybe a better error message for datasets that need manual downloading? @severo 

Maybe we can raise a specific excpetion as done from `load_dataset`...

Yes, ideally something like https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py#L81


The preview is now disabled (and a descriptive warning is displayed) for datasets requiring manual download. See:

![timit_asr-manual-download](https://user-images.githubusercontent.com/8515462/193578572-3d21b790-f848-4257-9e9b-7cab3d76a269.png)
Issue: datasets generate large arrow file
State: closed
Author: jax11235

Checking the large file in disk, and found the large cache file in the cifar10 data directory:

![image](https://user-images.githubusercontent.com/18533904/186830449-ba96cdeb-0fe8-4543-994d-2abe7145933f.png)

As we know, the size of cifar10 dataset is ~130MB, but the cache file has almost 30GB size, there may be some problems here.

Comments:
Hi ! The cache files are the results of all the transforms you applied to the dataset using `map` for example.
Did you run a transform that could potentially blow up the size of the dataset ?

@lhoestq,
I don't remember, but I can't imagine what kind of transform may generate data that grow over 200 times in size. 
I think maybe it doesn' matter, it's just cache after all.
Issue: Fix missing tags in dataset cards
State: closed
Author: albertvillanova

Fix missing tags in dataset cards:
- anli
- coarse_discourse
- commonsense_qa
- cos_e
- ilist
- lc_quad
- web_questions
- xsum

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: Add citation to ro_sts and ro_sts_parallel datasets
State: closed
Author: albertvillanova

This PR adds the citation information to `ro_sts_parallel` and `ro_sts_parallel` datasets, once they have replied our request for that information:
- https://github.com/dumitrescustefan/RO-STS/issues/4

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4892). All of your documentation changes will be reflected on that endpoint.
Issue: Fix missing tags in dataset cards
State: closed
Author: albertvillanova

Fix missing tags in dataset cards:
- aslg_pc12
- librispeech_lm
- mwsc
- opus100
- qasc
- quail
- squadshifts
- winograd_wsc

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833


Comments:
Issue: add Dataset.from_list
State: closed
Author: sanderland

As discussed in #4885 

I initially added this bit at the end, thinking filling this field was necessary as it is done in from_dict. 
However, it seems the constructor takes care of filling info when it is empty.
```
if info.features is None:
  info.features = Features(
      {
          col: generate_from_arrow_type(coldata.type)
          for col, coldata in zip(pa_table.column_names, pa_table.columns)
      }
  )
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova it seems tests fail on pyarrow 6, perhaps from_pylist is a v7 method? How do you usually handle these version differences?
Added something that at least works
Issue: Dataset Viewer issue for subjqa
State: closed
Author: lewtun

### Link

https://huggingface.co/datasets/subjqa

### Description

Getting the following error for this dataset:

```
Status code:   500
Exception:     Status500Error
Message:       2 or more items returned, instead of 1
```

Not sure what's causing it though 🤔 

### Owner

Yes

Comments:
It's a bug in the viewer, thanks for reporting it. We're hoping to update to a new version in the next few days which should fix it.

Fixed 

https://huggingface.co/datasets/subjqa

<img width="1040" alt="Capture d’écran 2022-09-08 à 10 23 26" src="https://user-images.githubusercontent.com/1676121/189073210-2a57ff88-8bb1-44bd-851e-0e75473cea3f.png">
Issue: Create dataset from list of dicts
State: closed
Author: sanderland

I often find myself with data from a variety of sources, and a list of dicts is very common among these.
However, converting this to a Dataset is a little awkward, requiring either

```Dataset.from_pandas(pd.DataFrame(formatted_training_data))```
Which can error out on some more exotic values as 2-d arrays for reasons that are not entirely clear
> ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column labels with type object')

Alternatively:
```Dataset.from_dict({k: [s[k] for s in formatted_training_data] for k in  formatted_training_data[0].keys()})```
Which works, but is a little ugly.

**Describe the solution you'd like**
Either `.from_dict` accepting a list of dicts, or a `.from_records` function accepting such.

I am happy to PR this, just wanted to check you are happy to accept this I haven't missed something obvious, and which of the solutions would be prefered.


Comments:
Hi @sanderland, thanks for your enhancement proposal.

I agree with you that this would be useful.

Please note that under the hood, we use PyArrow tables as backend:
- The implementation  of `Dataset.from_dict` uses the PyArrow `Table.from_pydict`

Therefore, I would suggest:
- Implementing `Dataset.from_list` using the PyArrow `Table.from_pylist`

What do you think?
Let's see if other people have other suggestions...

Thanks for the quick and positive reply @albertvillanova! 
`from_list` seems sensible. Have opened a PR so we can discuss details there.

Resolved via #4890.
Issue: Fix language tags resource file
State: closed
Author: albertvillanova

This PR fixes/updates/adds ALL language tags from IANA (as of 2022-08-08).

This PR also removes all BCP47 suffixes (the languages file only contains language subtags, i.e. ISO 639 1 or 2 codes; no script/region/variant suffixes). See:
- #4753

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4882). All of your documentation changes will be reflected on that endpoint.
Issue: Added names of less-studied languages
State: closed
Author: BenjaminGalliot

Added names of less-studied languages (nru – Narua and jya – Japhug) for existing datasets.

Comments:
OK, I removed Glottolog codes and only added ISO 639-3 ones. The former are for the moment in corpus card description, language details, and in subcorpora names.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4880). All of your documentation changes will be reflected on that endpoint.
Issue: Fix Citation Information section in dataset cards
State: closed
Author: albertvillanova

Fix Citation Information section in dataset cards:
- cc_news
- conllpp
- datacommons_factcheck
- gnad10
- id_panl_bppt
- jigsaw_toxicity_pred
- kinnews_kirnews
- kor_sarcasm
- makhzan
- reasoning_bg
- ro_sts
- ro_sts_parallel
- sanskrit_classic
- telugu_news
- thaiqa_squad
- wiki_movies

This PR partially fixes the Citation Information section in dataset cards. Subsequent PRs will follow to complete this task.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4879). All of your documentation changes will be reflected on that endpoint.
Issue: [not really a bug] `identical_ok` is deprecated in huggingface-hub's `upload_file`
State: closed
Author: severo

In the huggingface-hub dependency, the `identical_ok` argument has no effect in `upload_file` (and it will be removed soon)

See

https://github.com/huggingface/huggingface_hub/blob/43499582b19df1ed081a5b2bd7a364e9cacdc91d/src/huggingface_hub/hf_api.py#L2164-L2169

It's used here:

https://github.com/huggingface/datasets/blob/fcfcc951a73efbc677f9def9a8707d0af93d5890/src/datasets/dataset_dict.py#L1373-L1381

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4354-L4362

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4197-L4213

We should remove it.

Maybe the third code sample has an unexpected behavior since it uses the non-default value `identical_ok = False`, but the argument is ignored.

Comments:
Resolved via https://github.com/huggingface/datasets/pull/4937.
Issue: Multiple dataloader memory error
State: open
Author: cyk1337

For the use of multiple datasets and tasks, we use around more than 200+ dataloaders, then pass it into `dataloader1, dataloader2, ..., dataloader200=accelerate.prepare(dataloader1, dataloader2, ..., dataloader200)`
It causes the memory error when generating batches. Any solutions to it?

```bash
  File "/home/xxx/my_code/src/utils/data_utils.py", line 54, in generate_batch
    x = next(iterator)
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 301, in __iter__
    for batch in super().__iter__():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 28, in fetch
    data.append(next(self.dataset_iter))
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 249, in __iter__
    for element in self.dataset:
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 503, in __iter__
    for key, example in self._iter():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 500, in _iter
    yield from ex_iterable
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 231, in __iter__
    new_key = "_".join(str(key) for key in keys)
MemoryError
```

Comments:
Hi!

200+ data loaders is a lot. Have you tried to reduce the number of datasets by concatenating/interleaving the ones with the same structure/task (the API is `{concatenate_datasets/interleave_datasets}([dset1, ..., dset_N])`)?

Hi @mariosasko, thank you for your reply. I tried pre-concatenating different datasets into one, but one key need is to keep each batch the same data type. Considering that the concatenate-then-segment operation for prefetched samples may span across different data types after concatenating/interleaving (cuz different data sources are mixed), any solution to remain the same data source for each batch?

@cyk1337 have you found any solutions to it?
@mariosasko I tried with interleave_datasets to sample batches from two large datasets (wikipedia alike) and it results in out-of-memory error during data loading (16gpus, >1TB physical memory). Do you have any idea about it?
Issue: Docs for creating an audio dataset
State: closed
Author: stevhliu

This PR is a first draft of how to create audio datasets (`AudioFolder` and loading script). Feel free to let me know if there are any specificities I'm missing for this. 🙂

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Awesome thanks ! I think we can also encourage TAR archives as for image dataset scripts (feel free to copy paste some parts from there lol)

Thanks for all the great feedback @polinaeterna and @lhoestq! 🥰

I added all the other feedback, and I'll look into the `librivox-indonesia` script now!

If you don't mind, I'm taking over this PR since we'll do a release pretty soon

@lhoestq no, I do :D 

haha sorry ^^
Issue: Fix: wmt datasets - fix CWMT zh subsets
State: closed
Author: lhoestq

Fix https://github.com/huggingface/datasets/issues/4575

TODO: run `datasets-cli test`:
- [x] wmt17
- [x] wmt18
- [x] wmt19

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4871). All of your documentation changes will be reflected on that endpoint.
Issue: adding mafand to datasets
State: closed
Author: dadelani

I'm addding the MAFAND dataset by Masakhane based on the paper/repository below:

Paper: https://aclanthology.org/2022.naacl-main.223/
Code: https://github.com/masakhane-io/lafand-mt

Please, help merge this

Everything works except for creating dummy data file

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @dadelani, thanks for your awesome contribution!!! :heart: 

However, now we are using the Hub to add new datasets, instead of this GitHub repo. 

You could share this dataset under your Hub organization namespace: [Masakhane NLP](https://huggingface.co/masakhane). This way the dataset will be accessible using:
```python
ds = load_dataset("masakhane/mafand")
```

You have the procedure documented in our online docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Share](https://huggingface.co/docs/datasets/share)

Moreover, datasets shared on the Hub no longer need the dummy data files.

Please, feel free to ping me if you need any further guidance/support.

thank you for the comment. I have moved it to the Hub https://huggingface.co/datasets/masakhane/mafand

Great job, @dadelani!!

Please, note that in the README.md file, the YAML tags should be preceded and followed by three dashes `---`, so that they are properly parsed. See, e.g.: https://raw.githubusercontent.com/huggingface/datasets/main/templates/README.md

Also you could replace the line:
```
# Dataset Card for [Needs More Information]
```
with
```
# Dataset Card for MAFAND-MT
```

Great, thank you for the feedback. I have fixed both issues.