josteece/github-issues huggingface dataset:
None Type error for swda datasets
Opened by hannan72 (closed)

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10


Comments:
Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)
[LibriSpeech] Fix dev split local_extracted_archive for 'all' config
Opened by sanchit-gandhi (closed)

We define the keys for the `_DL_URLS` of the dev split as `dev.clean` and `dev.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L60-L61

These keys get forwarded to the `dl_manager` and thus the `local_extracted_archive`.

However, when calling `SplitGenerator` for the dev sets, we query the `local_extracted_archive` keys `validation.clean` and `validation.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L212
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L219

The consequence of this is that the `local_extracted_archive` arg passed to `_generate_examples` is always `None`, as the keys `validation.clean` and `validation.other` do not exists in the `local_extracted_archive`.

When defining the `audio_file` in `_generate_examples`, since `local_extracted_archive` is always `None`, we always omit the `local_extracted_archive` path from the `audio_file` path, **even** if in non-streaming mode:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L259-L263

Thus, `audio_file` will only ever be the streaming path (`audio_file`, not `os.path.join(local_extracted_archive, audio_file)`).

This PR fixes the `.get()` keys for the `local_extracted_archive` for the dev splits.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This PR fixes a bug introduced in:
- #4184
Fix CI reporting
Opened by albertvillanova (closed)

Fix CI so that it reports defaults (failed and error) besides the custom (xfailed and xpassed) in the test summary.

This PR fixes a regression introduced by:
- #4845

This introduced the reporting of xfailed and xpassed, but wrongly removed the reporting of the defaults failed and error.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Name the default config `default`
Opened by severo (closed)

Currently, if a dataset has no configuration, a default configuration is created from the dataset name.

For example, for a dataset loaded from the hub repository, such as https://huggingface.co/datasets/user/dataset (repo id is `user/dataset`), the default configuration will be `user--dataset`.

It might be easier to handle to set it to `default`, or another reserved word.

Comments:
Addressed in #5331.
Re-add code and und language tags
Opened by albertvillanova (closed)

This PR fixes the removal of 2 language tags done by:
- #4882

The tags are:
- "code": this is not a IANA tag but needed
- "und": this is one of the special scoped tags removed by 0d53202b9abce6fd0358cb00d06fcfd904b875af
  - used in "mc4" and "udhr" datasets

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Dataset Viewer issue for timit_asr
Opened by InayatUllah932 (closed)

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
Yes, the dataset viewer is based on `datasets`, and the following does not work:

```
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('timit_asr')
Downloading builder script: 7.48kB [00:00, 6.69MB/s]
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 354, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/timit_asr/43f9448dd5db58e95ee48a277f466481b151f112ea53e27f8173784da9254fb2/timit_asr.py", line 117, in _split_generators
    data_dir = os.path.abspath(os.path.expanduser(dl_manager.manual_dir))
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/posixpath.py", line 231, in expanduser
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

cc @huggingface/datasets 

Due to license restriction, this dataset needs manual downloading of the original data.

This information is in the dataset card: https://huggingface.co/datasets/timit_asr
> The dataset needs to be downloaded manually from https://catalog.ldc.upenn.edu/LDC93S1

Maybe a better error message for datasets that need manual downloading? @severo 

Maybe we can raise a specific excpetion as done from `load_dataset`...

Yes, ideally something like https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py#L81


The preview is now disabled (and a descriptive warning is displayed) for datasets requiring manual download. See:

![timit_asr-manual-download](https://user-images.githubusercontent.com/8515462/193578572-3d21b790-f848-4257-9e9b-7cab3d76a269.png)
datasets generate large arrow file
Opened by jax11235 (closed)

Checking the large file in disk, and found the large cache file in the cifar10 data directory:

![image](https://user-images.githubusercontent.com/18533904/186830449-ba96cdeb-0fe8-4543-994d-2abe7145933f.png)

As we know, the size of cifar10 dataset is ~130MB, but the cache file has almost 30GB size, there may be some problems here.

Comments:
Hi ! The cache files are the results of all the transforms you applied to the dataset using `map` for example.
Did you run a transform that could potentially blow up the size of the dataset ?

@lhoestq,
I don't remember, but I can't imagine what kind of transform may generate data that grow over 200 times in size. 
I think maybe it doesn' matter, it's just cache after all.
Fix missing tags in dataset cards
Opened by albertvillanova (closed)

Fix missing tags in dataset cards:
- anli
- coarse_discourse
- commonsense_qa
- cos_e
- ilist
- lc_quad
- web_questions
- xsum

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add citation to ro_sts and ro_sts_parallel datasets
Opened by albertvillanova (closed)

This PR adds the citation information to `ro_sts_parallel` and `ro_sts_parallel` datasets, once they have replied our request for that information:
- https://github.com/dumitrescustefan/RO-STS/issues/4

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4892). All of your documentation changes will be reflected on that endpoint.
Fix missing tags in dataset cards
Opened by albertvillanova (closed)

Fix missing tags in dataset cards:
- aslg_pc12
- librispeech_lm
- mwsc
- opus100
- qasc
- quail
- squadshifts
- winograd_wsc

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833


Comments:
add Dataset.from_list
Opened by sanderland (closed)

As discussed in #4885 

I initially added this bit at the end, thinking filling this field was necessary as it is done in from_dict. 
However, it seems the constructor takes care of filling info when it is empty.
```
if info.features is None:
  info.features = Features(
      {
          col: generate_from_arrow_type(coldata.type)
          for col, coldata in zip(pa_table.column_names, pa_table.columns)
      }
  )
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova it seems tests fail on pyarrow 6, perhaps from_pylist is a v7 method? How do you usually handle these version differences?
Added something that at least works
Dataset Viewer issue for subjqa
Opened by lewtun (closed)

### Link

https://huggingface.co/datasets/subjqa

### Description

Getting the following error for this dataset:

```
Status code:   500
Exception:     Status500Error
Message:       2 or more items returned, instead of 1
```

Not sure what's causing it though 🤔 

### Owner

Yes

Comments:
It's a bug in the viewer, thanks for reporting it. We're hoping to update to a new version in the next few days which should fix it.

Fixed 

https://huggingface.co/datasets/subjqa

<img width="1040" alt="Capture d’écran 2022-09-08 à 10 23 26" src="https://user-images.githubusercontent.com/1676121/189073210-2a57ff88-8bb1-44bd-851e-0e75473cea3f.png">
Create dataset from list of dicts
Opened by sanderland (closed)

I often find myself with data from a variety of sources, and a list of dicts is very common among these.
However, converting this to a Dataset is a little awkward, requiring either

```Dataset.from_pandas(pd.DataFrame(formatted_training_data))```
Which can error out on some more exotic values as 2-d arrays for reasons that are not entirely clear
> ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column labels with type object')

Alternatively:
```Dataset.from_dict({k: [s[k] for s in formatted_training_data] for k in  formatted_training_data[0].keys()})```
Which works, but is a little ugly.

**Describe the solution you'd like**
Either `.from_dict` accepting a list of dicts, or a `.from_records` function accepting such.

I am happy to PR this, just wanted to check you are happy to accept this I haven't missed something obvious, and which of the solutions would be prefered.


Comments:
Hi @sanderland, thanks for your enhancement proposal.

I agree with you that this would be useful.

Please note that under the hood, we use PyArrow tables as backend:
- The implementation  of `Dataset.from_dict` uses the PyArrow `Table.from_pydict`

Therefore, I would suggest:
- Implementing `Dataset.from_list` using the PyArrow `Table.from_pylist`

What do you think?
Let's see if other people have other suggestions...

Thanks for the quick and positive reply @albertvillanova! 
`from_list` seems sensible. Have opened a PR so we can discuss details there.

Resolved via #4890.
Fix language tags resource file
Opened by albertvillanova (closed)

This PR fixes/updates/adds ALL language tags from IANA (as of 2022-08-08).

This PR also removes all BCP47 suffixes (the languages file only contains language subtags, i.e. ISO 639 1 or 2 codes; no script/region/variant suffixes). See:
- #4753

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4882). All of your documentation changes will be reflected on that endpoint.
Added names of less-studied languages
Opened by BenjaminGalliot (closed)

Added names of less-studied languages (nru – Narua and jya – Japhug) for existing datasets.

Comments:
OK, I removed Glottolog codes and only added ISO 639-3 ones. The former are for the moment in corpus card description, language details, and in subcorpora names.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4880). All of your documentation changes will be reflected on that endpoint.
Fix Citation Information section in dataset cards
Opened by albertvillanova (closed)

Fix Citation Information section in dataset cards:
- cc_news
- conllpp
- datacommons_factcheck
- gnad10
- id_panl_bppt
- jigsaw_toxicity_pred
- kinnews_kirnews
- kor_sarcasm
- makhzan
- reasoning_bg
- ro_sts
- ro_sts_parallel
- sanskrit_classic
- telugu_news
- thaiqa_squad
- wiki_movies

This PR partially fixes the Citation Information section in dataset cards. Subsequent PRs will follow to complete this task.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4879). All of your documentation changes will be reflected on that endpoint.
[not really a bug] `identical_ok` is deprecated in huggingface-hub's `upload_file`
Opened by severo (closed)

In the huggingface-hub dependency, the `identical_ok` argument has no effect in `upload_file` (and it will be removed soon)

See

https://github.com/huggingface/huggingface_hub/blob/43499582b19df1ed081a5b2bd7a364e9cacdc91d/src/huggingface_hub/hf_api.py#L2164-L2169

It's used here:

https://github.com/huggingface/datasets/blob/fcfcc951a73efbc677f9def9a8707d0af93d5890/src/datasets/dataset_dict.py#L1373-L1381

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4354-L4362

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4197-L4213

We should remove it.

Maybe the third code sample has an unexpected behavior since it uses the non-default value `identical_ok = False`, but the argument is ignored.

Comments:
Resolved via https://github.com/huggingface/datasets/pull/4937.
Multiple dataloader memory error
Opened by cyk1337 (open)

For the use of multiple datasets and tasks, we use around more than 200+ dataloaders, then pass it into `dataloader1, dataloader2, ..., dataloader200=accelerate.prepare(dataloader1, dataloader2, ..., dataloader200)`
It causes the memory error when generating batches. Any solutions to it?

```bash
  File "/home/xxx/my_code/src/utils/data_utils.py", line 54, in generate_batch
    x = next(iterator)
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 301, in __iter__
    for batch in super().__iter__():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 28, in fetch
    data.append(next(self.dataset_iter))
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 249, in __iter__
    for element in self.dataset:
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 503, in __iter__
    for key, example in self._iter():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 500, in _iter
    yield from ex_iterable
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 231, in __iter__
    new_key = "_".join(str(key) for key in keys)
MemoryError
```

Comments:
Hi!

200+ data loaders is a lot. Have you tried to reduce the number of datasets by concatenating/interleaving the ones with the same structure/task (the API is `{concatenate_datasets/interleave_datasets}([dset1, ..., dset_N])`)?

Hi @mariosasko, thank you for your reply. I tried pre-concatenating different datasets into one, but one key need is to keep each batch the same data type. Considering that the concatenate-then-segment operation for prefetched samples may span across different data types after concatenating/interleaving (cuz different data sources are mixed), any solution to remain the same data source for each batch?

@cyk1337 have you found any solutions to it?
@mariosasko I tried with interleave_datasets to sample batches from two large datasets (wikipedia alike) and it results in out-of-memory error during data loading (16gpus, >1TB physical memory). Do you have any idea about it?
Docs for creating an audio dataset
Opened by stevhliu (closed)

This PR is a first draft of how to create audio datasets (`AudioFolder` and loading script). Feel free to let me know if there are any specificities I'm missing for this. 🙂

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Awesome thanks ! I think we can also encourage TAR archives as for image dataset scripts (feel free to copy paste some parts from there lol)

Thanks for all the great feedback @polinaeterna and @lhoestq! 🥰

I added all the other feedback, and I'll look into the `librivox-indonesia` script now!

If you don't mind, I'm taking over this PR since we'll do a release pretty soon

@lhoestq no, I do :D 

haha sorry ^^
Fix: wmt datasets - fix CWMT zh subsets
Opened by lhoestq (closed)

Fix https://github.com/huggingface/datasets/issues/4575

TODO: run `datasets-cli test`:
- [x] wmt17
- [x] wmt18
- [x] wmt19

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4871). All of your documentation changes will be reflected on that endpoint.
adding mafand to datasets
Opened by dadelani (closed)

I'm addding the MAFAND dataset by Masakhane based on the paper/repository below:

Paper: https://aclanthology.org/2022.naacl-main.223/
Code: https://github.com/masakhane-io/lafand-mt

Please, help merge this

Everything works except for creating dummy data file

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @dadelani, thanks for your awesome contribution!!! :heart: 

However, now we are using the Hub to add new datasets, instead of this GitHub repo. 

You could share this dataset under your Hub organization namespace: [Masakhane NLP](https://huggingface.co/masakhane). This way the dataset will be accessible using:
```python
ds = load_dataset("masakhane/mafand")
```

You have the procedure documented in our online docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Share](https://huggingface.co/docs/datasets/share)

Moreover, datasets shared on the Hub no longer need the dummy data files.

Please, feel free to ping me if you need any further guidance/support.

thank you for the comment. I have moved it to the Hub https://huggingface.co/datasets/masakhane/mafand

Great job, @dadelani!!

Please, note that in the README.md file, the YAML tags should be preceded and followed by three dashes `---`, so that they are properly parsed. See, e.g.: https://raw.githubusercontent.com/huggingface/datasets/main/templates/README.md

Also you could replace the line:
```
# Dataset Card for [Needs More Information]
```
with
```
# Dataset Card for MAFAND-MT
```

Great, thank you for the feedback. I have fixed both issues.
Dataset Viewer issue for MoritzLaurer/multilingual_nli
Opened by MoritzLaurer (closed)

### Link

_No response_

### Description

I've just uploaded a new dataset to the hub and the viewer does not work for some reason, see here: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

It displays the error: 
```
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

Weirdly enough the dataviewer works for an earlier version of the same dataset. The only difference is that it is smaller, but I'm not aware of other changes I have made: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli_test

Do you know why the dataviewer is not working?

### Owner

_No response_

Comments:
Thanks for reporting  @MoritzLaurer.

Currently, the dataset preview is working properly: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

Please note that when a dataset is modified, it might take some time until the preview is completely updated.

@severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Thanks for your response. You are right, its now working well. I had waited for 30 min or so and refreshed several times and thought there was some other error. Yeah, a different error message sounds like a good idea to avoid confusion. 

I'm closing this issue then.

> @severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Yes, it's a known issue, and we're about to ship a better version
Allow pathlib PoxisPath in Dataset.read_json
Opened by changjonathanc (open)

**Is your feature request related to a problem? Please describe.**
```
from pathlib import Path
from datasets import Dataset
ds = Dataset.read_json(Path('data.json'))
```
causes an error
```
AttributeError: 'PosixPath' object has no attribute 'decode'
```

**Describe the solution you'd like**

It should be able to accept PosixPath and read the json from inside.

Comments:
This same error will occur using `ds = datasets.load_dataset('json', data_files=['test.jsonl'])`

@cccntu I want to make a quick fix for this, but I am struggling to find where the json dataset builder is. Do you know?

@vvvm23 I think you mean think:
```python
ds = datasets.load_dataset('json', data_files=[Path('test.jsonl')])
```
And the place you want to modify is here:
```
utils/file_utils.py:64, in is_remote_url(url_or_filename)
     63 def is_remote_url(url_or_filename: str) -> bool:
---> 64     parsed = urlparse(url_or_filename)
     65     return parsed.scheme in ("http", "https", "s3", "gs", "hdfs", "ftp")
```

Probably just need to check first if `url_or_filename` is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Btw, I tried installing from main, and ran my code above and got a different error. Probably because the API have changed.
`AttributeError: module 'datasets' has no attribute 'read_json'`


> @vvvm23 I think you mean think:

You are correct, thanks!

> Probably just need to check first if url_or_filename is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Is PathLike sufficient, or should I check the file exists here? Or is that handled later?

I think here we just want to avoid passing Path to urlparse. A simpler solution is to add a str() call and convert the input to string before passing to the next step. No need to check anything.

Above PR should do your first suggestion. Hope that works for you, as I am going on holiday and won't be able to change much :wink: 

I just came across this problem with `DatasetDict.from_parquet()`. The error still seems to originate from the same place:
```
  File "/usr/local/lib/python3.12/dist-packages/datasets/utils/file_utils.py", line 88, in is_relative_path
    return urlparse(url_or_filename).scheme == "" and not os.path.isabs(url_or_filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
```
The PR looks like it would still solve everything. Is there a reason this got delayed?