helmo/github-issues huggingface dataset:
Datasets conflicts with fsspec 2024.9
State: open | Author: cw-igormorgado (NONE)

### Describe the bug

Installing both in latest versions are not possible

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`

But using older version of datasets is ok

`pip install "datasets==1.24.4" "fsspec==2024.9.0"`



### Steps to reproduce the bug

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`


### Expected behavior

install both versions.

### Environment info

debian 11.
python 3.10.15

Comments: Yes, I need to use the latest version of fsspec and datasets for my usecase. 
https://github.com/fsspec/s3fs/pull/888#issuecomment-2404204606
https://github.com/apache/arrow/issues/34363#issuecomment-2403553473

last version where things install without conflict is: 2.14.4

So this issue starts from:
https://github.com/huggingface/datasets/releases/tag/2.14.5
Audio preview in dataset viewer for audio array data without a path/filename
State: open | Author: Lauler (NONE)
Labels: name: enhancement

### Feature request

Huggingface has quite a comprehensive set of guides for [audio datasets](https://huggingface.co/docs/datasets/en/audio_dataset). It seems, however, all these guides assume the audio array data to be decoded/inserted into a HF dataset always originates from individual files. The [Audio-dataclass](https://github.com/huggingface/datasets/blob/3.0.1/src/datasets/features/audio.py#L20) appears designed with this assumption in mind. Looking at its source code it returns a dictionary with the keys `path`, `array` and `sampling_rate`. 

However, sometimes users may have different pipelines where they themselves decode the audio array. This feature request has to do with wishing some clarification in guides on whether it is possible, and in such case how users can insert already decoded audio array data into datasets (pandas DataFrame, HF dataset or whatever) that are later saved as parquet, and still get a functioning audio preview in the dataset viewer. 

Do I perhaps need to write a tempfile of my audio array slice to wav and capture the bytes object with `io.BytesIO` and pass that to `Audio()`? 

### Motivation

I'm working with large audio datasets, and my pipeline reads (decodes) audio from larger files, and slices the relevant portions of audio from that larger file based on metadata I have available. 

The pipeline is designed this way to avoid having to store multiple copies of data, and to avoid having to store tens of millions of small files. 

I tried [test-uploading parquet files](https://huggingface.co/datasets/Lauler/riksdagen_test) where I store the audio array data of decoded slices of audio in an `audio` column with a dictionary with the keys `path`, `array` and `sampling_rate`. But I don't know the secret sauce of what the Huggingface Hub expects and requires to be able to display audio previews correctly. 

### Your contribution

I could contribute a tool agnostic guide of creating HF audio datasets directly as parquet to the HF documentation if there is an interest. Provided you help me figure out the secret sauce of what the dataset viewer expects to display the preview correctly.
Pin multiprocess<0.70.1 to align with dill<0.3.9
State: closed | Author: albertvillanova (MEMBER)

Pin multiprocess<0.70.1 to align with dill<0.3.9.

Note that multiprocess-0.70.1 requires dill-0.3.9: https://github.com/uqfoundation/multiprocess/releases/tag/0.70.17

Fix #7186.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7188). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
shard_data_sources() got an unexpected keyword argument 'worker_id'
State: open | Author: Qinghao-Hu (NONE)

### Describe the bug

```
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 238, in __iter__
[rank0]:     for key_example in islice(self.generate_examples_fn(**gen_kwags), shard_example_idx_start, None):
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py", line 32, in _generate_examples
[rank0]:     for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
[rank0]:   File "/home/qinghao/workdir/doremi/doremi/dataloader.py", line 337, in take_data_generator
[rank0]:     for ex in ds:
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1791, in __iter__
[rank0]:     yield from self._iter_pytorch()
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1704, in _iter_pytorch
[rank0]:     ex_iterable = ex_iterable.shard_data_sources(worker_id=worker_info.id, num_workers=worker_info.num_workers)
[rank0]: TypeError: UpdatableRandomlyCyclingMultiSourcesExamplesIterable.shard_data_sources() got an unexpected keyword argument 'worker_id'
```

### Steps to reproduce the bug

IterableDataset cannot use

### Expected behavior

can work on datasets==2.10,  but will raise error for later versions.

### Environment info

datasets==3.0.1
pinning `dill<0.3.9` without pinning `multiprocess` 
State: closed | Author: shubhbapna (NONE)

### Describe the bug

The [latest `multiprocess` release](https://github.com/uqfoundation/multiprocess/releases/tag/0.70.17) requires `dill>=0.3.9` which causes issues when installing `datasets` without backtracking during package version resolution. Is it possible to add a pin for multiprocess so something like `multiprocess<=0.70.16` so that the `dill` version is compatible?

### Steps to reproduce the bug

NA

### Expected behavior

NA

### Environment info

NA
CI benchmarks are broken
State: closed | Author: albertvillanova (MEMBER)
Labels: name: maintenance

Since Aug 30, 2024, CI benchmarks are broken: https://github.com/huggingface/datasets/actions/runs/11108421214/job/30861323975
```
{"level":"error","message":"Resource not accessible by integration","name":"HttpError","request":{"body":"{\"body\":\"<details>\\n<summary>Show benchmarks</summary>\\n\\nPyArrow==8.0.0\\n\\n<details>\\n<summary>Show updated benchmarks!</summary>\\n\\n### Benchmark: benchmark_array_xd.json\\n\\n| metric | read_batch_formatted_as_numpy after write_array2d |
...
"headers":{"accept":"application/vnd.github.v3+json","authorization":"token [REDACTED]","content-type":"application/json; charset=utf-8","user-agent":"octokit-rest.js/18.0.0 octokit-core.js/3.6.0 Node.js/16.20.2 (linux; x64)"},"method":"POST","request":{"agent":{"_events":{},"_eventsCount":2,"cache":
...
"response":{"data":{"documentation_url":"https://docs.github.com/rest/issues/comments#create-an-issue-comment","message":"Resource not accessible by integration","status":"403"},
...
"stack":"HttpError: Resource not accessible by integration\n    at /usr/lib/node_modules/@dvcorg/cml/node_modules/@octokit/request/dist-node/index.js:86:21\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Job.doExecute (/usr/lib/node_modules/@dvcorg/cml/node_modules/bottleneck/light.js:405:18)","status":403}
```

Comments: Fixed by #7205
Pin dill<0.3.9 to fix CI
State: closed | Author: albertvillanova (MEMBER)

Pin dill<0.3.9 to fix CI for deps-latest.

Note that dill-0.3.9 was released yesterday Sep 29, 2024:
- https://pypi.org/project/dill/0.3.9/
- https://github.com/uqfoundation/dill/releases/tag/0.3.9

Fix #7183.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7184). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
CI is broken for deps-latest
State: closed | Author: albertvillanova (MEMBER)

See: https://github.com/huggingface/datasets/actions/runs/11106149906/job/30853879890

```
=========================== short test summary info ============================
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_caching_on_disk - AssertionError: Lists differ: [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}] != [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]

First differing element 1:
{'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}
{'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}

  [{'filename': '/tmp/tmp6xcyyjs4/dataset0.arrow'},
-  {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}]
?                                        ^^^^^ --------

+  {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]
?                                       ++++++++++ ^^  +
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - AssertionError: Lists differ: [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}] != [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]

First differing element 0:
{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}
{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}

- [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}]
?                                         ^^ -----------

+ [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]
?                                       +++++++++++  ^^
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: '52e56ee04ad92499' != '0a4f75cec280f634'
- 52e56ee04ad92499
+ 0a4f75cec280f634
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: 'a6bd2041ca63d6c0' != '517bf36b7eecdef5'
- a6bd2041ca63d6c0
+ 517bf36b7eecdef5
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NameError: name 'log' is not defined
= 11 failed, 2850 passed, 3 skipped, 23 warnings, 1 error in 191.06s (0:03:11) =
```
Support features in metadata configs
State: closed | Author: albertvillanova (MEMBER)

Support features in metadata configs, like:
```
configs:
  - config_name: default
    features:
      - name: id
        dtype: int64
      - name:  name
        dtype: string
      - name: score
        dtype: float64
```

This will allow to avoid inference of data types.

Currently, we allow passing this information in the `dataset_info` (instead of `configs`) field, but this is not intuitive and it is not properly documented.

TODO:
- [ ] Document usage

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7182). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The CI issue is unrelated:
- #7183
Fix datasets export to JSON
State: closed | Author: varadhbhatnagar (CONTRIBUTOR)


Comments: Linked Issue: #7037
Ideas: #7039 

@albertvillanova / @lhoestq any early feedback?

AFAIK there is no param `orient` in `load_dataset()`. So for orientations other than "records", the loading isn't very accurate. Any thoughts?

`orient = "split"` can also be handled. I will add the changes soon

Thanks for diving into this ! I don't think we want the JSON export to be that complex though, especially if people can do `ds.to_pandas().to_json(orient=...)`. Maybe we can just raise an error and suggest users to use pandas ? And also note that it loads the full dataset in memory so it's mainly for small scale datasets. The only acceptable option for large scale datasets is probably just JSON Lines anyway since it enables streaming.

@lhoestq Simply doing `ds.to_pandas().to_json(orient=...)` is not going to give any batching or multiprocessing benefits right? Also, which function are you referring to - when you say that its meant for small scale datasets only?

Yes indeed. Though I think it's fine since using something else than orient="lines" is only suitable/useful for small datasets. Or you know a case where a big dataset need to be in a format that is not orient="lines" ?

@lhoestq Let me close this PR and open another one where I will add an error message, as suggested here.

> Thanks for diving into this ! I don't think we want the JSON export to be that complex though, especially if people can do `ds.to_pandas().to_json(orient=...)`. Maybe we can just raise an error and suggest users to use pandas ? And also note that it loads the full dataset in memory so it's mainly for small scale datasets. The only acceptable option for large scale datasets is probably just JSON Lines anyway since it enables streaming.



Addressed here: #7273 
@lhoestq
[FSTimeoutError] load_dataset
State: closed | Author: cosmo3769 (NONE)

### Describe the bug

When using `load_dataset`to load [HuggingFaceM4/VQAv2](https://huggingface.co/datasets/HuggingFaceM4/VQAv2), I am getting `FSTimeoutError`. 

### Error

```
TimeoutError: 

The above exception was the direct cause of the following exception:

FSTimeoutError                            Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py](https://klh9mr78js-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240924-060116_RC00_678132060#) in sync(loop, func, timeout, *args, **kwargs)
     99     if isinstance(return_result, asyncio.TimeoutError):
    100         # suppress asyncio.TimeoutError, raise FSTimeoutError
--> 101         raise FSTimeoutError from return_result
    102     elif isinstance(return_result, BaseException):
    103         raise return_result

FSTimeoutError:
```

It usually fails around 5-6 GB.

<img width="847" alt="Screenshot 2024-09-26 at 9 10 19 PM" src="https://github.com/user-attachments/assets/ff91995a-fb55-4de6-8214-94025d6c8470">


### Steps to reproduce the bug

To reproduce it, run this in colab notebook:

```
!pip install -q -U datasets

from datasets import load_dataset
ds = load_dataset('HuggingFaceM4/VQAv2', split="train[:10%]")
```



### Expected behavior

It should download properly.

### Environment info

Using Colab Notebook.

Comments: Is this `FSTimeoutError` due to download network issue from remote resource (from where it is being accessed)?

It seems to happen for all datasets, not just a specific one, and especially for versions after 3.0. (3.0.0, 3.0.1 have this problem)

I had the same error on a different dataset, but after downgrading to datasets==2.21.0, the problem was solved.

Same as https://github.com/huggingface/datasets/issues/7164

This dataset is made of a python script that downloads data from elsewhere than HF, so availability depends on the original host. Ultimately it would be nice to host the files of this dataset on HF

in `datasets` <3.0 there were lots of mechanisms that got removed after the decision to make datasets with python loading scripts legacy for security and maintenance reasons (we only do very basic support now)

@lhoestq Thank you for the clarification! Closing the issue.

I'm getting this too, and also at 5 minutes.  But for `CSTR-Edinburgh/vctk`, so it's not just this dataset, it seems to be a timeout that was introduced and needs to be raised.  The progress bar was moving along just fine before the timeout, and I get more or less of it depending on how fast the network is.

You can change the `aiohttp` timeout from 5min to 1h like this:

```python
import datasets, aiohttp
dataset = datasets.load_dataset(
    dataset_name,
    storage_options={'client_kwargs': {'timeout': aiohttp.ClientTimeout(total=3600)}}
)
```

@JonasLoos Solution solved a download timeout error I received when downloading  `"HuggingFaceM4/VQAv2"` 🎉
Add torchdata as a regular test dependency
State: closed | Author: albertvillanova (MEMBER)

Add `torchdata` as a regular test dependency.

Note that previously, `torchdata` was installed from their repo and current main branch (0.10.0.dev) requires Python>=3.9.

Also note they made a recent release: 0.8.0 on Jul 31, 2024.

Fix #7171.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7172). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
CI is broken: No solution found when resolving dependencies
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

See: https://github.com/huggingface/datasets/actions/runs/11046967444/job/30687294297
```
Run uv pip install --system -r additional-tests-requirements.txt --no-deps
  × No solution found when resolving dependencies:
  ╰─▶ Because the current Python version (3.8.18) does not satisfy Python>=3.9
      and torchdata==0.10.0a0+1a98f21 depends on Python>=3.9, we can conclude
      that torchdata==0.10.0a0+1a98f21 cannot be used.
      And because only torchdata==0.10.0a0+1a98f21 is available and
      you require torchdata, we can conclude that your requirements are
      unsatisfiable.
Error: Process completed with exit code 1.
```
Support JSON lines with missing columns
State: closed | Author: albertvillanova (MEMBER)

Support JSON lines with missing columns.

Fix #7169.

The implemented test raised:
```
datasets.table.CastError: Couldn't cast
age: int64
to
{'age': Value(dtype='int32', id=None), 'name': Value(dtype='string', id=None)}
because column names don't match
```

Related to:
- #7160 
- #7162

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7170). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix docstring code example for distributed shuffle
State: closed | Author: lhoestq (MEMBER)

close https://github.com/huggingface/datasets/issues/7163

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7166). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix increase_load_count
State: closed | Author: lhoestq (MEMBER)

it was failing since 3.0 and therefore not updating download counts on HF or in our dashboard

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7165). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I tested a few load_dataset and they do show up in download stats now

Thanks for having noticed and fixed.
Set explicit seed in iterable dataset ddp shuffling example 
State: closed | Author: alex-hh (CONTRIBUTOR)

### Describe the bug

In the examples section of the iterable dataset docs https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDataset
the ddp example shuffles without seeding

```python
from datasets.distributed import split_dataset_by_node
ids = ds.to_iterable_dataset(num_shards=512)
ids = ids.shuffle(buffer_size=10_000)  # will shuffle the shards order and use a shuffle buffer when you start iterating
ids = split_dataset_by_node(ds, world_size=8, rank=0)  # will keep only 512 / 8 = 64 shards from the shuffled lists of shards when you start iterating
dataloader = torch.utils.data.DataLoader(ids, num_workers=4)  # will assign 64 / 4 = 16 shards from this node's list of shards to each worker when you start iterating
for example in ids:
    pass
```

This code would - I think - raise an error due to the lack of an explicit seed:

https://github.com/huggingface/datasets/blob/2eb4edb97e1a6af2ea62738ec58afbd3812fc66e/src/datasets/iterable_dataset.py#L1707-L1711


### Steps to reproduce the bug

Run example code

### Expected behavior

Add explicit seeding to example code

### Environment info

latest datasets

Comments: thanks for reporting !
Support JSON lines with empty struct
State: closed | Author: albertvillanova (MEMBER)

Support JSON lines with empty struct.

Fix #7161.

Related to:
- #7160

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7162). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
JSON lines with empty struct raise ArrowTypeError
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

JSON lines with empty struct raise ArrowTypeError: struct fields don't match or are in the wrong order

See example: https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5

> ArrowTypeError: struct fields don't match or are in the wrong order: Input fields: struct<> output fields: struct<pov_count: int64, update_count: int64, citation_needed_count: int64>

Related to:
- #7159
Support JSON lines with missing struct fields
State: closed | Author: albertvillanova (MEMBER)

Support JSON lines with missing struct fields.

Fix #7159.

The implemented test raised:
```
TypeError: Couldn't cast array of type
struct<age: int64>
to
{'age': Value(dtype='int32', id=None), 'name': Value(dtype='string', id=None)}
```

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7160). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
JSON lines with missing struct fields raise TypeError: Couldn't cast array
State: closed | Author: albertvillanova (MEMBER)
Labels: name: bug

JSON lines with missing struct fields raise TypeError: Couldn't cast array of type.

See example: https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5

One would expect that the struct missing fields are added with null values.

Comments: Hello,

I have still the same issue when loading the dataset with the new version:
[https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5](https://huggingface.co/datasets/wikimedia/structured-wikipedia/discussions/5)

I have downloaded and unzipped the wikimedia/structured-wikipedia dataset locally but when loading I have the same issue.

```
import datasets

dataset = datasets.load_dataset("/gpfsdsdir/dataset/HuggingFace/wikimedia/structured-wikipedia/20240916.fr")
```
```
TypeError: Couldn't cast array of type
struct<content_url: string, width: int64, height: int64, alternative_text: string>
to
{'content_url': Value(dtype='string', id=None), 'width': Value(dtype='int64', id=None), 'height': Value(dtype='int64', id=None)}

The above exception was the direct cause of the following exception:
```
My version of datasets is 3.0.1
Fix zero proba interleave datasets
State: closed | Author: lhoestq (MEMBER)

fix https://github.com/huggingface/datasets/issues/7147

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7157). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.