helmo/github-issues huggingface dataset:
Enable the Audio Feature to decode / read with an offset + duration
State: open | Author: TParcollet (NONE)
Labels: name: enhancement

### Feature request

For most large speech dataset, we do not wish to generate hundreds of millions of small audio samples. Instead, it is quite common to provide larger audio files with frame offset (soundfile start and stop arguments). We should be able to pass these arguments to Audio() (column ID corresponding in the dataset row).

### Motivation

I am currently generating a fairly big dataset to .parquet(). Unfortunately, it does not work because all existing functions load the whole .wav file corresponding to the row. All my attempts at bypassing this failed. We should be able to put in the Table only the bytes corresponding to what soundfile reads with an offset (and subset of the audio file). 

### Your contribution

I can totally test whatever code on my large dataset creation script. 

Comments: Hi ! What about having audio + start + duration columns and enable something like this ?

```python
for example in ds:
    array = example["audio"].read(start=example["start"], frames=example["duration"])
```

Hi @lhoestq, this would work with a file-based dataset but would be terrible for a sharded one as it would duplicate the large audio file many times. Also, very long audio files are not embedded very well in the parquet file, even with large_binary(). It crashed a few times for me until I switched to one sample == one file :-(
Faster parquet streaming + filters with predicate pushdown
State: closed | Author: lhoestq (MEMBER)

ParquetFragment.to_batches uses a buffered stream to read parquet data, which makes streaming faster (x2 on my laptop).

I also added the `filters` config parameter to support filtering with predicate pushdown, e.g.

```python
from datasets import load_dataset

filters = [('problem_source', '==', 'math')]
ds = load_dataset("nvidia/OpenMathInstruct-2", streaming=True, filters=filters)
first_example = next(iter(ds["train"]))
print(first_example["problem_source"])
# 'math'
```

cc @allisonwang-db this is a nice plus for usage in spark

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7309). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Creating new dataset from list loses information. (Audio Information Lost - either Datatype or Values).
State: open | Author: ai-nikolai (NONE)

### Describe the bug

When creating a dataset from a list of datapoints, information is lost of the individual items.

Specifically, when creating a dataset from a list of datapoints (from another dataset). Either the datatype is lost or the values are lost. See examples below. 

-> What is the best way to create a dataset from a list of datapoints?

---
e.g.:
**When running this code:**
```python
from datasets import load_dataset, Dataset
commonvoice_data = load_dataset("mozilla-foundation/common_voice_17_0", "it", split="test", streaming=True)
datapoint = next(iter(commonvoice_data))
out = [datapoint]
new_data = Dataset.from_list(out) #this loses datatype information
new_data2= Dataset.from_list(out,features=commonvoice_data.features) #this loses value information
```

**We get the following**:
---
1. `datapoint`: (the original datapoint)
```
'audio': {'path': 'it_test_0/common_voice_it_23606167.mp3', 'array': array([0.00000000e+00, 0.00000000e+00, 0.00000000e+00, ...,
       2.21619011e-05, 2.72628222e-05, 0.00000000e+00]), 'sampling_rate': 48000}
 ```
Original Dataset Features:
```
>>> commonvoice_data.features
'audio': Audio(sampling_rate=48000, mono=True, decode=True, id=None)
```
 - Here we see column "audio", has the proper values (both `path` & and `array`) and has the correct datatype (Audio).

 
 ----
 2. new_data[0]:
```
# Cannot be printed (as it prints the entire array).
```
New Dataset 1 Features:
```
>>> new_data.features
'audio': {'array': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None), 'path': Value(dtype='string', id=None), 'sampling_rate': Value(dtype='int64', id=None)}
```
 - Here we see that the column "audio", has the correct values, but is not the Audio datatype anymore.

---
3. new_data2[0]:
```
'audio': {'path': None, 'array': array([0., 0., 0., ..., 0., 0., 0.]), 'sampling_rate': 48000},
```
New Dataset 2 Features:
```
>>> new_data2.features
'audio': Audio(sampling_rate=48000, mono=True, decode=True, id=None),
```
 - Here we see that the column "audio", has the correct datatype, but all the array & path values were lost!



### Steps to reproduce the bug

## Run:
```python
from datasets import load_dataset, Dataset
commonvoice_data = load_dataset("mozilla-foundation/common_voice_17_0", "it", split="test", streaming=True)
datapoint = next(iter(commonvoice_data))
out = [datapoint]
new_data = Dataset.from_list(out) #this loses datatype information
new_data2= Dataset.from_list(out,features=commonvoice_data.features) #this loses value information
```





### Expected behavior

## Expected:
```datapoint == new_data[0]```

AND

```datapoint == new_data2[0]```

### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1
Build Documentation Test Fails Due to "Bad Credentials" Error
State: open | Author: ruidazeng (CONTRIBUTOR)

### Describe the bug

The `Build documentation / build / build_main_documentation (push)` job is consistently failing during the "Syncing repository" step. The error occurs when attempting to determine the default branch name, resulting in "Bad credentials" errors.

### Steps to reproduce the bug

1. Trigger the `build_main_documentation` job.
2. Observe the logs during the "Syncing repository" step.

### Expected behavior

The workflow should be able to retrieve the default branch name without encountering credential issues.

### Environment info

```plaintext
Syncing repository: huggingface/notebooks
Getting Git version info
Temporarily overriding HOME='/home/runner/work/_temp/00e62748-9940-4a4f-bbbc-eb2cda6d7ed6' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /home/runner/work/datasets/datasets/notebooks
Initializing the repository
Disabling automatic garbage collection
Setting up auth
Determining the default branch
  Retrieving the default branch name
  Bad credentials - https://docs.github.com/rest
  Waiting 20 seconds before trying again
  Retrieving the default branch name
  Bad credentials - https://docs.github.com/rest
  Waiting 19 seconds before trying again
  Retrieving the default branch name
  Error: Bad credentials - https://docs.github.com/rest
```

Comments: how were you able to fix this please?

> how were you able to fix this please?

I was not able to fix this.
Update iterable_dataset.py
State: closed | Author: lhoestq (MEMBER)

close https://github.com/huggingface/datasets/issues/7297

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7304). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
DataFilesNotFoundError for datasets LM1B
State: closed | Author: hml1996-fight (NONE)

### Describe the bug

Cannot load the dataset https://huggingface.co/datasets/billion-word-benchmark/lm1b

### Steps to reproduce the bug

`dataset = datasets.load_dataset('lm1b', split=split)`

### Expected behavior

`Traceback (most recent call last):
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/word_freq.py", line 13, in <module>
    train_data = DiffusionLoader(tokenizer=tokenizer).my_load(task_name='lm1b', splits=['train'])[0]
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/dataloader.py", line 20, in my_load
    return [self._load(task_name, name) for name in splits]
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/dataloader.py", line 20, in <listcomp>
    return [self._load(task_name, name) for name in splits]
  File "/home/hml/projects/DeepLearning/Generative_model/Diffusion-BERT/dataloader.py", line 13, in _load
    dataset = datasets.load_dataset('lm1b', split=split)
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 2594, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 2266, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 1827, in dataset_module_factory
    ).get_module()
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 1040, in get_module
    module_name, default_builder_kwargs = infer_module_for_data_files(
  File "/home/hml/.conda/envs/DB/lib/python3.10/site-packages/datasets/load.py", line 598, in infer_module_for_data_files
    raise DataFilesNotFoundError("No (supported) data files found" + (f" in {path}" if path else ""))
datasets.exceptions.DataFilesNotFoundError: No (supported) data files found in lm1b`

### Environment info

datasets: 2.20.0

Comments: Hi ! Can you try with a more recent version of `datasets` ? Also you might need to pass trust_remote_code=True since it's a script based dataset
Let server decide default repo visibility
State: closed | Author: Wauplin (CONTRIBUTOR)

Until now, all repos were public by default when created without passing the `private` argument. This meant that passing `private=False` or `private=None` was strictly the same. This is not the case anymore. Enterprise Hub offers organizations to set a default visibility setting for new repos. This is useful for organizations forbidding public repos for security matters. This PR mostly updates docstrings + default values so that `private=None` is always passed when users don't set it manually.

This PR doesn't create any breaking change. The real update has been done server-side when introducing the new Enterprise Hub feature. Related to https://github.com/huggingface/huggingface_hub/pull/2679.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7302). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

No need for a specific version of huggingface_hub to avoid a breaking change no (it's a server-side change)
update load_dataset doctring
State: closed | Author: lhoestq (MEMBER)

- remove canonical dataset name
- remove dataset script logic
- add streaming info
- clearer download and prepare steps

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7301). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: update elasticsearch version
State: closed | Author: ruidazeng (CONTRIBUTOR)

This should fix the `test_py311 (windows latest, deps-latest` errors.

```
=========================== short test summary info ===========================
ERROR tests/test_search.py - AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.
ERROR tests/test_search.py - AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead.
===== 2822 passed, 54 skipped, 10 warnings, 2 errors in 373.36s (0:06:13) =====
Error: Process completed with exit code 1.
```

The elasticsearch version used is `elasticsearch==7.9.1`, which is 4 years old and uses the removed `numpy.float_`.

elasticsearch fixed this in [https://github.com/elastic/elasticsearch-py/pull/2551](https://github.com/elastic/elasticsearch-py/pull/2551) and released in 8.15.0 (August 2024) and 7.17.12 (September 2024).


Comments: May I request a review @lhoestq

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7300). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Efficient Image Augmentation in Hugging Face Datasets
State: open | Author: fabiozappo (NONE)

### Describe the bug

 I'm using the Hugging Face datasets library to load images in batch and would like to apply a torchvision transform to solve the inconsistent image sizes in the dataset and apply some on the fly image augmentation. I can just think about using the collate_fn, but seems quite inefficient. 
 
 I'm new to the Hugging Face datasets library, I didn't find nothing in the documentation or the issues here on github.

Is there an existing way to add image transformations directly to the dataset loading pipeline? 

### Steps to reproduce the bug

from datasets import load_dataset
from torch.utils.data import DataLoader

```python
def collate_fn(batch):
    images = [item['image'] for item in batch]
    texts = [item['text'] for item in batch]
    return {
        'images': images,
        'texts': texts
    }

dataset = load_dataset("Yuki20/pokemon_caption", split="train")
dataloader = DataLoader(dataset, batch_size=4, collate_fn=collate_fn)

# Output shows varying image sizes:
# [(1280, 1280), (431, 431), (789, 789), (769, 769)]
```

### Expected behavior

I'm looking for a way to resize images on-the-fly when loading the dataset, similar to PyTorch's Dataset.__getitem__ functionality. This would be more efficient than handling resizing in the collate_fn.


### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.11.10
- `huggingface_hub` version: 0.26.2
- PyArrow version: 18.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
wrong return type for `IterableDataset.shard()`
State: closed | Author: ysngshn (NONE)

### Describe the bug

`IterableDataset.shard()` has the wrong typing for its return as `"Dataset"`. It should be `"IterableDataset"`. Makes my IDE unhappy.

### Steps to reproduce the bug

look at [the source code](https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py#L2668)?

### Expected behavior

Correct return type as `"IterableDataset"`

### Environment info

datasets==3.1.0

Comments: Oops my bad ! thanks for reporting
Updated inconsistent output in documentation examples for `ClassLabel`
State: closed | Author: sergiopaniego (MEMBER)

fix #7129 

@stevhliu

Comments: Updated! 😄 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7293). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq, can you help with this failing test please? 🙏
Why return_tensors='pt' doesn't work？
State: open | Author: bw-wang19 (NONE)

### Describe the bug

I tried to add input_ids to dataset with map(), and I used the return_tensors='pt', but why I got the callback with the type of List？
![image](https://github.com/user-attachments/assets/ab046e20-2174-4e91-9cd6-4a296a43e83c)


### Steps to reproduce the bug

![image](https://github.com/user-attachments/assets/5d504d4c-22c7-4742-99a1-9cab78739b17)

### Expected behavior

Sorry for this silly question, I'm noob on using this tool. But I think it should return a tensor value as I have used the protocol？
When I tokenize only one sentence using tokenized_input=tokenizer(input, return_tensors='pt' )，it does return in tensor type. Why doesn't it work in map()？

### Environment info

transformers>=4.41.2,<=4.45.0
datasets>=2.16.0,<=2.21.0
accelerate>=0.30.1,<=0.34.2
peft>=0.11.1,<=0.12.0
trl>=0.8.6,<=0.9.6
gradio>=4.0.0
pandas>=2.0.0
scipy
einops
sentencepiece
tiktoken
protobuf
uvicorn
pydantic
fastapi
sse-starlette
matplotlib>=3.7.0
fire
packaging
pyyaml
numpy<2.0.0


Comments: Hi ! `datasets` uses Arrow as storage backend which is agnostic to deep learning frameworks like torch. If you want to get torch tensors back, you need to do `dataset = dataset.with_format("torch")`

> Hi ! `datasets` uses Arrow as storage backend which is agnostic to deep learning frameworks like torch. If you want to get torch tensors back, you need to do `dataset = dataset.with_format("torch")`

It does work! Thanks for your suggestion!
`Dataset.save_to_disk` hangs when using num_proc > 1
State: open | Author: JohannesAck (NONE)

### Describe the bug

Hi, I'm encountered a small issue when saving datasets that led to the saving taking up to multiple hours.
Specifically, [`Dataset.save_to_disk`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.save_to_disk) is a lot slower when using `num_proc>1` than when using `num_proc=1`

The documentation mentions that "Multiprocessing is disabled by default.", but there is no explanation on how to enable it.

### Steps to reproduce the bug

```
import numpy as np
from datasets import Dataset

n_samples = int(4e6)
n_tokens_sample = 100
data_dict = {
    'tokens' : np.random.randint(0, 100, (n_samples, n_tokens_sample)),
}

dataset = Dataset.from_dict(data_dict)
dataset.save_to_disk('test_dataset', num_proc=1)
dataset.save_to_disk('test_dataset', num_proc=4)
dataset.save_to_disk('test_dataset', num_proc=8)
```

This results in:
```
>>> dataset.save_to_disk('test_dataset', num_proc=1)
Saving the dataset (7/7 shards): 100%|██████████████| 4000000/4000000 [00:17<00:00, 228075.15 examples/s]
>>> dataset.save_to_disk('test_dataset', num_proc=4)
Saving the dataset (7/7 shards): 100%|██████████████| 4000000/4000000 [01:49<00:00, 36583.75 examples/s]
>>> dataset.save_to_disk('test_dataset', num_proc=8)
Saving the dataset (8/8 shards): 100%|██████████████| 4000000/4000000 [02:11<00:00, 30518.43 examples/s]
```

With larger datasets it can take hours, but I didn't benchmark that for this bug report.

### Expected behavior

I would expect using `num_proc>1` to be faster instead of slower than `num_proc=1`. 

### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.2
- PyArrow version: 18.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1

Comments: I've met the same situations.

Here's my logs:
num_proc = 64, I stop it early as it cost **too** much time.
```
Saving the dataset (1540/4775 shards):  32%|███▏      | 47752224/147853764 [15:32:54<132:28:34, 209.89 examples/s]
Saving the dataset (1540/4775 shards):  32%|███▏      | 47754224/147853764 [15:32:54<78:14:37, 355.37 examples/s] 
Saving the dataset (1540/4775 shards):  32%|███▏      | 47755224/147853764 [15:32:59<93:43:45, 296.65 examples/s]
```

num_proc = 1(Not set num_proc parameter)
```
Saving the dataset (1753/4775 shards):  37%|███▋      | 54301556/147853764 [24:46<38:33, 40440.93 examples/s]
Saving the dataset (1753/4775 shards):  37%|███▋      | 54306556/147853764 [24:46<39:34, 39392.01 examples/s]
Saving the dataset (1753/4775 shards):  37%|███▋      | 54311520/147853764 [24:46<38:56, 40030.53 examples/s]
```

I check the conditions of CPUs and Memory I/O, I found that disk I/O was blocked, but CPU and memory usage was high. There should be some bugs in the code.
Support for identifier-based automated split construction
State: open | Author: alex-hh (CONTRIBUTOR)
Labels: name: enhancement

### Feature request

As far as I understand, automated construction of splits for hub datasets is currently based on either file names or directory structure ([as described here](https://huggingface.co/docs/datasets/en/repository_structure))

It would seem to be pretty useful to also allow splits to be based on identifiers of individual examples

This could be configured like 
{"split_name": {"column_name": [column values in split]}}

(This in turn requires unique 'index' columns, which could be explicitly supported or just assumed to be defined appropriately by the user).

I guess a potential downside would be that shards would end up spanning different splits - is this something that can be handled somehow? Would this only affect streaming from hub?

### Motivation

The main motivation would be that all data files could be stored in a single directory, and multiple sets of splits could be generated from the same data. This is often useful for large datasets with multiple distinct sets of splits.

This could all be configured via the README.md yaml configs

### Your contribution

May be able to contribute if it seems like a good idea

Comments: Hi ! You can already configure the README.md to have multiple sets of splits, e.g.

```yaml
configs:
- config_name: my_first_set_of_split
  data_files:
  - split: train
    path: *.csv
- config_name: my_second_set_of_split
  data_files:
  - split: train
    path: train-*.csv
  - split: test
    path: test-*.csv
```

Hi - I had something slightly different in mind:

Currently the yaml splits specified like this only allow specifying which filenames to pass to each split.
But what if I have a situation where I know which individual *training examples* I want to put in each split.

I could build split-specific files, however for large datasets with overlapping (e.g. multiple sets of) splits this could result in significant duplication of data.

I can see that this could actually be very much intended (i.e. to discourage overlapping splits), but wondered whether some support for handling splits based on individual identifiers is something that could be considered. 

This is not supported right now :/ Though you can load the data in two steps like this
```python
from datasets import load_dataset

full_dataset = load_dataset("username/dataset", split="train")
my_first_set_indices = load_dataset("username/dataset", "my_first_set_of_split", split="train")

my_first_set = full_dataset.select(my_first_set_indices["indices"])
```

you can create such a dataset by adapting this code for example
```python
# upload the full dataset
full_dataset.push_to_hub("username/dataset")
# then upload the indices for each set
DatasetDict({
    "train": Dataset.from_dict({"indices": [0, 1, 2, 3]}),
    "test": Dataset.from_dict({"indices": [4, 5]}),
}).push_to_hub("username/dataset", "my_first_set_of_split")
Concurrent loading in `load_from_disk` - `num_proc` as a param 
State: closed | Author: unography (NONE)
Labels: name: enhancement

### Feature request

https://github.com/huggingface/datasets/pull/6464 mentions a `num_proc` param while loading dataset from disk, but can't find that in the documentation and code anywhere

### Motivation

Make loading large datasets from disk faster

### Your contribution

Happy to contribute if given pointers
support for custom feature encoding/decoding
State: closed | Author: alex-hh (CONTRIBUTOR)

Fix for https://github.com/huggingface/datasets/issues/7220 as suggested in discussion, in preference to #7221 

(only concern would be on effect on type checking with custom feature types that aren't covered by FeatureType?)

Comments: @lhoestq 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7284). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.