NyssaW/github-issues huggingface dataset:
website broken: Create a new dataset repository, doesn't create a new repo in Firefox

### Describe the bug

This issue is also reported here:
https://discuss.huggingface.co/t/create-a-new-dataset-repository-broken-page/102644

This page is broken.
https://huggingface.co/new-dataset

I fill in the form with my text, and click `Create Dataset`.
![Screenshot 2024-08-16 at 15 55 37](https://github.com/user-attachments/assets/de16627b-7a55-4bcf-9f0b-a48227aabfe6)

Then the form gets wiped. And no repo got created. No error message visible in the developer console.
![Screenshot 2024-08-16 at 15 56 54](https://github.com/user-attachments/assets/0520164b-431c-40a5-9634-11fd62c4f4c3)

# Idea for improvement

For better UX, if the repo cannot be created, then show an error message, that something went wrong.

# Work around, that works for me

```python
from huggingface_hub import HfApi, HfFolder
repo_id = 'simon-arc-solve-fractal-v3'
api = HfApi()
username = api.whoami()['name']
repo_url = api.create_repo(repo_id=repo_id, exist_ok=True, private=True, repo_type="dataset")
```

### Steps to reproduce the bug

Go https://huggingface.co/new-dataset

Fill in the form.

Click `Create dataset`.

Now the form is cleared. And the page doesn't jump anywhere.

### Expected behavior

The moment the user clicks `Create dataset`, the repo gets created and the page jumps to the created repo.


### Environment info

Firefox 128.0.3 (64-bit)
macOS Sonoma 14.5


Comments:
I don't reproduce, I was able to create a new repo: https://huggingface.co/datasets/severo/reproduce-datasets-issues-7108. Can you confirm it's still broken?

I have just tried again.

Firefox: The `Create dataset` doesn't work. It has worked in the past. It's my preferred browser.

Chrome: The `Create dataset` works.

It seems to be a Firefox specific issue.

I have updated Firefox 129.0 (64 bit), and now the `Create dataset` is working again in Firefox.

UX: It would be nice with better error messages on HuggingFace.

maybe an issue with the cookie. cc @Wauplin @coyotte508
load_dataset broken in 2.21.0

### Describe the bug

`eval_set = datasets.load_dataset("tatsu-lab/alpaca_eval", "alpaca_eval_gpt4_baseline", trust_remote_code=True)` 
used to work till 2.20.0 but doesn't work in 2.21.0

In 2.20.0:
![Screenshot 2024-08-16 at 3 57 10 PM](https://github.com/user-attachments/assets/0516489b-8187-486d-bee8-88af3381dee9)


in 2.21.0:
![Screenshot 2024-08-16 at 3 57 24 PM](https://github.com/user-attachments/assets/bc257570-f461-41e4-8717-90a69ed7c24f)


### Steps to reproduce the bug

1. Spin up a new google collab
2. `pip install datasets==2.21.0`
3. `import datasets`
4. `eval_set = datasets.load_dataset("tatsu-lab/alpaca_eval", "alpaca_eval_gpt4_baseline", trust_remote_code=True)` 
5. Will throw an error.



### Expected behavior

Try steps 1-5 again but replace datasets version with 2.20.0, it will work

### Environment info



- `datasets` version: 2.21.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.5
- PyArrow version: 17.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2024.5.0


Comments:
There seems to be a PR related to the load_dataset path that went into 2.21.0 -- https://github.com/huggingface/datasets/pull/6862/files

Taking a look at it now

+1

Downgrading to 2.20.0 fixed my issue, hopefully helpful for others.

I tried adding a simple test to `test_load.py` with the alpaca eval dataset but the test didn't fail :(. 

So looks like this might have something to do with the environment? 

There was an issue with the script of the "tatsu-lab/alpaca_eval" dataset.

I was fixed with this PR: 
- [Fix FileNotFoundError](https://huggingface.co/datasets/tatsu-lab/alpaca_eval/discussions/2)

It should work now if you retry to load the dataset.
IterableDataset: cannot resolve features from list of numpy arrays

### Describe the bug

when resolve features of `IterableDataset`, got `pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values` error.

```
Traceback (most recent call last):
  File "test.py", line 6
    iter_ds = iter_ds._resolve_features()
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 2876, in _resolve_features
    features = _infer_features_from_batch(self.with_format(None)._head())
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 63, in _infer_features_from_batch
    pa_table = pa.Table.from_pydict(batch)
  File "pyarrow/table.pxi", line 1813, in pyarrow.lib._Tabular.from_pydict
  File "pyarrow/table.pxi", line 5339, in pyarrow.lib._from_pydict
  File "pyarrow/array.pxi", line 374, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 344, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 42, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
```

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

# create list of numpy
iter_ds = Dataset.from_dict({'a': [[[1, 2, 3], [1, 2, 3]]]}).to_iterable_dataset().map(lambda x: {'a': [np.array(x['a'])]})
iter_ds = iter_ds._resolve_features()  # errors here
```

### Expected behavior

features can be successfully resolved

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0

Comments:
Assign this issue to me under Hacktoberfest with hacktoberfest label inserted on the issue
Some of DownloadConfig's properties are always being overridden in load.py

### Describe the bug

The `extract_compressed_file` and `force_extract` properties of DownloadConfig are always being set to True in the function `dataset_module_factory` in the `load.py` file. This behavior is very annoying because data extracted will just be ignored the next time the dataset is loaded.

See this image below:
![image](https://github.com/user-attachments/assets/9e76ebb7-09b1-4c95-adc8-a959b536f93c)


### Steps to reproduce the bug

1. Have a local dataset that contains archived files (zip, tar.gz, etc)
2. Build a dataset loading script to download and extract these files
3. Run the load_dataset function with a DownloadConfig that specifically set `force_extract` to False
4. The extraction process will start no matter if the archives was extracted previously

### Expected behavior

The extraction process should not run when the archives were previously extracted and `force_extract` is set to False.

### Environment info

datasets==2.20.0
python3.9

Comments:
Add Arabic Docs to Datasets

Translate Docs into Arabic  issue-number : #7093
[Arabic Docs](https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx)
[English Docs](https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/en/index.mdx)

@stevhliu

Comments:
Add Arabic Docs to datasets

### Feature request

Add Arabic Docs to datasets
[Datasets Arabic](https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx)

### Motivation

@AhmedAlmaghz
https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx

### Your contribution

@AhmedAlmaghz
https://github.com/AhmedAlmaghz/datasets/blob/main/docs/source/ar/index.mdx

Comments:
The test test_move_script_doesnt_change_hash fails because it runs the 'python' command while the python executable has a different name

### Describe the bug

Tests should use the same pythin path as they are launched with, which in the case of FreeBSD is /usr/local/bin/python3.11

Failure:
```
                if err_filename is not None:
>                   raise child_exception_type(errno_num, err_msg, err_filename)
E                   FileNotFoundError: [Errno 2] No such file or directory: 'python'
```


### Steps to reproduce the bug

regular test run using PyTest

### Expected behavior

n/a

### Environment info

FreeBSD 14.1

Comments:
Unable to create dataset card for Lushootseed language

### Feature request

While I was creating the dataset which contained all documents from the Lushootseed Wikipedia, the dataset card asked me to enter which language the dataset was in. Since Lushootseed is a critically endangered language, it was not available as one of the options. Is it possible to allow entering languages that aren't available in the options?

### Motivation

I'd like to add more information about my dataset in the dataset card, and the language is one of the most important pieces of information, since the entire dataset is primarily concerned collecting Lushootseed documents.

### Your contribution

I can submit a pull request

Comments:
Thanks for reporting.

It is weird, because the language entry is in the list. See: https://github.com/huggingface/huggingface.js/blob/98e32f0ed4ee057a596f66a1dec738e5db9643d5/packages/languages/src/languages_iso_639_3.ts#L15186-L15189

I have reported the issue:
- https://github.com/huggingface/huggingface.js/issues/834



As explained in the reported issue above, the problem only appears in the autocomplete field: you can still enter the `lut` language directly in the markdown editor window.
load_dataset ignores cached datasets and tries to hit HF Hub, resulting in API rate limit errors

### Describe the bug

I have been running lm-eval-harness a lot which has results in an API rate limit. This seems strange, since all of the data should be cached locally. I have in fact verified this.

### Steps to reproduce the bug

1. Be Me
2. Run `load_dataset("TAUR-Lab/MuSR")`
3. Hit rate limit error
4. Dataset is in .cache/huggingface/datasets
5. ???

### Expected behavior

We should not run into API rate limits if we have cached the dataset

### Environment info

datasets 2.16.0
python 3.10.4

Comments:
I'm having the same issue - running into rate limits when doing hyperparameter tuning even though the dataset is supposed to be cached. I feel like this behaviour should at the very least be documented, but honestly you should just not be running into rate limits in the first place when the dataset is cached. It even happens when specifying a specific revision for the dataset, in which case AFAIK there should be no reason to be doing API requests if it's already cached (besides maybe a quick hash check but hitting rate limits for that in ~200 requests across 10 hours of use seems a bit ridiculous).

I was running into the same issue and solved it by upgrading `huggingface_hub` from `0.33.5` to `0.34.0`.
```bash
> python -m pip install huggingface_hub==0.34.0
```
More easily support streaming local files

### Feature request

Simplify downloading and streaming datasets locally. Specifically, perhaps add an option to `load_dataset(..., streaming="download_first")` or add better support for streaming symlinked or arrow files.

### Motivation

I have downloaded FineWeb-edu locally and currently trying to stream the dataset from the local files. I have both the raw parquet files using `hugginface-cli download --repo-type dataset HuggingFaceFW/fineweb-edu` and the processed arrow files using `load_dataset("HuggingFaceFW/fineweb-edu")`. 

Streaming the files locally does not work well for both file types for two different reasons.

**Arrow files**

When running `load_dataset("arrow", data_files={"train": "~/.cache/huggingface/datasets/HuggingFaceFW___fineweb-edu/default/0.0.0/5b89d1ea9319fe101b3cbdacd89a903aca1d6052/fineweb-edu-train-*.arrow"})` resolving the data files is fast, but because `arrow` is not included in the known [extensions file list](https://github.com/huggingface/datasets/blob/ce4a0c573920607bc6c814605734091b06b860e7/src/datasets/utils/file_utils.py#L738) , all files are opened and scanned to determine the compression type. Adding `arrow` to the known extension types resolves this issue. 

**Parquet files**

When running `load_dataset("arrow", data_files={"train": "~/.cache/huggingface/hub/dataset-HuggingFaceFW___fineweb-edu/snapshots/5b89d1ea9319fe101b3cbdacd89a903aca1d6052/data/CC-MAIN-*/train-*.parquet"})` the paths do not get resolved because the parquet files are symlinked from the blobs (which contain all files in case there are different versions). This occurs because the [pattern matching](https://github.com/huggingface/datasets/blob/ce4a0c573920607bc6c814605734091b06b860e7/src/datasets/data_files.py#L389) checks if the path is a file and does not check for symlinks. Symlinks (at least on my machine) are of type "other". 


### Your contribution

I have created a PR for fixing arrow file streaming and symlinks. However, I have not checked locally if the tests work or new tests need to be added.

IMO, the easiest option would be to add a `streaming=download_first` option, but I'm afraid that exceeds my current knowledge of how the datasets library works. https://github.com/huggingface/datasets/pull/7083

Comments:
Generating train split takes a long time

### Describe the bug

Loading a simple webdataset takes ~45 minutes.

### Steps to reproduce the bug

```
from datasets import load_dataset
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M")
```

### Expected behavior

The dataset should load immediately as it does when loaded through a normal indexed WebDataset loader. Generating splits should be optional and there should be a message showing how to disable it. 

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
@alexanderswerdlow 
When no specific split is mentioned, the load_dataset library will load all available splits of the dataset. For example, if a dataset has "train" and "test" splits, the load_dataset function will load both into the DatasetDict object.

![image](https://github.com/user-attachments/assets/379e6f57-7e1b-4cc3-bc36-dae3e878a51c)


The dataset PixArt-alpha/SAM-LLaVA-Captions10M may have been uploaded with different predefined splits (e.g., "train", "test", etc.), and by default, Hugging Face will load all splits unless you specifically request only one.

### If you want to load only a specific split (e.g., only the "train" set), you can specify it in the split parameter like this:
```python
from datasets import load_dataset
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M", split="train")
```

### You can also load multiple splits if needed:
```python
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M", split=["train", "test"])
```



@alexanderswerdlow, I will now work on this..

## Idea:
Whenever this code has ran:
```python
from datasets import load_dataset
dataset = load_dataset("PixArt-alpha/SAM-LLaVA-Captions10M")
```

It should show all the splits of the datasets, and user has to choose which one should be loaded before generating a split like this,,

![image](https://github.com/user-attachments/assets/8fbc604f-f0a5-4a59-a63e-aa4c26442c83)
column_names ignored by load_dataset() when loading CSV file

### Describe the bug

load_dataset() ignores the column_names kwarg when loading a CSV file.  Instead, it uses whatever values are on the first line of the file.


### Steps to reproduce the bug

Call `load_dataset` to load data from a CSV file and specify `column_names` kwarg.

### Expected behavior

The resulting dataset should have the specified column names **and** the first line of the file should be considered as data values.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.10.0-30-cloud-amd64-x86_64-with-glibc2.31
- Python version: 3.9.2
- `huggingface_hub` version: 0.24.2
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
I confirm that `column_names` values are not copied to `names` variable because in this case `CsvConfig.__post_init__` is not called: `CsvConfig` is instantiated with default values and afterwards the `config_kwargs` are used to overwrite its attributes.

@luismsgomes in the meantime, you can avoid the bug if you pass `names` instead of `column_names`.
how set_transform affects batch size?

### Describe the bug

I am trying to fine-tune w2v-bert for ASR task. Since my dataset is so big, I preferred to use the on-the-fly method with set_transform. So i change the preprocessing function to this:
```
def prepare_dataset(batch):
    input_features = processor(batch["audio"], sampling_rate=16000).input_features[0]
    input_length = len(input_features)
    labels = processor.tokenizer(batch["text"], padding=False).input_ids

    batch = {
        "input_features": [input_features],
        "input_length": [input_length],
        "labels": [labels]
    }
    
    return batch


train_ds.set_transform(prepare_dataset)
val_ds.set_transform(prepare_dataset)
```

After this, I also had to change the DataCollatorCTCWithPadding class like this:
```
@dataclass
class DataCollatorCTCWithPadding:
    processor: Wav2Vec2BertProcessor
    padding: Union[bool, str] = True

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # Separate input_features and labels
        input_features = [{"input_features": feature["input_features"][0]} for feature in features]
        labels = [feature["labels"][0] for feature in features]

        # Pad input features
        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            return_tensors="pt",
        )

        # Pad and process labels
        label_features = self.processor.tokenizer.pad(
            {"input_ids": labels},
            padding=self.padding,
            return_tensors="pt",
        )

        labels = label_features["input_ids"]
        attention_mask = label_features["attention_mask"]

        # Replace padding with -100 to ignore these tokens during loss calculation
        labels = labels.masked_fill(attention_mask.ne(1), -100)

        batch["labels"] = labels

        return batch
```
        
But now a strange thing is happening, no matter how much I increase the batch size, the amount of V-RAM GPU usage does not change, while the number of total steps in the progress-bar (logging) changes. Is this normal or have I made a mistake?

### Steps to reproduce the bug

i can share my code if needed

### Expected behavior

Equal to the batch size value, the set_transform function is applied to the dataset and given to the model as a batch.

### Environment info

all updated versions

Comments:
Convert_to_parquet fails for datasets with multiple configs

If the dataset has multiple configs, when using the `datasets-cli convert_to_parquet` command to avoid issues with the data viewer caused by loading scripts, the conversion process only successfully converts the data corresponding to the first config. When it starts converting the second config, it throws an error:

```
Traceback (most recent call last):
  File "/opt/anaconda3/envs/dl/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/convert_to_parquet.py", line 83, in run
    dataset.push_to_hub(
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/dataset_dict.py", line 1713, in push_to_hub
    api.create_branch(repo_id, branch=revision, token=token, repo_type="dataset", exist_ok=True)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 358, in hf_raise_for_status
    raise BadRequestError(message, response=response) from e
huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-669fc665-7c2e80d75f4337496ee95402;731fcdc7-0950-4eec-99cf-ce047b8d003f)

Bad request:
Invalid reference for a branch: refs/pr/1
```

Comments:
Many users have encountered the same issue, which has caused inconvenience.

https://discuss.huggingface.co/t/convert-to-parquet-fails-for-datasets-with-multiple-configs/86733


Thanks for reporting.

I will make the code more robust.

I have opened an issue in the huggingface-hub repo:
- https://github.com/huggingface/huggingface_hub/issues/2419

I am opening a PR to avoid calling `create_branch` if the branch already exists.
One subset per file in repo ?

Right now we consider all the files of a dataset to be the same data, e.g.
```
single_subset_dataset/
├── train0.jsonl
├── train1.jsonl
└── train2.jsonl
```
but in cases like this, each file is actually a different subset of the dataset and should be loaded separately
```
many_subsets_dataset/
├── animals.jsonl
├── trees.jsonl
└── metadata.jsonl
```

It would be nice to detect those subsets automatically using a simple heuristic. For example we can group files together if their paths names are the same except some digits ?

Comments:
Hi @lhoestq! I’ve opened a PR that addresses this issue
None values are skipped when reading jsonl in subobjects

### Describe the bug

I have been fighting against my machine since this morning only to find out this is some kind of a bug.

When loading a dataset composed of `metadata.jsonl`, if you have nullable values (Optional[str]), they can be ignored by the parser, shifting things around.

E.g., let's take this example

Here are two version of a same dataset:

[not-buggy.tar.gz](https://github.com/user-attachments/files/16333532/not-buggy.tar.gz)
[buggy.tar.gz](https://github.com/user-attachments/files/16333553/buggy.tar.gz)


### Steps to reproduce the bug

1. Load the `buggy.tar.gz` dataset
2. Print baseline of `dts = load_dataset("./data")["train"][0]["baselines]`
3. Load the `not-buggy.tar.gz` dataset
4. Print baseline of `dts = load_dataset("./data")["train"][0]["baselines]`

### Expected behavior

Both should have 4 baseline entries:
1. Buggy should have None followed by three lists
2. Non-Buggy should have four lists, and the first one should be an empty list.

One does not work, 2 works. Despite accepting None in another position than the first one.

### Environment info


- `datasets` version: 2.19.1
- Platform: Linux-6.5.0-44-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.0
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Comments:
Datasets.datafiles resolve_pattern `TypeError: can only concatenate tuple (not "str") to tuple`

### Describe the bug

in data_files.py, line 332, 
`fs,  _, _ = get_fs_token_paths(pattern, storage_options=storage_options)`
If we run the code on AWS, as fs.protocol will be a tuple like: `('file', 'local')`
So,  `isinstance(fs.protocol, str) == False` and
`protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""` will raise 
`TypeError: can only concatenate tuple (not "str") to tuple`.

### Steps to reproduce the bug

Steps to reproduce:
1. Run on a cloud server like AWS,
2. `import datasets.data_files as datafile`
3. datafile.resolve_pattern('path/to/dataset', '.')
4. `TypeError: can only concatenate tuple (not "str") to tuple`

### Expected behavior

Should return path of the dataset, with fs.protocol at the beginning

### Environment info

- `datasets` version: 2.14.0
- Platform: Linux-3.10.0-1160.119.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.19
- Huggingface_hub version: 0.23.5
- PyArrow version: 16.1.0
- Pandas version: 1.1.5

Comments:
Hi,

This issue was fixed in `datasets` 2.15.0:
- #6105

You will need to update your `datasets`:
```
pip install -U datasets
```

Duplicate of:
- #6100
Adding `Music` feature for symbolic music modality (MIDI, abc)

⚠️ (WIP) ⚠️

### What this PR does

This PR adds a `Music` feature for the symbolic music modality, in particular [MIDI](https://en.wikipedia.org/wiki/Musical_Instrument_Digital_Interface) and [abc](https://en.wikipedia.org/wiki/ABC_notation) files.

### Motivations

These two file formats are widely used in the [Music Information Retrieval (MIR)](https://en.wikipedia.org/wiki/Music_information_retrieval) for tasks such as music generation, music transcription, music synthesis or music transcription. Having a dedicated feature in the datasets library would allow to both encourage researchers to share datasets of this modality as well as making them more easily usable for end users, benefitting from the perks of the library.

These file formats are supported by [symusic](https://github.com/Yikai-Liao/symusic), a lightweight Python library with C bindings (using nanobind) allowing to efficiently read, write and manipulate them. The library is actively developed, and can in the future also implement other file formats such as [musicXML](https://en.wikipedia.org/wiki/MusicXML). As such, this PR relies on it.

The music data can then easily be tokenized with appropriate tokenizers such as [MidiTok](https://github.com/Natooz/MidiTok) or converted to pianorolls matrices by symusic.

**Jul 16th 2024:**

* the tests for the `Music` feature are currently failing due to non-supported access to the LazyBatch in `test_dataset_with_music_feature_map` and `test_dataset_with_music_feature_map_resample_music` (see TODOs). I am a beginner with pyArrow, I'll take any advice to make this work;
* additional tests including the `Music` feature with parquet and WebDataset should be implemented. As of right now, I am waiting for your feedback before taking further steps;
* a `MusicFolder` should also be implemented to comply with the usages of the `Image` and `Audio` features, waiting for your feedback too.

CCing @lhoestq and @albertvillanova 

Comments:
Save nparray as list

### Describe the bug

When I use the `map` function to convert images into features, datasets saves nparray as a list. Some people use the `set_format` function to convert the column back, but doesn't this lose precision?

### Steps to reproduce the bug

the map function
```python
def convert_image_to_features(inst, processor, image_dir):
    image_file = inst["image_url"]
    file = image_file.split("/")[-1]
    image_path = os.path.join(image_dir, file)
    image = Image.open(image_path)
    image = image.convert("RGBA")

    inst["pixel_values"] = processor(images=image, return_tensors="np")["pixel_values"]
    return inst
```

main function
```python
map_fun = partial(
      convert_image_to_features, processor=processor, image_dir=image_dir
  )
ds = ds.map(map_fun, batched=False, num_proc=20)
print(type(ds[0]["pixel_values"])
```

### Expected behavior

(type < list>) 

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-4.19.91-009.ali4000.alios7.x86_64-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.23.4
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0

Comments:
In addition, when I use `set_format ` and index the ds, the following error occurs:
the code
```python
ds.set_format(type="np", colums="pixel_values")
```
error
<img width="918" alt="image" src="https://github.com/user-attachments/assets/b28bbff2-20ea-4d28-ab62-b4ed2d944996">


>  Some people use the set_format function to convert the column back, but doesn't this lose precision?

Under the hood the data is saved in Arrow format using the same precision as your numpy arrays?
By default the Arrow data is read as python lists, but you can indeed read them back as numpy arrays with the same precision

(you can fix your second issue by fixing the typo `colums` -> `columns`)

> (you can fix your second issue by fixing the typo `colums` -> `columns`)

You are right, I was careless. Thank you.

> > Some people use the set_format function to convert the column back, but doesn't this lose precision?
> 
> Under the hood the data is saved in Arrow format using the same precision as your numpy arrays? By default the Arrow data is read as python lists, but you can indeed read them back as numpy arrays with the same precision

Yes, after testing I found that there was no loss of precision. Thanks again for your answer.
ImportError: numpy.core.multiarray when using `filter`

### Describe the bug

I can't apply the filter method on my dataset. 

### Steps to reproduce the bug

The following snippet generates a bug: 

```python
from datasets import load_dataset 

ami = load_dataset('kamilakesbi/ami', 'ihm')

ami['train'].filter(
    lambda example: example["file_name"] == 'EN2001a'
)
```

I get the following error: 

`ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).`

### Expected behavior

It should work properly! 

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
Could you please check your `numpy` version?

I got this issue while using numpy version 2.0. 

I solved it by switching back to numpy 1.26.0 :) 

We recently added support for numpy 2.0, but it is not released yet.

Ok I see, thanks! I think we can close this issue for now as switching back to version 1.26.0 solves the problem :)
`sort` after `filter` unreasonably  slow

### Describe the bug

as the tittle says ...

### Steps to reproduce the bug

`sort` seems to be normal.

```python
from datasets import Dataset
import random

nums = [{"k":random.choice(range(0,1000))} for _ in range(100000)]

ds = Dataset.from_list(nums)

print("start sort")

ds = ds.sort("k")

print("finish sort")
```

but `sort` after `filter` is extremely slow.

```python
from datasets import Dataset
import random

nums = [{"k":random.choice(range(0,1000))} for _ in range(100000)]

ds = Dataset.from_list(nums)

ds = ds.filter(lambda x:x > 100, input_columns="k")

print("start sort")

ds = ds.sort("k")

print("finish sort")
```


### Expected behavior

Is this a bug, or is it a misuse of the `sort` function?

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2023.10.0

Comments:
`filter` add an indices mapping on top of the dataset, so `sort` has to gather all the rows that are kept to form a new Arrow table and sort the table. Gathering all the rows can take some time, but is a necessary step. You can try calling `ds = ds.flatten_indices()` before sorting to remove the indices mapping.

> `filter` add an indices mapping on top of the dataset, so `sort` has to gather all the rows that are kept to form a new Arrow table and sort the table. Gathering all the rows can take some time, but is a necessary step. You can try calling `ds = ds.flatten_indices()` before sorting to remove the indices mapping.

This worked, thank you so much.
load `streaming=True` dataset with downloaded cache

### Describe the bug

We build a dataset which contains several hdf5 files and write a script using `h5py` to generate the dataset. The hdf5 files are large and the processed dataset cache takes more disk space. So we hope to try streaming iterable dataset. Unfortunately, `h5py` can't convert a remote URL into a hdf5 file descriptor. So we use `fsspec` as an interface like below:

```python
def _generate_examples(self, filepath, split):
        for file in filepath:
            with fsspec.open(file, "rb") as fs:
                with h5py.File(fs, "r") as fp:
                    # for event_id in sorted(list(fp.keys())):
                    event_ids = list(fp.keys())
                    ......
```

### Steps to reproduce the bug

The `fsspec` works, but it takes 10+ min to print the first 10 examples, which is even longer than the downloading time. I'm not sure if it just caches the whole hdf5 file and generates the examples.

### Expected behavior

So does the following make sense so far?

1. download the files
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True)
```

2. load the iterable dataset faster (using the raw file cache at path `.cache/huggingface/datasets/downloads`)
```python
dataset = datasets.load('path/to/myscripts', split="train", name="event", trust_remote_code=True, streaming=true)
```

I made some tests, but the code above can't get the expected result. I'm not sure if this is supported. I also find the issue #6327 . It seemed similar to mine, but I couldn't find a solution.

### Environment info

- `datasets` = 2.18.0
- `h5py` = 3.10.0
- `fsspec` = 2023.10.0

Comments:
When you pass `streaming=True`, the cache is ignored. The remote data URL is used instead and the data is streamed from the remote server.

Thanks for your reply! So is there any solution to get my expected behavior besides clone the whole repo ? Or could I adjust my script to load the downloaded arrow files and generate the dataset streamingly?
Fix export to JSON when dataset larger than batch size

Fix export to JSON (`lines=False`) when dataset larger than batch size.

Fix #7037.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7039). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The test before confirms the bug.

There are different possible solutions to this issue:
- the easiest would be to write multiple JSON files, one for each batch; this solution can be done in parallel if `num_proc` is passed
- alternatively, we could tweak the writing and remove the extra `[` and `]` characters; this solution will only be valid if `orient="records"`
- others?

@albertvillanova I was planning to approach it in the second way for `orient="records"` , `orient="values"` and `orient="index"`. For `orient="split"`, the columns and index can be written in one go and the data can be written in streaming manner. For `orient="columns"`, each column can be written in a streaming way.  LMK if I should go ahead with this.

> The test before confirms the bug.
> 
> There are different possible solutions to this issue:
> 
>     * the easiest would be to write multiple JSON files, one for each batch; this solution can be done in parallel if `num_proc` is passed
> 
>     * alternatively, we could tweak the writing and remove the extra `[` and `]` characters; this solution will only be valid if `orient="records"`
> 
>     * others?
A bug of Dataset.to_json() function

### Describe the bug

When using the Dataset.to_json() function, an unexpected error occurs if the parameter is set to lines=False. The stored data should be in the form of a list, but it actually turns into multiple lists, which causes an error when reading the data again.
The reason is that to_json() writes to the file in several segments based on the batch size. This is not a problem when lines=True, but it is incorrect when lines=False, because writing in several times will produce multiple lists(when len(dataset) > batch_size).

### Steps to reproduce the bug

try this code:
```python
from datasets import load_dataset
import json

train_dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base")["train"]
output_path = "./harmless-base_hftojs.json"
print(len(train_dataset))
train_dataset.to_json(output_path, lines=False, force_ascii=False, indent=2)

with open(output_path, encoding="utf-8") as f:
    data = json.loads(f.read())

```
it raise error: json.decoder.JSONDecodeError: Extra data: line 4003 column 1 (char 1373709)

Extra square brackets have appeared here:
<img width="265" alt="image" src="https://github.com/huggingface/datasets/assets/26499566/81492332-386d-42e8-88d1-b6d4ae3682cc">


### Expected behavior

The code runs normally.

### Environment info

datasets=2.20.0

Comments:
Thanks for reporting, @LinglingGreat.

I confirm this is a bug.

@albertvillanova I would like to take a shot at this if you aren't working on it currently. Let me know!
Docs are not generated when a parameter defaults to a NamedSplit value

While generating the docs, we get an error when some parameter defaults to a `NamedSplit` value, like:
```python
def call_function(split=Split.TRAIN):
    ...
```
The error is: ValueError: Equality not supported between split train and <class 'inspect._empty'>

See: https://github.com/huggingface/datasets/actions/runs/9869660902/job/27254359863?pr=7015
```
Building the MDX files:  97%|█████████▋| 58/60 [00:00<00:00, 91.94it/s] 
Traceback (most recent call last):
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 197, in build_mdx_files
    content, new_anchors, source_files, errors = resolve_autodoc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 123, in resolve_autodoc
    doc = autodoc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 499, in autodoc
    method_doc, check = document_object(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 395, in document_object
    signature = format_signature(obj)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 126, in format_signature
    if param.default != inspect._empty:
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/datasets/splits.py", line 136, in __ne__
    return not self.__eq__(other)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/datasets/splits.py", line 379, in __eq__
    raise ValueError(f"Equality not supported between split {self} and {other}")
ValueError: Equality not supported between split train and <class 'inspect._empty'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/datasets/datasets/.venv/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/commands/build.py", line 102, in build_command
    build_doc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 367, in build_doc
    anchors_mapping, source_files_mapping = build_mdx_files(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 230, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Equality not supported between split train and <class 'inspect._empty'>
```

Comments:
`from_generator` does not allow to specify the split name

### Describe the bug

I'm building train, dev, and test using `from_generator`; however, in all three cases, the logger prints `Generating train split:`
It's not possible to change the split name since it seems to be hardcoded: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/generator/generator.py

### Steps to reproduce the bug

```
In [1]: from datasets import Dataset

In [2]: def gen():
   ...:     yield {"pokemon": "bulbasaur", "type": "grass"}
   ...: 

In [3]: ds = Dataset.from_generator(gen)
Generating train split: 1 examples [00:00, 133.89 examples/s]
```

### Expected behavior

It should be possible to specify any split name

### Environment info

- `datasets` version: 2.19.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- `huggingface_hub` version: 0.23.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0

Comments:
Thanks for reporting, @pminervini.

I agree we should give the option to define the split name.

Indeed, there is a PR that addresses precisely this issue:
- #7015

I am reviewing it.

Booom! thank you guys :)
Register `.zstd` extension for zstd-compressed files

For example, https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0 dataset files have `.zstd` extension which is currently ignored (only `.zst` is registered).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7032). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@albertvillanova hm I don't know tbh, it's just that "mlfoundations/dclm-baseline-1.0" dataset contains [files](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0/tree/main/global-shard_01_of_10/local-shard_0_of_10) with this extension and these files seem to be valid 

not sure why CI is failing but seems to be unrelated to this pr? can I merge @lhoestq @albertvillanova ?

yes you can merge, the CI failure is unrelated (surely an issue with hub-ci)

ah why not, you could try opening a PR

btw there is a channel with them at (internal) https://app.slack.com/client/T1RCG4490/C079AKTV11P if you want to let them know

@lhoestq, your previous comment was addressed to me or Polina?

@polinaeterna let me know if it is OK for you.

I opened https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0/discussions/7

Should we close this PR then?
Add option to disable progress bar when reading a dataset ("Loading dataset from disk")

### Feature request

Add an option in load_from_disk to disable the progress bar even if the number of files is larger than 16.

### Motivation

I am reading a lot of datasets that it creates lots of logs.

<img width="1432" alt="image" src="https://github.com/huggingface/datasets/assets/57996478/8d4bbf03-6b89-44b6-937c-932f01b4eb2a">


### Your contribution

Seems like an easy fix to make.  I can create a PR if necessary.

Comments:
You can disable progress bars for all of `datasets` with `disable_progress_bars`. [Link](https://huggingface.co/docs/datasets/en/package_reference/utilities#datasets.enable_progress_bars)

So you could do something like:

```python
from datasets import load_from_disk, enable_progress_bars, disable_progress_bars

disable_progress_bars()
# Your code
load_from_disk(....)

enable_progress_bars()
```


Thank you! Closing the issue.
load_dataset on AWS lambda throws OSError(30, 'Read-only file system') error

### Describe the bug

I'm using AWS lambda to run a python application. I run the `load_dataset` function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory. 

### Steps to reproduce the bug

```python
d = load_dataset(
      path=hugging_face_link,
      split=split,
      token=token,
      cache_dir="/tmp/hugging_face_cache",
 )
```

### Expected behavior

Everything written to the file system as part of the load_datasets function should be in the /tmp directory. 

### Environment info

datasets version: 2.16.1
Platform: Linux-5.10.216-225.855.amzn2.x86_64-x86_64-with-glibc2.26
Python version: 3.11.9
huggingface_hub version: 0.19.4
PyArrow version: 16.1.0
Pandas version: 2.2.2
fsspec version: 2023.10.0

Comments:
hi ! can you share the full stack trace ? this should help locate what files is not written in the cache_dir
Streaming dataset not returning data

### Describe the bug

I'm deciding to post here because I'm still not sure what the issue is, or if I am using IterableDatasets wrongly.
I'm following the guide on here https://huggingface.co/learn/cookbook/en/fine_tuning_code_llm_on_single_gpu pretty much to a tee and have verified that it works when I'm fine-tuning on the provided dataset.

However, I'm doing some data preprocessing steps (filtering out entries), when I try to swap out the dataset for mine, it fails to train. However, I eventually fixed this by simply setting `stream=False` in `load_dataset`.

Coud this be some sort of network / firewall issue I'm facing?

### Steps to reproduce the bug

I made a post with greater description about how I reproduced this problem before I found my workaround: https://discuss.huggingface.co/t/problem-with-custom-iterator-of-streaming-dataset-not-returning-anything/94551

Here is the problematic dataset snippet, which works when streaming=False (and with buffer keyword removed from shuffle)

```
commitpackft = load_dataset(
    "chargoddard/commitpack-ft-instruct", split="train", streaming=True
).filter(lambda example: example["language"] == "Python")
 
 
def form_template(example):
    """Forms a template for each example following the alpaca format for CommitPack"""
    example["content"] = (
        "### Human: " + example["instruction"] + " " + example["input"] + " ### Assistant: " + example["output"]
    )
    return example
 
 
dataset = commitpackft.map(
    form_template,
    remove_columns=["id", "language", "license", "instruction", "input", "output"],
).shuffle(
    seed=42, buffer_size=10000
)  # remove everything since its all inside "content" now
validation_data = dataset.take(4000)
train_data = dataset.skip(4000)
```

The annoying part about this is that it only fails during training and I don't know when it will fail, except that it always fails during evaluation.

### Expected behavior

The expected behavior is that I should be able to get something from the iterator when called instead of getting nothing / stuck in a loop somewhere.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.4.0-121-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0



Comments:
Casting list array to fixed size list raises error

When trying to cast a list array to fixed size list, an AttributeError is raised:
> AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'

Steps to reproduce the bug:
```python
import pyarrow as pa
from datasets.table import array_cast

arr = pa.array([[0, 1]])
array_cast(arr, pa.list_(pa.int64(), 2))
```

Stack trace:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-6cb90a1d8216> in <module>
      3 
      4 arr = pa.array([[0, 1]])
----> 5 array_cast(arr, pa.list_(pa.int64(), 2))

~/huggingface/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1802             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1803         else:
-> 1804             return func(array, *args, **kwargs)
   1805 
   1806     return wrapper

~/huggingface/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_primitive_to_str, allow_decimal_to_str)
   1920                 else:
   1921                     array_values = array.values[
-> 1922                         array.offset * pa_type.length : (array.offset + len(array)) * pa_type.length
   1923                     ]
   1924                     return pa.FixedSizeListArray.from_arrays(_c(array_values, pa_type.value_type), pa_type.list_size)

AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'
```

Comments: