jamesLeeeeeee/datasets-github-issues huggingface dataset:
`load_dataset(...,save_infos=True)` not working without loading script
Author: morganveyret | Pull request: no | State: open

### Describe the bug

It seems that saving a dataset infos back into the card file is not working for datasets without a loading script.

After tracking the problem a bit it looks like saving the infos uses `Builder.get_imported_module_dir()` as its destination directory.
Internally this is a call to `inspect.getfile()` but since the actual builder class used is dynamically created (cf. `datasets.load.configure_builder_class`) this method actually return te path to the parent builder class (e.g. `datasets.packaged_modules.json.JSON`). 



### Steps to reproduce the bug

1. Have a local dataset without any loading script
2. Make sure there are no dataset infos in the README.md
3. Load with `save_infos=True`
4. No change in the dataset README.md
5. A new README.md file is created in the directory of the parent builder class (e.g. for json in `.../site-packages/datasets/packaged_modules/json/README.md`)

### Expected behavior

The dataset README.md should be updated and no file should be created in the python environment.

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.6.0
load_dataset imageflder for aws s3 path 
Author: segalinc | Pull request: no | State: open

### Feature request

I would like to load a dataset from S3 using the imagefolder option 
something like 
`dataset = datasets.load_dataset('imagefolder', data_dir='s3://.../lsun/train/bedroom', fs=S3FileSystem(), streaming=True)  `

### Motivation

no need of data_files

### Your contribution

no experience with this
429 Client Error
Author: sasaadi | Pull request: no | State: open

Hello, I was downloading the following dataset and after 20% of data was downloaded, I started getting error 429. It is not resolved since a few days. How should I resolve it?
Thanks

Dataset:
https://huggingface.co/datasets/cerebras/SlimPajama-627B

Error:
`requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk1/example_train_3300.jsonl.zst`
Update builder hash with info
Author: lhoestq | Pull request: yes | State: closed

Currently if you change the `dataset_info` of a dataset (e.g. in the YAML part of the README.md), the cache ignores this change.

This is problematic because you want to regenerate a dataset if you change the features or the split sizes for example (e.g. after push_to_hub)

Ideally we should take the resolved files into account as well but this will be for another PR
FileNotFoundError: [Errno 2] No such file or directory: 'nul'
Author: amanyara | Pull request: no | State: closed

### Describe the bug

it seems that sth wrong with my terrible "bug body" life, When i run this code, "import datasets"
i meet this error FileNotFoundError: [Errno 2] No such file or directory: 'nul'
![image](https://github.com/huggingface/datasets/assets/73683903/3973c120-ebb1-42b7-bede-b9de053e861d)
![image](https://github.com/huggingface/datasets/assets/73683903/0496adff-a7a7-4dcb-929e-ec11ede71f04)


### Steps to reproduce the bug

1.import datasets

### Expected behavior

i just run a single line code and stuct in this bug 

### Environment info

OS: Windows10
Datasets==2.15.0
python=3.10
using torchrun, save_to_disk suddenly shows SIGTERM
Author: Ariya12138 | Pull request: no | State: open

### Describe the bug

When I run my code using the "torchrun" command, when the code reaches the "save_to_disk" part, suddenly I get the following warning and error messages:
Because the dataset is too large, the "save_to_disk" function splits it into 70 parts for saving. However, an error occurs suddenly when it reaches the 14th shard.
WARNING: torch.distributed.elastic.multiprocessing.api: Sending process 2224968 closing signal SIGTERM
ERROR: torch.distributed.elastic.multiprocessing.api: failed (exitcode: -7). traceback: Signal 7 (SIGBUS) received by PID 2224967.

### Steps to reproduce the bug

ds_shard = ds_shard.map(map_fn, *args, **kwargs)
ds_shard.save_to_disk(ds_shard_filepaths[rank])

Saving the dataset (14/70 shards):  20%|██        | 875350/4376702 [00:19<01:53, 30863.15 examples/s]
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2224968 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 2224967) of binary: /home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/python
Traceback (most recent call last):
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
==========================================================
run.py FAILED
----------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
----------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-12-08_20:09:04
  rank      : 0 (local_rank: 0)
  exitcode  : -7 (pid: 2224967)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2224967



### Expected behavior

I hope it can save successfully without any issues, but it seems there is a problem.

### Environment info

`datasets` version: 2.14.6
- Platform: Linux-4.19.90-24.4.v2101.ky10.aarch64-aarch64-with-glibc2.28
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 14.0.0
- Pandas version: 2.1.2
Add IterableDataset `__repr__`
Author: lhoestq | Pull request: yes | State: closed

Example for glue sst2:

Dataset

```
DatasetDict({
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
})
```

IterableDataset (new)

```
IterableDatasetDict({
    test: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    train: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    validation: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
})
```

IterableDataset (before)

```
{'test': <datasets.iterable_dataset.IterableDataset object at 0x130d421f0>, 'train': <datasets.iterable_dataset.IterableDataset object at 0x136f3aaf0>, 'validation': <datasets.iterable_dataset.IterableDataset object at 0x136f4b100>}
{'sentence': 'hide new secretions from the parental units ', 'label': 0, 'idx': 0}
```
CI on windows is broken: PermissionError
Author: albertvillanova | Pull request: no | State: closed

See: https://github.com/huggingface/datasets/actions/runs/7104781624/job/19340572394
```
FAILED tests/test_load.py::test_loading_from_the_datasets_hub - NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpfcnps56i\\hf-internal-testing___dataset_with_script\\default\\0.0.0\\c240e2be3370bdbd\\dataset_with_script-train.arrow'
```
Deprecate Beam API and download from HF GCS bucket
Author: mariosasko | Pull request: yes | State: open

Deprecate the Beam API and download from the HF GCS bucked. 

TODO:
- [ ] Deprecate the Beam-based [`wikipedia`](https://huggingface.co/datasets/wikipedia) in favor of [`wikimedia/wikipedia`](https://huggingface.co/datasets/wikimedia/wikipedia) ([Hub PR](https://huggingface.co/datasets/wikipedia/discussions/19))
- [ ] Make [`natural_questions`](https://huggingface.co/datasets/natural_questions) a no-code dataset ([Hub PR](https://huggingface.co/datasets/natural_questions/discussions/7))
- [ ] Make [`wiki40b`](https://huggingface.co/datasets/wiki40b) a no-code dataset ([Hub PR](https://huggingface.co/datasets/wiki40b/discussions/5))
- [ ] Make [`wiki_dpr`](https://huggingface.co/datasets/wiki_dpr) an Arrow-based dataset
If an image in a dataset is corrupted, we get unescapable error
Author: chigozienri | Pull request: no | State: open

### Describe the bug

Example discussed in detail here: https://huggingface.co/datasets/sasha/birdsnap/discussions/1

### Steps to reproduce the bug

```
from datasets import load_dataset, VerificationMode
dataset = load_dataset(
    'sasha/birdsnap',
    split="train",
    verification_mode=VerificationMode.ALL_CHECKS,
    streaming=True # I recommend using streaming=True when reproducing, as this dataset is large
)
for idx, row in enumerate(dataset):
    # Iterating to 9287 took 7 minutes for me
    # If you already have the data locally cached and set streaming=False, you see the same error just by with dataset[9287]
    pass
    # error at 9287 OSError: image file is truncated (45 bytes not processed)
    # note that we can't avoid the error using a try/except + continue inside the loop
```

### Expected behavior

Able to escape errors in casting to Image() without killing the whole loop

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
New version release request
Author: LZHgrla | Pull request: no | State: closed

### Feature request

Hi!
I am using `datasets` in library `xtuner` and am highly interested in the features introduced since v2.15.0.

To avoid installation from source in our pypi wheels, we are eagerly waiting for the new release. So, Does your team have a new release plan for v2.15.1 and could you please share it with us?

Thanks very much!


### Motivation

.

### Your contribution

.
Can't align optional features of struct
Author: Dref360 | Pull request: no | State: closed

### Describe the bug

Hello!

I'm currently experiencing an issue where I can't concatenate datasets if an inner field of a Feature is Optional.

I have a column named `speaker`, and this holds some information about a speaker.

```python
@dataclass
class Speaker:
   name: str
   email: Optional[str]
```

If I have two datasets, one happens to have `email` always None, then I get `The features can't be aligned because the key email of features`




### Steps to reproduce the bug

You can run the following script:

```python

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
concatenate_datasets([ds, ds2])

>>>The features can't be aligned because the key speaker of features {'speaker': {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)}} has unexpected type - {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)} (expected either {'email': Value(dtype='null', id=None), 'name': Value(dtype='string', id=None)} or Value("null").

```

### Expected behavior

I think this should work; if two top-level columns were in the same situation it would properly cast to `string`.

```python
ds = Dataset.from_dict({'email': [None, None]})
ds2 = Dataset.from_dict({'email': ['abc@aol.com', 'one@yahoo.com']})
concatenate_datasets([ds, ds2])
>>> # Works!
```

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
- `fsspec` version: 2023.6.0

I would be happy to fix this issue.
Fix shard retry mechanism in `push_to_hub`
Author: mariosasko | Pull request: yes | State: closed

When it fails, `preupload_lfs_files` throws a [`RuntimeError`](https://github.com/huggingface/huggingface_hub/blob/5eefebee2c150a2df950ab710db350e96c711433/src/huggingface_hub/_commit_api.py#L402) error and chains the original HTTP error. This PR modifies the retry mechanism's error handling to account for that.

Fix https://github.com/huggingface/datasets/issues/6392
Retrieve cached datasets that were pushed to hub when offline
Author: lhoestq | Pull request: yes | State: open

I drafted the logic to retrieve a no-script dataset in the cache.
For example it can reload datasets that were pushed to hub if they exist in the cache.

example:

```python
>>> Dataset.from_dict({"a": [1, 2]}).push_to_hub("lhoestq/tmp")
>>> load_dataset("lhoestq/tmp")
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

and later, without connection:

```python
>>> load_dataset("lhoestq/tmp")
Using the latest cached version of the dataset from /Users/quentinlhoest/.cache/huggingface/datasets/lhoestq___tmp/*/*/0b3caccda1725efb(last modified on Wed Nov 29 16:50:27 2023) since it couldn't be found locally at lhoestq/tmp.
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

fix https://github.com/huggingface/datasets/issues/3547

## Implementation details (EDITED)

I continued in https://github.com/huggingface/datasets/pull/6493, see the changes there

TODO:
- [x] tests
- [ ] compatible with https://github.com/huggingface/datasets/pull/6458
Lazy data files resolution
Author: lhoestq | Pull request: yes | State: closed

Related to discussion at https://github.com/huggingface/datasets/pull/6255

this makes this code run in 2sec instead of >10sec

```python
from datasets import load_dataset

ds = load_dataset("glue", "sst2", streaming=True, trust_remote_code=False)
```

For some datasets with many configs and files it can be up to 100x faster.
This is particularly important now that some datasets will be loaded from the Parquet export instead of the scripts.

The data files are only resolved in the builder `__init__`. To do so I added DataFilesPatternsList and DataFilesPatternsDict that have `.resolve()` to return resolved DataFilesList and DataFilesDict
`TypeError`: huggingface_hub.hf_file_system.HfFileSystem.find() got multiple values for keyword argument 'maxdepth'
Author: wasertech | Pull request: no | State: closed

### Describe the bug

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Steps to reproduce the bug

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Expected behavior

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Environment info

Please see https://github.com/huggingface/huggingface_hub/issues/1872
Refactor `dill` logic
Author: mariosasko | Pull request: yes | State: closed

Refactor the `dill` logic to make it easier to maintain (and fix some issues along the way)

It makes the following improvements to the serialization API:
* consistent order of a `dict`'s keys
* support for hashing `torch.compile`-ed modules and functions
* deprecates `datasets.fingerprint.hashregister` as the `hashregister`-ed reducers are never invoked anyways (does not support nested data as `pickle`/`dill` do)


~~TODO: optimize hashing of `pa.Table` and `datasets.table.Table`~~ The `pa_array.to_string` approach is faster for large arrays because it outputs the first 10 and last 10 elements (by default). The problem is that this can produce identical hashes for non-identical arrays if their differing elements get ellipsed...
 
Fix https://github.com/huggingface/datasets/issues/6440, fix https://github.com/huggingface/datasets/issues/5839
Unable to read "marsyas/gtzan" data
Author: gerald-wrona | Pull request: no | State: closed

Hi, this is my code and the error:
```
from datasets import load_dataset
gtzan = load_dataset("marsyas/gtzan", "all")
```
[error_trace.txt](https://github.com/huggingface/datasets/files/13464397/error_trace.txt)
[audio_yml.txt](https://github.com/huggingface/datasets/files/13464410/audio_yml.txt)
Python 3.11.5
Jupyter Notebook 6.5.4
Windows 10

I'm able to download and work with other datasets, but not this one. For example, both these below work fine:
```
from datasets import load_dataset
dataset = load_dataset("facebook/voxpopuli", "pl", split="train", streaming=True)
minds = load_dataset("PolyAI/minds14", name="en-US", split="train")
```
Thanks for your help

https://huggingface.co/datasets/marsyas/gtzan/tree/main
Support multiple image/audio columns in ImageFolder/AudioFolder
Author: severo | Pull request: no | State: closed

### Feature request

Have a metadata.csv file with multiple columns that point to relative image or audio files.

### Motivation

Currently, ImageFolder allows one column, called `file_name`, pointing to relative image files. On the same model, AudioFolder allows one column, called `file_name`, pointing to relative audio files.

But it's not possible to have two image columns, or to have two audio column, or to have one audio column and one image column.

### Your contribution

no specific contribution
Use parquet export if possible
Author: lhoestq | Pull request: yes | State: closed

The idea is to make this code work for datasets with scripts if they have a Parquet export

```python
ds = load_dataset("squad", trust_remote_code=False)
```

And more generally, it means we use the Parquet export whenever it's possible (it's safer and faster than dataset scripts).

I also added a `config.USE_PARQUET_EXPORT` variable to use in the datasets-server parquet conversion job

- [x] Needs https://github.com/huggingface/datasets/pull/6429 to be merged first

cc @severo I use the /parquet and /info endpoints from datasets-server
Support one dataset loader per config when using YAML
Author: severo | Pull request: no | State: open

### Feature request

See https://huggingface.co/datasets/datasets-examples/doc-unsupported-1

I would like to use CSV loader for the "csv" config, JSONL loader for the "jsonl" config, etc.

### Motivation

It would be more flexible for the users

### Your contribution

No specific contribution
Speech Commands v2 dataset doesn't match AST-v2 config
Author: vymao | Pull request: no | State: closed

### Describe the bug

[According](https://huggingface.co/MIT/ast-finetuned-speech-commands-v2) to `MIT/ast-finetuned-speech-commands-v2`, the model was trained on the Speech Commands v2 dataset. However, while the model config says the model should have 35 class labels, the dataset itself has 36 class labels. Moreover, the class labels themselves don't match between the model config and the dataset. It is difficult to reproduce the data used to fine tune `MIT/ast-finetuned-speech-commands-v2`.

### Steps to reproduce the bug

```
>>> model = ASTForAudioClassification.from_pretrained("MIT/ast-finetuned-speech-commands-v2")
>>> model.config.id2label
{0: 'backward', 1: 'follow', 2: 'five', 3: 'bed', 4: 'zero', 5: 'on', 6: 'learn', 7: 'two', 8: 'house', 9: 'tree', 10: 'dog', 11: 'stop', 12: 'seven', 13: 'eight', 14: 'down', 15: 'six', 16: 'forward', 17: 'cat', 18: 'right', 19: 'visual', 20: 'four', 21: 'wow', 22: 'no', 23: 'nine', 24: 'off', 25: 'three', 26: 'left', 27: 'marvin', 28: 'yes', 29: 'up', 30: 'sheila', 31: 'happy', 32: 'bird', 33: 'go', 34: 'one'}

>>> dataset = load_dataset("speech_commands", "v0.02", split="test")
>>> torch.unique(torch.Tensor(dataset['label']))
tensor([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
        14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.,
        28., 29., 30., 31., 32., 33., 34., 35.])
```
If you try to explore the [dataset itself](https://huggingface.co/datasets/speech_commands/viewer/v0.02/test), you can see that the id to label does not match what is provided by `model.config.id2label`.


### Expected behavior

The labels should match completely and there should be the same number of label classes between the model config and the dataset itself.

### Environment info

datasets = 2.14.6, transformers =  4.33.3
Use `filelock` package for file locking
Author: mariosasko | Pull request: yes | State: closed

Use the `filelock` package instead of `datasets.utils.filelock` for file locking to be consistent with `huggingface_hub` and not to be responsible for improving the `filelock` capabilities 🙂. 

(Reverts https://github.com/huggingface/datasets/pull/859, but these `INFO` logs are not printed by default (anymore?), so this should be okay)
Remove `Table.__getstate__` and `Table.__setstate__`
Author: LZHgrla | Pull request: yes | State: closed

When using distributed training, the code of `os.remove(filename)` may be executed separately by each rank, leading to `FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmprxxxxxxx.arrow'`

```python
from torch import distributed as dist

if dist.get_rank() == 0:
    dataset = process_dataset(*args, **kwargs)
    objects = [dataset]
else:
    objects = [None]
dist.broadcast_object_list(objects, src=0)
dataset = objects[0]
```
Trouble loading image folder with additional features -  metadata file ignored
Author: linoytsaban | Pull request: no | State: closed

### Describe the bug

Loading image folder with a caption column using `load_dataset(<image_folder_path>)` doesn't load the captions. 

When loading a local image folder with captions using `datasets==2.13.0`
```
from datasets import load_dataset
data = load_dataset(<image_folder_path>)
data.column_names
```
yields 
`{'train': ['image', 'prompt']}`

but when using `datasets==2.15.0`
yeilds 
`{'train': ['image']}`

Putting the images and `metadata.jsonl` file into a nested `train` folder **or** loading with `load_dataset("imagefolder", data_dir=<image_folder_path>)` solves the issue and
yields 
`{'train': ['image', 'prompt']}`

### Steps to reproduce the bug

1. create a folder `<image_folder_path>` that contains images and a metadata file with additional features- e.g. "prompt"
2. run:
```
from datasets import load_dataset
data = load_dataset("<image_folder_path>")
data.column_names
```

### Expected behavior

`{'train': ['image', 'prompt']}`

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Trouble Loading a Gated Dataset For User with Granted Permission
Author: e-trop | Pull request: no | State: closed

### Describe the bug

I have granted permissions to several users to access a gated huggingface dataset. The users accepted the invite and when trying to load the dataset using their access token they get 
`FileNotFoundError: Couldn't find a dataset script at .....` . Also when they try to click the url link for the dataset they get a 404 error.

### Steps to reproduce the bug

1. Grant access to gated dataset for specific users 
2. Users accept invitation
3. Users login to hugging face hub using cli login
4. Users run load_dataset

### Expected behavior

Dataset is loaded normally for users who were granted access to the gated dataset.

### Environment info

datasets==2.15.0