dalssy/data_del huggingface dataset:
GitHub issue: Refactor `dill` logic
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-11-27 20:01:25

Refactor the `dill` logic to make it easier to maintain (and fix some issues along the way)

It makes the following improvements to the serialization API:
* consistent order of a `dict`'s keys
* support for hashing `torch.compile`-ed modules and functions
* deprecates `datasets.fingerprint.hashregister` as the `hashregister`-ed reducers are never invoked anyways (does not support nested data as `pickle`/`dill` do)


~~TODO: optimize hashing of `pa.Table` and `datasets.table.Table`~~ The `pa_array.to_string` approach is faster for large arrays because it outputs the first 10 and last 10 elements (by default). The problem is that this can produce identical hashes for non-identical arrays if their differing elements get ellipsed...
 
Fix https://github.com/huggingface/datasets/issues/6440, fix https://github.com/huggingface/datasets/issues/5839
GitHub issue: Unable to read "marsyas/gtzan" data
Author: gerald-wrona (NONE)
State: closed | Created: 2023-11-25 15:13:17

Hi, this is my code and the error:
```
from datasets import load_dataset
gtzan = load_dataset("marsyas/gtzan", "all")
```
[error_trace.txt](https://github.com/huggingface/datasets/files/13464397/error_trace.txt)
[audio_yml.txt](https://github.com/huggingface/datasets/files/13464410/audio_yml.txt)
Python 3.11.5
Jupyter Notebook 6.5.4
Windows 10

I'm able to download and work with other datasets, but not this one. For example, both these below work fine:
```
from datasets import load_dataset
dataset = load_dataset("facebook/voxpopuli", "pl", split="train", streaming=True)
minds = load_dataset("PolyAI/minds14", name="en-US", split="train")
```
Thanks for your help

https://huggingface.co/datasets/marsyas/gtzan/tree/main
GitHub issue: Support multiple image/audio columns in ImageFolder/AudioFolder
Author: severo (COLLABORATOR)
State: closed | Created: 2023-11-24 10:34:09
Labels: name: duplicate, name: enhancement

### Feature request

Have a metadata.csv file with multiple columns that point to relative image or audio files.

### Motivation

Currently, ImageFolder allows one column, called `file_name`, pointing to relative image files. On the same model, AudioFolder allows one column, called `file_name`, pointing to relative audio files.

But it's not possible to have two image columns, or to have two audio column, or to have one audio column and one image column.

### Your contribution

no specific contribution
GitHub issue: Use parquet export if possible
Author: lhoestq (MEMBER)
State: closed | Created: 2023-11-23 17:31:57

The idea is to make this code work for datasets with scripts if they have a Parquet export

```python
ds = load_dataset("squad", trust_remote_code=False)
```

And more generally, it means we use the Parquet export whenever it's possible (it's safer and faster than dataset scripts).

I also added a `config.USE_PARQUET_EXPORT` variable to use in the datasets-server parquet conversion job

- [x] Needs https://github.com/huggingface/datasets/pull/6429 to be merged first

cc @severo I use the /parquet and /info endpoints from datasets-server
GitHub issue: Support one dataset loader per config when using YAML
Author: severo (COLLABORATOR)
State: open | Created: 2023-11-23 13:03:07
Labels: name: enhancement

### Feature request

See https://huggingface.co/datasets/datasets-examples/doc-unsupported-1

I would like to use CSV loader for the "csv" config, JSONL loader for the "jsonl" config, etc.

### Motivation

It would be more flexible for the users

### Your contribution

No specific contribution
GitHub issue: Speech Commands v2 dataset doesn't match AST-v2 config
Author: vymao (NONE)
State: closed | Created: 2023-11-22 20:46:36

### Describe the bug

[According](https://huggingface.co/MIT/ast-finetuned-speech-commands-v2) to `MIT/ast-finetuned-speech-commands-v2`, the model was trained on the Speech Commands v2 dataset. However, while the model config says the model should have 35 class labels, the dataset itself has 36 class labels. Moreover, the class labels themselves don't match between the model config and the dataset. It is difficult to reproduce the data used to fine tune `MIT/ast-finetuned-speech-commands-v2`.

### Steps to reproduce the bug

```
>>> model = ASTForAudioClassification.from_pretrained("MIT/ast-finetuned-speech-commands-v2")
>>> model.config.id2label
{0: 'backward', 1: 'follow', 2: 'five', 3: 'bed', 4: 'zero', 5: 'on', 6: 'learn', 7: 'two', 8: 'house', 9: 'tree', 10: 'dog', 11: 'stop', 12: 'seven', 13: 'eight', 14: 'down', 15: 'six', 16: 'forward', 17: 'cat', 18: 'right', 19: 'visual', 20: 'four', 21: 'wow', 22: 'no', 23: 'nine', 24: 'off', 25: 'three', 26: 'left', 27: 'marvin', 28: 'yes', 29: 'up', 30: 'sheila', 31: 'happy', 32: 'bird', 33: 'go', 34: 'one'}

>>> dataset = load_dataset("speech_commands", "v0.02", split="test")
>>> torch.unique(torch.Tensor(dataset['label']))
tensor([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,
        14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.,
        28., 29., 30., 31., 32., 33., 34., 35.])
```
If you try to explore the [dataset itself](https://huggingface.co/datasets/speech_commands/viewer/v0.02/test), you can see that the id to label does not match what is provided by `model.config.id2label`.


### Expected behavior

The labels should match completely and there should be the same number of label classes between the model config and the dataset itself.

### Environment info

datasets = 2.14.6, transformers =  4.33.3
GitHub issue: Use `filelock` package for file locking
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-11-22 19:04:45

Use the `filelock` package instead of `datasets.utils.filelock` for file locking to be consistent with `huggingface_hub` and not to be responsible for improving the `filelock` capabilities 🙂. 

(Reverts https://github.com/huggingface/datasets/pull/859, but these `INFO` logs are not printed by default (anymore?), so this should be okay)
GitHub issue: Remove `Table.__getstate__` and `Table.__setstate__`
Author: LZHgrla (CONTRIBUTOR)
State: closed | Created: 2023-11-22 17:55:10

When using distributed training, the code of `os.remove(filename)` may be executed separately by each rank, leading to `FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmprxxxxxxx.arrow'`

```python
from torch import distributed as dist

if dist.get_rank() == 0:
    dataset = process_dataset(*args, **kwargs)
    objects = [dataset]
else:
    objects = [None]
dist.broadcast_object_list(objects, src=0)
dataset = objects[0]
```
GitHub issue: Trouble loading image folder with additional features -  metadata file ignored
Author: linoytsaban (NONE)
State: closed | Created: 2023-11-22 11:01:35

### Describe the bug

Loading image folder with a caption column using `load_dataset(<image_folder_path>)` doesn't load the captions. 

When loading a local image folder with captions using `datasets==2.13.0`
```
from datasets import load_dataset
data = load_dataset(<image_folder_path>)
data.column_names
```
yields 
`{'train': ['image', 'prompt']}`

but when using `datasets==2.15.0`
yeilds 
`{'train': ['image']}`

Putting the images and `metadata.jsonl` file into a nested `train` folder **or** loading with `load_dataset("imagefolder", data_dir=<image_folder_path>)` solves the issue and
yields 
`{'train': ['image', 'prompt']}`

### Steps to reproduce the bug

1. create a folder `<image_folder_path>` that contains images and a metadata file with additional features- e.g. "prompt"
2. run:
```
from datasets import load_dataset
data = load_dataset("<image_folder_path>")
data.column_names
```

### Expected behavior

`{'train': ['image', 'prompt']}`

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
GitHub issue: Trouble Loading a Gated Dataset For User with Granted Permission
Author: e-trop (NONE)
State: closed | Created: 2023-11-21 19:24:36

### Describe the bug

I have granted permissions to several users to access a gated huggingface dataset. The users accepted the invite and when trying to load the dataset using their access token they get 
`FileNotFoundError: Couldn't find a dataset script at .....` . Also when they try to click the url link for the dataset they get a 404 error.

### Steps to reproduce the bug

1. Grant access to gated dataset for specific users 
2. Users accept invitation
3. Users login to hugging face hub using cli login
4. Users run load_dataset

### Expected behavior

Dataset is loaded normally for users who were granted access to the gated dataset.

### Environment info

datasets==2.15.0
GitHub issue: `.map` not hashing under python 3.9
Author: changyeli (NONE)
State: closed | Created: 2023-11-21 15:14:54

### Describe the bug

The `.map` function cannot hash under python 3.9. Tried to use [the solution here](https://github.com/huggingface/datasets/issues/4521#issuecomment-1205166653), but still get the same message: 


`Parameter 'function'=<function map_to_pred at 0x7fa0b49ead30> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.`

### Steps to reproduce the bug

```python
def map_to_pred(batch):
    """
    Perform inference on an audio batch

    Parameters:
        batch (dict): A dictionary containing audio data and other related information.

    Returns:
        dict: The input batch dictionary with added prediction and transcription fields.
    """
    audio = batch['audio']
    input_features = processor(
        audio['array'], sampling_rate=audio['sampling_rate'], return_tensors="pt").input_features
    input_features = input_features.to('cuda')
    with torch.no_grad():
        predicted_ids = model.generate(input_features)
    preds = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
    batch['prediction'] = processor.tokenizer._normalize(preds)
    batch["transcription"] = processor.tokenizer._normalize(batch['transcription'])
    return batch

MODEL_CARD = "openai/whisper-small"
MODEL_NAME = MODEL_CARD.rsplit('/', maxsplit=1)[-1]
model = WhisperForConditionalGeneration.from_pretrained(MODEL_CARD)
processor = AutoProcessor.from_pretrained(
MODEL_CARD, language="english", task="transcribe")
model = torch.compile(model)
dt = load_dataset("audiofolder", data_dir=config['DATA']['dataset'], split="test")
dt = dt.cast_column("audio", Audio(sampling_rate=16000))
result = coraal_dt.map(map_to_pred, num_proc=16)
```

### Expected behavior

Hashed and cached dataset starts inferencing

### Environment info

- `transformers` version: 4.35.0
- Platform: Linux-5.14.0-284.30.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
GitHub issue: Support GeoParquet
Author: severo (COLLABORATOR)
State: open | Created: 2023-11-20 11:54:58
Labels: name: enhancement

### Feature request

Support the GeoParquet format

### Motivation

GeoParquet (https://geoparquet.org/) is a common format for sharing vectorial geospatial data on the cloud, along with "traditional" data columns.

It would be nice to be able to load this format with datasets, and more generally, in the Datasets Hub (see https://huggingface.co/datasets/joshuasundance/govgis_nov2023-slim-spatial/discussions/1).

### Your contribution

I would be happy to help work on a PR (but I don't think I can do one on my own).

Also, we have to define what we want to support:
- load all the columns, but get the "geospatial" column in text-only mode for now
- or, fully support the spatial features, maybe taking inspiration from (or depending upon) https://geopandas.org/en/stable/index.html (which itself depends on https://fiona.readthedocs.io/en/stable/, which requires a local install of https://gdal.org/)
GitHub issue: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Author: kopyl (NONE)
State: closed | Created: 2023-11-19 04:21:16

### Describe the bug

1. I ran dataset mapping with `num_proc=6` in it and got this error:
`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method`

I can't actually find a way to run multi-GPU dataset mapping. Can you help?

### Steps to reproduce the bug

1. Rund SDXL training with `num_proc=6`: https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py

### Expected behavior

Should work well

### Environment info

6x A100 SXM, Linux
GitHub issue: Use `ruff` for formatting
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-11-17 16:53:22

Use `ruff` instead of `black` for formatting to be consistent with `transformers` ([PR](https://github.com/huggingface/transformers/pull/27144)) and `huggingface_hub` ([PR 1](https://github.com/huggingface/huggingface_hub/pull/1783) and [PR 2](https://github.com/huggingface/huggingface_hub/pull/1789)).
GitHub issue: Better `tqdm` wrapper
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-11-17 15:45:15

This PR aligns the `tqdm` logic with `huggingface_hub` (without introducing breaking changes), as the current one is error-prone.

Additionally, it improves the doc page about the `datasets`' utilities, and the handling of local `fsspec` paths in `cached_path`.
 
Fix #6409
GitHub issue: load_dataset does not load all of the data in my input file
Author: demongolem-biz2 (NONE)
State: open | Created: 2023-11-17 14:28:50

### Describe the bug

I have 127 elements in my input dataset.  When I do a len on the dataset after loaded, it is only 124 elements.

### Steps to reproduce the bug

    train_dataset = nlp.load_dataset(data_args.dataset_path, name=data_args.qg_format, split=nlp.Split.TRAIN)
    valid_dataset = nlp.load_dataset(data_args.dataset_path, name=data_args.qg_format, split=nlp.Split.VALIDATION)
    logger.info(len(train_dataset))
    logger.info(len(valid_dataset))

Both train and valid input are 127 items.  However, they both only load 124 items.  The input format is in json.  At the end of the day, I am trying to create .pt files.  

### Expected behavior

I see all 127 elements in my dataset when performing len

### Environment info

Python 3.10.  CentOS operating system.  nlp==0.40, datasets==2.14.5, transformers==4.26.1
GitHub issue: Add trust_remote_code argument
Author: lhoestq (MEMBER)
State: closed | Created: 2023-11-16 12:12:54

Draft about adding `trust_remote_code` to `load_dataset`.

```python
ds = load_dataset(..., trust_remote_code=True)  # run remote code (current default)
```

It would default to `True` (current behavior) and in the next major release it will prompt the user to check the code before running it (we'll communicate on this before doing it of course).

```python
# in the future
ds = load_dataset(...)  # prompt the user to check the code before running it (future default)
ds = load_dataset(..., trust_remote_code=True)  # run remote code
ds = load_dataset(..., trust_remote_code=False)  # disallow remote code
```

Related to https://github.com/huggingface/datasets/issues/6400

Will do a separate PR to use the parquet export when possible
GitHub issue: More robust temporary directory deletion
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-11-15 19:06:42

While fixing the Windows errors in #6362, I noticed that `PermissionError` can still easily be thrown on the session exit by the temporary cache directory's finalizer (we would also have to keep track of intermediate datasets, copies, etc.). ~~Due to the low usage of `datasets` on Windows, this PR takes a simpler approach to the issue than https://github.com/huggingface/datasets/pull/2403 - it tries to delete the temporary cache directory, and if this fails, logs a warning message about using a `delete-temp-cache` CLI command to delete it manually. The problematic references are freed after the session exits, so the CLI command should then succeed.~~ This PR implements `Dataset.__setstate__` to register datasets with temporary cache files for deletion.
GitHub issue: Fix deprecation warning when building conda package
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-11-15 18:00:11

When building/releasing conda package, we get this deprecation warning:
```
/usr/share/miniconda/envs/build-datasets/bin/conda-build:11: DeprecationWarning: conda_build.cli.main_build.main is deprecated and will be removed in 4.0.0. Use `conda build` instead.
```

This PR fixes the deprecation warning by using `conda build` instead.
GitHub issue: [docs] troubleshooting guide
Author: MKhalusova (CONTRIBUTOR)
State: closed | Created: 2023-11-15 17:28:14

Hi all! This is a PR adding a troubleshooting guide for Datasets docs. 
I went through the library's GitHub Issues and Forum questions and identified a few issues that are common enough that I think it would be valuable to include them in the troubleshooting guide. These are: 
- creating a dataset from a folder and not following the required format
- authentication issues when using `push_to_hub`
- `Too Many Requests` with `push_to_hub`
- Pickling issues when using Dataset.from_generator()

There's also a section on asking for help. Please let me know if there are other common issues or advice that we can include here.
GitHub issue: Fix conda release by adding pyarrow-hotfix dependency
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-11-15 14:57:12

Fix conda release by adding pyarrow-hotfix dependency.

Note that conda release failed in latest 2.14.7 release: https://github.com/huggingface/datasets/actions/runs/6874667214/job/18696761723
```
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/test_tmp/run_test.py", line 2, in <module>
    import datasets
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/__init__.py", line 22, in <module>
    from .arrow_dataset import Dataset
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 67, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import Features, Image, Value
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/features/__init__.py", line 18, in <module>
    from .features import Array2D, Array3D, Array4D, Array5D, ClassLabel, Features, Sequence, Value
  File "/usr/share/miniconda/envs/build-datasets/conda-bld/datasets_1700036460222/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.12/site-packages/datasets/features/features.py", line 34, in <module>
    import pyarrow_hotfix  # noqa: F401  # to fix vulnerability on pyarrow<14.0.1
    ^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pyarrow_hotfix'
```
GitHub issue: Allow to choose the `writer_batch_size` when using `save_to_disk`
Author: NathanGodey (NONE)
State: open | Created: 2023-11-15 11:18:34
Labels: name: enhancement

### Feature request

Add an argument in `save_to_disk` regarding batch size, which would be passed to `shard` and other methods.

### Motivation

The `Dataset.save_to_disk` method currently calls `shard` without passing a `writer_batch_size` argument, thus implicitly using the default value (1000). This can result in RAM saturation when using a lot of processes on long text sequences or other modalities, or for specific IO configs.

### Your contribution

I would be glad to submit a PR, as long as it does not imply extensive tests refactoring.
GitHub issue: Set `usedforsecurity=False` in hashlib methods (FIPS compliance)
Author: Wauplin (CONTRIBUTOR)
State: closed | Created: 2023-11-14 10:47:09

Related to https://github.com/huggingface/transformers/issues/27034 and https://github.com/huggingface/huggingface_hub/pull/1782.

**TL;DR:** `hashlib` is not a secure library for cryptography-related stuff. We are only using `hashlib` for non-security-related purposes in `datasets` so it's fine. From Python 3.9 we set can `usedforsecurity=False` in any `hashlib` method which is mandatory for companies that forbid the use of `hashlib` for security purposes. This PR fixes that.

**Note:** before merging this we need to release a new tokenizers version that would allow the newest `huggingface_hub` version (see https://github.com/huggingface/tokenizers/pull/1385). Otherwise it might create friction to users that want to install `datasets` + `tokenizers` at the same time.
GitHub issue: User token is printed out!
Author: mohsen-goodarzi (NONE)
State: closed | Created: 2023-11-14 10:01:34

This line prints user token on command line! Is it safe?

https://github.com/huggingface/datasets/blob/12ebe695b4748c5a26e08b44ed51955f74f5801d/src/datasets/load.py#L2091
GitHub issue: Fix dependency conflict within CI build documentation
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-11-14 09:52:51

Manually fix dependency conflict on `typing-extensions` version originated by `apache-beam` + `pydantic` (now a dependency of `huggingface-hub`).
This is a temporary hot fix of our CI build documentation until we stop using `apache-beam`.
Fix #6406.
GitHub issue: using DownloadManager to download from local filesystem and disable_progress_bar, there will be an exception 
Author: neiblegy (NONE)
State: closed | Created: 2023-11-14 04:21:01

### Describe the bug

i'm using datasets.download.download_manager.DownloadManager to download files like "file:///a/b/c.txt", and i disable_progress_bar() to disable bar. there will be an exception as follows:

`AttributeError: 'function' object has no attribute 'close'
Exception ignored in: <function TqdmCallback.__del__ at 0x7fa8683d84c0>
Traceback (most recent call last):
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/fsspec/callbacks.py", line 233, in __del__
    self.tqdm.close()`

i check your source code in datasets/utils/file_utils.py:348 you define TqdmCallback derive from fsspec.callbacks.TqdmCallback

but in the newest fsspec code [https://github.com/fsspec/filesystem_spec/blob/master/fsspec/callbacks.py](url) , line 146, in this case, _DEFAULT_CALLBACK will take effect, but in line 234, it calls "close()" function which _DEFAULT_CALLBACK don't have such thing.

so i think the class "TqdmCallback" in datasets/utils/file_utils.py may override "__del__" function or report this bug to fsspec.



### Steps to reproduce the bug

as i said

### Expected behavior

no exception

### Environment info

datasets: 2.14.4
python: 3.9
platform: x86_64
GitHub issue: `IterableDataset` lost but not keep columns when map function adding columns with names in `remove_columns`
Author: shmily326 (NONE)
State: open | Created: 2023-11-14 03:12:08

### Describe the bug

IterableDataset lost but not keep columns when map function adding columns with names in remove_columns,
Dataset not.

May be related to the code below:
https://github.com/huggingface/datasets/blob/06c3ffb8d068b6307b247164b10f7c7311cefed4/src/datasets/iterable_dataset.py#L750-L756

### Steps to reproduce the bug

```python
dataset: IterableDataset = load_dataset("Anthropic/hh-rlhf", streaming=True, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx}   
```

```python
# when load_dataset with streaming=False, the column_names are kept: 
dataset: Dataset = load_dataset("Anthropic/hh-rlhf", streaming=False, split="train")
column_names = list(next(iter(dataset)).keys())  #  ['chosen', 'rejected']
# map_fn will return dict {"chosen": xxx, "rejected": xxx, "prompt": xxx, "history": xxxx}
dataset = dataset.map(map_fn, batched=True, remove_columns=column_names)
next(iter(dataset))
# output
# {'prompt': 'xxx, 'history': xxx, "chosen": xxx, "rejected": xxx}   
```


### Expected behavior

IterableDataset keep columns when map function adding columns with names in remove_columns

### Environment info

datasets==2.14.6
GitHub issue: CI Build PR Documentation is broken: ImportError: cannot import name 'TypeAliasType' from 'typing_extensions'
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-11-13 11:36:10

Our CI Build PR Documentation is broken. See: https://github.com/huggingface/datasets/actions/runs/6799554060/job/18486828777?pr=6390
```
ImportError: cannot import name 'TypeAliasType' from 'typing_extensions'
```
GitHub issue: ConfigNamesError on a simple CSV file
Author: severo (COLLABORATOR)
State: closed | Created: 2023-11-13 10:28:29
Labels: name: bug

See https://huggingface.co/datasets/Nguyendo1999/mmath/discussions/1

```
Error code:   ConfigNamesError
Exception:    TypeError
Message:      __init__() missing 1 required positional argument: 'dtype'
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/dataset/config_names.py", line 65, in compute_config_names_response
                  for config in sorted(get_dataset_config_names(path=dataset, token=hf_token))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
                  dataset_module = dataset_module_factory(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1512, in dataset_module_factory
                  raise e1 from None
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1489, in dataset_module_factory
                  return HubDatasetModuleFactoryWithoutScript(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1039, in get_module
                  dataset_infos = DatasetInfosDict.from_dataset_card_data(dataset_card_data)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/info.py", line 468, in from_dataset_card_data
                  dataset_info = DatasetInfo._from_yaml_dict(dataset_card_data["dataset_info"])
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/info.py", line 399, in _from_yaml_dict
                  yaml_data["features"] = Features._from_yaml_list(yaml_data["features"])
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1838, in _from_yaml_list
                  return cls.from_dict(from_yaml_inner(yaml_data))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1690, in from_dict
                  obj = generate_from_dict(dic)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1345, in generate_from_dict
                  return {key: generate_from_dict(value) for key, value in obj.items()}
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1345, in <dictcomp>
                  return {key: generate_from_dict(value) for key, value in obj.items()}
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1353, in generate_from_dict
                  return class_type(**{k: v for k, v in obj.items() if k in field_names})
              TypeError: __init__() missing 1 required positional argument: 'dtype'
```

This is the CSV file: https://huggingface.co/datasets/Nguyendo1999/mmath/blob/dbcdd7c2c6fc447f852ec136a7532292802bb46f/math_train.csv
GitHub issue: Support pyarrow 14.0.1 and fix vulnerability CVE-2023-47248
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-11-13 09:15:39

Support `pyarrow` 14.0.1 and fix vulnerability [CVE-2023-47248](https://github.com/advisories/GHSA-5wvp-7f3h-6wmm).

Fix #6396.
GitHub issue: Cannot import datasets on google colab (python 3.10.12)
Author: nabilaannisa (NONE)
State: closed | Created: 2023-11-13 08:14:43

### Describe the bug

I'm trying A full colab demo notebook of zero-shot-distillation from https://github.com/huggingface/transformers/tree/main/examples/research_projects/zero-shot-distillation but i got this type of error when importing  datasets on my google colab (python version is 3.10.12)

![image](https://github.com/huggingface/datasets/assets/15389235/6f7758a2-681d-4436-87d0-5e557838e368)

I found the same problem that have been solved in [#3326 ] but it seem still error on the google colab. I can't try on my local using jupyter notebook because of my laptop resource doesn't fulfill the requirements. 

Please can anyone help me solve this problem. Thank you 😅 

### Steps to reproduce the bug

Error:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-8-b6e092f83978>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from datasets import load_dataset
      2 
      3 # Print all the available datasets
      4 from huggingface_hub import list_datasets
      5 print([dataset.id for dataset in list_datasets()])

6 frames
[/usr/lib/python3.10/functools.py](https://localhost:8080/#) in update_wrapper(wrapper, wrapped, assigned, updated)
     59     # Issue #17482: set __wrapped__ last so we don't inadvertently copy it
     60     # from the wrapped function when updating __dict__
---> 61     wrapper.__wrapped__ = wrapped
     62     # Return the wrapper so this can be used as a decorator via partial()
     63     return wrapper

AttributeError: readonly attribute
```

### Expected behavior

Run success on Google Colab (free)

### Environment info

Windows 11 x64, Google Colab free
GitHub issue: Safely load datasets by disabling execution of dataset loading script
Author: irenedea (NONE)
State: closed | Created: 2023-11-10 23:48:29
Labels: name: enhancement

### Feature request

Is there a way to disable execution of dataset loading script using `load_dataset`? This is a security vulnerability that could lead to arbitrary code execution. 

Any suggested workarounds are welcome as well. 

### Motivation

This is a security vulnerability that could lead to arbitrary code execution. 

### Your contribution

n/a
GitHub issue: TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array
Author: y-hwang (NONE)
State: open | Created: 2023-11-10 20:48:46

### Describe the bug

Hi, I am preprocessing a large custom dataset with numpy arrays. I am running into this TypeError during writing in a dataset.map() function. I've tried decreasing writer batch size, but this error persists. This error does not occur for smaller datasets. 



Thank you!



### Steps to reproduce the bug

Traceback (most recent call last):
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1354, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3493, in _map_single
    writer.write_batch(batch)
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_writer.py", line 555, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 243, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/arrow_writer.py", line 184, in __arrow_array__
    out = numpy_to_pyarrow_listarray(data)
  File "/n/home12/yhwang/.conda/envs/lib/python3.10/site-packages/datasets/features/features.py", line 1394, in numpy_to_pyarrow_listarray
    values = pa.ListArray.from_arrays(offsets, values)
  File "pyarrow/array.pxi", line 2004, in pyarrow.lib.ListArray.from_arrays
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

### Expected behavior

Type should not be a ChunkedArray

### Environment info

datasets v2.14.5
arrow v1.2.3
pyarrow v12.0.1
GitHub issue: Raise a different exception for inexisting dataset vs files without known extension
Author: severo (COLLABORATOR)
State: closed | Created: 2023-11-10 13:22:14

See https://github.com/huggingface/datasets-server/issues/2082#issuecomment-1805716557

We have the same error for:
- https://huggingface.co/datasets/severo/a_dataset_that_does_not_exist: a dataset that does not exist
- https://huggingface.co/datasets/severo/test_files_without_extension: a dataset with files without a known extension

```
>>> import datasets
>>> datasets.get_dataset_config_names('severo/a_dataset_that_does_not_exist')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1508, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /home/slesage/hf/datasets-server/services/worker/severo/a_dataset_that_does_not_exist/a_dataset_that_does_not_exist.py or any data file in the same directory. Couldn't find 'severo/a_dataset_that_does_not_exist' on the Hugging Face Hub either: FileNotFoundError: Dataset 'severo/a_dataset_that_does_not_exist' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with `huggingface-cli login`.
>>> datasets.get_dataset_config_names('severo/test_files_without_extension')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 351, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1508, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /home/slesage/hf/datasets-server/services/worker/severo/test_files_without_extension/test_files_without_extension.py or any data file in the same directory. Couldn't find 'severo/test_files_without_extension' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in severo/test_files_without_extension.
```

To differentiate, we must parse the error message (only the end is different). We should have a different exception for these two errors.
GitHub issue: Add ability to set lock type
Author: leoleoasd (NONE)
State: closed | Created: 2023-11-09 22:12:30
Labels: name: enhancement

### Feature request

Allow setting file lock type, maybe from an environment variable

Currently, it only depends on whether fnctl is available:

https://github.com/huggingface/datasets/blob/12ebe695b4748c5a26e08b44ed51955f74f5801d/src/datasets/utils/filelock.py#L463-L470C16

### Motivation

In my environment, flock isn't supported on a network attached drive

### Your contribution

I'll be happy to submit a pr.
GitHub issue: TorchFormatter images (H, W, C) instead of (C, H, W) format
Author: Modexus (CONTRIBUTOR)
State: closed | Created: 2023-11-09 16:02:15

### Describe the bug

Using .set_format("torch") leads to images having shape (H, W, C), the same as in numpy. 
However, pytorch normally uses (C, H, W) format.

Maybe I'm missing something but this makes the format a lot less useful as I then have to permute it anyways.
If not using the format it is possible to directly use torchvision transforms but any non-transformed value will not be a tensor.

Is there a reason for this choice?

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Audio, Image
images = ["path/to/image.png"] * 10
features = Features({"image": Image()})
ds = Dataset.from_dict({"image": images}, features=features) 
ds = ds.with_format("torch")
ds[0]["image"].shape
```
```python
torch.Size([512, 512, 4])
```

### Expected behavior

```python
from datasets import Dataset, Features, Audio, Image
images = ["path/to/image.png"] * 10
features = Features({"image": Image()})
ds = Dataset.from_dict({"image": images}, features=features) 
ds = ds.with_format("torch")
ds[0]["image"].shape
```
```python
torch.Size([4, 512, 512])
```

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.5.9-100.fc37.x86_64-x86_64-with-glibc2.31
- Python version: 3.11.6
- Huggingface_hub version: 0.18.0
- PyArrow version: 14.0.1
- Pandas version: 2.1.2
GitHub issue: Filter occasionally hangs
Author: dakinggg (NONE)
State: open | Created: 2023-11-09 06:18:30

### Describe the bug

A call to `.filter` occasionally hangs (after the filter is complete, according to tqdm)

There is a trace produced
```
Exception ignored in: <function Dataset.__del__ at 0x7efb48130c10>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/datasets/arrow_dataset.py", line 1366, in __del__
    if hasattr(self, "_indices"):
  File "/usr/lib/python3/dist-packages/composer/core/engine.py", line 123, in sigterm_handler
    sys.exit(128 + signal)
SystemExit: 143
```

but I'm not sure if the trace is actually from `datasets`, or from surrounding code that is trying to clean up after datasets gets stuck.

Unfortunately I can't reproduce this issue anywhere close to reliably. It happens infrequently when using `num_procs > 1`. Anecdotally I started seeing it when using larger datasets (~10M samples).

### Steps to reproduce the bug

N/A see description

### Expected behavior

map/filter calls always complete sucessfully

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.2
GitHub issue: Webdataset dataset builder
Author: lhoestq (MEMBER)
State: closed | Created: 2023-11-08 17:31:59

Allow `load_dataset` to support the Webdataset format.

It allows users to download/stream data from local files or from the Hugging Face Hub.

Moreover it will enable the Dataset Viewer for Webdataset datasets on HF.

## Implementation details

- I added a new Webdataset builder
- dataset with TAR files are now read using the Webdataset builder
- Basic decoding from `webdataset` is used by default, except unsafe ones like pickle
- HF authentication support is done with `xopen`

## TODOS

- [x] tests
- [x] docs
GitHub issue: handle future deprecation argument
Author: winglian (CONTRIBUTOR)
State: closed | Created: 2023-11-08 14:21:25

getting this error:
```
/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/datasets/table.py:1387: FutureWarning: promote has been superseded by mode='default'.
  return cls._concat_blocks(pa_tables_to_concat_vertically, axis=0)
```

Since datasets supports arrow greater than 8.0.0, we need to handle both cases.

[Arrow v14 docs](https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html)
[Arrow v13 docs](https://arrow.apache.org/docs/13.0/python/generated/pyarrow.concat_tables.html)
GitHub issue: How to create 3d medical imgae dataset?
Author: QingYunA (NONE)
State: open | Created: 2023-11-07 11:27:36
Labels: name: enhancement

### Feature request

I am newer to huggingface, after i look up `datasets` docs, I can't find how to create the dataset contains 3d medical image (ends with '.mhd', '.dcm', '.nii')

### Motivation

help us to upload 3d medical dataset to huggingface!

### Your contribution

I'll submit a PR if I find a way to add this feature
GitHub issue: How to load existing downloaded dataset ?
Author: liming-ai (NONE)
State: closed | Created: 2023-11-06 22:51:44
Labels: name: enhancement

Hi @mariosasko @lhoestq  @katielink 

Thanks for your contribution and hard work.

### Feature request

First, I download a dataset as normal by:
```
from datasets import load_dataset

dataset = load_dataset('username/data_name', cache_dir='data')
```

The dataset format in `data` directory will be:
```
-data
  |-data_name
    |-test-00000-of-00001-bf4c733542e35fcb.parquet
    |-train-00000-of-00001-2a1df75c6bce91ab.parquet
```


Then I use SCP to clone this dataset into another machine, and then try:
```
from datasets import load_dataset

dataset = load_dataset('data/data_name')  # load from local path
```

This leads to re-generating training and validation split for each time, and the disk quota will be duplicated occupation.

How can I just load the dataset without generating and saving these splits again?

### Motivation

I do not want to download the same dataset in two machines, scp is much faster and better than HuggingFace API. I hope we can directly load the downloaded datasets (.parquest)

### Your contribution

Please refer to the feature
GitHub issue: Formatting overhead
Author: d-miketa (NONE)
State: closed | Created: 2023-11-06 19:06:38

### Describe the bug

Hi! I very recently noticed that my training time is dominated by batch formatting. Using Lightning's profilers, I located the bottleneck within `datasets.formatting.formatting` and then narrowed it down with `line-profiler`. It turns out that almost all of the overhead is due to creating new instances of `self.python_arrow_extractor`. I admit I'm confused why that could be the case - as far as I can tell there's no complex `__init__` logic to execute.

![image](https://github.com/huggingface/datasets/assets/320321/5e022e0b-0d21-43d0-8e6f-9e641142e96b)

### Steps to reproduce the bug

1. Set up a dataset `ds` with potentially several (4+) columns (not sure if this is necessary, but it did at one point of the investigation make overhead worse)
2. Process it using a custom transform, `ds = ds.with_transform(transform_func)`
3. Decorate this function https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py#L512 with `@profile` from https://pypi.org/project/line-profiler/
4. Profile with `$ kernprof -l script_to_profile.py`

### Expected behavior

Batch formatting should have acceptable overhead.

### Environment info

```
datasets=2.14.6
pyarrow=14.0.0
```
GitHub issue: Get an error when i try to concatenate the squad dataset with my own dataset
Author: CCDXDX (NONE)
State: closed | Created: 2023-11-06 14:29:22

### Describe the bug

Hello,

I'm new here and I need to concatenate the squad dataset with my own dataset i created. I find the following error when i try to do it: Traceback (most recent call last):

  Cell In[9], line 1
    concatenated_dataset = concatenate_datasets([train_dataset, dataset1])

  File ~\anaconda3\Lib\site-packages\datasets\combine.py:213 in concatenate_datasets
    return _concatenate_map_style_datasets(dsets, info=info, split=split, axis=axis)

  File ~\anaconda3\Lib\site-packages\datasets\arrow_dataset.py:6002 in _concatenate_map_style_datasets
    _check_if_features_can_be_aligned([dset.features for dset in dsets])

  File ~\anaconda3\Lib\site-packages\datasets\features\features.py:2122 in _check_if_features_can_be_aligned
    raise ValueError(

ValueError: The features can't be aligned because the key answers of features {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)} has unexpected type - Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None) (expected either {'answer_start': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'text': Value(dtype='string', id=None)} or Value("null").

### Steps to reproduce the bug

```python

from huggingface_hub import notebook_login
from datasets import load_dataset



notebook_login("mymailadresse", "mypassword")
squad = load_dataset("squad", split="train[:5000]")
squad = squad.train_test_split(test_size=0.2)
dataset1 = squad["train"]




import json

mybase = [
    {
        "id": "1",
        "context": "She lives in Nantes",
        "question": "Where does she live?",
        "answers": {
            "text": "Nantes",
            "answer_start": [13],
        }
    }
]




# Save the data to a JSON file
json_file_path = r"C:\Users\mypath\thefile.json"
with open(json_file_path, "w", encoding= "utf-8") as json_file:
    json.dump(mybase, json_file, indent=4)




# Load the JSON file as a dataset
custom_dataset = load_dataset("json", data_files=json_file_path)


# Access the train split
train_dataset = custom_dataset["train"]


from datasets import concatenate_datasets


# Concatenate the datasets
concatenated_dataset = concatenate_datasets([train_dataset, dataset1])
```

### Expected behavior

I would expect the two datasets to be concatenated without error. The len(dataset1) is equal to 4000 and the len(train_dataset) is equal to 1 so I would exepect concatenated_dataset to be created and having lenght 4001.

### Environment info

Python 3.11.4 and using windows

Thank you for your help