konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
remove_columns is not in-place but the doc shows it is in-place

### Describe the bug

The doc of `datasets` v2.17.0/v2.17.1 shows that `remove_columns` is in-place. [link](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)

In the text classification example of transformers v4.38.1, the columns are not removed.
https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Steps to reproduce the bug

https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Expected behavior

Actually remove the columns.

### Environment info

1. datasets v2.17.0
2. transformers v4.38.1

Good catch! I've opened a PR with a fix in the `transformers` repo.

@mariosasko Thanks!

Will the doc of `datasets` be updated?

I find some possible mistakes in doc about whether `remove_columns` is in-place.
1. [You can also remove a column using map() with remove_columns but the present method is in-place (doesn’t copy the data to a new dataset) and is thus faster.](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.Dataset.remove_columns)
2. [You can also remove a column using Dataset.map() with remove_columns but the present method is in-place (doesn’t copy the data to a new dataset) and is thus faster.](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)
3. [🤗 Datasets also has a remove_columns() function which is faster because it doesn’t copy the data of the remaining columns.](https://huggingface.co/docs/datasets/v2.17.1/en/process#map)

I've linked a PR that will fix the usage in the `datasets` docs.
Support JSON file with an array of strings

Support loading a dataset from a JSON file with an array of strings.

See: https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1

https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1 has been fixed, but how can we check if there are other datasets with the same error, in datasets-server's database? I don't know how to get the list of erroneous cache entries, since we only copied `Error code:   JobManagerCrashedError`, but not the traceback in `details`... Do you remember the error message, or the underlying exception, we had?
__add__ for Dataset, IterableDataset

It's too cumbersome to write this command every time we perform a dataset merging operation. ```pythonfrom datasets import concatenate_datasets``` We have added a simple `__add__` magic method to each class using `concatenate_datasets.`

```python
from datasets import load_dataset

bookcorpus = load_dataset("bookcorpus", split="train")
wiki = load_dataset("wikimedia/wikipedia", "20231101.ab", split="train")
wiki = wiki.remove_columns([col for col in wiki.column_names if col != "text"])  # only keep the 'text' column

bookcorpus + wiki

#Dataset({
#   features: ['text'],
#   num_rows: 74004228
#})
#Dataset({
#    features: ['text'],
#    num_rows: 6152
#})
#Dataset({
#   features: ['text'],
#    num_rows: 74010380
#})
```

Hi! You can find a reason why we are against this feature in https://github.com/huggingface/datasets/issues/3449. 

> It's too cumbersome to write this command every time we perform a dataset merging operation

Explicit is better than implicit, so this isn't a good enough reason. 

Thanks for the effort nonetheless :)!
Enhancement: Enable loading TSV files in load_dataset()

Fix #6691

Hi @harsh1504660,

Thanks for your work, but this functionality already exists. See my comment in the corresponding issue: https://github.com/huggingface/datasets/issues/6691#issuecomment-1963449923

Next time you would like to contribute, I would suggest you take on an issue that is previously validated by one of the maintainers. Thanks anyway.
load_dataset() does not support tsv

### Feature request

the load_dataset() for local functions support file types like csv, json etc but not of type tsv (tab separated values).

### Motivation

cant easily load files of type tsv, have to convert them to another type like csv then load

### Your contribution

Can try by raising a PR with a little help, currently went through the code but didn't fully understand

#self-assign

Hi @dipsivenkatesh,

Please note that this functionality is already implemented. Our CSV builder uses `pandas.read_csv` under the hood, and you can pass the parameter `delimiter="\t"` to read TSV files.

See the list of CSV config parameters in our docs: https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.packaged_modules.csv.CsvConfig
Question: Is there any way for uploading a large image dataset?

I am uploading an image dataset like this:
```
dataset = load_dataset(
    "json",
    data_files={"train": "data/custom_dataset/train.json", "validation": "data/custom_dataset/val.json"},
)
dataset = dataset.cast_column("images", Sequence(Image()))
dataset.push_to_hub("StanfordAIMI/custom_dataset", max_shard_size="1GB")
```
where it takes a long time in the `Map` process. Do you think I can use multi-processing to map all the image data to the memory first? For the `Map()` function, I can set `num_proc`. But for `push_to_hub` and `cast_column`, I can not find it.

Thanks in advance!

Best,

```
import pandas as pd
from datasets import Dataset, Image

# Read the CSV file
data = pd.read_csv("XXXX.csv")

# Create a Hugging Face Dataset
dataset = Dataset.from_pandas(data)
dataset = dataset.cast_column("file_name", Image())

# Upload to Hugging Face Hub (make sure authentication is set up)
dataset.push_to_hub("XXXXX"")
```

stuck in "Casting the dataset
![截屏2024-05-02 11 44 50](https://github.com/huggingface/datasets/assets/48406770/dc012dc5-16f6-4fd5-9e02-1b705c552c5b)
"
Node.js 16 GitHub Actions are deprecated

`Node.js` 16 GitHub Actions are deprecated. See: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

We should update them to Node 20.

See warnings in our CI, e.g.: https://github.com/huggingface/datasets/actions/runs/7957295009?pr=6678
> Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-python@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
Can't Read List of JSON Files Properly

### Describe the bug

Trying to read a bunch of JSON files into Dataset class but default approach doesn't work. I don't get why it works when I read it one by one but not when I pass as a list :man_shrugging: 

The code fails with 

```
ArrowInvalid: JSON parse error: Invalid value. in row 0  
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte 
DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

This doesn't work

```
from datasets import Dataset

# dir contains 100 json files.
Dataset.from_json("/PUT SOME PATH HERE/*")
```

This works:
```
from datasets import concatenate_datasets

ls_ds = []

for file in list_of_json_files:
    ls_ds.append(Dataset.from_json(file))

ds = concatenate_datasets(ls_ds)

```





### Expected behavior

I expect this to read json files properly as error is not clear

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Found the issue, if there are other files in the directory, it gets caught into this `*` so essentially it should be `*.json`. Could we possibly to check for list of files to make sure the pattern matches json files and raise error if not?

I don't think we should filter for `*.json` as this might silently remove desired files for many users. And this could be a major breaking change for many organizations.

You could do the globbing yourself which would keep the code clean.

```python
from glob import glob

Dataset.from_json(glob('folder/*.json'))
```

I think it should still be fine to log a warning message in case the folder contains different files? I also don't get why would this be breaking as in the end using `from_FILE_TYPE` should be able to read a specific file type only. Maybe some other use case I am not aware of but since globbing or this case not mentioned anywhere in the doc, I spent quite a bit of time trying to figure out where the issue was. Just making sure it's clear for users.
Allow image model (color conversion) to be specified as part of datasets Image() decode

### Feature request

Typical torchvision / torch Datasets in image applications apply color conversion in the Dataset portion of the code as part of image decode, separately from the image transform stack. This is true for PIL.Image where convert is usually called in dataset, for native torchvision https://pytorch.org/vision/main/generated/torchvision.io.decode_jpeg.html, and similarly in tensorflow.data pipelines decode_jpeg or https://www.tensorflow.org/api_docs/python/tf/io/decode_and_crop_jpeg have a channels arg that allows controlling the image mode in the decode step.

datasets currently requires this pattern (from [examples](https://huggingface.co/docs/datasets/main/en/image_process)):
```
from torchvision.transforms import Compose, ColorJitter, ToTensor

jitter = Compose(
    [
         ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.7),
         ToTensor(),
    ]
)

def transforms(examples):
    examples["pixel_values"] = [jitter(image.convert("RGB")) for image in examples["image"]]
    return examples
```




### Motivation

It would be nice to be able to handle `image.convert("RGB")` (or other modes) in the decode step, before applying torchvision transforms, this would reduce differences in code when handling pipelines that can handle torchvision, webdatset, or hf datasets with fewer code differences and without needing to handle image mode argument passing in two different stages of the pipelines...

### Your contribution

Can do a PR with guidance on how mode should be passed / set on the dataset.

It would be a great addition indeed :)

This can be implemented the same way we have `sampling_rate` for Audio(): we just add a new parameter to the Image() type and take this parameter into account in `Image.decode_example`

EDIT: adding an example of how it can be used:

```python
ds = ds.cast_column("image", Image(mode=...))
```
Depprcated Overview.ipynb Link to new Quickstart Notebook invalid

### Describe the bug

For the dreprecated notebook found  [here](https://github.com/huggingface/datasets/blob/main/notebooks/Overview.ipynb). The link to the new notebook is broken. 

### Steps to reproduce the bug

Click the [Quickstart notebook](https://github.com/huggingface/notebooks/blob/main/datasets_doc/quickstart.ipynb) link in the notebook.

### Expected behavior

I believe is it suposed to link [here](https://github.com/huggingface/notebooks/blob/main/datasets_doc/en/quickstart.ipynb). That is mentioned in the readme.

### Environment info

Colab

Good catch! Feel free to open a PR to fix the link.
IterableDataset `set_epoch` is ignored when DataLoader `persistent_workers=True`

### Describe the bug

When persistent workers are enabled, the epoch that's set via the IterableDataset instance held by the training process is ignored by the workers as they are disconnected across processes.

PyTorch samplers for non-iterable datasets have a mechanism to sync this, datasets.IterableDataset does not.

In my own use of IterableDatasets I usually track the epoch count which crosses process boundaries in a multiprocessing.Value 

### Steps to reproduce the bug

Use a streaming dataset (Iterable) w/ the recommended pattern below and `persistent_workers=True` in the torch DataLoader. 

```
for epoch in range(epochs):
    shuffled_dataset.set_epoch(epoch)
    for example in shuffled_dataset:
        ...
```

### Expected behavior

When the canonical bit of code above is used with `num_workers > 0` and `persistent_workers=True`, the epoch set via `set_epoch()` is propagated to the IterableDataset instances in the worker processes


### Environment info

N/A
ValueError

### Describe the bug

ValueError                                Traceback (most recent call last)
[<ipython-input-11-9b99bc80ec23>](https://localhost:8080/#) in <cell line: 11>()
      9 import numpy as np
     10 import matplotlib.pyplot as plt
---> 11 from datasets import DatasetDict, Dataset
     12 from transformers import AutoTokenizer, AutoModelForSequenceClassification
     13 from transformers import Trainer, TrainingArguments

5 frames
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     16 __version__ = "2.17.0"
     17 
---> 18 from .arrow_dataset import Dataset
     19 from .arrow_reader import ReadInstruction
     20 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     65 
     66 from . import config
---> 67 from .arrow_reader import ArrowReader
     68 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     69 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     27 
     28 import pyarrow as pa
---> 29 import pyarrow.parquet as pq
     30 from tqdm.contrib.concurrent import thread_map
     31 

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/__init__.py](https://localhost:8080/#) in <module>
     18 # flake8: noqa
     19 
---> 20 from .core import *

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/core.py](https://localhost:8080/#) in <module>
     34 import pyarrow as pa
     35 import pyarrow.lib as lib
---> 36 import pyarrow._parquet as _parquet
     37 
     38 from pyarrow._parquet import (ParquetReader, Statistics,  # noqa

/usr/local/lib/python3.10/dist-packages/pyarrow/_parquet.pyx in init pyarrow._parquet()

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Expected behavior

Resolve the binary incompatibility

### Environment info

Google Colab Note book

Hi @prashanth19bolukonda,

You have to restart the notebook runtime session after the installation of `datasets`.

Duplicate of:
- #5923

Thank you soo much

On Fri, Feb 16, 2024 at 8:14 PM Albert Villanova del Moral <
***@***.***> wrote:

> Closed #6670 <https://github.com/huggingface/datasets/issues/6670> as
> completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6670#event-11829788289>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A2Y44YDQOBUFUWMR4C5O3QTYT5WDJAVCNFSM6AAAAABDL24S5SVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRHAZDSNZYHAZDQOI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Default config for squad is incorrect

### Describe the bug

If you download Squad, it will download the plain_text version, but the config still specifies "default", so if you set the offline mode the cache will try to look it up according to the config_id which is "default" and this will say;
ValueError: Couldn't find cache for squad for config 'default'
Available configs in the cache: ['plain_text']

### Steps to reproduce the bug

1. export HF_DATASETS_OFFLINE=0
2. load_dataset("squad")
3. export HF_DATASETS_OFFLINE=1
4. load_dataset("squad")

### Expected behavior

We should change the config_name I guess? 

### Environment info

linux, latest version of datasets

you can try: pip install datasets==2.16.1
`write_examples_on_file` and `write_batch` are broken in `ArrowWriter`

### Describe the bug

`write_examples_on_file` and `write_batch` are broken in `ArrowWriter` since #6636. The order between the columns and the schema is not preserved anymore. So these functions don't work anymore unless the order happens to align well.

### Steps to reproduce the bug

Try to do `write_batch` with anything that has many columns, and it's likely to break.

### Expected behavior

I expect these functions to work, instead of it trying to cast a column to its incorrect type.

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-5.15.0-1040-aws-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0

Thanks for reporting! I've left some comments on the PR on how to fix this recent change rather than reverting it.

> Thanks for reporting! I've left some comments on the PR on how to fix this recent change rather than reverting it.

I feel that'd be good, but it'd be great to release a hotfix ASAP (a revert is a fast thing to do) so people can continue using this library and then focus on still applying the improvement.

Fixed by #6664
Import error on Google Colab

### Describe the bug

Cannot be imported on Google Colab, the import throws the following error:
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

1. `! pip install -U datasets`
2. `import datasets`

### Expected behavior

Should be possible to use the library

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0

Hi! This can happen if an incompatible `pyarrow` version (`pyarrow<12.0.0`) has been imported before the `datasets` installation and the Colab session hasn't been restarted afterward. To avoid the error, go to "Runtime -> Restart session" after `!pip install -U datasets` and before `import datasets`, or insert the `import os; os.kill(os.getpid(), 9)` cell between `!pip install -U datasets` and `import datasets` to do the same programmatically.

One possible cause might be the one pointed out by @mariosasko above, and you get the following warning on Colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```

On the other hand, if the old version of `pyarrow` is not previously imported (before the installation of `datasets`), the reported issue here is not reproducible: `datasets` can be installed, imported and used on Colab.

Duplicate of:
- #5923

Google Colab now pre-installs PyArrow 14.0.2, making this issue unlikely to happen. So, I'm unpinning it.
Release not pushed to conda channel

### Describe the bug

The github actions step to publish the release 2.17.0 to conda channel has failed due to expired token. Can some one please update the anaconda token rerun the failed action? @albertvillanova ?
 
![image](https://github.com/huggingface/datasets/assets/7138162/1b56ad3d-7643-4778-9cce-4bf531717700)


### Steps to reproduce the bug

Please see this actions [link](https://github.com/huggingface/datasets/actions/runs/7842473662)

### Expected behavior

The action runs successfully and the latest release is pushed to HuggingFace conda channel

### Environment info

Not applicable.

Thanks for reporting, @atulsaurav.

We are investigating the issue. 

I can't fix this issue because I do not appear as a team member of the huggingface datasets project: https://anaconda.org/huggingface/datasets

@lhoestq could you please add `datasets` team members to the corresponding Anaconda project?

Once this done, I could recreate and update the Anaconda token, as mentioned above it seems the current one has expired.

I think @LysandreJik has access ?

FYI it failed for 2.18.0 too: https://github.com/huggingface/datasets/actions/runs/8117132330/job/22188677936

We updated the token and I re-ran the conda releases :)
Error when loading a big local json file

### Describe the bug

When trying to load big json files from a local directory, `load_dataset` throws the following error

```
Traceback (most recent call last):
  File "/miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/builder.py", line 1989, in _prepare_split_single
    writer.write_table(table)
  File "miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/arrow_writer.py", line 573, in write_table
    pa_table = pa_table.combine_chunks()
  File "pyarrow/table.pxi", line 3638, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
```

### Steps to reproduce the bug

1. Download a big file, e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-train.json.gz`
2. Load it like `data = load_dataset("json", data_files=["nq-train.json"], split="train")`

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-train.json"], split="train")
```

A similarly formatted but smaller file, e.g. e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-dev.json.gz` is loaded without issues

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-dev.json"], split="train")
```

### Expected behavior

It should load normally

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0

I get similar when dealing with a large jsonl file (6k lines), 

> TypeError: Couldn't cast array of type timestamp[us] to null

Yet when I split it into 1k lines, files, load_dataset works fine!

https://github.com/huggingface/course/issues/692



What's the proposed solution? :-)
Slice splits support for datasets.load_from_disk

### Feature request

Support for slice splits in `datasets.load_from_disk`, similar to how it's already supported for `datasets.load_dataset`.

### Motivation

Slice splits are convienient in a numer of cases - adding support to `datasets.load_from_disk` would make working with local datasets easier and homogenize the APIs of load_from_disk and load_dataset. 

### Your contribution

Sure, if the devs think the feature request is sensible.
Update loading.mdx to include "jsonl" file loading.

* A small update to the documentation, noting the ability to load jsonl files. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6647). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks for adding the explicit loading command.
> 
> However, I would move it just below, where we present the JSON-Lines example.
> 
> * Maybe adding that this format is called JSON-Lines
> * Add the example after the JSON-Lines data example
> 
> https://github.com/huggingface/datasets/blob/14d9afbb7ae1b787c450261ca0ff374551993031/docs/source/loading.mdx#L135-L138

Thank you @albertvillanova for the feedback! I moved the jsonl file loading example to a more appropriate location.
Support fsspec 2023.12

Support fsspec 2023.12 by handling previous and new glob behavior.

The pinned fsspec version range dependency conflict has been affecting several of our users in https://github.com/iterative/dvc. I've opened an initial PR that I think should resolve the glob behavior changes with using datasets + the latest fsspec release.

Please let us know if there's any other fsspec related behavior in datasets that needs to be updated to get 2024.2 supported, we'd like to get this conflict resolved as quickly as possible and we're willing to contribute any additional work that's required here.

cc @dberenbaum
Differently dataset object saved than it is loaded.

### Describe the bug

Differently sized object is saved than it is loaded.

### Steps to reproduce the bug

Hi, I save dataset in a following way:
```
        dataset = load_dataset("json",
                               data_files={
                                   "train": os.path.join(input_folder, f"{task_meta_type}_{task_type}_train.jsonl"),
                                   "test": os.path.join(input_folder, f"{task_meta_type}_{task_type}_test.jsonl")})
        print(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
        print(f"Length of train dataset: {len(dataset['train'])}")
        print(f"Length of test dataset: {len(dataset['test'])}")
        dataset.save_to_disk(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
```

this yields output
```
.data/hf_dataset/propaganda_zanr
Length of train dataset: 7642
Length of test dataset: 1000
```

Everything looks fine.

Then I load the dataset
```python
from datasets import load_dataset

dataset_path = ".data/hf_dataset/propaganda_zanr"
dataset = load_dataset(dataset_path)

print(f"Length of train dataset: {len(dataset['train'])}")
print(f"Length of test dataset: {len(dataset['test'])}")

```

this prints
```
Generating train split: 1 examples [00:00, 72.10 examples/s]
Generating test split: 1 examples [00:00, 100.69 examples/s]
Length of train dataset: 1
Length of test dataset: 1
```

I dont' understand :(


### Expected behavior

same object is loaded

### Environment info

datasets==2.16.1

I see now, that I have to use `load_from_disk`, in order to load dataset properly, not `load_dataset`. Why is this behavior split? Why do we need both, `load_dataset` and `load_from_disk`?

Unless answered, I believe this might be helpful for other hf datasets newbies.

Anyway, made a `load_dataset` compatible dataset in a following way. I created a directory, and just copied jsonl there as `train.jsonl/test.jsonl`.
```python
output_folder = os.path.join(args.output_folder, f"{task_meta_type}_{task_type}")
os.makedirs(output_folder, exist_ok=True)
file = f"{task_meta_type}_{task_type}_train.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "train.jsonl"))
# now test
file = f"{task_meta_type}_{task_type}_test.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "test.jsonl"))
```


Hi @MFajcik, 

You can find information about save_to_disk/load_from_disk in our docs:
- https://huggingface.co/docs/datasets/v2.16.1/en/process#save
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.save_to_disk
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.load_from_disk
Sign Language Support

### Feature request

Currently, there are only several Sign Language labels, I would like to propose adding all the Signed Languages as new labels which are described in this ISO standard: https://www.evertype.com/standards/iso639/sign-language.html

### Motivation

Datasets currently only have labels for several signed languages. There are more signed languages in the world. Furthermore, some signed languages that have a lot of online data cannot be found because of this reason (for instance, German Sign Language, and there is no German Sign Language label on huggingface datasets even though there are a lot of readily available sign language datasets exist for German Sign Language, which are used very frequently in Sign Language Processing papers, and models.)

### Your contribution

I can submit a PR for this as well, adding the ISO codes and languages to the labels in datasets.
'with_format' is extremely slow when used together with 'interleave_datasets' or 'shuffle' on IterableDatasets

### Describe the bug

If you:

1. Interleave two iterable datasets together with the interleave_datasets function, or shuffle an iterable dataset
2. Set the output format to torch tensors with .with_format('torch')

Then iterating through the dataset becomes over 100x slower than it is if you don't apply the torch formatting.

### Steps to reproduce the bug

```python
import datasets
import torch
from tqdm import tqdm

rand_a = torch.randn(3,224,224)
rand_b = torch.randn(3,224,224)

a = torch.stack([rand_a] * 1000)
b = torch.stack([rand_b] * 1000)

features = datasets.Features({"tensor": datasets.Array3D(shape=(3,224,224), dtype="float32")})
ds_a = datasets.Dataset.from_dict({"tensor": a}, features=features).to_iterable_dataset()
ds_b = datasets.Dataset.from_dict({"tensor": b}, features=features).to_iterable_dataset()

# Iterating through either dataset with torch formatting is really fast (2000it/s on my machine)
for example in tqdm(ds_a.with_format('torch')):
    pass

# Iterating through either dataset shuffled is also pretty fast (100it/s on my machine)
for example in tqdm(ds_a.shuffle()):
    pass

# Iterating through this interleaved dataset is pretty fast (200it/s on my machine)
ds_fast = datasets.interleave_datasets([ds_a, ds_b])
for example in tqdm(ds_fast):
    pass

# Iterating through either dataset with torch formatting *after shuffling* is really slow... (<2it/s on my machine)
for example in tqdm(ds_a.shuffle().with_format('torch')):
    pass

# Iterating through this torch formatted interleaved dataset is also really slow (<2it/s on my machine)...
ds_slow = datasets.interleave_datasets([ds_a, ds_b]).with_format('torch')
for example in tqdm(ds_slow):
    pass

# Even doing this is way faster!! (70it/s on my machine)
for example in tqdm(ds_fast):
    test = torch.tensor(example['tensor'])

```

### Expected behavior

Applying torch formatting to the interleaved dataset shouldn't increase the time taken to iterate through the dataset by very much, since even explicitly converting every example is over 70x faster than calling .with_format('torch').

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.38
- Python version: 3.11.6
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


The "torch" formatting is usually fast because we do zero-copy conversion from the Arrow data on your disk to Torch tensors. However IterableDataset shuffling seems to do data copies that slow down the pipeline, and it shuffles python objects instead of Arrow data.

To fix this we need to implement `BufferShuffledExamplesIterable.iter_arrow()` (same as regular `BufferShuffledExamplesIterable.__iter__()` but yields Arrow tables)

https://github.com/huggingface/datasets/blob/b7d854b7fd3e9a330e21b76ee8421d4a7ebb4a7a/src/datasets/iterable_dataset.py#L968-L974
multi-GPU map does not work

### Describe the bug

Here is the code for single-GPU processing: https://pastebin.com/bfmEeK2y
Here is the code for multi-GPU processing: https://pastebin.com/gQ7i5AQy

Here is the video showing that the multi-GPU mapping does not work as expected (there are so many things wrong here, it's better to watch the 3-minute video than explain here):

https://youtu.be/RNbdPkSppc4

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

x2 RTX A4000

This should now be fixed by https://github.com/huggingface/datasets/pull/6550 and updated with https://github.com/huggingface/datasets/pull/6646

Feel free to re-open if you're still having issues :)
wiki_dpr.py error (ID mismatch between lines {id} and vector {vec_id}

### Describe the bug

I'm trying to run a rag example, and the dataset is wiki_dpr.
wiki_dpr download and extracting have been completed successfully.
However, at the generating train split stage, an error from wiki_dpr.py keeps popping up.

Especially in "_generate_examples" :
1. The following error occurs in the line **id, text, title = line.strip().split("\t")**
ValueError: not enough values ​​to unpack (expected 3, got 2)
-> This part handles exceptions so that even if an error occurs, it passes.

2. **ID mismatch between lines {id} and vector {vec_id}**
This error seems to occur at the line " assert int(id) == int(vec_id),".
After I handled the exception in the split error, generating train split progressed to 80%, but an id mismatch error occurred at about the 16200000th vector id.

Debugging is even more difficult because it takes a long time to download and split wiki_dpr. I need help. thank you in advance!!

### Steps to reproduce the bug

Occurs in the generating train split step when running the rag example in the transformers repository.
Specifically, it is an error in wiki_dpr.py.

### Expected behavior

.

### Environment info

python 3.8

Thanks for reporting, @kiehls90.

As this seems an issue with the specific "wiki_dpr" dataset, I am transferring the issue to the corresponding dataset page: https://huggingface.co/datasets/wiki_dpr/discussions/13
While importing load_dataset from datasets 

### Describe the bug

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' this is the error i received

### Steps to reproduce the bug

from datasets import load_dataset

### Expected behavior

No errors

### Environment info

python 3.11.5

Hi! Can you please share the error's stack trace so we can see where it comes from?

We cannot reproduce the issue and we do not have enough information: environment info (need to run `datasets-cli env`), stack trace,...

I am closing the issue. Feel free to reopen it (with additional information) if the problem persists.

Yeah 👍

On Tue, 6 Feb 2024 at 2:56 PM, Albert Villanova del Moral <
***@***.***> wrote:

> We cannot reproduce the issue and we do not have enough information:
> environment info (need to run datasets-cli env), stack trace,...
>
> I am closing the issue. Feel free to reopen it (with additional
> information) if the problem persists.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6618#issuecomment-1929102334>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ASS4PJ3XOIIWISPY3VX3QRTYSHZK5AVCNFSM6AAAAABCL3BT4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRZGEYDEMZTGQ>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Please downgrade the version of urllib3 if you have the same issue:

!pip install urllib3==1.25.11

> Please downgrade the version of urllib3 if you have the same issue:
> 
> !pip install urllib3==1.25.11

This worked for me. Thanks.

I use python 3.11 and datasets==2.20.0. Downgrading urllib3 to 1.25.11 worked in my case.
`datasets/downloads` cleanup tool

### Feature request

Splitting off https://github.com/huggingface/huggingface_hub/issues/1997 - currently `huggingface-cli delete-cache` doesn't take care of cleaning `datasets` temp files

e.g. I discovered having millions of files under `datasets/downloads` cache, I had to do:

```
sudo find /data/huggingface/datasets/downloads -type f -mtime +3 -exec rm {} \+
sudo find /data/huggingface/datasets/downloads -type d -empty -delete
```
 
could the cleanup be integrated into `huggingface-cli` or a different tool provided to keep the folders tidy and not consume inodes and space 

e.g. there were tens of thousands of `.lock` files - I don't know why they never get removed - lock files should be temporary for the duration of the operation requiring the lock and not remain after the operation finished, IMHO.

Also I think one should be able to nuke `datasets/downloads` w/o hurting the cache, but I think there are some datasets that rely on files extracted under this dir - or at least they did in the past - which is very difficult to manage since one has no idea what is safe to delete and what not.

Thank you

@Wauplin (requested to be tagged)
cnn_dailymail repeats itself

### Describe the bug

When I try to load `cnn_dailymail` dataset, it takes longer than usual and when I checked the dataset it's 3x bigger than it's supposed to be.

Check https://huggingface.co/datasets/cnn_dailymail: it says 287k rows for train. But when I check length of train split it says 861339.

Also I checked data:

```
>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][287113]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][574226]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."
```

The datasets seems to be updated 6 days ago to convert it to Parquet. Probably, there is some issue with backward compatability.



### Steps to reproduce the bug

1.
```
from datasets import load_dataset
ds = load_dataset('cnn_dailymail', '3.0.0')
len(ds['train'])
```

### Expected behavior

It should not repeat itself.

### Environment info

datasets==2.13.2
Python==3.7.13

Hi ! We recently updated `cnn_dailymail` and now `datasets>=2.14` is needed to load it.

You can update `datasets` with

```
pip install -U datasets
```
cast_column to Sequence(subfeatures_dict) has err

### Describe the bug

I am working with the following demo code:
```
from datasets import load_dataset
from datasets.features import Sequence, Value, ClassLabel, Features

ais_dataset = load_dataset("/data/ryan.gao/ais_dataset_cache/raw/1978/")
ais_dataset = ais_dataset["train"]

def add_class(example):
    example["my_labeled_bbox"] = {"bbox": [100,100,200,200], "label": "cat"}
    return example

ais_dataset = ais_dataset.map(add_class, batched=False, num_proc=32)
ais_dataset = ais_dataset.cast_column("my_labeled_bbox", Sequence(
                                                        {
                                                            "bbox": Sequence(Value(dtype="int64")),
                                                            "label": ClassLabel(names=["cat", "dog"])
                                                        }))

print(ais_dataset[0])
```
However, executing this code results in an error:

```
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2111, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
int64
to
Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)
```
Upon examining the source code in datasets/table.py at line 2035:

```
        if isinstance(feature, Sequence) and isinstance(feature.feature, dict):
            feature = {
                name: Sequence(subfeature, length=feature.length) for name, subfeature in feature.feature.items()
            }
```

I noticed that if subfeature is of type Sequence, the code results in Sequence(Sequence(...), ...) and Sequence(ClassLabel(...), ...), which appears to be the source of the error.


### Steps to reproduce the bug

run my demo code

### Expected behavior

no exception

### Environment info

python 3.9 
datasets: 2.16.1

Hi! You are passing the wrong feature type to `cast_column`. This is the fixed call:
```python
ais_dataset = ais_dataset.cast_column("my_labeled_bbox", {"bbox": Sequence(Value(dtype="int64")), "label": ClassLabel(names=["cat", "dog"])})
```

> Hi! You are passing the wrong feature type to `cast_column`. This is the fixed call:
> 
> ```python
> ais_dataset = ais_dataset.cast_column("my_labeled_bbox", {"bbox": Sequence(Value(dtype="int64")), "label": ClassLabel(names=["cat", "dog"])})
> ```

thanks
Wrong path for cache directory in offline mode

### Describe the bug

Dear huggingfacers,

I'm trying to use a subset of the-stack dataset. When I run the command the first time

```
dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )
```

It downloads the files and caches them normally.

Nevertheless, since my compute nodes are not online (`HF_DATASETS_OFFLINE=1`) . Whenever I try to run the command again, the library is passing the wrong cache path:


`Cache directory for the-stack doesn't exist at /Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data%2Ffortran-data_dir=data%2Ffortran`

when the right path is:

`'/Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data\%2Ffortran`

Not sure why those redundancies are included in the path.  If I try adding the correct path through the the cache_dir argument it throws an error:

ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'bigcode/the-stack': Offline mode is enabled.


Your help with this issue is greatly appreciated. Thanks a lot for the great work.


### Steps to reproduce the bug

1: 

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


2: 

`HF_DATASETS_OFFLINE=1`

3:

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


### Expected behavior

being able to use the cached data

### Environment info

several different systems

+1

same error in 2.16.1

@kongjiellx any luck with the issue?

I opened https://github.com/huggingface/datasets/pull/6632 to fix this issue. Once it's merged we'll do a new release of `datasets`

Thanks @lhoestq !
ELI5 no longer available, but referenced in example code

Here, an example code is given:
https://huggingface.co/docs/transformers/tasks/language_modeling

This code + article references the ELI5 dataset.
ELI5 is no longer available, as the ELI5 dataset page states: https://huggingface.co/datasets/eli5
"Defunct: Dataset "eli5" is defunct and no longer accessible due to unavailability of the source data.
Reddit recently [changed the terms of access](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) to its API, making the source data for this dataset unavailable.
"

Please change the example code to use a different dataset.

Addressed in https://github.com/huggingface/transformers/pull/28715.
Transform fingerprint collisions due to setting fixed random seed

### Describe the bug

The transform fingerprinting logic relies on the `random` library for random bits when the function is not hashable (e.g. bound methods as used in `trl`: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L356). This causes collisions when the training code sets a fixed random seed, which is common practice: https://github.com/huggingface/alignment-handbook/blob/main/recipes/zephyr-7b-beta/sft/config_full.yaml#L45.

This results in fingerprint collisions which leads to silently loading incorrect cache files corresponding to completely different datasets.

### Steps to reproduce the bug

n/a

### Expected behavior

Use `uuid` v4 instead of `random.getrandbits()`

### Environment info

`datasets` main branch

I've opened a PR with a fix.

I don't think the PR fixes the root cause, since it still relies on the `random` library which will often have its seed fixed. I think the builtin `uuid.uuid4()` is a better choice: https://docs.python.org/3/library/uuid.html
datasets map `cache_file_name` does not work

### Describe the bug

In the documentation `datasets.Dataset.map` arg `cache_file_name` is said to be a string, but it doesn't work.

### Steps to reproduce the bug

1. pick a dataset
2. write a map function
3. do `ds.map(..., cache_file_name='some_filename')`
4. it crashes

### Expected behavior

It will tell you the filename you specified does not exist or it will generate a new file and tell you the filename does not exist.

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2

Unfortunately, I'm unable to reproduce this error. Can you share the reproducer?

```
ds = datasets.Dataset.from_dict(dict(a=[i for i in range(100)]))
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-fn") # this worked
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-folder/filename") # this failed
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-folder/") # this failed


FileNotFoundError: [Errno 2] No such file or directory: '/tmp/whatever-folder/tmp1_izxvoo'
```

It will fail if the filename parents do not exists. If we have `os.makedirs("/tmp/whatever-folder")`, then it worked.

Maybe add the `mkdir -p` into the map function?
Index error when data is large

### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2

I'm facing this problem while doing my translation of [mteb/stackexchange-clustering](https://huggingface.co/datasets/mteb/stackexchange-clustering). each row has lots of samples (up to 100k samples), because in this dataset, each row represent multiple clusters.
my hack is to setting `max_shard_size` to 20Gb or even larger
```py
final_dataset.push_to_hub(
        output_dataset, 
        private=True,
        max_shard_size="20GB"  # This will ensure appropriate sharding based on data size
    )
```
It will work, but depends on your data size.
Easy way to segment into 30s snippets given an m4a file and a vtt file 

### Feature request

Uploading datasets is straightforward thanks to the ability to push Audio to hub. However, it would be nice if the data (text and audio) could be segmented when being pushed (if not possible already).

### Motivation

It's easy to create a vtt file from an audio file. If there could be auto-segmenting, this would make the creation of datasets much faster.

### Your contribution

I have made a custom script to do this but it's not all that clean - uses librosa and pydub.

Hi! Non-generic data processing is out of this library's scope, so it's downstream libraries/users' responsibility to implement such logic.

That's fair. Thanks