konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Remove columns in interable dataset

### Feature request

Right now, remove_columns() produces a NotImplementedError for iterable style datasets

### Motivation

It would be great to have the same functionality irrespective of whether one is using an iterable or a map-style dataset

### Your contribution

hope and courage.

Transferring this issue as it's related to the 🤗 Datasets library 

Hi @surya-narayanan! Could you provide some code snippet?

This method has been recently added to the `IterableDataset`, so you need to update the `datasets`' installation (`pip install -U datasets`) to use it.
Can't add iterable datasets to a Dataset Dict.

### System Info

standard env

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Get the following error:

TypeError: Values in `DatasetDict` should be of type `Dataset` but got type '<class 'datasets.iterable_dataset.IterableDataset'>'


### Expected behavior

should be able to add iterable datasets to a dataset dict

Transferring as this is relating to the 🤗 Datasets library

You need to use `IterableDatasetDict` instead of `DatasetDict` for iterable datasets.
Make models/functions optimized with `torch.compile` hashable

As reported in https://github.com/huggingface/datasets/issues/5819, hashing functions/transforms that reference a model, or a function, optimized with `torch.compile` currently fails due to them not being picklable (the concrete error can be found in the linked issue).

The solutions to consider:
1. hashing/pickling the original, uncompiled version of a compiled model/function (attributes `_orig_mod`/`_torchdynamo_orig_callable`) (less precise than the 2nd option as it ignores the other params of `torch.compute`)
2. wait for https://github.com/pytorch/pytorch/issues/101107 to be resolved
Is uint8 supported?

### Describe the bug

I expect the dataset to store the data in the `uint8` data type, but it's returning `int64` instead.
While I've found that `datasets` doesn't yet support float16 (https://github.com/huggingface/datasets/issues/4981), I'm wondering if this is the case for other data types as well.
Is there a way to store vector data as `uint8` and then upload it to the hub?

### Steps to reproduce the bug

```python
from datasets import Features, Dataset, Sequence, Value
import numpy as np

dataset = Dataset.from_dict(
    {"vector": [np.array([0, 1, 2], dtype=np.uint8)]}, features=Features({"vector": Sequence(Value("uint8"))})
).with_format("numpy")

print(dataset[0]["vector"].dtype)
```

### Expected behavior

Expected: `uint8`
Actual: `int64`

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-12.1-x86_64-i386-64bit
- Python version: 3.8.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Hi ! The numpy formatting detaults to int64 and float32 - but you can use uint8 using
```python
ds = ds.with_format("numpy", dtype=np.uint8)
```

Related to https://github.com/huggingface/datasets/issues/5517.

Thank you!
By setting `ds.with_format("numpy", dtype=np.uint8)`, the dataset returns the data in `uint8`.

However, `with_format` and `set_format` seem to cast the data on-the-fly.
I want to reduce the dataset size by using `uint8` instead of `int64` and I observe no difference between using `int64` and `uint8` for the vector.
Is there any way to actually store the data in `uint8` and save the disk space and the downloading time when loaded from the hub?


If the feature type is `Value("uint8")` then it's written an uint8 on disk using the uint8 Arrow dtype.

e.g.
```python
ds = Dataset.from_dict({"a": range(10)}, features=Features({"a": Value("uint8")}))
ds.data.nbytes
# 10
```

Oh, I understand now.
The data was stored in `uint8` from the beginning (when the dataset returns `int64`).

Thank you for your time!
My question is fully resolved.
(mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

### Describe the bug

M2 MBP can't run
```python
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Steps to reproduce the bug

1. Use M2 MBP
2. Python 3.10.10 from pyenv
3. Run 
```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Expected behavior

Be able to run normally

### Environment info

```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```
OSX: 13.2
CPU: M2


Can you paste the error stack trace?

That is weird. I can't reproduce it again after reboot.
```python
In [2]: import platform

In [3]: platform.platform()
Out[3]: 'macOS-13.2-arm64-arm-64bit'

In [4]: from datasets import load_dataset
   ...:
   ...: jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
Found cached dataset parquet (/Users/sarit/.cache/huggingface/datasets/nomic-ai___parquet/nomic-ai--gpt4all-j-prompt-generations-a3b62015e2e52043/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 63.25it/s]
```
Stream data concatenation issue

### Describe the bug

I am not able to concatenate the augmentation of the stream data. I am using the latest version of dataset.

ValueError: The features can't be aligned because the key audio of features {'audio_id': Value(dtype='string', 
id=None), 'audio': {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'path': 
Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)}, 'transcript': Value(dtype='string', 
id=None)} has unexpected type - {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 
'path': Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)} (expected either 
Audio(sampling_rate=16000, mono=True, decode=True, id=None) or Value("null").

### Steps to reproduce the bug

dataset = load_dataset("tobiolatunji/afrispeech-200", "all", streaming=True).shuffle(seed=42)
dataset_cln = dataset.remove_columns(['speaker_id', 'path', 'age_group', 'gender', 'accent', 'domain', 'country', 'duration'])
dataset_cln = dataset_cln.cast_column("audio", Audio(sampling_rate=16000))
from audiomentations import AddGaussianNoise,Compose,Gain,OneOf,PitchShift,PolarityInversion,TimeStretch

augmentation = Compose([
    AddGaussianNoise(min_amplitude=0.005, max_amplitude=0.015, p=0.2)
])

def augment_dataset(batch):
    audio = batch["audio"]
    audio["array"] = augmentation(audio["array"], sample_rate=audio["sampling_rate"])    
    return batch

augmented_dataset_cln = dataset_cln['train'].map(augment_dataset)
dataset_cln['train'] = interleave_datasets([dataset_cln['train'], augmented_dataset_cln])
dataset_cln['train'] = dataset_cln['train'].shuffle(seed=42)

### Expected behavior

I should be able to merge as sampling rate is same.

### Environment info

import datasets
import transformers
import accelerate
print(datasets.__version__)
print(transformers.__version__)
print(torch.__version__)
print(evaluate.__version__)
print(accelerate.__version__)

2.12.0
4.28.1
2.0.0
0.4.0
0.18.0

Hi! 

You can call `map` as follows to avoid the error:
```python
augmented_dataset_cln = dataset_cln['train'].map(augment_dataset, features=dataset_cln['train'].features)
```

Thanks it is solved

Hi! 
I have run into the same problem with you. Could you please let me know how you solve it? Thanks!
load json dataset interrupt when dtype cast problem occured

### Describe the bug

i have a json like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}, 
    ....
]
，which have several problematic rows data like row 2, then i load it with datasets.load_dataset('json', data_files=['xx.json'], split='train'), it will report like this:
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file 'C:\Users\gawinjunwu\Downloads\test\data\a.json' with error <class 'pyarrow.lib.ArrowInvalid'>: Could not convert '2' with type str: tried to convert to int64
Traceback (most recent call last):
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1858, in _prepare_split_single
    for _, table in generator:
  File "D:\Python3.9\lib\site-packages\datasets\packaged_modules\json\json.py", line 146, in _generate_tables
    raise ValueError(f"Not able to read records in the JSON file at {file}.") from None
ValueError: Not able to read records in the JSON file at C:\Users\gawinjunwu\Downloads\test\data\a.json.     

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\gawinjunwu\Downloads\test\scripts\a.py", line 4, in <module>
    ds = load_dataset('json', data_dir='data', split='train')
  File "D:\Python3.9\lib\site-packages\datasets\load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset.
Could datasets skip those problematic data row? 

### Steps to reproduce the bug

prepare a json file like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}
]
then use datasets.load_dataset('json', dir_files=['xxx.json']) to load the json file

### Expected behavior

skip the problematic data row and load row1 and row3

### Environment info

python3.9

Indeed the JSON dataset builder raises an error when it encounters an unexpected type.

There's an old PR open to add away to ignore such elements though, if it can help: https://github.com/huggingface/datasets/pull/2838
[2.12.0] DatasetDict.save_to_disk not saving to S3

### Describe the bug

When trying to save a `DatasetDict` to a private S3 bucket using `save_to_disk`, the artifacts are instead saved locally, and not in the S3 bucket.

I have tried using the deprecated `fs` as well as the `storage_options` arguments and I get the same results.

### Steps to reproduce the bug

1. Create a DatsetDict `dataset`
2. Create a S3FileSystem object
`s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)`
3. Save using `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", storage_options=s3.storage_options)` or `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", fs=s3)`
4. Check the corresponding S3 bucket and verify nothing has been uploaded
5. Check the path at f"{s3_bucket}/{s3_dir}/{dataset_name}" and verify that files have been saved there

### Expected behavior

Artifacts are uploaded at the f"{s3_bucket}/{s3_dir}/{dataset_name}" S3 location.

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-x86_64-i386-64bit
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

Hi ! Can you try adding the `s3://` prefix ?
```python
f"s3://{s3_bucket}/{s3_dir}/{dataset_name}"
```

Ugh, yeah that was it. Thank you!

Hi @thejamesmarq, by any chance, did you use multiprocessing `num_proc > 1` when saving your dataset on the s3 bucket ? I'm struggling making it work in a multiprocessing setting while everything works fine with one processor.
Incomplete docstring for `BuilderConfig`

Hi guys !
I stumbled upon this docstring while working on a project.
Some of the attributes have missing descriptions.

https://github.com/huggingface/datasets/blob/bc5fef5b6d91f009e4101684adcb374df2c170f6/src/datasets/builder.py#L104-L117

Thanks for reporting! You are more than welcome to improve `BuilderConfig`'s docstring.

This class serves an identical purpose as `tensorflow_datasets`'s `BuilderConfig`, and its docstring is [here](https://github.com/tensorflow/datasets/blob/a95e38b5bb018312c3d3720619c2a8ef83ebf57f/tensorflow_datasets/core/dataset_builder.py#L81), so feel free to re-use parts of it.
Ability to update a dataset

### Feature request

The ability to load a dataset, add or change something, and save it back to disk.

Maybe it's possible, but I can't work out how to do it, e.g. this fails:

```py
import datasets

dataset = datasets.load_from_disk("data/test1")
dataset = dataset.add_item({"text": "A new item"})
dataset.save_to_disk("data/test1")
```

With the error:
```
PermissionError: Tried to overwrite /mnt/c/Users/david/py/learning/mini_projects/data_sorting_and_filtering/data/test1 but a dataset can't overwrite itself.
```


### Motivation

My use case is that I want to process a dataset in a particular way but it doesn't fit in memory if I do it in one go. So I want to perform a loop and at each step in the loop, process one shard and append it to an ever-growing dataset. The code in the loop will load a dataset, add some rows, then save it again.

Maybe I'm just thinking about things incorrectly and there's a better approach. FWIW I can't use `dataset.map()` to do the task because that doesn't work with `num_proc` when adding rows, so is confined to a single process which is too slow.

The only other way I can think of is to create a new file each time, but surely that's not how people do this sort of thing.

### Your contribution

na

This [reply](https://discuss.huggingface.co/t/how-do-i-add-things-rows-to-an-already-saved-dataset/27423) from @mariosasko on the forums may be useful :)

In this case, I think we can avoid the `PermissionError` by unpacking the underlying `ConcatenationTable` and saving only the newly added data blocks (in new files).

Thanks @stevhliu and @mariosasko , so saving to individual files then loading them later, concatenating again and saving again is the recommended way. Good to know.

Question that I hope doesn't sound rude: is this sort of thing (processing a dataset that doesn't fit in memory) outside of `datasets`'s core area of focus? Are there other tools you would recommend to do this sort of thing that play nice with `datasets`? Or is it just that I've found myself in a niche situation that hasn't specifically been catered for?
Easy way to create a Kaggle dataset from a Huggingface dataset?

I'm not sure whether this is more appropriately addressed with HuggingFace or Kaggle.  I would like to somehow directly create a Kaggle dataset from a HuggingFace Dataset.

While Kaggle does provide the option to create a dataset from a URI, that URI must point to a single file.  For example:

![image](https://user-images.githubusercontent.com/5355286/235792394-7c559d07-4aff-45b7-ad2b-9c5280c88415.png)


Is there some mechanism from huggingface to represent a dataset (such as that from `load_dataset('wmt14', 'de-en', split='train')` as a single file?  Or, some other way to get that into a Kaggle dataset so that I can use the huggingface `datasets` module to process and consume it inside of a Kaggle notebook?

Thanks in advance!


Hi @hrbigelow , I'm no expert for such a question so I'll ping @lhoestq from the `datasets` library (also this issue could be moved there if someone with permission can do it :) )

Hi ! Many datasets are made of several files, and how they are parsed often requires a python script. Because of that, datasets like wmt14 are not available as a single file on HF. Though you can create this file using `datasets`:

```python
from datasets import load_dataset

ds = load_dataset("wmt14", "de-en", split="train")

ds.to_json("wmt14-train.json")
# OR to parquet, which is compressed:
# ds.to_parquet("wmt14-train.parquet")
```

We are also working on providing parquet exports for all datasets, but wmt14 is not supported yet (we're rolling it out for datasets <1GB first). They're usually available in the `refs/convert/parquet` branch (empty for wmt14):

<img width="267" alt="image" src="https://user-images.githubusercontent.com/42851186/235878909-7339f5a4-be19-4ada-85d8-8a50d23acf35.png">


also cc @nateraw for visibility on this (and cc @osanseviero too)

I've requested support for creating a Kaggle dataset from an imported HF dataset repo on their "forum" here: https://www.kaggle.com/discussions/product-feedback/427142 (upvotes appreciated 🙂)
Cannot shuffle interleaved IterableDataset with "all_exhausted" stopping strategy

### Describe the bug

Shuffling interleaved `IterableDataset` with "all_exhausted" strategy yields non-exhaustive sampling.

### Steps to reproduce the bug

```py
from datasets import IterableDataset, interleave_datasets

def gen(bias, length):
  for i in range(length):
    yield dict(a=bias+i)

seed = 42
probabilities = [0.2, 0.6, 0.2]
d1 = IterableDataset.from_generator(lambda: gen(0, 3))
d2 = IterableDataset.from_generator(lambda: gen(10, 4))
d3 = IterableDataset.from_generator(lambda: gen(20, 3))
ds = interleave_datasets([d1, d2, d3], probabilities=probabilities, seed=seed, stopping_strategy='all_exhausted')
ds = ds.shuffle(buffer_size=1000)
for x in ds:
  print(x)
```

This code produces 
```
{'a': 0}
{'a': 22}
{'a': 20}
{'a': 21}
{'a': 10}
{'a': 1}
```

### Expected behavior

It should produce a longer list of examples to exhaust all the datasets.
If you comment out the shuffle line, it will exhaust all the datasets properly.
Here is the output if you comment out shuffling:
```
{'a': 10}
{'a': 11}
{'a': 20}
{'a': 12}
{'a': 0}
{'a': 21}
{'a': 13}
{'a': 10}
{'a': 1}
{'a': 11}
{'a': 12}
{'a': 22}
{'a': 13}
{'a': 20}
{'a': 10}
{'a': 11}
{'a': 12}
{'a': 2}
```

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

This was run on Google Colab.
load_dataset: TypeError: 'NoneType' object is not callable, on local dataset filename changes

### Describe the bug

I've adapted Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py) to train using a local dataset, which has been working. Upon changing the filenames of the `.json` & `.py` files in my local dataset directory, `dataset = load_dataset(path_or_dataset)["train"]` throws the error:

```python
2023-04-30 09:10:52 INFO [training.trainer] Loading dataset from dushowxa-characters
Traceback (most recent call last):
  File "/data/dushowxa-dolly/train_dushowxa.py", line 26, in <module>
    load_training_dataset()
  File "/data/dushowxa-dolly/training/trainer.py", line 89, in load_training_dataset
    dataset = load_dataset(path_or_dataset)["train"]
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1773, in load_dataset
    builder_instance = load_dataset_builder(
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1528, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
```
The local dataset filenames were of the form `dushowxa-characters/expanse-dushowxa-characters.json` and are now of the form `dushowxa-characters/dushowxa-characters.json` (the word `expanse-` was removed from the filenames). Is this perhaps a dataset caching issue?

I have attempted to manually clear caches, but to no effect:

```sh
rm -rfv ~/.cache/huggingface/datasets/*
rm -rfv ~/.cache/huggingface/modules/* 
```

### Steps to reproduce the bug

Run `python3 train_dushowxa.py` (adapted from Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py)).

### Expected behavior

Training succeeds as before local dataset filenames were changed.

### Environment info

Ubuntu 22.04, Python 3.10.6, venv
```python
accelerate>=0.16.0,<1
click>=8.0.4,<9
datasets>=2.10.0,<3
deepspeed>=0.9.0,<1
transformers[torch]>=4.28.1,<5
langchain>=0.0.139
```

This error means a `DatasetBuilder` subclass that generates the dataset could not be found inside the script, so make sure `dushowxa-characters/dushowxa-characters.py `is a valid dataset script (assuming `path_or_dataset` is `dushowxa-characters`)

Also, we should improve the error to make it more obvious what the problem is.

from datasets import load_dataset
lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

or  
configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True)

**both  error:**
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 2131, in load_dataset
    builder_instance = load_dataset_builder(
  File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 1888, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
wiki_dpr details for Open Domain Question Answering tasks

Hey guys!

Thanks for creating the wiki_dpr dataset!

I am currently trying to combine wiki_dpr and my own datasets. but I don't know how to make the embedding value the same way as wiki_dpr.

As an experiment, I embeds the text of id="7" of wiki_dpr, but this result was very different from wiki_dpr.

Hi ! I don't remember exactly how it was done, but maybe you have to embed `f"{title}<sep>{text}"` ?

Using a HF tokenizer it corresponds to doing
```python
tokenized = tokenizer(titles, texts)
```
Support parallelized downloading in load_dataset with Spark

As proposed in https://github.com/huggingface/datasets/issues/5798, this adds support to parallelized downloading in `load_dataset` with Spark, which can speed up the process by distributing the workload to worker nodes.

Parallelizing dataset processing is not supported in this PR.

Hi @lhoestq or other maintainers, this is ready for review, could you please take a look?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5807). All of your documentation changes will be reflected on that endpoint.

Per the discussion in #5798, will implement with `joblibspark` instead.
Improve `Create a dataset` tutorial 

Our [tutorial on how to create a dataset](https://huggingface.co/docs/datasets/create_dataset) is a bit misleading. 
1. In **Folder-based builders** section it says that we have two folder-based builders as standard builders, but we also have similar builders (that can be created from directory with data of required format) for `csv`, `json/jsonl`, `parquet` and `txt` files. We have info about these loaders in separate [guide for loading](https://huggingface.co/docs/datasets/loading#local-and-remote-files) but it's worth briefly mentioning them in the beginning tutorial because they are more common and for consistency. Would be helpful to add the link to the full guide.
2. **From local files** section lists methods for creating a dataset from in-memory data which are also described in [loading guide](https://huggingface.co/docs/datasets/loading#inmemory-data).  

Maybe we should actually rethink and restructure this tutorial somehow.

I can work on this. The link to the tutorial seems to be broken though @polinaeterna. 

@isunitha98selvan would be great, thank you! which link are you talking about? I think it should work: https://huggingface.co/docs/datasets/create_dataset

Hey I don't mind working on this issue. From my understanding, we want to let the reader know that they can build datasets from `csv`, `json/jsonl`, `parquet` and `txt` files in the **folder-based builders** section and include a link to the full guide. Then in the **from local files** section, we just want to list the methods from in-memory data section such as `.from_dict()`. 

Hey @polinaeterna, I have a pull request for this issue. Can you review and see if it needs any changes?
Files downloaded to cache do not respect umask

As reported by @stas00, files downloaded to the cache do not respect umask:
```bash
$ ls -l /path/to/cache/datasets/downloads/
-rw------- 1 uername    username    150M Apr 25 16:41 5e646c1d600f065adaeb134e536f6f2f296a6d804bd1f0e1fdcd20ee28c185c6 
```

Related to:
- #2065
load_dataset is case sentitive?

### Describe the bug

load_dataset() function is case sensitive?

### Steps to reproduce the bug

The following two code, get totally different behavior.

1. load_dataset('mbzuai/bactrian-x','en')

2. load_dataset('MBZUAI/Bactrian-X','en')

### Expected behavior

Compare 1 and 2.
1 will download all 52 subsets, shell output:
```Downloading and preparing dataset json/MBZUAI--bactrian-X to xxx```
2 will only download single subset, shell output
```Downloading and preparing dataset bactrian-x/en to xxx```


### Environment info

Python 3.10.11
datasets Version: 2.11.0

Hi @haonan-li , thank you for the report! It seems to be a bug on the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) site, there is even no such dataset as `mbzuai/bactrian-x` on the Hub.  I opened and [issue](https://github.com/huggingface/huggingface_hub/issues/1453) there.

I think `load_dataset("mbzuai/bactrian-x")` shouldn't be loaded at all and raise an error but because of [this fallback](https://github.com/huggingface/datasets/blob/main/src/datasets/load.py#L1194) to packaged loaders when no other options are applicable, it loads the dataset with standard `json` loader instead of the custom loading script.
CI ZeroDivisionError

Sometimes when running our CI on Windows, we get a ZeroDivisionError:
```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_frugalscore - ZeroDivisionError: float division by zero
```

See for example:
- https://github.com/huggingface/datasets/actions/runs/4809358266/jobs/8560513110
- https://github.com/huggingface/datasets/actions/runs/4798359836/jobs/8536573688

```
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

split = 'test', start_time = 1682516718.8236516, num_samples = 2, num_steps = 1

    def speed_metrics(split, start_time, num_samples=None, num_steps=None):
        """
        Measure and return speed performance metrics.
    
        This function requires a time snapshot `start_time` before the operation to be measured starts and this function
        should be run immediately after the operation to be measured has completed.
    
        Args:
        - split: name to prefix metric (like train, eval, test...)
        - start_time: operation start time
        - num_samples: number of samples processed
        """
        runtime = time.time() - start_time
        result = {f"{split}_runtime": round(runtime, 4)}
        if num_samples is not None:
>           samples_per_second = num_samples / runtime
E           ZeroDivisionError: float division by zero

C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\transformers\trainer_utils.py:354: ZeroDivisionError
```

Hello!
This issue seems to have been fixed in https://github.com/huggingface/transformers/pull/24049 
I was looking for my first issue to work on when I noticed this; not sure if there is a specific protocol for suggesting to close an issue.

Thanks for informing, @zeppdev. I am closing this issue.

Fixed by:
- huggingface/transformers#24049
IterableDataset.with_format("torch") not working

### Describe the bug

After calling the with_format("torch") method on an IterableDataset instance, the data format is unchanged.

### Steps to reproduce the bug

```python
from datasets import IterableDataset
def gen():
    for i in range(4):
        yield {"a": [i] * 4}
dataset = IterableDataset.from_generator(gen).with_format("torch")
next(iter(dataset))
```

### Expected behavior

`{"a": torch.tensor([0, 0, 0, 0])}` is expected, but `{"a": [0, 0, 0, 0]}` is observed.

### Environment info

```bash
platform==ubuntu 22.04.01
python==3.10.9
datasets==2.11.0
```

Hi ! Thanks for reporting, I'm working on it ;)
TIFF/TIF support 

### Feature request

I currently have a dataset (with tiff and json files) where I have to do this:

`wget path_to_data/images.zip && unzip images.zip`

`wget path_to_data/annotations.zip && unzip annotations.zip`
 
 Would it make sense a contribution that supports these type of files? 

### Motivation

instead of using `load_dataset` have to use wget as these files are not supported for annotations with JSON and images with TIFF files. 


Additionally to this, the PIL formatting from datasets does not read correctly the image channels with TIFF format, besides multichannel adaptation might be necessary as well (as my data e.g has more than 3 channels)

### Your contribution

1. Support TIFF images over multi channel format
2. Support JSON annotations

The issue with multichannel TIFF images has already been reported in Pillow (https://github.com/python-pillow/Pillow/issues/1888). We can't do much about it on our side.

Still, to avoid the error, you can bypass the default Pillow decoding and define a custom one as follows:
```python
import tifffile # pip install tifffile

dset = dset.cast_column("image", datasets.Image(decode=False))

def decode_mutlichannel_tiff(batch):
    batch["image"] = [tifffile.imread(image["path"]) for image in batch["image"]]
    return batch

dset.set_transform(decode_mutlichannel_tiff)
```

Regarding the annotations, in which format are they? In the COCO format? I think this is a bit too specific to have a built-in loader for it.

This snippet is awesome! I know I probably should have gotten deeper in to the docs to find cast_column and set_transform, but perhaps a link ushering folks to that documentation or even this thread somewhere in https://huggingface.co/docs/datasets/image_load would be helpful? Thanks again for the snippet

We have a section on custom decoding [here](https://huggingface.co/docs/datasets/process#format-transform) (for the audio case though)

Btw, we can close this issue as it should be addressed in Pillow rather than here. 

For sure, if image based stuff becomes a priority I think guiding folks to an image decoder section would be really helpful, but im just one dev :) and I know priorities gotta be balanced so no worries. Thanks again for the snippet, agreed we can close
Offset overflow while doing regex on a text column

### Describe the bug

`ArrowInvalid: offset overflow while concatenating arrays`

Same error as [here](https://github.com/huggingface/datasets/issues/615)

### Steps to reproduce the bug


Steps to reproduce: (dataset is a few GB big so try in colab maybe)

```
import datasets
import re

ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')

def get_text_caption(example):
    regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
    example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
    return example

ds = ds.map(get_text_caption)
```

I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.

### Expected behavior

Dataset should have a new column with processed text

### Environment info

Datasets version - 2.11.0

Hi! This looks like an Arrow bug, but it can be avoided by reducing the `writer_batch_size`.

(`ds = ds.map(get_text_caption, writer_batch_size=100)` in Colab runs without issues)


@mariosasko I ran into this problem with load_dataset. What should I do

@AisingioroHao0  You can also pass the `writer_batch_size` parameter to `load_dataset`, e.g., `load_dataset("mnist", writer_batch_size=100)`

@mariosasko How do I determine the optimal size of write_batch_size? My training is sometimes fast and sometimes slow. Is it because write_batch_size is too small? Each batch of the current dataloader should be the same size. I preprocessed the dataset using map

@aihao2000 It's unlikely `writer_batch_size` is the problem. You can use the following code to profile the training loop (e.g., on a subset of data) and find slow parts:
```python
import cProfile, pstats

with cProfile.Profile() as profiler:
    ... # training loop code

stats = pstats.Stats(profiler).sort_stats("cumtime")
stats.print_stats()
```


@nishanthcgit ok，thanks.Recently I found dataset.with_transform to be faster and more stable with multiple processes

@mariosasko Is the larger the num_proc of load_dataset within the number of cpu cores, the better? Then the num_proc of data_loader is the number of cpu cores/number of training processes
Error using `load_datasets`

### Describe the bug

I tried to load a dataset using the `datasets` library in a conda jupyter notebook and got the below error. 

```
ImportError: dlopen(/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib
  Referenced from: <65B094A2-59D7-31AC-A966-4DB9E11D2A15> /Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so
  Reason: tried: '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file, not in dyld cache)
```

### Steps to reproduce the bug

Run the `load_datasets` function

### Expected behavior

I expected the dataset to be loaded into my notebook.

### Environment info

name: review_sense
channels:
  - apple
  - conda-forge
dependencies:
  - python=3.8
  - pip>=19.0
  - jupyter
  - tensorflow-deps
  #- scikit-learn
  #- scipy
  - pandas
  - pandas-datareader
  - matplotlib
  - pillow
  - tqdm
  - requests
  - h5py
  - pyyaml
  - flask
  - boto3
  - ipykernel
  - seaborn
  - pip:
      - tensorflow-macos==2.9
      - tensorflow-metal==0.5.0
      - bayesian-optimization
      - gym
      - kaggle
      - huggingface_hub
      - datasets
      - numpy
      - huggingface


It looks like an issue with your installation of scipy, can you try reinstalling it ?

Sorry for the late reply, but that worked @lhoestq . Thanks for the assist.
Schrödinger's dataset_dict

### Describe the bug

If you use load_dataset('json', data_files="path/test.json"), it will return DatasetDict({train:...}).

And if you use load_dataset("path"), it will return DatasetDict({test:...}).

Why can't the output behavior be unified?

### Steps to reproduce the bug

as description above.

### Expected behavior

consistent predictable output.

### Environment info

'2.11.0'

Hi ! Passing `data_files="path/test.json"` is equivalent to `data_files={"train": ["path/test.json"]}`, that's why you end up with a train split. If you don't pass `data_files=`, then split names are inferred from the data files names
Use Pandas' `read_json` in the JSON builder

Instead of PyArrow's `read_json`, we should use `pd.read_json` in the JSON builder for consistency with the CSV and SQL builders (e.g., to address https://github.com/huggingface/datasets/issues/5725).

In Pandas2.0, to get the same performance, we can set the `engine` to "pyarrow". The issue is that Colab still doesn't install Pandas 2.0 by default, so I think it's best to wait for this to be resolved on their side to avoid downgrading decoding performance in scenarios when Pandas 2.0 is not installed.
ArrowDataset.save_to_disk lost some logic of remote

### Describe the bug

https://github.com/huggingface/datasets/blob/e7ce0ac60c7efc10886471932854903a7c19f172/src/datasets/arrow_dataset.py#L1371

Here is the bug point, when I want to save from a `DatasetDict` class and the items of the instance is like `[('train', Dataset({features: ..., num_rows: ...}))]` , there is no guarantee that there exists a directory name `train` under `dataset_dict_path`.

### Steps to reproduce the bug

1. Mock a DatasetDict with items like what I said.
2. using save_to_disk with storage_options, u can use local sftp. code may like below

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset.save_to_disk('sftp:///tmp', storage_options={'host': 'localhost', 'username': 'admin'})
```
I suppose u can reproduce the bug by these steps. 

### Expected behavior

Should create the folder if it does not exists, just like we do locally.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-6.2.10-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.13.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

We just fixed this on `main` and will do a new release soon :)
train_dataset does not implement __len__

when train using data  precessored by the datasets, I get follow warning and it leads to that I can not set epoch numbers:
`ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.`

Thanks for reporting, @v-yunbin.

Could you please give more details, the steps to reproduce the bug, the complete error back trace and the environment information (`datasets-cli env`)?

this is a detail error info from transformers：
```
Traceback (most recent call last):
  File "finetune.py", line 177, in <module>
    whisper_finetune(traindir,devdir,outdir)
  File "finetune.py", line 161, in whisper_finetune
    trainer = Seq2SeqTrainer(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 56, in __init__
    super().__init__(
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/transformers/trainer.py", line 567, in __init__
    raise ValueError(
ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.
```


How did you create `train_dataset`? The `datasets` library does not appear in your stack trace.

We need more information in order to reproduce the issue...

```
def asr_dataset(traindir,devdir):
    we_voice = IterableDatasetDict()
    #we_voice["train"] = load_from_disk(traindir,streaming=True)
    #we_voice["test"]= load_from_disk(devdir,streaming=True)
    we_voice["train"] = load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train",streaming=True)
    #print(load_dataset("csv",data_files=os.path.join(traindir,"train.csv"),split="train"))
    we_voice["test"] = load_dataset("csv",data_files=os.path.join(devdir,"dev.csv"), split="train",streaming=True)
    we_voice = we_voice.remove_columns(["id"])
    we_voice = we_voice.cast_column("audio", Audio(sampling_rate=16000))
    return we_voice

```

As you are using iterable datasets (`streaming=True`), their length is not defined.

You should:
- Either use non-iterable datasets, which have a defined length: use `DatasetDict` and not passing `streaming=True`
- Or pass `args.max_steps` to the `Trainer`

I don't know how to give a reasonable args.max_steps...........................

Then you should not use streaming.

@albertvillanova I think @v-yunbin, myself, and others might be slightly confused about max_steps and how it differs from num_train_epochs.

@lkurlandski A **step** is referring to optimizer's update after back propagation, and it's associated with a batch of data. For example, if a dataset contains 64 examples and you have an overall batch size of 4, then an epoch will have 64/4=16 batches. Therefore, in one epoch, you will have 16 optimizer **steps**.
Tiktoken tokenizers are not pickable 

### Describe the bug

Since tiktoken tokenizer is not pickable, it is not possible to use it inside `dataset.map()` with multiprocessing enabled. However, you [made](https://github.com/huggingface/datasets/issues/5536) tiktoken's tokenizers pickable in `datasets==2.10.0` for caching. For some reason, this logic does not work in dataset processing and raises `TypeError: cannot pickle 'builtins.CoreBPE' object`


### Steps to reproduce the bug

```
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
    num_proc=2,
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

starts processing dataset 

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-1021-oracle-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.0

Thanks for reporting, @markovalexander.

Unfortunately, I'm not able to reproduce the issue: the `tiktoken` tokenizer can be used within `Dataset.map`, both in my local machine and in a Colab notebook: https://colab.research.google.com/drive/1DhJroZgk0sNFJ2Mrz-jYgrmh9jblXaCG?usp=sharing

Are you sure you are using `datasets` version 2.11.0?
How to use Distill-BERT with different datasets?

### Describe the bug

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Steps to reproduce the bug

I recently read [this](https://huggingface.co/docs/transformers/quicktour#train-with-tensorflow:~:text=The%20most%20important%20thing%20to%20remember%20is%20you%20need%20to%20instantiate%20a%20tokenizer%20with%20the%20same%20model%20name%20to%20ensure%20you%E2%80%99re%20using%20the%20same%20tokenization%20rules%20a%20model%20was%20pretrained%20with.) and was wondering how to use distill-BERT (which is pre-trained with imdb dataset) with a different dataset (for eg. [this](https://huggingface.co/datasets/yhavinga/imdb_dutch) dataset)?

### Expected behavior

Distill-BERT should work with different datasets.

### Environment info

- `datasets` version: 1.12.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 11.0.0

Closing this one in favor of the same issue opened in the `transformers` repo.
Not able to load the pile

### Describe the bug

Got this error when I am trying to load the pile dataset 

```
TypeError: Couldn't cast array of type
struct<file: string, id: string>
to
{'id': Value(dtype='string', id=None)}

```


### Steps to reproduce the bug

Please visit the following sample notebook 

https://colab.research.google.com/drive/1JHcjawcHL6QHhi5VcqYd07W2QCEj2nWK#scrollTo=ulJP3eJCI-tB

### Expected behavior

The pile should work

### Environment info

-  `datasets` version: 2.11.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Thanks for reporting, @surya-narayanan.

I see you already started a discussion about this on the Community tab of the corresponding dataset: https://huggingface.co/datasets/EleutherAI/the_pile/discussions/10
Let's continue the discussion there!
Can I load in list of list of dict format? 

### Feature request

my jsonl dataset has following format:
```
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
[{'input':xxx, 'output':xxx},{'input:xxx,'output':xxx},...]
```

I try to use `datasets.load_dataset('json', data_files=path)` or `datasets.Dataset.from_json`, it raises
```
  File "site-packages/datasets/arrow_dataset.py", line 1078, in from_json
    ).read()
  File "site-packages/datasets/io/json.py", line 59, in read
    self.builder.download_and_prepare(
  File "site-packages/datasets/builder.py", line 872, in download_and_prepare
    self._download_and_prepare(
  File "site-packages/datasets/builder.py", line 967, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "site-packages/datasets/builder.py", line 1749, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "site-packages/datasets/builder.py", line 1892, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Motivation

I wanna use features like `Datasets.map` or `Datasets.shuffle`, so i need the dataset in memory to be `arrow_dataset.Datasets` format

### Your contribution

PR

Thanks for reporting, @LZY-the-boys.

Could you please give more details about what is your intended dataset structure? What are the names of the columns and the value of each row?

Currently, the JSON-Lines format is supported:
- Each line correspond to one row of the dataset
- Each line is composed of one JSON object, where the names are the names of the columns, and the values are the values for the row-column pair.