NyssaW/github-issues huggingface dataset:
`map` with `num_proc` > 1 leads to OOM

### Describe the bug

When running `map` on parquet dataset loaded from local machine, the RAM usage increases linearly eventually leading to OOM. I was wondering if I should I save the `cache_file` after every n steps in order to prevent this?

### Steps to reproduce the bug



```
ds = load_dataset("parquet", data_files=dataset_path, split="train")
ds = ds.shard(num_shards=4, index=0)

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
ds = ds.map(prepare_dataset,
            num_proc=32,
            writer_batch_size=1000,
            keep_in_memory=False,
            desc="preprocess dataset")
```

```
def prepare_dataset(batch):
    # load audio
    sample = batch["audio"]

    inputs = feature_extractor(sample["array"], sampling_rate=16000)
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(sample["array"].squeeze())

    return batch
```

### Expected behavior

It shouldn't run into OOM problem.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0

Comments:
Hi ! You can try to reduce `writer_batch_size`. It corresponds to the number of samples that stay in RAM before being flushed to disk
Allow deleting a subset/config from a no-script dataset

As proposed by @BramVanroy, it would be neat to have this functionality through the API.

Comments:
Probably best to implement this as a CLI command?

Thanks for your comment, @mariosasko. Or maybe both (in Python and as CLI command)? The Python command would be just the reverse of `push_to_hub`...

I am working on a draft implementation, so we can discuss about the API and UX.

2nd time I've come here in the past few days so for reference this doesn't work for me at the moment (token auth error despite successful `whoami` and all auths applied) so I resorted to [reverting](https://discuss.huggingface.co/t/how-to-revert-to-an-earlier-commit-on-a-repo/65735/2) to a previous commit (after installing git lfs)

```
# git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/USER/REPO
cd REPO
git reset --hard HASH
git push --force
```
Make convert_to_parquet CLI command create script branch

As proposed by @severo, maybe we should add this functionality as well to the CLI command to convert a script-dataset to Parquet. See: https://github.com/huggingface/datasets/pull/6795#discussion_r1562819168
> When providing support, we sometimes suggest that users store their script in a script branch. What do you think of this alternative to deleting the files?

Comments:
got fileNotFound

### Describe the bug

When I use load_dataset to load the nyanko7/danbooru2023 data set, the cache is read in the form of a symlink. There may be a problem with the arrow_dataset initialization process and I get FileNotFoundError: [Errno 2] No such file or directory: '2945000.jpg'

### Steps to reproduce the bug

#code show as below
from datasets import load_dataset
data = load_dataset("nyanko7/danbooru2023",cache_dir=<symlink>)
data["train"][0]

### Expected behavior

I should get this result:
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=365x256 at 0x7FB730CB4070>, 'label': 0}

### Environment info

datasets==2.12.0
python==3.10.14


Comments:
Hi! I'll open a PR on the Hub to fix this, but please use the Hub's [Community tab](https://huggingface.co/datasets/nyanko7/danbooru2023/discussions) to report such issues in the future.

I've opened a [PR](https://huggingface.co/datasets/nyanko7/danbooru2023/discussions/8) in the repo, so let's continue the discussion there
`DatasetBuilder._split_generators` incomplete type annotation

### Describe the bug

The [`DatasetBuilder._split_generators`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/builder.py#L1449) function has currently the following signature:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: DownloadManager):
        ...
```

However, the `dl_manager` argument can also be of type [`StreamingDownloadManager`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/download/streaming_download_manager.py#L962), which has different functionality. For example, the `download` function doesn't download, but rather just returns the given url(s). 

I suggest changing the function signature to:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: Union[DownloadManager, StreamingDownloadManager]):
        ...
```

and also adjust the docstring accordingly.

I would like to create a Pull Request to fix this, and have the following questions:
* Are there also other options than `DownloadManager`, and `StreamingDownloadManager`?
* Should this also be changed in other functions?

### Steps to reproduce the bug

Minimal example to print the different class names:

```python
import tempfile
from datasets import load_dataset

example = b'''
from datasets import GeneratorBasedBuilder, DatasetInfo, Features, Value, SplitGenerator

class Test(GeneratorBasedBuilder):
    def _info(self):
        return DatasetInfo(features=Features({"x": Value("int64")}))
    def _split_generators(self, dl_manager):
        print(type(dl_manager))
        return [SplitGenerator('test')]
    def _generate_examples(self):
        yield 0, {'x': 42}
'''

with tempfile.NamedTemporaryFile(suffix='.py') as f:
    f.write(example)
    f.flush()
    load_dataset(f.name, streaming=False)
    load_dataset(f.name, streaming=True)
```

### Expected behavior

complete type annotations

### Environment info

/

Comments:
Good catch! Feel free to open a PR with the suggested fix :).

There is also the [`MockDownloadManager`](https://github.com/JonasLoos/datasets/blob/main/src/datasets/download/mock_download_manager.py#L33), which seems like it might get passed here too. However, to me, it doesn't really seem relevant to the users of the datasets library, so I would just ignore it. What do you think, @mariosasko?

The API (`dummy_data` CLI command ) that uses the `MockDownloadManager` has been deprecated, so ignoring it sounds good!
Fix CI test_load_dataset_distributed_with_script

Fix #6796.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6797). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Finally:
- the initial issue seems it was temporary
- there is a different issue now 

```
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport_errors_on_wrong_sha - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_with_script - AssertionError: assert 'dataset_with_script' == 'parquet'
  
  - parquet
  + dataset_with_script
```
CI is broken due to hf-internal-testing/dataset_with_script

CI is broken for test_load_dataset_distributed_with_script. See: https://github.com/huggingface/datasets/actions/runs/8614926216/job/23609378127
```
FAILED tests/test_load.py::test_load_dataset_distributed_with_script[None] - assert False
 +  where False = all(<generator object test_load_dataset_distributed_with_script.<locals>.<genexpr> at 0x7f0c741de3b0>)
FAILED tests/test_load.py::test_load_dataset_distributed_with_script[force_redownload] - assert False
 +  where False = all(<generator object test_load_dataset_distributed_with_script.<locals>.<genexpr> at 0x7f0be45f6ea0>)
```

Comments:
Finally:
- the initial issue seems it was temporary
- there is a different issue now: https://github.com/huggingface/datasets/actions/runs/8627153993/job/23646584590?pr=6797
```
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport_errors_on_wrong_sha - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_with_script - AssertionError: assert 'dataset_with_script' == 'parquet'
  
  - parquet
  + dataset_with_script
```

Maybe related to `hf-internal-testing/dataset_with_script` dataset: https://huggingface.co/datasets/hf-internal-testing/dataset_with_script

This URL: https://datasets-server.huggingface.co/parquet?dataset=hf-internal-testing/dataset_with_script
raises:
> {"error":"The dataset viewer doesn't support this dataset because it runs arbitrary python code. Please open a discussion in the discussion tab if you think this is an error and tag @lhoestq and @severo."}

Was there a recent change on the Hub enforcing this behavior?

OK, I just saw this PR:
- https://github.com/huggingface/dataset-viewer/pull/2689

Once merged and deployed, it should fix the issue.

Once the script-dataset has been allowed in the dataset-viewer, we should fix our test to make the CI pass.

I am addressing this.
Loading just one particular split is not possible for imagenet-1k

### Describe the bug

I'd expect the following code to download just the validation split but instead I get all data on my disk (train, test and validation splits)

`
from datasets import load_dataset

dataset = load_dataset("imagenet-1k", split="validation", trust_remote_code=True)
`

Is it expected to work like that? 

### Steps to reproduce the bug

1. Install the required libraries (python, datasets, huggingface_hub)
2. Login using huggingface cli
2. Run the code in the description

### Expected behavior

Just a single (validation) split should be downloaded.

### Environment info

python: 3.12.2
datasets: 2.18.0
huggingface_hub: 0.22.2

Comments:
+1

+1
`add_faiss_index` raises ValueError: not enough values to unpack (expected 2, got 1)

### Describe the bug

Calling `add_faiss_index` on a `Dataset` with a column argument raises a ValueError. The following is the trace

```python
    214 def replacement_add(self, x):
    215     """Adds vectors to the index.
    216     The index must be trained before vectors can be added to it.
    217     The vectors are implicitly numbered in sequence. When `n` vectors are
   (...)
    224         `dtype` must be float32.
    225     """
--> 227     n, d = x.shape
    228     assert d == self.d
    229     x = np.ascontiguousarray(x, dtype='float32')

ValueError: not enough values to unpack (expected 2, got 1)
```

### Steps to reproduce the bug

1. Load any dataset like `ds = datasets.load_dataset("wikimedia/wikipedia", "20231101.en")["train"]`
2. Add an FAISS index on any column `ds.add_faiss_index('title')`

### Expected behavior

The index should be created

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
- Python version: 3.9.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
- `faiss-cpu` version: 1.8.0

Comments:
I realized I was passing a string column to this instead of float. Is it possible to add a warning or error to prevent users from falsely believing there's a bug?

Hello!

I agree that we could add some safeguards around the type of `ds[column]`. At least for FAISS, we need the column to be made of embeddings as FAISS doesn't perform the embeddings itself.

I can propose a PR sometime this week.

@Dref360 thanks for the initiative!
A Question About the Map Function

### Describe the bug

Hello,

I have a question regarding the map function in the Hugging Face datasets.

The situation is as follows: when I load a jsonl file using load_dataset(..., streaming=False), and then utilize the map function to process it, I specify that the returned example should be of type Torch.tensor. However, I noticed that after applying the map function, the datatype automatically changes to List, which leads to errors in my program.

I attempted to use load_dataset(..., streaming=True), and this issue no longer occurs. I'm not entirely clear on why this happens. Could you please provide some insights into this?

### Steps to reproduce the bug

1.dataset =  load_dataset(xxx, streaming = False)
2. dataset.map(function), function will return torch.Tensor.
3. you will find the format of data in dataset is List.

### Expected behavior

I expected to receieve the format of data is torch.Tensor.

### Environment info

2.18.0

Comments:
All data is saved in the arrow format on disk.
If you return a tensor it gets converted to arrow before saving to disk when using map.

To get a tensor when you access data elements you can use `dataset.set_format("pt")`.
Note that this just changes how the data is loaded, not how it is stored.

> All data is saved in the arrow format on disk. If you return a tensor it gets converted to arrow before saving to disk when using map.
> 
> To get a tensor when you access data elements you can use `dataset.set_format("pt")`. Note that this just changes how the data is loaded, not how it is stored.

Thank you very much for your explanation, I understand what you mean now. So you're saying that when streaming=True, there's no need to convert it to the arrow format and save it to disk. But if we directly load all formats and then convert them into the arrow format after passing through the map function, it will convert torch.Tensor into a List. I see.
Remove get_inferred_type from ArrowWriter write_batch

Inferring the type seems to be unnecessary given that the pyarrow array has already been created.
Because pyarrow array creation is sometimes extremely slow this doubles the time write_batch takes.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6781). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Close in favor of #6786.
Dataset.to_csv() missing commas in columns with lists

### Describe the bug

The `to_csv()` method does not output commas in lists.  So when the Dataset is loaded back in the data structure of the column with a list is not correct.

Here's an example:

Obviously, it's not as trivial as inserting commas in the list, since its a comma-separated file.  But hopefully there's a way to export the list in a way that it'll be imported by `load_dataset()` correctly.

### Steps to reproduce the bug

Here's some code to reproduce the bug:


```python
from datasets import Dataset

ds = Dataset.from_dict(
    {
        "pokemon": ["bulbasaur", "squirtle"],
        "type": ["grass", "water"]
    }
)

def ascii_to_hex(text):
    return [ord(c) for c in text]

ds = ds.map(lambda x: {"int": ascii_to_hex(x['pokemon'])})

ds.to_csv('../output/temp.csv')
```

temp.csv then contains:
```

### Expected behavior

ACTUAL OUTPUT:
```
pokemon,type,int
bulbasaur,grass,[ 98 117 108  98  97 115  97 117 114]
squirtle,water,[115 113 117 105 114 116 108 101]
```

EXPECTED OUTPUT:
```
pokemon,type,int
bulbasaur,grass,[98, 117, 108,  98,  97, 115,  97, 117, 114]
squirtle,water,[115, 113, 117, 105, 114, 116, 108, 101]
```

or probably something more like this since it's a CSV file:
```
pokemon,type,int
bulbasaur,grass,"[98, 117, 108,  98,  97, 115,  97, 117, 114]"
squirtle,water,"[115, 113, 117, 105, 114, 116, 108, 101]"
```

### Environment info

### Package Version
Name: datasets
Version: 2.16.1

### Python
version: 3.10.12


### OS Info
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
...
UBUNTU_CODENAME=jammy

Comments:
Hello!

This is due to how pandas write numpy arrays to csv. [Source](https://stackoverflow.com/questions/54753179/to-csv-saves-np-array-as-string-instead-of-as-a-list)
To fix this, you can convert them to list yourselves.

```python
df = ds.to_pandas()
df['int'] = df['int'].apply(lambda arr: list(arr))
df.to_csv(index=False, '../output/temp.csv')
```

I think it would be good if `datasets` would do the conversion itself, but it's a breaking change and I would wait for the greenlight from someone from HF.
Generating split is very slow when Image format is PNG

### Describe the bug

When I create a dataset, it gets stuck while generating cached data. 
The image format is PNG, and it will not get stuck when the image format is jpeg.

![image](https://github.com/huggingface/datasets/assets/22740819/3b888fd8-e6d6-488f-b828-95a8f206a152)

After debugging, I know that it is because of the `pa.array` operation in [arrow_writer](https://github.com/huggingface/datasets/blob/2.13.0/src/datasets/arrow_writer.py#L553), but i don't why.

### Steps to reproduce the bug

```
from datasets import Dataset

def generator(lines):
    for line in lines:
        img = Image.open(open(line["url"], "rb"))
        # print(img.format)  # "PNG"
        yield {
            "image": img,
        }

lines = open(dataset_path, "r")
dataset = Dataset.from_generator(
    generator,
    gen_kwargs={"lines": lines}
)
```

### Expected behavior

Generating split done.

### Environment info

datasets    2.13.0

Comments:
I think this is due to the speed of reading a `png` image using pillow compared to a `jpg` image.
Notably the same is true with `tiff`, it is even faster than `jpg` in my case.
(Willing to PR) Datasets with custom python objects

### Feature request

Hi thanks for the library! I would like to have a huggingface Dataset, and one of its column is custom (non-serializable) Python objects. For example, a minimal code:

```
class MyClass:
    pass

dataset = datasets.Dataset.from_list([
    dict(a=MyClass(), b='hello'),
])
```

It gives error:

```
ArrowInvalid: Could not convert <__main__.MyClass object at 0x7a852830d050> with type MyClass: did not recognize Python value type when inferring an Arrow data type
```

I guess it is because Dataset forces to convert everything into arrow format. However, is there any ways to make the scenario work? Thanks!

### Motivation

(see above)

### Your contribution

Yes, I am happy to PR!

Cross-posted: https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050?u=fzyzcjy

EDIT: possibly related https://github.com/huggingface/datasets/issues/5766

Comments:
Compatibility issue between s3fs, fsspec, and datasets

### Describe the bug

Here is the full error stack when installing:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires fsspec[http]<=2024.2.0,>=2023.1.0, but you have fsspec 2024.3.1 which is incompatible.
Successfully installed aiobotocore-2.12.1 aioitertools-0.11.0 botocore-1.34.51 fsspec-2024.3.1 jmespath-1.0.1 s3fs-2024.3.1 urllib3-2.0.7 wrapt-1.16.0
```
When I install with pip, pip allows this error to exist while still installing s3fs, but this error breaks poetry, since poetry will refuse to install s3fs because of the dependency conflict.

Maybe I'm missing something so maybe it's not a bug but some mistake on my end? Any input would be helpful. Thanks!

### Steps to reproduce the bug

1. conda create -n tmp python=3.10 -y
2. conda activate tmp
3. pip install datasets
4. pip install s3fs

### Expected behavior

I would expect there to be no error.

### Environment info

MacOS (ARM), Python3.10, conda 23.11.0. 

Comments:
Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.

> Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.

Thanks so much! My inexperience with pip is showing 😆  🙈 

> Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.

you are awesome bro

Hey, the suggestion by @mariosasko unfortunately only address this issue via pip. The original message was about poetry and I am still facing a dependency conflict with that.

The following command complains first about `fsspec` (`... no versions of fsspec match ...`) and then I get an error.

Command:
`poetry add datasets s3fs` 

Error: 
` ... your project ... depends on both datasets (^3.1.0) and s3fs (^2024.10.0), version solving failed`

Installing first `s3fs` and then the rest of the huggingface libraries, like `datasets`, also did not help.
load_dataset can't work with symbolic links

### Feature request

Enable the `load_dataset` function to load local datasets with symbolic links. 

E.g, this dataset can be loaded:
├── example_dataset/
│   ├── data/
│   │   ├── train/
│   │   │   ├── file0
│   │   │   ├── file1
│   │   ├── dev/
│   │   │   ├── file2
│   │   │   ├── file3
│   ├── metadata.csv

while this dataset can't:
├── example_dataset_symlink/
│   ├── data/
│   │   ├── train/
│   │   │   ├── sym0 -> file0
│   │   │   ├── sym1 -> file1
│   │   ├── dev/
│   │   │   ├── sym2 -> file2
│   │   │   ├── sym3 -> file3
│   ├── metadata.csv

I have created an example dataset in order to reproduce the problem:

1. Unzip `example_dataset.zip`.
2. Run `no_symlink.sh`. Training should start without issues. 
3. Run `symlink.sh`. You will see that all four examples will be in train split, instead of having two examples in train and two examples in dev. The script won't load the correct audio files.

[example_dataset.zip](https://github.com/huggingface/datasets/files/14807053/example_dataset.zip)

### Motivation

I have a very large dataset locally. Instead of initiating training on the entire dataset, I need to start training on smaller subsets of the data. Due to the purpose of the experiments I am running, I will need to create many smaller datasets with overlapping data. Instead of copying the all the files for each subset, I would prefer copying symbolic links of the data. This way, the memory usage would not significantly increase beyond the initial dataset size.

Advantages of this approach:

- It would leave a smaller memory footprint on the hard drive
- Creating smaller datasets would be much faster

### Your contribution

I would gladly contribute, if this is something useful to the community. It seems like a simple change of code, something like `file_path = os.path.realpath(file_path)` should be added before loading the files. If anyone has insights on how to incorporate this functionality, I would greatly appreciate your knowledge and input.

Comments:
In fact,You can use a hard link instead of a symbolic link.Hard link works
Fix issue with case sensitivity when loading dataset from local cache

When a dataset with upper-cases in its name is first loaded using `load_dataset()`, the local cache directory is created with all lowercase letters. 

However, upon subsequent loads, the current version attempts to locate the cache directory using the dataset's original name, which includes uppercase letters. This discrepancy can lead to confusion and, particularly in offline mode, results in errors. 

### Reproduce
```bash
~$ python            
Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
>>> quit()

~$ export HF_DATASETS_OFFLINE=1 
~$ python                      
Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'locuslab/TOFU': Offline mode is enabled.
>>> 
```

I fix this issue by lowering the dataset name (`.lower()`) when generating cache_dir.

Comments:
I also need this feature for ["Cnam-LMSSC/vibravox "](https://huggingface.co/datasets/Cnam-LMSSC/vibravox)


EDIT: Upgrading to `2.19.0` fixed my problem thanks to [this PR](https://github.com/huggingface/datasets/pull/6754)
Persistent multi-process Pool

### Feature request

Running .map and filter functions with `num_procs` consecutively instantiates several multiprocessing pools iteratively. 

As instantiating a Pool is very resource intensive it can be a bottleneck to performing iteratively filtering. 

My ideas:
1. There should be an option to declare `persistent_workers` similar to pytorch DataLoader. Downside would be that would be complex to determine the correct resource allocation and deallocation of the pool. i.e. the dataset can outlive the utility of the pool. 
2. Provide a pool as an argument. Downside would be the expertise required by the user. Upside, is that there is better resource management.

### Motivation

Is really slow to iteratively perform map and filter operations on a dataset.

### Your contribution

If approved I could integrate it. I would need to know what method would be most suitable to implement from the two options above.

Comments:
Passing `sample_by` to `load_dataset` when loading text data does not work

### Describe the bug

I have a dataset that consists of a bunch of text files, each representing an example. There is an undocumented `sample_by` argument for the `TextConfig` class that is used by `Text` to decide whether to split files into lines, paragraphs or take them whole. Passing `sample_by=“document”` to `load_dataset` results in files getting split into lines regardless. I have edited `src/datasets/packaged_modules/text/text.py` for myself to switch the default and it works fine.

As a side note, the `if-else` for `sample_by` will silently load an empty dataset if someone makes a typo in the argument, which is not ideal.

### Steps to reproduce the bug

1. Prepare data as a bunch of files in a directory.
2. Load that data via `load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)`.
3. Inspect the resultant dataset — every item should have the form of `{“text”: <a line from a file>}`.

### Expected behavior

`load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)` should result in a dataset with items of the form `{“text”: <one document>}`.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-1046-nvidia-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0

Comments:
Thanks for reporting! We are working on a fix.
Test disabling transformers containers in docs CI

Related to https://github.com/huggingface/doc-builder/pull/487 and [internal slack thread](https://huggingface.slack.com/archives/C04F8N7FQNL/p1711384899462349?thread_ts=1711041424.720769&cid=C04F8N7FQNL). There is now a `custom_container` option when building docs in CI. When set to `""` (instead of `"huggingface/transformers-doc-builder"` by default), we don't run the CI inside a container, therefore saving ~2min of download time. The plan is to test disabling the transformers container on a few "big" repo and if everything works correctly, we will stop making it the default container. More details on https://github.com/huggingface/doc-builder/pull/487.

cc @mishig25 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6757). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

On slack it was mentioned that it was actually slower for `datasets`, should we close this one or am I missing something ?

@lhoestq I converted to draft. Want to make some more tests and will let you know
Support SQLite files?

### Feature request

Support loading a dataset from a SQLite file

https://huggingface.co/datasets/severo/test_iris_sqlite/tree/main

### Motivation

SQLite is a popular file format.

### Your contribution

See discussion on slack: https://huggingface.slack.com/archives/C04L6P8KNQ5/p1702481859117909 (internal)

In particular: a SQLite file can contain multiple tables, which might be matched to multiple configs. Maybe the detail of splits and configs should be defined in the README YAML, or use the same format as for ZIP files: `Iris.sqlite::Iris`. 

See dataset here: https://huggingface.co/datasets/severo/test_iris_sqlite

Note: should we also support DuckDB files?

Comments:
You can use `Dataset.from_sql(path_to_sql_file)` already. Though we haven't added the Sql dataset builder to the `_PACKAGED_DATASETS_MODULES` list or in `_EXTENSION_TO_MODULE` to map `.sqlite` to the Sql dataset builder

This would allow to load a dataset repository with a `.sqlite` file using `load_dataset` and enable the Dataset Viewer

Considering `Dataset.from_sql`'s (extremely) low usage, I don't think many users are interested in using this format for their datasets. Also, SQLite files are hard/impossible to stream efficiently and require custom logic to define splits/subsets, so IMO we shouldn't encourage people to use SQLite on the Hub.

@severo Do you have some real-world examples of datasets published in this format?

No. Indeed, it seems better to explicitly not support sqlite
Small typo on the documentation

### Describe the bug

There is a small typo on https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938

It should be `caching is enabled`. 

### Steps to reproduce the bug

Please visit 

https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938


### Expected behavior

`caching is enabled`

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-5.15.0-101-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.10.0

Comments:
Thanks for reporting @fostiropoulos! I've edited your comment to fix the link to the problematic line.


@mariosasko can i take this up?

#self-assign
Precision being changed from float16 to float32 unexpectedly

### Describe the bug

I'm loading a HuggingFace Dataset for images. 

I'm running a preprocessing (map operation) step that runs a few operations, one of them being conversion to float16.  The Dataset features also say that the 'img' is of type float16.  Whenever I take an image from that HuggingFace Dataset instance, the type turns out to be float32.  



### Steps to reproduce the bug

```python
import torchvision.transforms.v2 as transforms
from datasets import load_dataset

dataset = load_dataset('cifar10', split='test')
dataset = dataset.with_format("torch")

data_transform = transforms.Compose([transforms.Resize((32, 32)), 
                                     transforms.ToDtype(torch.float16, scale=True),
                                     transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
                                     ])
def _preprocess(examples):
    # Permutes from (BS x H x W x C) to (BS x C x H x W)
    images = torch.permute(examples['img'], (0, 3, 2, 1))
    examples['img'] = data_transform(images)
    return examples

dataset = dataset.map(_preprocess, batched=True, batch_size=8)
```

Now at this point the dataset.features are showing float16 which is great because that's what I want. 

```python
print(data_loader.features['img'])

Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float16', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None)
```

But when I try to sample an image from this dataloader; I'm getting a float32 image, when I'm expecting float16:
```python
print(next(iter(data_loader))['img'].dtype)

torch.float32
```

### Expected behavior

I'm expecting the images loaded after the transformation to stay in float16.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.9
- `huggingface_hub` version: 0.21.4
- PyArrow version: 14.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0

Comments:
This is because of the formatter (`torch` in this case).
It defaults to `float32`.

You can load it in `float16` using `dataset.set_format("torch", dtype=torch.float16)`.

I am having a similar issue when _building_ a dataset given `float16`-valued data.

```py
from datasets import Features, Value, Dataset

d = Dataset.from_dict({"data": np.ones(4, dtype="float16")}, features=Features({"data": Value("float16")}))
type(d["data"][0])  # returns <class 'float'>
```

Note that I specifically indicate that my data is `float16`-valued in the `features` argument above
Use 'with' operator for some download functions

Some functions in `streaming_download_manager.py` are not closing the file they open which lead to `Unclosed file` warnings in our code. This fixes a few of them.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6751). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I was mistaken on the intent of those functions, closing the PR.
Strange slicing behavior

### Describe the bug

I have loaded a dataset, and then slice first 300 samples using `:` ops, however, the resulting dataset is not expected, as the output below:

```bash
len(dataset)=1050324
len(dataset[:300])=2
len(dataset[0:300])=2
len(dataset.select(range(300)))=300
```

### Steps to reproduce the bug

load a dataset then:
```bash
    dataset = load_from_disk(args.train_data_dir)

    print(f"{len(dataset)=}", flush=True)
    print(f"{len(dataset[:300])=}", flush=True)
    print(f"{len(dataset[0:300])=}", flush=True)
    print(f"{len(dataset.select(range(300)))=}", flush=True)
```

### Expected behavior

```bash
len(dataset)=1050324
len(dataset[:300])=300
len(dataset[0:300])=300
len(dataset.select(range(300)))=300
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- `huggingface_hub` version: 0.20.2
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0

Comments:
As explained in the [docs](https://huggingface.co/docs/datasets/v2.18.0/en/access#slicing), slicing a `Dataset` returns a dictionary that maps its column names to their values. So, `len(dataset[:300])=2` is expected, assuming your dataset has 2 columns (the returned dict has 2 keys, but each value in the dict has 300 items).
`
Option to disable file locking

### Feature request

Commands such as `load_dataset` creates file locks with `filelock.FileLock`. It would be good if there was a way to disable this.

### Motivation

File locking doesn't work on all file-systems (in my case NFS mounted Weka). If the `cache_dir` only had small files then it would be possible to point to local disk and the problem would be solved. However, as cache_dir is both where the small info files are written and the processed datasets are put this isn't a feasible solution.

Considering https://github.com/huggingface/datasets/issues/6395 I still do think this is something that belongs in HuggingFace. The possibility to control packages separately is valuable. It might be that a user has their dataset on a file-system that doesn't support file-locking while they are using file locking on local disk to control some other type of access.

### Your contribution

My suggested solution:
```
diff --git a/src/datasets/utils/_filelock.py b/src/datasets/utils/_filelock.py
index 19620e6e..58f41a02 100644
--- a/src/datasets/utils/_filelock.py
+++ b/src/datasets/utils/_filelock.py
@@ -18,11 +18,15 @@
 import os
 
 from filelock import FileLock as FileLock_
-from filelock import UnixFileLock
+from filelock import SoftFileLock, UnixFileLock
 from filelock import __version__ as _filelock_version
 from packaging import version
 
 
+if os.getenv('HF_USE_SOFTFILELOCK', 'false').lower() in ('true', '1'):
+    FileLock_ = SoftFileLock
+
+
 class FileLock(FileLock_):
     """
     A `filelock.FileLock` initializer that handles long paths.
```



Comments:
Support for loading geotiff files as a part of the ImageFolder

### Feature request

Request for adding rasterio support to load geotiff as a part of ImageFolder, instead of using PIL

### Motivation

As of now, there are many datasets in HuggingFace Hub which are predominantly focussed towards RemoteSensing or are from RemoteSensing. The current ImageFolder (if I have understood correctly) uses PIL. This is not really optimized because mostly these datasets have images with many channels and additional metadata. Using PIL makes one loose it unless we provide a custom script. Hence, maybe an API could be added to have this in common?

### Your contribution

If the issue is accepted - i can contribute the code, because I would like to have it automated and generalised.

Comments:
Dict feature is non-nullable while nested dict feature is

When i try to create a `Dataset` object with None values inside a dict column, like this: 
```python
from datasets import Dataset, Features, Value

Dataset.from_dict(
    {
        "dict": [{"a": 0, "b": 0}, None],
    }, features=Features(
        {"dict": {"a": Value("int16"), "b": Value("int16")}}
    )
)
```
i get `ValueError: Got None but expected a dictionary instead`.


At the same time, having None in _nested_ dict feature works, for example, this doesn't throw any errors:

```python
from datasets import Dataset, Features, Value, Sequence

dataset = Dataset.from_dict(
    {
        "list_dict": [[{"a": 0, "b": 0}], None],
        "sequence_dict": [[{"a": 0, "b": 0}], None],
    }, features=Features({
        "list_dict": [{"a": Value("int16"), "b": Value("int16")}],
        "sequence_dict": Sequence({"a": Value("int16"), "b": Value("int16")}),
    })
)
```
Other types of features also seem to be nullable (but I haven't checked all of them).

Version of `datasets` is the latest atm (2.18.0)


Is this an expected behavior or a bug?

Comments:
It looks like a bug, by default every feature should be nullable.

I've linked a PR with a fix :)

@mariosasko awesome thank you!
Mosaic Streaming (MDS) Support

### Feature request

I'm a huge fan of the current HF Datasets `webdataset` integration (especially the built-in streaming support). However, I'd love to upload some robotics and multimodal datasets I've processed for use with [Mosaic Streaming](https://docs.mosaicml.com/projects/streaming/en/stable/), specifically their [MDS Format](https://docs.mosaicml.com/projects/streaming/en/stable/fundamentals/dataset_format.html#mds). 

Because the shard files have similar semantics to WebDataset, I'm hoping that adding such support won't be too much trouble?

### Motivation

One of the downsides with WebDataset is a lack of out-of-the-box determinism (especially for large-scale training and reproducibility), easy job resumption, and the ability to quickly debug / visualize individual examples. 

Mosaic Streaming provides a [great interface for this out of the box](https://docs.mosaicml.com/projects/streaming/en/stable/#key-features), so I'd love to see it supported in HF Datasets.

### Your contribution

Happy to help test things / provide example data. Can potentially submit a PR if maintainers could point me to the necessary WebDataset logic / steps for adding a new streaming format!

Comments:
Hi ! that would be great :) Though note that `datasets` doesn't implement format-specific resuming when streaming, so in general I think it's better if users can use the mosaic-streaming library to read their MDS datasets. I wonder if they support `hf://` paths though...

Anyway for those interested, the code for WebDataset is a single file here: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py.

It implements `_split_generators` that downloads files and returns the lists of splits (train/validation/test) and `_split_generators` to generate examples (dicts) from the downloaded files. Streaming is automatically supported by making download steps lazy and by extending `open()` to work with remote URLs.

edit: we could try with datasets like https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining
EmptyDatasetError when loading dataset downloaded with HuggingFace cli

### Describe the bug

I am using a cluster that does not have access to the internet when given a job. I tried downloading the dataset using the huggingface-cli command and then loading it with load_dataset but I get an error:

```raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None```

The dataset I'm using is "lmsys/chatbot_arena_conversations". The folder structure is

- README.md
- data
  - train-00000-of-00001-cced8514c7ed782a.parquet



### Steps to reproduce the bug

1. Download dataset using HuggingFace CLI: ```huggingface-cli download lmsys/chatbot_arena_conversations --local-dir ./lmsys/chatbot_arena_conversations```
2. In Python
```
from datasets import load_dataset
load_dataset("lmsys/chatbot_arena_conversations")
```

### Expected behavior

Should return a Dataset Dict in the form of
```
DatasetDict({
    train: Dataset({
        features: [...],
        num_rows: 33,000
    })
})
```
### Environment info

Python 3.11.5
Datasets 2.18.0
Transformers 4.38.2
Pytorch 2.2.0
Pyarrow 15.0.1
Rocky Linux release 8.9 (Green Obsidian)


Comments:
Hi! `datasets` is not compatible with `huggingface_hub`'s cache structure, hence the error.

You can track https://github.com/huggingface/datasets/issues/5080 to get notified when this is implemented.
Unexpected behavior when using load_dataset with streaming=True in a for loop

### Describe the bug

### My Code
```
from datasets import load_dataset
res=[]
for i in [0,1]:
    di=load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
        ).map(lambda x: {"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

### Unexpected Behavior
Data in `res[0]` has `source=1`. However the expected value is 0.

### FYI
I further switch `streaming` to `False`. And the output value is as expected (0). So there may exist bugs in setting `streaming=True` in a for loop.

### Environment
Python 3.8.0
datasets==2.18.0
transformers==4.28.1

### Steps to reproduce the bug

1. Create a Json file with any content.
2. Run the provided code.
3. Switch `streaming` to `False` and run again to see the expected behavior.

### Expected behavior

The expected behavior is the data are mapped with its corresponding value in the for loop.

### Environment info

Python 3.8.0

datasets==2.18.0
transformers==4.28.1

Ubuntu 20.04

Comments:
This is normal behavior in python when using `lambda`: the `i` defined in your `lambda` refers to the global variable `i` in your loop, and `i` equals to `1` when you run your `for e in res[0]` line.

You should pass `fn_kwargs` that will be passed to your `lambda` instead of using the global variable:

```python
from datasets import load_dataset

res=[]
for i in [0,1]:
    di = load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
    ).map(lambda x, source: {"source": source}, fn_kwargs={"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

This doesn't happen in non-streaming since in that case `map` is executed while the variable `i` has the right value. In streaming mode, `map` is executed on-the-fly when you iterate on the dataset.

Thank you very much for your answer. I think this issue can be closed now.
Request for a comparison of huggingface datasets compared with other data format especially webdataset

### Feature request

Request for a comparison of huggingface datasets compared with other data format especially webdataset

### Motivation

I see huggingface datasets uses Apache Arrow as its backend, it seems to be great, but I'm curious about how it is good compared with other dataset format, like webdataset, what's the pros/cons of them.

### Your contribution

More information

Comments:
Dataset with loading script does not work in renamed repos

### Describe the bug

My data repository was first called `BramVanroy/hplt-mono-v1-2` but I then renamed to use underscores instead of dashes. However, it seems that `datasets` retrieves the old repo name when it checks whether the repo contains data loading scripts in this line.

https://github.com/huggingface/datasets/blob/6fb6c834f008996c994b0a86c3808d0a33d44525/src/datasets/load.py#L1845

When I print `filename` it returns `hplt-mono-v1-2.py` but the files in the repo are of course `['.gitattributes', 'README.md', 'hplt_mono_v1_2.py']`. So the `filename` is the original reponame instead of the renamed one.

I am not sure if this is a caching issue or not or how I can resolve it.

### Steps to reproduce the bug

```
from datasets import load_dataset


ds = load_dataset(
    "BramVanroy/hplt-mono-v1-2",
    "ky",
    trust_remote_code=True
)
```

### Expected behavior

That the most recent repo name is used when `filename` is generated.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0


Comments:
Hi,do you know how to load the dataset from local file now?

              Hi, if I want to load the dataset from local file, then how to specify the configuration name?

_Originally posted by @WHU-gentle in https://github.com/huggingface/datasets/issues/2976#issuecomment-1333455222_
            

Comments:

@Gera001
# Loading Dataset from Local Files Using 🤗Hugging Face.

To load a dataset from local files using the Hugging Face datasets library, you can use the `load_dataset` function.

```
from datasets import load_dataset
dataset = load_dataset('csv', data_files={'train': 'path/to/train.csv',
                                         'test': 'path/to/test.csv'})
```

Reference to [HF Datasets docs for loading from local](https://huggingface.co/docs/datasets/en/loading#csv).       

@albertvillanova
this issue can be closed here.

like this: from datasets import load_from_disk
dataset = load_from_disk(data_path)


@ge00009 
> like this: from datasets import load_from_disk dataset = load_from_disk(data_path)

Loads a dataset that was previously saved using `save_to_disk()`.

Reference link:
https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_from_disk.example