dalssy/data_del huggingface dataset:
GitHub issue: CI fails on Windows for test_delete_from_hub and test_xgetsize_private due to new-line character
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-02 07:37:03
Labels: name: bug

CI fails on Windows for test_delete_from_hub after the merge of:
- #6820

This is weird because the CI was green in the PR branch before merging to main.

```
FAILED tests/test_hub.py::test_delete_from_hub - AssertionError: assert [CommitOperat...\r\n---\r\n')] == [CommitOperat...in/*\n---\n')]
  
  At index 1 diff: CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=b'---\r\nconfigs:\r\n- config_name: cats\r\n  data_files:\r\n  - split: train\r\n    path: cats/train/*\r\n---\r\n') != CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=b'---\nconfigs:\n- config_name: cats\n  data_files:\n  - split: train\n    path: cats/train/*\n---\n')
  
  Full diff:
    [
        CommitOperationDelete(
            path_in_repo='dogs/train/0000.csv',
            is_folder=False,
        ),
        CommitOperationAdd(
            path_in_repo='README.md',
  -         path_or_fileobj=b'---\nconfigs:\n- config_name: cats\n  data_files:\n '
  ?                                                                       --------
  +         path_or_fileobj=b'---\r\nconfigs:\r\n- config_name: cats\r\n  data_f'
  ?                               ++          ++                     ++
  -                         b' - split: train\n    path: cats/train/*\n---\n',
  ?                                                                   ^^^^^^ -
  +                         b'iles:\r\n  - split: train\r\n    path: cats/train/*\r'
  ?                           ++++++++++                ++                        ^
  +                         b'\n---\r\n',
        ),
    ]
```
GitHub issue: Wrong example of usage when config name is missing for community script-datasets
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-02 06:59:39
Labels: name: bug

As reported by @Wauplin, when loading a community dataset with script, there is a bug in the example of usage of the error message if the dataset has multiple configs (and no default config) and the user does not pass any config. For example:
```python
>>> ds = load_dataset("google/fleurs")
ValueError: Config name is missing.
Please pick one among the available configs: ['af_za', 'am_et', 'ar_eg', 'as_in', 'ast_es', 'az_az', 'be_by', 'bg_bg', 'bn_in', 'bs_ba', 'ca_es', 'ceb_ph', 'ckb_iq', 'cmn_hans_cn', 'cs_cz', 'cy_gb', 'da_dk', 'de_de', 'el_gr', 'en_us', 'es_419', 'et_ee', 'fa_ir', 'ff_sn', 'fi_fi', 'fil_ph', 'fr_fr', 'ga_ie', 'gl_es', 'gu_in', 'ha_ng', 'he_il', 'hi_in', 'hr_hr', 'hu_hu', 'hy_am', 'id_id', 'ig_ng', 'is_is', 'it_it', 'ja_jp', 'jv_id', 'ka_ge', 'kam_ke', 'kea_cv', 'kk_kz', 'km_kh', 'kn_in', 'ko_kr', 'ky_kg', 'lb_lu', 'lg_ug', 'ln_cd', 'lo_la', 'lt_lt', 'luo_ke', 'lv_lv', 'mi_nz', 'mk_mk', 'ml_in', 'mn_mn', 'mr_in', 'ms_my', 'mt_mt', 'my_mm', 'nb_no', 'ne_np', 'nl_nl', 'nso_za', 'ny_mw', 'oc_fr', 'om_et', 'or_in', 'pa_in', 'pl_pl', 'ps_af', 'pt_br', 'ro_ro', 'ru_ru', 'sd_in', 'sk_sk', 'sl_si', 'sn_zw', 'so_so', 'sr_rs', 'sv_se', 'sw_ke', 'ta_in', 'te_in', 'tg_tj', 'th_th', 'tr_tr', 'uk_ua', 'umb_ao', 'ur_pk', 'uz_uz', 'vi_vn', 'wo_sn', 'xh_za', 'yo_ng', 'yue_hant_hk', 'zu_za', 'all']
Example of usage:
	`load_dataset('fleurs', 'af_za')`
```

Note the example of usage in the error message suggests loading "fleurs" instead of "google/fleurs".
GitHub issue: Support soft links for load_datasets imagefolder
Author: billytcl (NONE)
State: open | Created: 2024-04-30 22:14:29
Labels: name: enhancement

### Feature request

Load_dataset from a folder of images doesn't seem to support soft links. It would be nice if it did, especially during methods development where image folders are being curated.

### Motivation

Images are coming from a complex variety of sources and we'd like to be able to soft link directly from the originating folders as opposed to copying. Having a copy of the file ensures that there may be issues with image versioning as well as having double the amount of required disk space.

### Your contribution

N/A
GitHub issue: Write token isn't working while pushing to datasets
Author: zaibutcooler (NONE)
State: closed | Created: 2024-04-30 21:18:20

### Describe the bug

<img width="1001" alt="Screenshot 2024-05-01 at 3 37 06 AM" src="https://github.com/huggingface/datasets/assets/130903099/00fcf12c-fcc1-4749-8592-d263d4efcbcc">

As you can see I logged in to my account and the write token is valid.

But I can't upload on my main account and I am getting that error. It was okay on my test account at first try.

(I refreshed the token, tried a new token but still doesn't work)

### Steps to reproduce the bug

1. I loaded a dataset.

2. I logged in using both cli and huggingface_hub

3. I pushed to my down dataset
(It went well without any issues on my test account)

### Expected behavior

It should have gone smoothly and this is not even my first time uploading to huggingface datasets

### Environment info

colab, dataset (tried multiple versions)
GitHub issue: load_dataset('emotion')    UnicodeDecodeError
Author: L-Block-C (NONE)
State: open | Created: 2024-04-30 09:25:01

### Describe the bug

**emotions = load_dataset('emotion')**

_UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte_

### Steps to reproduce the bug

load_dataset('emotion') 

### Expected behavior

succese

### Environment info

py3.10
transformers  4.41.0.dev0
datasets     2.19.0
GitHub issue: Problem loading voxpopuli dataset 
Author: Namangarg110 (NONE)
State: closed | Created: 2024-04-29 16:46:51

### Describe the bug

```
Exception has occurred: FileNotFoundError
Couldn't find file at https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/{'en': 'data/en/asr_train.tsv'}
```

Error in logic for link url creation. The link should be  https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/en/asr_train.tsv 

Basically there should be links directly under ```metadata["train"]```, not under ```metadata["train"][self.config.languages[0]]```

same for audio urls

### Steps to reproduce the bug

```
from datasets import load_dataset

dataset = load_dataset("facebook/voxpopuli","en")
```

### Expected behavior

Dataset should be loaded successfully.

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.15.0-1041-aws-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.12.2
GitHub issue: [Streaming] Only load requested splits without resolving files for the other splits
Author: lhoestq (MEMBER)
State: open | Created: 2024-04-29 09:49:32

e.g. [thangvip](https://huggingface.co/thangvip)/[cosmopedia_vi_math](https://huggingface.co/datasets/thangvip/cosmopedia_vi_math) has 300 splits and it takes a very long time to load only one split.

This is due to `load_dataset()` resolving the files of all the splits even if only one is needed.

In `dataset-viewer` the splits are loaded in different jobs so it results in 300 jobs that resolve 300 splits -> 90k calls to `/paths-info`
GitHub issue: Unimaginable super slow iteration
Author: rangehow (NONE)
State: closed | Created: 2024-04-28 05:24:14

### Describe the bug

Assuming there is a dataset with 52000 sentences, each with a length of 500, it takes 20 seconds to extract a sentence from the dataset……？Is there something wrong with my iteration?

### Steps to reproduce the bug

```python
import datasets
import time
import random

num_rows = 52000
num_cols = 500

random_input = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]
random_output = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]


s=time.time()
d={'random_input':random_input,'random_output':random_output}
dataset=datasets.Dataset.from_dict(d)
print('from dict',time.time()-s)
print(dataset)


for i in range(len(dataset)):
    aa=time.time()
    a,b=dataset['random_input'][i],dataset['random_output'][i]
    print(time.time()-aa)

```

corresponding output
```bash
from dict 9.215498685836792
Dataset({
    features: ['random_input', 'random_output'],
    num_rows: 52000
})
19.129778146743774
19.329464197158813
19.27668261528015
19.28557538986206
19.247620582580566
19.624247074127197
19.28673791885376
19.301053047180176
19.290496110916138
19.291821718215942
19.357765197753906

```

### Expected behavior

Under normal circumstances, iteration should be very rapid as it does not involve the main tasks other than getting items

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
GitHub issue: Datasets with files with colon : in filenames cannot be used on Windows
Author: jacobjennings (NONE)
State: open | Created: 2024-04-26 00:14:16

### Describe the bug

Datasets (such as https://huggingface.co/datasets/MLCommons/peoples_speech) cannot be used on Windows due to the fact that windows does not allow colons ":" in filenames. These should be converted into alternative strings.



### Steps to reproduce the bug

1. Attempt to run load_dataset on MLCommons/peoples_speech

### Expected behavior

Does not crash during extraction

### Environment info

Windows 11, NTFS filesystem, Python 3.12
GitHub issue: Delete uploaded files from the UI
Author: saicharan2804 (NONE)
State: open | Created: 2024-04-25 22:33:57
Labels: name: enhancement

### Feature request

Once a file is uploaded and the commit is made, I am unable to delete individual files without completely deleting the whole dataset via the website UI.

### Motivation

Would be a useful addition

### Your contribution

Would love to help out with some guidance
GitHub issue: Remove token arg from CLI examples
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-04-25 14:00:38

As suggested by @Wauplin, see: https://github.com/huggingface/datasets/pull/6831#discussion_r1579492603
> I would not advertise the --token arg in the example as this shouldn't be the recommended way (best to login with env variable or huggingface-cli login)
GitHub issue: Cannot use cached dataset without Internet connection (or when servers are down)
Author: DionisMuzenitov (NONE)
State: open | Created: 2024-04-25 10:48:20

### Describe the bug

I want to be able to use cached dataset from HuggingFace even when I have no Internet connection (or when HuggingFace servers are down, or my company has network issues).  
The problem why I can't use it:
`data_files` argument from `datasets.load_dataset()` function get it updates from the server before calculating hash for caching. As a result, when I run the same code with and without Internet I get different dataset configuration directory name.

### Steps to reproduce the bug

```
import datasets

c4_dataset = datasets.load_dataset(
    path="allenai/c4",
    data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
    split="train",
    cache_dir="/datesets/cache",
    download_mode="reuse_cache_if_exists",
    token=False,
)
```
1. Run this code with the Internet.
2. Run the same code without the Internet.

### Expected behavior

When running without the Internet connection, the loader should be able to get dataset from cache

### Environment info

- `datasets` version: 2.19.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2
GitHub issue: ExpectedMoreSplits error on load_dataset when upgrading to 2.19.0
Author: ebsmothers (NONE)
State: open | Created: 2024-04-24 21:52:35

### Describe the bug

Hi there, thanks for the great library! We have been using it a lot in torchtune and it's been a huge help for us. 

Regarding the bug: the same call to `load_dataset` errors with `ExpectedMoreSplits` in 2.19.0 after working fine in 2.18.0. Full details given in the repro below.

### Steps to reproduce the bug

On 2.18.0, things work fine:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.18.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

On 2.19.0, they do not:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.19.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

The stack trace I see from the 2.19.0 version of load_dataset can be seen [here](https://gist.github.com/ebsmothers/f9b1f1949bee7030a8d7bb8a491550d2).

(Maybe unsurprising but) notably if I do not delete the cache first I am able to load the dataset successfully. So based on this I suspect the cause is somewhere in the download logic.

### Expected behavior

Download the dataset successfully :)

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34
- Python version: 3.11.9
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1
GitHub issue: largelisttype not supported (.from_polars())
Author: Modexus (CONTRIBUTOR)
State: closed | Created: 2024-04-24 11:33:43

### Describe the bug

The following code fails because LargeListType is not supported.
This is especially a problem for .from_polars since polars uses LargeListType.


### Steps to reproduce the bug

```python
import datasets
import polars as pl

df = pl.DataFrame({"list": [[]]})
datasets.Dataset.from_polars(df)
```

### Expected behavior

Convert LargeListType to list.

### Environment info

- `datasets` version: 2.19.1.dev0
- Platform: Linux-6.8.7-200.fc39.x86_64-x86_64-with-glibc2.38
- Python version: 3.12.2
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2024.3.1
GitHub issue: Super slow iteration with trivial custom transform
Author: xslittlegrass (NONE)
State: open | Created: 2024-04-23 20:40:59

### Describe the bug

Dataset is 10X slower when applying trivial transforms:

```
import time
import numpy as np
from datasets import Dataset, Features, Array2D

a = np.zeros((800, 800))
a = np.stack([a] * 1000)
features = Features({"a": Array2D(shape=(800, 800), dtype="uint8")})

ds1 = Dataset.from_dict({"a": a}, features=features).with_format('numpy')

def transform(batch):
	return batch

ds2 = ds1.with_transform(transform)

%time sum(1 for _ in ds1)
%time sum(1 for _ in ds2)
```

```
CPU times: user 472 ms, sys: 319 ms, total: 791 ms
Wall time: 794 ms
CPU times: user 9.32 s, sys: 443 ms, total: 9.76 s
Wall time: 9.78 s
```

In my real code I'm using set_transform to apply some post-processing on-the-fly for the 2d array, but it significantly slows down the dataset even if the transform itself is trivial.

Related issue: https://github.com/huggingface/datasets/issues/5841

### Steps to reproduce the bug

Use code in the description to reproduce.

### Expected behavior

Trivial custom transform in the example should not slowdown the dataset iteration.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-79-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2
GitHub issue: Support downloading specific splits in `load_dataset`
Author: mariosasko (COLLABORATOR)
State: open | Created: 2024-04-23 12:32:27

This PR builds on https://github.com/huggingface/datasets/pull/6639 to support downloading only the specified splits in `load_dataset`. For this to work, a builder's `_split_generators` need to be able to accept the requested splits (as a list) via a `splits` argument to avoid processing the non-requested ones. Also, the builder has to define a `_available_splits` method that lists all the possible `splits` values.

Close https://github.com/huggingface/datasets/issues/4101, close https://github.com/huggingface/datasets/issues/2538 (I'm probably missing some)

Should also make it possible to address https://github.com/huggingface/datasets/issues/6793
GitHub issue: Add a doc page for the convert_to_parquet CLI
Author: severo (COLLABORATOR)
State: closed | Created: 2024-04-23 09:49:04
Labels: name: documentation

Follow-up to https://github.com/huggingface/datasets/pull/6795. Useful for https://github.com/huggingface/dataset-viewer/issues/2742. cc @albertvillanova
GitHub issue: Load and save from/to disk no longer accept pathlib.Path
Author: albertvillanova (MEMBER)
State: open | Created: 2024-04-23 09:44:45
Labels: name: bug

Reported by @vttrifonov at https://github.com/huggingface/datasets/pull/6704#issuecomment-2071168296:
> This change is breaking in 
> https://github.com/huggingface/datasets/blob/f96e74d5c633cd5435dd526adb4a74631eb05c43/src/datasets/arrow_dataset.py#L1515
> when the input is `pathlib.Path`. The issue is that `url_to_fs` expects a `str` and cannot deal with `Path`. `get_fs_token_paths` converts to `str` so it is not a problem

This change was introduced in:
- #6704
GitHub issue: Loading a remote dataset fails in the last release (v2.19.0)
Author: zrthxn (NONE)
State: open | Created: 2024-04-19 21:11:58

While loading a dataset with multiple splits I get an error saying `Couldn't find file at <URL>`

I am loading the dataset like so, nothing out of the ordinary.
This dataset needs a token to access it.
```
token="hf_myhftoken-sdhbdsjgkhbd"
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token=token)
```

I get the following error
![Screenshot 2024-04-19 at 11 03 07 PM](https://github.com/huggingface/datasets/assets/35369637/8dce757f-08ff-45dd-85b5-890fced7c5bc)

Now you can see that the URL that it is trying to reach has the JSON object of the dataset split appended to the base URL. I think this may be due to a newly introduced issue.

I did not have this issue with the previous version of the datasets. Everything was fine for me yesterday and after the release 12 hours ago, this seems to have broken. Also, the dataset in question runs custom code and I checked and there have been no commits to the dataset on Huggingface in 6 months.

### Steps to reproduce the bug

Since this happened with one particular dataset for me, I am listing steps to use that dataset.
1. Open https://huggingface.co/datasets/speechcolab/gigaspeech and fill the form to get access.
2. Create a token on your huggingface account with read access.
3. Run the following line, substituing `<your_token_here>` with your token.
```
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token="<your_token_here>")
```

### Expected behavior

Be able to load the dataset in question.

### Environment info

datasets == 2.19.0
python == 3.10
kernel == Linux 6.1.58+
GitHub issue: Fix parquet export infos
Author: lhoestq (MEMBER)
State: closed | Created: 2024-04-18 10:21:41

Don't use the parquet export infos when USE_PARQUET_EXPORT is False.

Otherwise the `datasets-server` might reuse erroneous data when re-running a job

this follows https://github.com/huggingface/datasets/pull/6714
GitHub issue: Give more details in `DataFilesNotFoundError` when getting the config names
Author: severo (COLLABORATOR)
State: open | Created: 2024-04-17 11:19:47
Labels: name: enhancement

### Feature request

After https://huggingface.co/datasets/cis-lmu/Glot500/commit/39060e01272ff228cc0ce1d31ae53789cacae8c3, the dataset viewer gives the following error:

```
{
  "error": "Cannot get the config names for the dataset.",
  "cause_exception": "DataFilesNotFoundError",
  "cause_message": "No (supported) data files found in cis-lmu/Glot500",
  "cause_traceback": [
    "Traceback (most recent call last):\n",
    " File \"/src/services/worker/src/worker/job_runners/dataset/config_names.py\", line 73, in compute_config_names_response\n config_names = get_dataset_config_names(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py\", line 347, in get_dataset_config_names\n dataset_module = dataset_module_factory(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1873, in dataset_module_factory\n raise e1 from None\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1854, in dataset_module_factory\n return HubDatasetModuleFactoryWithoutScript(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1245, in get_module\n module_name, default_builder_kwargs = infer_module_for_data_files(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 595, in infer_module_for_data_files\n raise DataFilesNotFoundError(\"No (supported) data files found\" + (f\" in {path}\" if path else \"\"))\n",
    "datasets.exceptions.DataFilesNotFoundError: No (supported) data files found in cis-lmu/Glot500\n"
  ]
}
```

because the deleted files were still listed in the README, see https://huggingface.co/datasets/cis-lmu/Glot500/discussions/4

Ideally, the error message would include the name of the first configuration with missing files, to help the user understand how to fix it. Here, it would tell that configuration `aze_Ethi` has no supported data files, instead of telling that the `cis-lmu/Glot500` *dataset* has no supported data files (which is not true).

### Motivation

Giving more detail in the error would help the Datasets Hub users to debug why the dataset viewer does not work.

### Your contribution

Not sure how to best fix this, as there are a lot of loops on the dataset configs in the traceback methods. "maybe" it would be easier to handle if the code was completely isolating each config.
GitHub issue: Improve typing of Dataset.search, matching definition
Author: Dref360 (CONTRIBUTOR)
State: closed | Created: 2024-04-16 14:53:39

Previously, the output of `score, indices = Dataset.search(...)` would be numpy arrays.

The definition in `SearchResult` is a `List[int]` so this PR now matched the expected type.

The previous behavior is a bit annoying as `Dataset.__getitem__` doesn't support `numpy.int64` which forced me to convert `indices` to int eg:

```python
score, indices = ds.search(...)
item = ds[int(indices[0])]
```
GitHub issue: `map` with `num_proc` > 1 leads to OOM
Author: bhavitvyamalik (CONTRIBUTOR)
State: open | Created: 2024-04-16 11:56:03

### Describe the bug

When running `map` on parquet dataset loaded from local machine, the RAM usage increases linearly eventually leading to OOM. I was wondering if I should I save the `cache_file` after every n steps in order to prevent this?

### Steps to reproduce the bug



```
ds = load_dataset("parquet", data_files=dataset_path, split="train")
ds = ds.shard(num_shards=4, index=0)

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
ds = ds.map(prepare_dataset,
            num_proc=32,
            writer_batch_size=1000,
            keep_in_memory=False,
            desc="preprocess dataset")
```

```
def prepare_dataset(batch):
    # load audio
    sample = batch["audio"]

    inputs = feature_extractor(sample["array"], sampling_rate=16000)
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(sample["array"].squeeze())

    return batch
```

### Expected behavior

It shouldn't run into OOM problem.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0
GitHub issue: Make convert_to_parquet CLI command create script branch
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-04-15 07:47:26

Make convert_to_parquet CLI command create a "script" branch and keep the script file on it.

This PR proposes the simplest UX approach: whenever `--revision` is not explicitly passed (i.e., when the script is in the main branch), try to create a "script" branch from the "main" branch; if the "script" branch exists already, then do nothing.

Follow-up of:
- #6795

Close #6808.

CC: @severo
GitHub issue: Make convert_to_parquet CLI command create script branch
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-04-15 06:46:07
Labels: name: enhancement

As proposed by @severo, maybe we should add this functionality as well to the CLI command to convert a script-dataset to Parquet. See: https://github.com/huggingface/datasets/pull/6795#discussion_r1562819168
> When providing support, we sometimes suggest that users store their script in a script branch. What do you think of this alternative to deleting the files?
GitHub issue: Fix hf-internal-testing/dataset_with_script commit SHA in CI test
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-04-12 08:47:50

Fix test using latest commit SHA in hf-internal-testing/dataset_with_script dataset: https://huggingface.co/datasets/hf-internal-testing/dataset_with_script/commits/refs%2Fconvert%2Fparquet

Fix #6796.
GitHub issue: Batched mapping of existing string column casts boolean to string
Author: starmpcc (NONE)
State: closed | Created: 2024-04-12 04:21:41

### Describe the bug

Let the dataset contain a column named 'a', which is of the string type.
If 'a' is converted to a boolean using batched mapping, the mapper automatically casts the boolean to a string (e.g., True -> 'true').
It only happens when the original column and the mapped column name are identical.

Thank you!

### Steps to reproduce the bug

```python
from datasets import Dataset

dset = Dataset.from_dict({'a': ['11', '22']})
dset = dset.map(lambda x: {'a': [True for _ in x['a']]}, batched=True)
print(dset['a'])
```
```
> ['true', 'true']
```

### Expected behavior

[True, True]

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2023.12.2
GitHub issue: Fix typo in docs (upload CLI)
Author: Wauplin (CONTRIBUTOR)
State: closed | Created: 2024-04-11 10:05:05

Related to https://huggingface.slack.com/archives/C04RG8YRVB8/p1712643948574129 (interal)

Positional args must be placed before optional args.

Feel free to merge whenever it's ready.
GitHub issue: got fileNotFound
Author: laoniandisko (NONE)
State: closed | Created: 2024-04-11 04:57:41

### Describe the bug

When I use load_dataset to load the nyanko7/danbooru2023 data set, the cache is read in the form of a symlink. There may be a problem with the arrow_dataset initialization process and I get FileNotFoundError: [Errno 2] No such file or directory: '2945000.jpg'

### Steps to reproduce the bug

#code show as below
from datasets import load_dataset
data = load_dataset("nyanko7/danbooru2023",cache_dir=<symlink>)
data["train"][0]

### Expected behavior

I should get this result:
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=365x256 at 0x7FB730CB4070>, 'label': 0}

### Environment info

datasets==2.12.0
python==3.10.14
GitHub issue: `DatasetBuilder._split_generators` incomplete type annotation
Author: JonasLoos (CONTRIBUTOR)
State: closed | Created: 2024-04-10 14:38:50

### Describe the bug

The [`DatasetBuilder._split_generators`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/builder.py#L1449) function has currently the following signature:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: DownloadManager):
        ...
```

However, the `dl_manager` argument can also be of type [`StreamingDownloadManager`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/download/streaming_download_manager.py#L962), which has different functionality. For example, the `download` function doesn't download, but rather just returns the given url(s). 

I suggest changing the function signature to:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: Union[DownloadManager, StreamingDownloadManager]):
        ...
```

and also adjust the docstring accordingly.

I would like to create a Pull Request to fix this, and have the following questions:
* Are there also other options than `DownloadManager`, and `StreamingDownloadManager`?
* Should this also be changed in other functions?

### Steps to reproduce the bug

Minimal example to print the different class names:

```python
import tempfile
from datasets import load_dataset

example = b'''
from datasets import GeneratorBasedBuilder, DatasetInfo, Features, Value, SplitGenerator

class Test(GeneratorBasedBuilder):
    def _info(self):
        return DatasetInfo(features=Features({"x": Value("int64")}))
    def _split_generators(self, dl_manager):
        print(type(dl_manager))
        return [SplitGenerator('test')]
    def _generate_examples(self):
        yield 0, {'x': 42}
'''

with tempfile.NamedTemporaryFile(suffix='.py') as f:
    f.write(example)
    f.flush()
    load_dataset(f.name, streaming=False)
    load_dataset(f.name, streaming=True)
```

### Expected behavior

complete type annotations

### Environment info

/
GitHub issue: CI is broken due to hf-internal-testing/dataset_with_script
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-04-10 06:56:02
Labels: name: bug

CI is broken for test_load_dataset_distributed_with_script. See: https://github.com/huggingface/datasets/actions/runs/8614926216/job/23609378127
```
FAILED tests/test_load.py::test_load_dataset_distributed_with_script[None] - assert False
 +  where False = all(<generator object test_load_dataset_distributed_with_script.<locals>.<genexpr> at 0x7f0c741de3b0>)
FAILED tests/test_load.py::test_load_dataset_distributed_with_script[force_redownload] - assert False
 +  where False = all(<generator object test_load_dataset_distributed_with_script.<locals>.<genexpr> at 0x7f0be45f6ea0>)
```
GitHub issue: Multithreaded downloads
Author: lhoestq (MEMBER)
State: closed | Created: 2024-04-09 11:13:19

...for faster dataset download when there are many many small files (e.g. imagefolder, audiofolder)

### Behcnmark

for example on [lhoestq/tmp-images-writer_batch_size](https://hf.co/datasets/lhoestq/tmp-images-writer_batch_size) (128 images)

|    | duration of the download step in `load_dataset()` |
|--| ----------------------------------------------------------------------|
| Before | 58s |
| Now | 3s |

This should fix issues with the Dataset Viewer taking too much time to show up for imagefolder/audiofolder datasets.

### Implementation details

The main change is in the `DownloadManager`:

```diff
- download_func = partial(self._download, download_config=download_config)
+ download_func = partial(self._download_batched, download_config=download_config)
downloaded_path_or_paths = map_nested(
    download_func,
    url_or_urls,
    map_tuple=True,
    num_proc=download_config.num_proc,
    desc="Downloading data files",
+   batched=True,
+   batch_size=-1,
)
```

and `_download_batched` is a multithreaded function.

I only enable multithreading if there are more than 16 files and files are small though, otherwise the progress bar that counts the number of downloaded files is not fluid (updating when a big batch of big files are done downloading). To do so I simply check if the first file is smaller than 20MB.

I also had to tweak `map_nested` to support batching. In particular it slices the data correctly if the user also enables multiprocessing.
GitHub issue: Loading just one particular split is not possible for imagenet-1k
Author: PaulPSta (NONE)
State: open | Created: 2024-04-08 14:39:14

### Describe the bug

I'd expect the following code to download just the validation split but instead I get all data on my disk (train, test and validation splits)

`
from datasets import load_dataset

dataset = load_dataset("imagenet-1k", split="validation", trust_remote_code=True)
`

Is it expected to work like that? 

### Steps to reproduce the bug

1. Install the required libraries (python, datasets, huggingface_hub)
2. Login using huggingface cli
2. Run the code in the description

### Expected behavior

Just a single (validation) split should be downloaded.

### Environment info

python: 3.12.2
datasets: 2.18.0
huggingface_hub: 0.22.2
GitHub issue: Fix cache conflict in `_check_legacy_cache2`
Author: lhoestq (MEMBER)
State: closed | Created: 2024-04-08 14:05:42

It was reloading from the wrong cache dir because of a bug in `_check_legacy_cache2`. This function should not trigger if there are config_kwars like `sample_by=`

fix https://github.com/huggingface/datasets/issues/6758
GitHub issue: `add_faiss_index` raises ValueError: not enough values to unpack (expected 2, got 1)
Author: NeuralFlux (NONE)
State: closed | Created: 2024-04-08 01:57:03

### Describe the bug

Calling `add_faiss_index` on a `Dataset` with a column argument raises a ValueError. The following is the trace

```python
    214 def replacement_add(self, x):
    215     """Adds vectors to the index.
    216     The index must be trained before vectors can be added to it.
    217     The vectors are implicitly numbered in sequence. When `n` vectors are
   (...)
    224         `dtype` must be float32.
    225     """
--> 227     n, d = x.shape
    228     assert d == self.d
    229     x = np.ascontiguousarray(x, dtype='float32')

ValueError: not enough values to unpack (expected 2, got 1)
```

### Steps to reproduce the bug

1. Load any dataset like `ds = datasets.load_dataset("wikimedia/wikipedia", "20231101.en")["train"]`
2. Add an FAISS index on any column `ds.add_faiss_index('title')`

### Expected behavior

The index should be created

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
- Python version: 3.9.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
- `faiss-cpu` version: 1.8.0
GitHub issue: PyArrow 'Memory mapping file failed: Cannot allocate memory' bug
Author: lasuomela (NONE)
State: open | Created: 2024-04-07 19:25:39

### Describe the bug

Hello,

I've been struggling with a problem using Huggingface datasets caused by PyArrow memory allocation. I finally managed to solve it, and thought to document it since similar issues have been raised here before (https://github.com/huggingface/datasets/issues/5710, https://github.com/huggingface/datasets/issues/6176).

In my case, I was trying to load ~70k dataset files from disk using `datasets.load_from_disk(data_path)` (meaning 70k repeated calls to load_from_disk). This triggered an (uninformative) exception around 64k loaded files:
```
  File "pyarrow/io.pxi", line 1053, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 1000, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```
Despite system RAM usage being very low. After a lot of digging around, I discovered that my Ubuntu machine had a limit on the maximum number of memory mapped files in `/proc/sys/vm/max_map_count` set to 65530, which was causing my data loader to crash. Increasing the limit in the file (`echo <new_mmap_size> | sudo tee /proc/sys/vm/max_map_count`) made the issue go away.

While this isn't a bug as such in either Datasets or PyArrow, this behavior can be very confusing to users. Maybe this should be mentioned in documentation? I suspect the other issues raised here about memory mapping OOM errors could actually be consequence of system configuration.

Br,
Lauri

### Steps to reproduce the bug

```
import numpy as np
import pyarrow as pa
import tqdm

# Write some data to disk
arr = pa.array(np.arange(100))
schema = pa.schema([
    pa.field('nums', arr.type)
])
with pa.OSFile('arraydata.arrow', 'wb') as sink:
    with pa.ipc.new_file(sink, schema=schema) as writer:
        batch = pa.record_batch([arr], schema=schema)
        writer.write(batch)

# Number of times to open the memory map
nums = 70000

# Read the data back
arrays = [pa.memory_map('arraydata.arrow', 'r') for _ in tqdm.tqdm(range(nums))]
```

### Expected behavior

No errors.

### Environment info

datasets: 2.18.0
pyarrow: 15.0.0
GitHub issue: Issue with map
Author: Nsohko (NONE)
State: open | Created: 2024-04-07 02:52:06

### Describe the bug

Map has been taking extremely long to preprocess my data.

It seems to process 1000 examples (which it does really fast in about 10 seconds), then it hangs for a good 1-2 minutes, before it moves on to the next batch of 1000 examples.

It also keeps eating up my hard drive space for some reason by creating a file named tmp1335llua that is over 300GB.

Trying to set num_proc to be >1 also gives me the following error: NameError: name 'processor' is not defined

Please advise on how I could optimise this?

### Steps to reproduce the bug

In general, I have been using map as per normal. Here is a snippet of my code:
````
###########################        DATASET LOADING AND PREP        #########################

def load_custom_dataset(split):
    ds = []
    if split == 'train':
        for dset in args.train_datasets:
            ds.append(load_from_disk(dset))
    if split == 'test':
        for dset in args.test_datasets:
            ds.append(load_from_disk(dset))

    ds_to_return = concatenate_datasets(ds)
    ds_to_return = ds_to_return.shuffle(seed=22)
    return ds_to_return



def prepare_dataset(batch):
    # load and (possibly) resample audio data to 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array
    batch["input_features"] = processor.feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]
    # compute input length of audio sample in seconds
    batch["input_length"] = len(audio["array"]) / audio["sampling_rate"]

    # optional pre-processing steps
    transcription = batch["sentence"]
    if do_lower_case:
        transcription = transcription.lower()
    if do_remove_punctuation:
        transcription = normalizer(transcription).strip()

    # encode target text to label ids
    batch["labels"] = processor.tokenizer(transcription).input_ids
    return batch

print('DATASET PREPARATION IN PROGRESS...')

# case 3: combine_and_shuffle is true, only train provided
# load train datasets
train_set = load_custom_dataset('train')

 # split dataset
raw_dataset = DatasetDict()
raw_dataset = train_set.train_test_split(test_size = args.test_size, shuffle=True, seed=42)

raw_dataset = raw_dataset.cast_column("audio", Audio(sampling_rate=args.sampling_rate))

print("Before Map:")
print(raw_dataset)

raw_dataset = raw_dataset.map(prepare_dataset, num_proc=1)

print("After Map:")
print(raw_dataset)
````


### Expected behavior

Based on the speed at which map is processing examples, I would expect a 5-6 hours completion for all mapping

However, because it hangs every 1000 examples, I instead roughly estimate it would take about  40 hours!

Moreover, i cant even finish the map because it keeps exponentially eating up my hard drive space

### Environment info

- `datasets` version: 2.18.0
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.14
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
GitHub issue: A Question About the Map Function
Author: Klein-Lan (NONE)
State: closed | Created: 2024-04-06 11:45:23

### Describe the bug

Hello,

I have a question regarding the map function in the Hugging Face datasets.

The situation is as follows: when I load a jsonl file using load_dataset(..., streaming=False), and then utilize the map function to process it, I specify that the returned example should be of type Torch.tensor. However, I noticed that after applying the map function, the datatype automatically changes to List, which leads to errors in my program.

I attempted to use load_dataset(..., streaming=True), and this issue no longer occurs. I'm not entirely clear on why this happens. Could you please provide some insights into this?

### Steps to reproduce the bug

1.dataset =  load_dataset(xxx, streaming = False)
2. dataset.map(function), function will return torch.Tensor.
3. you will find the format of data in dataset is List.

### Expected behavior

I expected to receieve the format of data is torch.Tensor.

### Environment info

2.18.0
GitHub issue: TimeoutError in map
Author: Jiaxin-Wen (CONTRIBUTOR)
State: open | Created: 2024-04-06 06:25:39

### Describe the bug

```python
from datasets import Dataset

def worker(example):
    while True:
        continue
    example['a'] = 100
    return example

data = Dataset.from_list([{"a": 1}, {"a": 2}])
data = data.map(worker)
print(data[0])
```

I'm implementing a worker function whose runtime will depend on specific examples (e.g., while most examples take 0.01s in worker, several examples may take 50s).

Therefore, I would like to know how the current implementation will handle those subprocesses that require a long (e.g., >= 5min) or even infinite time.

I notice that the current implementation set a timeout of 0.05 second
https://github.com/huggingface/datasets/blob/c3ddb1ef00334a6f973679a51e783905fbc9ef0b/src/datasets/utils/py_utils.py#L674

However, this example code still gets stuck.

### Steps to reproduce the bug

run the example above

### Expected behavior

I want to set a default worker to handle these timeout cases, instead of getting stuck

### Environment info

main branch version
GitHub issue: Make Image cast storage faster
Author: Modexus (CONTRIBUTOR)
State: open | Created: 2024-04-05 17:00:46

PR for issue #6782.
Makes `cast_storage` of the `Image` class faster by removing the slow call to `.pylist`.
Instead directly convert each `ListArray` item to either `Array2DExtensionType` or `Array3DExtensionType`.

This also preserves the `dtype` removing the warning if the array is already `uint8`.
GitHub issue: rename datasets-server to dataset-viewer
Author: severo (COLLABORATOR)
State: closed | Created: 2024-04-05 16:37:05

See https://github.com/huggingface/dataset-viewer/issues/2650

Tell me if it's OK, or if it's a breaking change that must be handled differently.

Also note that the docs page is still https://huggingface.co/docs/datasets-server/, so I didn't change it.

And the API URL is still https://datasets-server.huggingface.co/ (and [might always be](https://github.com/huggingface/dataset-viewer/issues/2666)), so I let it too.
GitHub issue: Extract data on the fly in packaged builders
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2024-04-05 16:12:25

Instead of waiting for data files to be extracted in the packaged builders, we can prepend the compression prefix and extract them as they are being read (using `fsspec`). This saves disk space (deleting extracted archives is not set by default) and slightly speeds up dataset generation (less disk reads)
GitHub issue: Fix CI
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2024-04-04 17:45:04

Updates the `wmt_t2t` test to pin the `revision` to the version with a loading script (cc @albertvillanova). 

Additionally, it replaces the occurrences of the `lhoestq/test` repo id with `hf-internal-testing/dataset_with_script` and re-enables logging checks in the `Dataset.from_sql` tests.
GitHub issue: Install dependencies with `uv` in CI
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2024-04-04 17:02:51

`diffusers` (https://github.com/huggingface/diffusers/pull/7116) and `huggingface_hub` (https://github.com/huggingface/huggingface_hub/pull/2072) also use `uv` to install their dependencies, so we can do the same here.

It seems to make the "Install dependencies" step in the `ubuntu` jobs 5-8x faster and 1.5-2x in the `windows` one.

Besides introducing `uv` in CI, this PR bumps the `tensorflow` minimal version requirement to align with Transformers and simplifies the SpaCy hashing tests (use blank language models instead of the pre-trained ones)
GitHub issue: Dataset.to_csv() missing commas in columns with lists
Author: mpickard-dataprof (NONE)
State: open | Created: 2024-04-04 16:46:13

### Describe the bug

The `to_csv()` method does not output commas in lists.  So when the Dataset is loaded back in the data structure of the column with a list is not correct.

Here's an example:

Obviously, it's not as trivial as inserting commas in the list, since its a comma-separated file.  But hopefully there's a way to export the list in a way that it'll be imported by `load_dataset()` correctly.

### Steps to reproduce the bug

Here's some code to reproduce the bug:


```python
from datasets import Dataset

ds = Dataset.from_dict(
    {
        "pokemon": ["bulbasaur", "squirtle"],
        "type": ["grass", "water"]
    }
)

def ascii_to_hex(text):
    return [ord(c) for c in text]

ds = ds.map(lambda x: {"int": ascii_to_hex(x['pokemon'])})

ds.to_csv('../output/temp.csv')
```

temp.csv then contains:
```

### Expected behavior

ACTUAL OUTPUT:
```
pokemon,type,int
bulbasaur,grass,[ 98 117 108  98  97 115  97 117 114]
squirtle,water,[115 113 117 105 114 116 108 101]
```

EXPECTED OUTPUT:
```
pokemon,type,int
bulbasaur,grass,[98, 117, 108,  98,  97, 115,  97, 117, 114]
squirtle,water,[115, 113, 117, 105, 114, 116, 108, 101]
```

or probably something more like this since it's a CSV file:
```
pokemon,type,int
bulbasaur,grass,"[98, 117, 108,  98,  97, 115,  97, 117, 114]"
squirtle,water,"[115, 113, 117, 105, 114, 116, 108, 101]"
```

### Environment info

### Package Version
Name: datasets
Version: 2.16.1

### Python
version: 3.10.12


### OS Info
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
...
UBUNTU_CODENAME=jammy