SurAyush/github_issues huggingface dataset:
Issue: Write token isn't working while pushing to datasets
State: closed
Author: zaibutcooler

### Describe the bug

<img width="1001" alt="Screenshot 2024-05-01 at 3 37 06 AM" src="https://github.com/huggingface/datasets/assets/130903099/00fcf12c-fcc1-4749-8592-d263d4efcbcc">

As you can see I logged in to my account and the write token is valid.

But I can't upload on my main account and I am getting that error. It was okay on my test account at first try.

(I refreshed the token, tried a new token but still doesn't work)

### Steps to reproduce the bug

1. I loaded a dataset.

2. I logged in using both cli and huggingface_hub

3. I pushed to my down dataset
(It went well without any issues on my test account)

### Expected behavior

It should have gone smoothly and this is not even my first time uploading to huggingface datasets

### Environment info

colab, dataset (tried multiple versions)

Comments:
Issue: Problem loading voxpopuli dataset 
State: closed
Author: Namangarg110

### Describe the bug

```
Exception has occurred: FileNotFoundError
Couldn't find file at https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/{'en': 'data/en/asr_train.tsv'}
```

Error in logic for link url creation. The link should be  https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/en/asr_train.tsv 

Basically there should be links directly under ```metadata["train"]```, not under ```metadata["train"][self.config.languages[0]]```

same for audio urls

### Steps to reproduce the bug

```
from datasets import load_dataset

dataset = load_dataset("facebook/voxpopuli","en")
```

### Expected behavior

Dataset should be loaded successfully.

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.15.0-1041-aws-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.12.2

Comments:
Version 2.18 works without problem.

@Namangarg110  @mohsen-goodarzi The bug appears because the number of urls is less than 16 and the algorithm is meant to work on the previously created mode for a single url as stated on line 314: https://github.com/huggingface/datasets/blob/1bf8a46cc7b096d5c547ea3794f6a4b6c31ea762/src/datasets/download/download_manager.py#L314

In addition, previously `map_nested` function was supported without batching and it is meant to be the default performance. 

One of the shortest walk-arounds would be changing the part of the manager with the current setting:
```
        if len(url_or_urls) >= 16:
            download_func = partial(self._download_batched, download_config=download_config)
        else:
            download_func = partial(self._download_single, download_config=download_config)

        start_time = datetime.now()
        with stack_multiprocessing_download_progress_bars():
            downloaded_path_or_paths = map_nested(
                download_func,
                url_or_urls,
                map_tuple=True,
                num_proc=download_config.num_proc,
                desc="Downloading data files",
                batched=True if len(url_or_urls) >= 16 else False,
                batch_size=-1,
            )
```

I would suggest to consider other datasets for similar issues and make a pull-request. 

Thanks for reporting @Namangarg110 and thanks for the investigation @MilanaShhanukova.

Apparently, there is an issue with the download functionality.
I am proposing a fix.
Issue: fix webdataset filename split
State: closed
Author: Bowser1704

use `os.path.splitext` to parse field_name.

fix filename which has dot. like:

```
a.b.jpeg
a.b.txt
```

Comments:
Hi ! This was fixed recently in https://github.com/huggingface/datasets/pull/6904 and https://github.com/huggingface/datasets/pull/6931
Issue: [Streaming] Only load requested splits without resolving files for the other splits
State: open
Author: lhoestq

e.g. [thangvip](https://huggingface.co/thangvip)/[cosmopedia_vi_math](https://huggingface.co/datasets/thangvip/cosmopedia_vi_math) has 300 splits and it takes a very long time to load only one split.

This is due to `load_dataset()` resolving the files of all the splits even if only one is needed.

In `dataset-viewer` the splits are loaded in different jobs so it results in 300 jobs that resolve 300 splits -> 90k calls to `/paths-info`

Comments:
This should help fixing this issue: https://github.com/huggingface/datasets/pull/6832

I'm having a similar issue when using splices:
<img width="947" alt="image" src="https://github.com/huggingface/datasets/assets/28941213/2153faac-e1fe-4b6d-a79b-30b2699407e8">
<img width="823" alt="image" src="https://github.com/huggingface/datasets/assets/28941213/80919eca-eb6c-407d-8070-52642fdcee54">
<img width="914" alt="image" src="https://github.com/huggingface/datasets/assets/28941213/5219c201-e22e-4536-acc3-a922677785ff">


It seems to be downloading, loading, and generating splits using the entire dataset.
Issue: Unimaginable super slow iteration
State: closed
Author: rangehow

### Describe the bug

Assuming there is a dataset with 52000 sentences, each with a length of 500, it takes 20 seconds to extract a sentence from the dataset……？Is there something wrong with my iteration?

### Steps to reproduce the bug

```python
import datasets
import time
import random

num_rows = 52000
num_cols = 500

random_input = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]
random_output = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]


s=time.time()
d={'random_input':random_input,'random_output':random_output}
dataset=datasets.Dataset.from_dict(d)
print('from dict',time.time()-s)
print(dataset)


for i in range(len(dataset)):
    aa=time.time()
    a,b=dataset['random_input'][i],dataset['random_output'][i]
    print(time.time()-aa)

```

corresponding output
```bash
from dict 9.215498685836792
Dataset({
    features: ['random_input', 'random_output'],
    num_rows: 52000
})
19.129778146743774
19.329464197158813
19.27668261528015
19.28557538986206
19.247620582580566
19.624247074127197
19.28673791885376
19.301053047180176
19.290496110916138
19.291821718215942
19.357765197753906

```

### Expected behavior

Under normal circumstances, iteration should be very rapid as it does not involve the main tasks other than getting items

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0

Comments:
In every iteration you load the full "random_input" column in memory, only then to access it's i-th element.

You can try using this instead

a,b=dataset[i]['random_input'],dataset[i]['random_output']
Issue: Retry on HF Hub error when streaming
State: closed
Author: mariosasko

Retry on the `huggingface_hub`'s `HfHubHTTPError` in the streaming mode.

Fix #6843 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6844). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Wauplin This PR is indeed not needed as explained in https://github.com/huggingface/datasets/issues/6843#issuecomment-2079630389. 

So, I'm closing it.
Issue: Datasets with files with colon : in filenames cannot be used on Windows
State: open
Author: jacobjennings

### Describe the bug

Datasets (such as https://huggingface.co/datasets/MLCommons/peoples_speech) cannot be used on Windows due to the fact that windows does not allow colons ":" in filenames. These should be converted into alternative strings.



### Steps to reproduce the bug

1. Attempt to run load_dataset on MLCommons/peoples_speech

### Expected behavior

Does not crash during extraction

### Environment info

Windows 11, NTFS filesystem, Python 3.12


Comments:
Issue: Delete uploaded files from the UI
State: open
Author: saicharan2804

### Feature request

Once a file is uploaded and the commit is made, I am unable to delete individual files without completely deleting the whole dataset via the website UI.

### Motivation

Would be a useful addition

### Your contribution

Would love to help out with some guidance

Comments:
This is super late, but if you click on any directory, you can delete the directory using a "Delete Directory" button in the top right of the interface, and similarly, if you click on any file, you can delete the file using a "Delete File" button in the top right.
Issue: ExpectedMoreSplits error on load_dataset when upgrading to 2.19.0
State: open
Author: ebsmothers

### Describe the bug

Hi there, thanks for the great library! We have been using it a lot in torchtune and it's been a huge help for us. 

Regarding the bug: the same call to `load_dataset` errors with `ExpectedMoreSplits` in 2.19.0 after working fine in 2.18.0. Full details given in the repro below.

### Steps to reproduce the bug

On 2.18.0, things work fine:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.18.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

On 2.19.0, they do not:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.19.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

The stack trace I see from the 2.19.0 version of load_dataset can be seen [here](https://gist.github.com/ebsmothers/f9b1f1949bee7030a8d7bb8a491550d2).

(Maybe unsurprising but) notably if I do not delete the cache first I am able to load the dataset successfully. So based on this I suspect the cause is somewhere in the download logic.

### Expected behavior

Download the dataset successfully :)

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34
- Python version: 3.11.9
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1

Comments:
Get same error on same datasets too.

+1

same error
Issue: Support pyarrow LargeListType
State: closed
Author: Modexus

Fixes #6834 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fixed the conversion from `pyarrow` to `python` `Sequence` features. 

There is still an issue that if `features` are passed the `Sequence` always forces conversion to `ListArray`.
This probably causes issues if the `LargeListArray` is actually needed.

There doesn't seem to be a great solution since this list is created solely on the `schema` for `Sequence`.
One solution would be to always use `LargeListArray` instead.


I am retaking this PR because we would like to have this feature implemented.
Issue: largelisttype not supported (.from_polars())
State: closed
Author: Modexus

### Describe the bug

The following code fails because LargeListType is not supported.
This is especially a problem for .from_polars since polars uses LargeListType.


### Steps to reproduce the bug

```python
import datasets
import polars as pl

df = pl.DataFrame({"list": [[]]})
datasets.Dataset.from_polars(df)
```

### Expected behavior

Convert LargeListType to list.

### Environment info

- `datasets` version: 2.19.1.dev0
- Platform: Linux-6.8.7-200.fc39.x86_64-x86_64-with-glibc2.38
- Python version: 3.12.2
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2024.3.1

Comments:
Issue: Support downloading specific splits in `load_dataset`
State: open
Author: mariosasko

This PR builds on https://github.com/huggingface/datasets/pull/6639 to support downloading only the specified splits in `load_dataset`. For this to work, a builder's `_split_generators` need to be able to accept the requested splits (as a list) via a `splits` argument to avoid processing the non-requested ones. Also, the builder has to define a `_available_splits` method that lists all the possible `splits` values.

Close https://github.com/huggingface/datasets/issues/4101, close https://github.com/huggingface/datasets/issues/2538 (I'm probably missing some)

Should also make it possible to address https://github.com/huggingface/datasets/issues/6793

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6832). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Friendly ping on this! This feature would be really helpful and useful to me (and likely others with limited download speed and storage space!). Thanks so much!

No one is working on this atm afaik :/

No worries! I've patched the ImageNet dataset in: <https://huggingface.co/datasets/ILSVRC/imagenet-1k/blob/refs%2Fpr%2F20/imagenet-1k.py> 

Together with:

```python
dataset = load_dataset(
        "ILSVRC/imagenet-1k",
        split="validation",
        data_files={"val": "data/val_images.tar.gz"},
        revision="refs/pr/20",
        trust_remote_code=True,
        download_config=DownloadConfig(resume_download=True),
        verification_mode=VerificationMode.NO_CHECKS,
    )
```

It only downloads the validation set this way (NO_CHECKS is a bit annoying because I'd rather have md5 checks, but I guess I can't have everything) ^^' The patch is not perfect, but it does the job.
Issue: Add docs about the CLI
State: closed
Author: albertvillanova

Add docs about the CLI.

Close #6830.

CC: @severo 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6831). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Concretely, the docs about convert_to_parquet are here: https://moon-ci-docs.huggingface.co/docs/datasets/pr_6831/en/cli#convert-to-parquet

There is an issue with the example snippet when copy/pasting it: the leading shell dollar sign is also copied. I guess they will not like to fix it in the backend: currently they only support Python code snippets (with leading `>>>` or `...`), as they appear in the IPython interactive console.

What do you suggest, @severo?
Issue: Load and save from/to disk no longer accept pathlib.Path
State: open
Author: albertvillanova

Reported by @vttrifonov at https://github.com/huggingface/datasets/pull/6704#issuecomment-2071168296:
> This change is breaking in 
> https://github.com/huggingface/datasets/blob/f96e74d5c633cd5435dd526adb4a74631eb05c43/src/datasets/arrow_dataset.py#L1515
> when the input is `pathlib.Path`. The issue is that `url_to_fs` expects a `str` and cannot deal with `Path`. `get_fs_token_paths` converts to `str` so it is not a problem

This change was introduced in:
- #6704 

Comments:
Issue: Loading a remote dataset fails in the last release (v2.19.0)
State: open
Author: zrthxn

While loading a dataset with multiple splits I get an error saying `Couldn't find file at <URL>`

I am loading the dataset like so, nothing out of the ordinary.
This dataset needs a token to access it.
```
token="hf_myhftoken-sdhbdsjgkhbd"
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token=token)
```

I get the following error
![Screenshot 2024-04-19 at 11 03 07 PM](https://github.com/huggingface/datasets/assets/35369637/8dce757f-08ff-45dd-85b5-890fced7c5bc)

Now you can see that the URL that it is trying to reach has the JSON object of the dataset split appended to the base URL. I think this may be due to a newly introduced issue.

I did not have this issue with the previous version of the datasets. Everything was fine for me yesterday and after the release 12 hours ago, this seems to have broken. Also, the dataset in question runs custom code and I checked and there have been no commits to the dataset on Huggingface in 6 months.

### Steps to reproduce the bug

Since this happened with one particular dataset for me, I am listing steps to use that dataset.
1. Open https://huggingface.co/datasets/speechcolab/gigaspeech and fill the form to get access.
2. Create a token on your huggingface account with read access.
3. Run the following line, substituing `<your_token_here>` with your token.
```
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token="<your_token_here>")
```

### Expected behavior

Be able to load the dataset in question.

### Environment info

datasets == 2.19.0
python == 3.10
kernel == Linux 6.1.58+

Comments:
Issue: Give more details in `DataFilesNotFoundError` when getting the config names
State: open
Author: severo

### Feature request

After https://huggingface.co/datasets/cis-lmu/Glot500/commit/39060e01272ff228cc0ce1d31ae53789cacae8c3, the dataset viewer gives the following error:

```
{
  "error": "Cannot get the config names for the dataset.",
  "cause_exception": "DataFilesNotFoundError",
  "cause_message": "No (supported) data files found in cis-lmu/Glot500",
  "cause_traceback": [
    "Traceback (most recent call last):\n",
    " File \"/src/services/worker/src/worker/job_runners/dataset/config_names.py\", line 73, in compute_config_names_response\n config_names = get_dataset_config_names(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py\", line 347, in get_dataset_config_names\n dataset_module = dataset_module_factory(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1873, in dataset_module_factory\n raise e1 from None\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1854, in dataset_module_factory\n return HubDatasetModuleFactoryWithoutScript(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1245, in get_module\n module_name, default_builder_kwargs = infer_module_for_data_files(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 595, in infer_module_for_data_files\n raise DataFilesNotFoundError(\"No (supported) data files found\" + (f\" in {path}\" if path else \"\"))\n",
    "datasets.exceptions.DataFilesNotFoundError: No (supported) data files found in cis-lmu/Glot500\n"
  ]
}
```

because the deleted files were still listed in the README, see https://huggingface.co/datasets/cis-lmu/Glot500/discussions/4

Ideally, the error message would include the name of the first configuration with missing files, to help the user understand how to fix it. Here, it would tell that configuration `aze_Ethi` has no supported data files, instead of telling that the `cis-lmu/Glot500` *dataset* has no supported data files (which is not true).

### Motivation

Giving more detail in the error would help the Datasets Hub users to debug why the dataset viewer does not work.

### Your contribution

Not sure how to best fix this, as there are a lot of loops on the dataset configs in the traceback methods. "maybe" it would be easier to handle if the code was completely isolating each config.

Comments:
Issue: Improve typing of Dataset.search, matching definition
State: closed
Author: Dref360

Previously, the output of `score, indices = Dataset.search(...)` would be numpy arrays.

The definition in `SearchResult` is a `List[int]` so this PR now matched the expected type.

The previous behavior is a bit annoying as `Dataset.__getitem__` doesn't support `numpy.int64` which forced me to convert `indices` to int eg:

```python
score, indices = ds.search(...)
item = ds[int(indices[0])]
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6816). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi! This is a breaking change. A better solution is to check for "indexable" types in `__getitem__` to support keys such as `np.int64`:
```python
import operator

def _query_table_with_indices_mapping(...):  # or _query_table
    ...
    try:
        operator.index(key)
    except TypeError:
        pass
 
    _raise_bad_key_type(key)
```

Sounds good! We should still update type annotations for SearchResult in my opinion.
Issue: `map` with `num_proc` > 1 leads to OOM
State: open
Author: bhavitvyamalik

### Describe the bug

When running `map` on parquet dataset loaded from local machine, the RAM usage increases linearly eventually leading to OOM. I was wondering if I should I save the `cache_file` after every n steps in order to prevent this?

### Steps to reproduce the bug



```
ds = load_dataset("parquet", data_files=dataset_path, split="train")
ds = ds.shard(num_shards=4, index=0)

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
ds = ds.map(prepare_dataset,
            num_proc=32,
            writer_batch_size=1000,
            keep_in_memory=False,
            desc="preprocess dataset")
```

```
def prepare_dataset(batch):
    # load audio
    sample = batch["audio"]

    inputs = feature_extractor(sample["array"], sampling_rate=16000)
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(sample["array"].squeeze())

    return batch
```

### Expected behavior

It shouldn't run into OOM problem.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0

Comments:
Hi ! You can try to reduce `writer_batch_size`. It corresponds to the number of samples that stay in RAM before being flushed to disk
Issue: Allow deleting a subset/config from a no-script dataset
State: closed
Author: albertvillanova

As proposed by @BramVanroy, it would be neat to have this functionality through the API.

Comments:
Probably best to implement this as a CLI command?

Thanks for your comment, @mariosasko. Or maybe both (in Python and as CLI command)? The Python command would be just the reverse of `push_to_hub`...

I am working on a draft implementation, so we can discuss about the API and UX.

2nd time I've come here in the past few days so for reference this doesn't work for me at the moment (token auth error despite successful `whoami` and all auths applied) so I resorted to [reverting](https://discuss.huggingface.co/t/how-to-revert-to-an-earlier-commit-on-a-repo/65735/2) to a previous commit (after installing git lfs)

```
# git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/USER/REPO
cd REPO
git reset --hard HASH
git push --force
```
Issue: Make convert_to_parquet CLI command create script branch
State: closed
Author: albertvillanova

As proposed by @severo, maybe we should add this functionality as well to the CLI command to convert a script-dataset to Parquet. See: https://github.com/huggingface/datasets/pull/6795#discussion_r1562819168
> When providing support, we sometimes suggest that users store their script in a script branch. What do you think of this alternative to deleting the files?

Comments: