liuqingwen/github-issues huggingface dataset:
Load Data Sets Too Slow In Train Seq2seq Model

### Describe the bug


step 'Generating train split' in load_dataset is too slow：
![image](https://github.com/huggingface/datasets/assets/19569322/d9b08eee-95fe-4741-a346-b70416c948f8)


### Steps to reproduce the bug

Data： own data，16K16B Mono wav
Oficial Script:[ run_speech_recognition_seq2seq.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py)
Add Code：
    if data_args.data_path is not None:
        print(data_args.data_path)
        raw_datasets = load_dataset("audiofolder", data_dir=data_args.data_path, cache_dir=model_args.cache_dir)
        raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
        raw_datasets = raw_datasets["train"].train_test_split(test_size=0.005, shuffle=True)
 （change cache_dir to other path ，ex:/DATA/cache）

### Expected behavior

load data fast,at least 1000+
`Generating train split: 387875 examples [32:24:45, 1154.83 examples/s]`

### Environment info


- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-149-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Comments:
Hi ! you can speed it up using multiprocessing by passing `num_proc=` to `load_dataset()`

already did，but not useful for step Generating train split，it works in step "Resolving data files" & "Downloading data files" 

@mariosasko some advice ， thanks！

I met the same problem,  terrible  experience
Modify levels of some logging messages

Some warning messages didn't quite sound like warnings so I modified their logging levels to info.

Comments:
Length of table does not accurately reflect the split
Labels: name: wontfix
description: This will not be worked on

### Describe the bug

I load a Huggingface Dataset and do `train_test_split`. I'm expecting the underlying table for the dataset to also be split, but it's not.

### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/8068268/83e5768f-8b4c-422a-945c-832a7585afff)


### Expected behavior

The expected behavior is when `len(hf_dataset["train"].data)` should match the length of the train split, and not be the entire unsplit dataset.

### Environment info

datasets 2.10.1
python 3.10.11

Comments:
As already replied by @lhoestq (private channel):
> `.train_test_split` (as well as `.shard`, `.select`) doesn't create a new arrow table to save time and disk space. Instead, it uses an indices mapping on top of the table that locate which examples are part of train or test.

This is an optimization that we don't plan to "fix", so I'm closing this issue.
add support for storage_options for load_dataset API

to solve the issue in #5880 

1.  add s3 support in the link check step, previous we only check `http` and `https`,

2. change the parameter of `use_auth_token` to `download_config` to support both `storage_options` and `use_auth_token` parameter when trying to handle(list, open, read, etc,.) the remote files.

3.  integrate the check part's duplicate code to make adding or deleting other sources easier.

Comments:
hi @lhoestq,
I saw some errors in my test and found all the failed reasons are `FileNotFoundError`  about `test_load_streaming_private_dataset_with_zipped_data` and `test_load_dataset_private_zipped_images` in `test_load.py `, I run pytest on my own Wins and Ubuntu system all the test in `test_load.py ` are succeed. could you help me to check the test environment of our server?

`2023-06-08T16:50:48.0828281Z FAILED tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_txt_data-16862429577813\repo_zipped_txt_data-16862429577813.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813`
`2023-06-08T16:50:48.0830602Z FAILED tests/test_load.py::test_load_dataset_private_zipped_images[False-False] - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_img_data-16862429594168\repo_zipped_img_data-16862429594168.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168`

I just re-ran the CI, hopefully it's fixed

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5919). All of your documentation changes will be reflected on that endpoint.

> I just re-ran the CI, hopefully it's fixed

I just checked, still has the same error, maybe need someone to fix it

I think the issue comes from this PR somehow, since the CI fail is related to loading private repositories and this PR touches authentication related code. Let me check what's the issue, and I'll also review your PR later (sorry I don't have a ton of bandwidth atm)
File not found for audio dataset

### Describe the bug

After loading an audio dataset, and looking at a sample entry, the `path` element, which is supposed to be the path to the audio file, doesn't actually exist.


### Steps to reproduce the bug

Run bug.py:

```py
import os.path

from datasets import load_dataset

def run() -> None:
    cv13 = load_dataset(
        "mozilla-foundation/common_voice_13_0",
        "hi",
        split="train",
    )

    print(cv13[0])
    audio_file = cv13[0]["path"]
    if not os.path.exists(audio_file):
        raise ValueError(f'File {audio_file} does not exist.')

if __name__ == "__main__":
    run()
```

The result (on my machine):

```json
{'client_id': '0f018a99663f33afbb7d38aee281fb1afcfd07f9e7acd00383f604e1e17c38d6ed8adf1bd2ccbf927a52c5adefb8ac4b158ce27a7c2ed9581e71202eb302dfb3', 'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'audio': {'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'array': array([ 6.46234854e-26, -1.35709319e-25, -8.07793567e-26, ...,
        1.06425944e-07,  4.46417090e-08,  2.61451660e-09]), 'sampling_rate': 48000}, 'sentence': 'हमने उसका जन्मदिन मनाया।', 'up_votes': 2, 'down_votes': 0, 'age': '', 'gender': '', 'accent': '', 'locale': 'hi', 'segment': '' ', 'variant': ''}
```

```txt
Traceback (most recent call last):
  File "F:\eo-reco\bug.py", line 18, in <module>
    run()
  File "F:\eo-reco\bug.py", line 15, in run
    raise ValueError(f'File {audio_file} does not exist.')
ValueError: File C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\common_voice_hi_26008353.mp3 does not exist.
```


### Expected behavior

The `path` element points to the correct file, which happens to be:

```
C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\hi_train_0\common_voice_hi_26008353.mp3
```

That is, there's an extra directory `hi_train_0` that is not in the `path` element.


### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
- 

Comments:
load_dataset () did not work for loading local files either
Cannot use both set_format and set_transform

### Describe the bug

I need to process some data using the set_transform method but I also need the data to be formatted for pytorch before processing it.

I don't see anywhere in the documentation something that says that both methods cannot be used at the same time.

### Steps to reproduce the bug

```
from datasets import load_dataset
ds = load_dataset("mnist", split="train")
ds.set_format(type="torch")
def transform(entry):
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```

### Expected behavior

It should print the pytorch tensor image as a double, but it errors because "entry" in the transform function doesn't receive a pytorch tensor to begin with, it receives a PIL Image -> entry.double() errors because entry isn't a pytorch tensor.

### Environment info
Latest versions.


### Note:
It would be at least handy to have access to a function that can do the dataset.set_format in the set_transform function.

Something like:
```
from datasets import load_dataset, do_format
ds = load_dataset("mnist", split="train")
def transform(entry):
    entry = do_format(entry, type="torch")
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```

Comments:
Currently, it's not possible to chain `set_format`/`set_transform` calls (plus, this is a breaking change if we decide to implement it), so I see two possible solutions:
* using `set_format`/`set_transform` for the 1st transform and then passing the transformed example/batch to the 2nd transform
* implementing and registering a custom formatter (the relevant code is [here](https://github.com/huggingface/datasets/tree/main/src/datasets/formatting))

Btw, your example requires a single `set_format` call:
```python
ds.set_format("torch", columns=["image"], output_all_columns=True, dtype=torch.double)
```

Hey Mario,
Thanks, for getting back to me. the toDouble was just an example my real life case requires many more transforms.

What do you mean by:
> using set_format/set_transform for the 1st transform and then passing the transformed example/batch to the 2nd transform

How would that go, I thought you can't chain them?

As for the custom formatter, is it possible to reference an existing formatter, in my case `torch_formatter` inside of my custom formatter?

maybe I can inherit from it and just call `super.recursive_tensorize()`?

> How would that go, I thought you can't chain them?

Yes, they cannot be chained. This is what I meant:
```python
ds.set_transform(first_transform)
# calling the 2nd transform on each accessed batch
second_transform(ds[2:3])
```

> As for the custom formatter, is it possible to reference an existing formatter, in my case torch_formatter inside of my custom formatter?
>
>maybe I can inherit from it and just call super.recursive_tensorize()?

Yes, subclassing makes the most sense.

Great, thank you for the details.
Relax `ci.yml` trigger for `pull_request` based on modified paths

## What's in this PR?

As of a previous PR at #5902, I've seen that the CI was automatically trigger on any file, in that case when modifying a Jupyter Notebook (.ipynb), which IMO could be skipped, as the modification on the Jupyter Notebook has no effect/impact on the `ci.yml` outcome. So this PR controls the paths that trigger the `ci.yml` to avoid wasting resources when not needed.

## What's pending in this PR?

I would like to confirm whether this should affect both `push` and `pull_request`, since just modifications in those files won't change the `ci.yml` outcome, so maybe it's worth skipping it too in the `push` trigger.

Comments:
Also this could be extended to the rest of the GitHub Action `yml` files, so let me know whether you want me to have a look into it! 🤗

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5903). All of your documentation changes will be reflected on that endpoint.
HuggingFace does not cache downloaded files aggressively/early enough

### Describe the bug

I wrote the following script:

```
import datasets

dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
```

I ran it and spent 90 minutes downloading a 20GB file. Then I saw:

```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.3G/20.3G [1:30:29<00:00, 3.73MB/s]
Traceback (most recent call last):
  File "/home/jack/Code/Projects/Transformers/Codebase/main.py", line 5, in <module>
    dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 883, in download_and_prepare
    self._save_info()
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 2037, in _save_info
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```

And the 20GB of data was seemingly instantly gone forever, because when I ran the script again, it had to do the download again.

### Steps to reproduce the bug

See above

### Expected behavior

See above

### Environment info

datasets 2.10.1
Python 3.10

Comments:
User access requests with manual review do not notify the dataset owner

### Describe the bug

When a user access requests are enabled, and new requests are set to Manual Review, the dataset owner should be notified of the pending requests. However, instead, currently nothing happens, and so the dataset request can go unanswered for quite some time until the owner happens to check that particular dataset's Settings pane.

### Steps to reproduce the bug

1. Enable a dataset's user access requests
2. Set to Manual Review
3. Ask another HF user to request access to the dataset
4. Dataset owner is not notified

### Expected behavior

The dataset owner should receive some kind of notification, perhaps in their HF site inbox, or by email, when a dataset access request is made and manual review is enabled.

### Environment info

n/a

Comments:
cc @SBrandeis
HuggingsFace dataset example give error

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/1328316/1f4f0086-3db9-4c79-906b-05a375357cce)


![image](https://github.com/huggingface/datasets/assets/1328316/733ebd3d-89b9-4ece-b80a-00ab5b0a4122)


### Steps to reproduce the bug

Use link as reference document written https://colab.research.google.com/github/huggingface/datasets/blob/main/notebooks/Overview.ipynb#scrollTo=biqDH9vpvSVz

```python
# Now let's train our model
device = 'cuda' if torch.cuda.is_available() else 'cpu'

model.train().to(device)
for i, batch in enumerate(dataloader):
    batch.to(device)
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    model.zero_grad()
    print(f'Step {i} - loss: {loss:.3}')
    if i > 5:
        break
```

Error

```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-44-7040b885f382>](https://localhost:8080/#) in <cell line: 5>()
      5 for i, batch in enumerate(dataloader):
      6     batch.to(device)
----> 7     outputs = model(**batch)
      8     loss = outputs.loss
      9     loss.backward()

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: DistilBertForQuestionAnswering.forward() got an unexpected keyword argument 'token_type_ids'
```

https://github.com/huggingface/datasets/assets/1328316/5d8b1d61-9337-4d59-8423-4f37f834c156



### Expected behavior

Run success on Google Colab (free)

### Environment info

Windows 11 x64, Google Colab free (my Google Drive just empty about 200 MB, but I don't think it cause problem)

Comments:
Nice catch @donhuvy, that's because some models don't need the `token_type_ids`, as in this case, as the example is using `distilbert-base-cased`, and according to the DistilBert documentation at https://huggingface.co/transformers/v3.0.2/model_doc/distilbert.html, `DistilBert doesn’t have token_type_ids, you don’t need to indicate which token belongs to which segment. Just separate your segments with the separation token tokenizer.sep_token (or [SEP])`. `token_type_ids` are neither required in some other well known models such as RoBERTa. 

Here the issue comes due to a mismatch between the tokenizer and the model, as the Colab is using a BERT tokenizer (`bert-base-cased`), while the model is a DistilBERT (`distilbert-base-cased`), so aligning the tokenizer and the model solves it!

#self-assign

@donhuvy I've created https://github.com/huggingface/datasets/pull/5902 to solve it! 🤗
Use work-stealing algorithm when parallel computing
Labels: name: enhancement
description: New feature or request

### Feature request

when i used Dataset.map api to process data concurrently, i found that
 it gets slower and slower as it gets closer to completion. Then i read the source code of arrow_dataset.py and found that it shard the dataset and use multiprocessing pool to execute each shard.It may cause the slowest task to drag out the entire program's execution time,especially when processing huge dataset.

### Motivation

using work-stealing algorithm instead of sharding and parallel computing to optimize performance. 

### Your contribution

just an idea.

Comments:
Alternatively we could set the number of shards to be a factor than the number of processes (current they're equal) - this way it will be less likely to end up with a shard that is significantly slower than all the other ones.
Prefetching for IterableDataset
Labels: name: enhancement
description: New feature or request

### Feature request

Add support for prefetching the next n batches through iterabledataset to reduce batch loading bottleneck in training loop.

### Motivation

The primary motivation behind this is to use hardware accelerators alongside a streaming dataset. This is required when you are in a low ram or low disk space setting as well as quick iteration where you're iterating though different accelerator environments (e.x changing ec2 instances quickly to figure out batch/sec for a particular architecture). 

Currently, using the IterableDataset results in accelerators becoming basically useless due to the massive bottleneck induced by the dataset lazy loading/transform/mapping.

I've considered two alternatives:
PyTorch dataloader that handles this. However, I'm using jax, and I believe this is a piece of functionality that should live in the stream class.

Replicating the "num_workers" part of the PyTorch DataLoader to eagerly load batches and apply the transform so Arrow caching will automatically cache results and make them accessible.

### Your contribution

I may or may not have time to do this. Currently, I've written the basic multiprocessor approach to handle the eager DataLoader for my own use case with code that's not integrated to datasets. I'd definitely see this as being the default over the regular Dataset for most people given that they wouldn't have to wait on the datasets while also not worrying about performance.

Comments:
Very cool! Do you have a link to the code that you're using to eagerly fetch the data? Would also be interested in hacking around something here for pre-fetching iterable datasets

I ended up just switching back to the pytorch dataloader and using it's multiprocessing functionality to handle this :(. I'm just not that familiar with python multiprocessing to get something to work in jupyter (kept having weird behaviors happening with zombies living after the cell finished).

Ultimately settled on using webdataset to circumvent huggingface datasets entirely. Would definitely switch back if: https://github.com/huggingface/datasets/issues/5337 was resolved.

Hi! You can combine `datasets` with `torchdata` to prefetch `IterableDataset`'s samples:
```python
from datasets import load_dataset
from torchdata.datapipes.iter import IterableWrapper, HuggingFaceHubReader
from torch.utils.data import DataLoader

ds = load_dataset("sst", split="train", streaming=True)
# processing...
dp = IterableWrapper(ds)
dp = dp.prefetch(100)
dl = DataLoader(dp, batch_size=8)

i = iter(dl)
next(i)
```

Hey @mariosasko! Thanks for the tip here - introducing prefetch with `torchdata` didn't really give me any performance difference vs not prefetching, but the concept is definitely one that could be really beneficial. Are there any benchmarks that show the speed-up you can get with `torchdata`'s prefetch just for comparison?
Request for text deduplication feature
Labels: name: enhancement
description: New feature or request

### Feature request

It would be great if there would be support for high performance, highly scalable text deduplication algorithms as part of the datasets library.

### Motivation

Motivated by this blog post https://huggingface.co/blog/dedup and this library https://github.com/google-research/deduplicate-text-datasets, but slightly frustrated by how its not very easy to work with these tools I am proposing this feature.

### Your contribution

I would be happy to contribute to the development effort of this feature. would love to collaborate with others in the development effort. 

Comments:
The "exact match" deduplication will be possible when we resolve https://github.com/huggingface/datasets/issues/2514 (first, https://github.com/apache/arrow/issues/30950 needs to be addressed on the Arrow side). In the meantime, you can use Polars or DuckDB (e.g., via [datasets-sql](https://github.com/mariosasko/datasets_sql)).

Fuzzy deduplication is out-of-scope for now ([splink](https://github.com/moj-analytical-services/splink) is probably the best tool for it).

This library can be an intermediate solution : https://github.com/ChenghaoMou/text-dedup/tree/main
Why split slicing doesn't behave like list slicing ?
Labels: name: duplicate
description: This issue or pull request already exists

### Describe the bug

If I want to get the first 10 samples of my dataset, I can do :

```
ds = datasets.load_dataset('mnist', split='train[:10]')
```

But if I exceed the number of samples in the dataset, an exception is raised : 

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

> ValueError: Requested slice [:999999999] incompatible with 60000 examples.

### Steps to reproduce the bug

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

### Expected behavior

I would expect it to behave like python lists (no exception raised, the whole list is kept) : 

```
d = list(range(1000))[:999999]
print(len(d))  # > 1000
```

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments:
A duplicate of https://github.com/huggingface/datasets/issues/1774