kardwalker/my_dataset huggingface dataset:
GitHub issue #5896: HuggingFace does not cache downloaded files aggressively/early enough
State: closed | Created: 2023-05-25T15:14:36Z

### Describe the bug

I wrote the following script:

```
import datasets

dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
```

I ran it and spent 90 minutes downloading a 20GB file. Then I saw:

```
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20.3G/20.3G [1:30:29<00:00, 3.73MB/s]
Traceback (most recent call last):
  File "/home/jack/Code/Projects/Transformers/Codebase/main.py", line 5, in <module>
    dataset = datasets.load.load_dataset("wikipedia", "20220301.en", split="train[:10000]")
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 883, in download_and_prepare
    self._save_info()
  File "/home/jack/.local/lib/python3.10/site-packages/datasets/builder.py", line 2037, in _save_info
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```

And the 20GB of data was seemingly instantly gone forever, because when I ran the script again, it had to do the download again.

### Steps to reproduce the bug

See above

### Expected behavior

See above

### Environment info

datasets 2.10.1
Python 3.10
GitHub issue #5895: The dir name and split strings are confused when loading ArmelR/stack-exchange-instruction dataset
State: closed | Created: 2023-05-25T09:39:06Z

### Describe the bug

When I load the ArmelR/stack-exchange-instruction dataset, I encounter a bug that may be raised by confusing the dir name string and the split string about the dataset. 

When I use the script "datasets.load_dataset('ArmelR/stack-exchange-instruction', data_dir="data/finetune", split="train", use_auth_token=True)", it fails. But it succeeds when I add the "streaming = True" parameter. 

The website of the dataset is https://huggingface.co/datasets/ArmelR/stack-exchange-instruction/ .

The traceback logs are as below: 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/builder.py", line 1706, in _prepare_split
    split_info = self.info.splits[split_generator.name]
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/splits.py", line 530, in __getitem__
    instructions = make_file_instructions(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/arrow_reader.py", line 112, in make_file_instructions
    name2filenames = {
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/arrow_reader.py", line 113, in <dictcomp>
    info.name: filenames_for_dataset_split(
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/naming.py", line 70, in filenames_for_dataset_split
    prefix = filename_prefix_for_split(dataset_name, split)
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/site-packages/datasets/naming.py", line 54, in filename_prefix_for_split
    if os.path.basename(name) != name:
  File "/home/xxx/miniconda3/envs/code/lib/python3.9/posixpath.py", line 142, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType





### Steps to reproduce the bug

1. import datasets library function: ```from datasets import load_dataset```
2. load dataset: ```ds=load_dataset('ArmelR/stack-exchange-instruction', data_dir="data/finetune", split="train", use_auth_token=True)```

### Expected behavior

The dataset can be loaded successfully without the streaming setting. 

### Environment info

Linux, 
python=3.9
datasets=2.12.0
GitHub issue #5892: User access requests with manual review do not notify the dataset owner
State: closed | Created: 2023-05-23T17:27:46Z

### Describe the bug

When a user access requests are enabled, and new requests are set to Manual Review, the dataset owner should be notified of the pending requests. However, instead, currently nothing happens, and so the dataset request can go unanswered for quite some time until the owner happens to check that particular dataset's Settings pane.

### Steps to reproduce the bug

1. Enable a dataset's user access requests
2. Set to Manual Review
3. Ask another HF user to request access to the dataset
4. Dataset owner is not notified

### Expected behavior

The dataset owner should receive some kind of notification, perhaps in their HF site inbox, or by email, when a dataset access request is made and manual review is enabled.

### Environment info

n/a
GitHub issue #5889: Token Alignment for input and output data over train and test batch/dataset.
State: open | Created: 2023-05-23T15:58:55Z

`data`
> DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: 4500
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: 500
    })
})

**# input (in-correct sentence)**
`data['train'][0]['input']`
**>>** 'We are meet sunday 10am12pmET in Crown Heights Brooklyn New York'
**# output (correct sentence)**
`data['train'][0]['output']`
**>>** 'We meet Sundays 10am-12pmET in Crown Heights, Brooklyn, New York.'

**I Want to align the output tokens with input**

```
`# tokenize both inputs and targets
def tokenize_fn(batch):
    # tokenize the input sequence first
    # this populates input_ids, attention_mask, etc.

    tokenized_inputs = tokenizer(
    batch['input']
    )
    
    labels_batch = tokenizer.tokenize(batch['output']) # original targets

    aligned_labels_batch = []
    for i, labels in enumerate(labels_batch):
        word_ids = tokenized_inputs[i].word_ids()
        aligned_labels_batch.append(align_targets(labels, word_ids))  # align_targets is another user defined function which is been called here

    # recall: the 'target' must be stored in key called 'labels'
    tokenized_inputs['labels'] = aligned_labels_batch

    return tokenized_inputs`
```
```
data.map(
  tokenize_fn,
  batched=True,
  remove_columns=data['train'].column_names,
)
```

When this user defined function is mapped to every records of train and test batch am getting following error:

**1.** **raise DatasetTransformationNotAllowedError(
   3457         "Using `.map` in batched mode on a dataset with attached indexes is allowed only if it doesn't create or remove existing examples. You can first run `.drop_index() to remove your index and then re-add it."**

**2.** **TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]**
GitHub issue #5887: HuggingsFace dataset example give error
State: closed | Created: 2023-05-23T14:09:05Z

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/1328316/1f4f0086-3db9-4c79-906b-05a375357cce)


![image](https://github.com/huggingface/datasets/assets/1328316/733ebd3d-89b9-4ece-b80a-00ab5b0a4122)


### Steps to reproduce the bug

Use link as reference document written https://colab.research.google.com/github/huggingface/datasets/blob/main/notebooks/Overview.ipynb#scrollTo=biqDH9vpvSVz

```python
# Now let's train our model
device = 'cuda' if torch.cuda.is_available() else 'cpu'

model.train().to(device)
for i, batch in enumerate(dataloader):
    batch.to(device)
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    model.zero_grad()
    print(f'Step {i} - loss: {loss:.3}')
    if i > 5:
        break
```

Error

```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-44-7040b885f382>](https://localhost:8080/#) in <cell line: 5>()
      5 for i, batch in enumerate(dataloader):
      6     batch.to(device)
----> 7     outputs = model(**batch)
      8     loss = outputs.loss
      9     loss.backward()

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: DistilBertForQuestionAnswering.forward() got an unexpected keyword argument 'token_type_ids'
```

https://github.com/huggingface/datasets/assets/1328316/5d8b1d61-9337-4d59-8423-4f37f834c156



### Expected behavior

Run success on Google Colab (free)

### Environment info

Windows 11 x64, Google Colab free (my Google Drive just empty about 200 MB, but I don't think it cause problem)
GitHub issue #5886: Use work-stealing algorithm when parallel computing
State: open | Created: 2023-05-23T03:08:44Z

### Feature request

when i used Dataset.map api to process data concurrently, i found that
 it gets slower and slower as it gets closer to completion. Then i read the source code of arrow_dataset.py and found that it shard the dataset and use multiprocessing pool to execute each shard.It may cause the slowest task to drag out the entire program's execution time,especially when processing huge dataset.

### Motivation

using work-stealing algorithm instead of sharding and parallel computing to optimize performance. 

### Your contribution

just an idea.
GitHub issue #5888: A way to upload and visualize .mp4 files (millions of them) as part of a dataset
State: open | Created: 2023-05-22T18:05:26Z

**Is your feature request related to a problem? Please describe.**
I recently chose to use huggingface hub as the home for a large multi modal dataset I've been building. https://huggingface.co/datasets/Antreas/TALI

It combines images, text, audio and video. Now, I could very easily upload a dataset made via datasets.Dataset.from_generator, as long as it did not include video files. I found that including .mp4 files in the entries would not auto-upload those files. 

Hence I tried to upload them myself. I quickly found out that uploading many small files is a very bad way to use git lfs, and that it would take ages, so, I resorted to using 7z to pack them all up. But then I had a new problem.

My dataset had a size of 1.9TB. Trying to upload such a large file with the default huggingface_hub API always resulted in time outs etc. So I decided to split the large files into chunks of 5GB each and reupload. 

So, eventually it all worked out. But now the dataset can't be properly and natively used by the datasets API because of all the needed preprocessing -- and furthermore the hub is unable to visualize things. 

**Describe the solution you'd like**
A native way to upload large datasets that include .mp4 or other video types.

**Describe alternatives you've considered**
Already explained earlier

**Additional context**
https://huggingface.co/datasets/Antreas/TALI
GitHub issue #5884: `Dataset.to_tf_dataset` fails when strings cannot be encoded as `np.bytes_`
State: closed | Created: 2023-05-22T12:03:06Z

### Describe the bug

When loading any dataset that contains a column with strings that are not ASCII-compatible, looping over those records raises the following exception e.g. for `é` character `UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)`.

### Steps to reproduce the bug

Running the following script will eventually fail, when reaching to the batch that contains non-ASCII compatible strings.

```python
from datasets import load_dataset

ds = load_dataset("imdb", split="train")
tfds = ds.to_tf_dataset(batch_size=16)

for batch in tfds:
  print(batch)
>>> UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)
```

### Expected behavior

The following script to run properly, making sure that the strings are either `numpy.unicode_` or `numpy.string` instead of `numpy.bytes_` since some characters are not ASCII compatible and that would lead to an issue when applying the `map`.

```python
from datasets import load_dataset

ds = load_dataset("imdb", split="train")
tfds = ds.to_tf_dataset(batch_size=16)

for batch in tfds:
  print(batch)
```

### Environment info

- `datasets` version: 2.12.1.dev0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5883: Fix string-encoding, make `batch_size` optional, and minor improvements in `Dataset.to_tf_dataset`
State: closed | Created: 2023-05-22T11:51:07Z

## What's in this PR?

This PR addresses some minor fixes and general improvements in the `to_tf_dataset` method of `datasets.Dataset`, to convert a 🤗HuggingFace Dataset as a TensorFlow Dataset.

The main bug solved in this PR comes with the string-encoding, since for safety purposes the internal conversion of `numpy.arrays` when `dtype` is unicode/string, is to convert it into `numpy.bytes`, more information in the docstring of https://github.com/tensorflow/tensorflow/blob/388d952114e59a1aeda440ed4737b29f8b7c6e8a/tensorflow/python/ops/script_ops.py#L210. That's triggered when using `tensorflow.numpy_function` as it's applying another type cast besides the one that `datasets` does, so the casting is applied at least twice per entry/batch. So this means that the definition of the `numpy.unicode_` dtype when the data in the batch is a string, is ignored, and replaced by `numpy.bytes_`.

Besides that, some other minor things have been fixed:

* Made `batch_size` an optional parameter in `to_tf_dataset`
* Map the `tensorflow` output dtypes just once, and not in every `tf.function` call during `map`
* Keep `numpy` formatting in the `datasets.Dataset` if already formatted like it, no need to format it again as `numpy`
* Docstring indentation in `dataset_to_tf` and `multiprocess_dataset_to_tf`

## What's missing in this PR?

I can include some integration tests if needed, to validate that `batch_size` is optional, and that the tensors in the TF-Dataset can be looped over with no issues as before.
GitHub issue #5880: load_dataset from s3 file system through streaming can't not iterate data 
State: open | Created: 2023-05-22T07:40:27Z

### Describe the bug

I have a JSON file in my s3 file system(minio), I can use load_dataset to get the file link, but I can't iterate it
<img width="816" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/cc0778d3-36f3-45b5-ac68-4e7c664c2ed0">
<img width="1144" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/76872af3-8b3c-42ff-9f55-528c920a7af1">

we can change 4 lines to fix this bug, you can check whether it is ok for us.
<img width="941" alt="image" src="https://github.com/huggingface/datasets/assets/59083384/5a22155a-ece7-496c-8506-047e5c235cd3">

### Steps to reproduce the bug

1. storage a file in you s3 file system
2. use load_dataset to read it through streaming
3. iterate it

### Expected behavior

can iterate it successfully

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
GitHub issue #5878: Prefetching for IterableDataset
State: open | Created: 2023-05-20T15:25:40Z

### Feature request

Add support for prefetching the next n batches through iterabledataset to reduce batch loading bottleneck in training loop.

### Motivation

The primary motivation behind this is to use hardware accelerators alongside a streaming dataset. This is required when you are in a low ram or low disk space setting as well as quick iteration where you're iterating though different accelerator environments (e.x changing ec2 instances quickly to figure out batch/sec for a particular architecture). 

Currently, using the IterableDataset results in accelerators becoming basically useless due to the massive bottleneck induced by the dataset lazy loading/transform/mapping.

I've considered two alternatives:
PyTorch dataloader that handles this. However, I'm using jax, and I believe this is a piece of functionality that should live in the stream class.

Replicating the "num_workers" part of the PyTorch DataLoader to eagerly load batches and apply the transform so Arrow caching will automatically cache results and make them accessible.

### Your contribution

I may or may not have time to do this. Currently, I've written the basic multiprocessor approach to handle the eager DataLoader for my own use case with code that's not integrated to datasets. I'd definitely see this as being the default over the regular Dataset for most people given that they wouldn't have to wait on the datasets while also not worrying about performance.
GitHub issue #5877: Request for text deduplication feature
State: open | Created: 2023-05-20T01:56:00Z

### Feature request

It would be great if there would be support for high performance, highly scalable text deduplication algorithms as part of the datasets library.

### Motivation

Motivated by this blog post https://huggingface.co/blog/dedup and this library https://github.com/google-research/deduplicate-text-datasets, but slightly frustrated by how its not very easy to work with these tools I am proposing this feature.

### Your contribution

I would be happy to contribute to the development effort of this feature. would love to collaborate with others in the development effort.
GitHub issue #5876: Incompatibility with DataLab
State: closed | Created: 2023-05-20T01:39:11Z

### Describe the bug

Hello,
I am currently working on a project where both [DataLab](https://github.com/ExpressAI/DataLab) and [datasets](https://github.com/huggingface/datasets) are subdependencies.
I noticed that I cannot import both libraries, as they both register FileSystems in `fsspec`, expecting the FileSystems not being registered before.

When running the code below, I get the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\datalabs\__init__.py", line 28, in <module>
    from datalabs.arrow_dataset import concatenate_datasets, Dataset
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\datalabs\arrow_dataset.py", line 60, in <module>
    from datalabs.arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\datalabs\arrow_writer.py", line 28, in <module>
    from datalabs.features import (
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\datalabs\features\__init__.py", line 2, in <module>
    from datalabs.features.audio import Audio
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\datalabs\features\audio.py", line 21, in <module>
    from datalabs.utils.streaming_download_manager import xopen
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\datalabs\utils\streaming_download_manager.py", line 16, in <module>
    from datalabs.filesystems import COMPRESSION_FILESYSTEMS
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\datalabs\filesystems\__init__.py", line 37, in <module>
    fsspec.register_implementation(fs_class.protocol, fs_class)
  File "C:\Users\Bened\anaconda3\envs\ner-eval-dashboard2\lib\site-packages\fsspec\registry.py", line 51, in register_implementation
    raise ValueError(
ValueError: Name (bz2) already in the registry and clobber is False
```

I think as simple solution would be to just set `clobber=True` in https://github.com/huggingface/datasets/blob/main/src/datasets/filesystems/__init__.py#L28. This allows the register to discard previous registrations. This should work, as the datalabs FileSystems are copies of the datasets FileSystems. However, I don't know if it is guaranteed to be compatible with other libraries that might use the same protocols.


I am linking the symmetric issue on [DataLab](https://github.com/ExpressAI/DataLab/issues/425) as ideally the issue is solved in both libraries the same way. Otherwise, it could lead to different behaviors depending on which library gets imported first.


### Steps to reproduce the bug

1. Run `pip install datalabs==0.4.15 datasets==2.12.0`
2. Run the following python code:
   ```
      import datalabs
      import datasets
   ```

### Expected behavior

It should be possible to import both libraries without getting a Value Error

### Environment info

datalabs==0.4.15
datasets==2.12.0
GitHub issue #5875: Why split slicing doesn't behave like list slicing ?
State: closed | Created: 2023-05-19T07:21:10Z

### Describe the bug

If I want to get the first 10 samples of my dataset, I can do :

```
ds = datasets.load_dataset('mnist', split='train[:10]')
```

But if I exceed the number of samples in the dataset, an exception is raised : 

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

> ValueError: Requested slice [:999999999] incompatible with 60000 examples.

### Steps to reproduce the bug

```
ds = datasets.load_dataset('mnist', split='train[:999999999]')
```

### Expected behavior

I would expect it to behave like python lists (no exception raised, the whole list is kept) : 

```
d = list(range(1000))[:999999]
print(len(d))  # > 1000
```

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue #5874: Using as_dataset on a "parquet" builder  
State: closed | Created: 2023-05-18T14:09:03Z

### Describe the bug

I used a custom builder to ``download_and_prepare`` a dataset. The first (very minor) issue is that the doc seems to suggest ``download_and_prepare`` will return the dataset, while it does not ([builder.py](https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py#L718-L738)).
```
        >>> from datasets import load_dataset_builder
        >>> builder = load_dataset_builder("rotten_tomatoes")
        >>> ds = builder.download_and_prepare("./output_dir", file_format="parquet")
```

The main issue I am facing is loading the dataset from those parquet files. I used the `as_dataset` method suggested by the doc, however it returns:
`
FileNotFoundError: [Errno 2] Failed to open local file 'output_dir/__main__-train-00000-of-00245.arrow'. Detail:
[errno 2] No such file or directory.
` 

### Steps to reproduce the bug

1. Create a custom builder of some sort: `builder = CustomBuilder()`.
2. Run `download_and_prepare` with the parquet format: `builder.download_and_prepare("./output_dir", file_format="parquet")`.
3. Run `dataset = builder.as_dataset()`.  

### Expected behavior

I guess I'd expect `as_dataset` to generate the dataset in arrow format if it has to, or to suggest an alternative way to load the dataset (I've also tried other methods with `load_dataset` to no avail, probably due to misunderstandings on my part). 

### Environment info

```
- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-1027-gcp-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.14.1
- PyArrow version: 8.0.0
- Pandas version: 1.5.3
```
GitHub issue #5873: Allow setting the environment variable for the lock file path
State: open | Created: 2023-05-17T07:10:02Z

### Feature request

Add an environment variable to replace the default lock file path.

### Motivation

Usually, dataset path is a read-only path while the lock file needs to be modified each time. It would be convenient if the path can be reset individually.

### Your contribution

```/src/datasets/utils/filelock.py
class UnixFileLock(BaseFileLock):
    def __init__(self, lock_file, timeout=-1, max_filename_length=None):
        #-------------------
        if os.getenv('DS_TMP_PATH'):
            file_name = str(lock_file).split('/')[-1]
            dataset_tmp_path = os.getenv('DS_TMP_PATH')
            lock_file = os.path.join(dataset_tmp_path, file_name)
        #-------------------
        max_filename_length = os.statvfs(os.path.dirname(lock_file)).f_namemax
        super().__init__(lock_file, timeout=timeout, max_filename_length=max_filename_length)
```
A simple demo is as upper. Thanks.
GitHub issue #5871: data configuration hash suffix depends on uncanonicalized data_dir
State: closed | Created: 2023-05-16T18:56:04Z

### Describe the bug

I am working with the `recipe_nlg` dataset, which requires manual download. Once it's downloaded, I've noticed that the hash in the custom data configuration is different if I add a trailing `/` to my `data_dir`. It took me a while to notice that the hashes were different, and to understand that that was the cause of my dataset being processed anew instead of the cached version being used.

### Steps to reproduce the bug

1. Follow the steps to manually download the `recipe_nlg` dataset to `/data/recipenlg`.
2. Load it using `load_dataset`, once without a trailing slash and once with one:

    ```python
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg")
    Using custom data configuration default-082278caeea85765
    Downloading and preparing dataset recipe_nlg/default to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Dataset recipe_nlg downloaded and prepared to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74. Subsequent calls will reuse this data.
    100%|███████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.10s/it]
    DatasetDict({
        train: Dataset({
            features: ['id', 'title', 'ingredients', 'directions', 'link', 'source', 'ner'],
            num_rows: 2231142
        })
    })
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg/")
    Using custom data configuration default-83e87680785d0493
    Downloading and preparing dataset recipe_nlg/default to /home/user/.cache/huggingface/datasets/recipe_nlg/default-83e87680785d0493/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Generating train split:   1%|          | 12701/2231142 [00:04<13:15, 2790.25 examples/s
    ^C
    ```

3. Observe that the hash suffix in the custom data configuration changes due to the altered string.

### Expected behavior

I think I would expect the hash to remain constant if it actually points to the same location on disk. I would expect the use of `os.path.normpath` to canonicalize the paths.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-147-generic-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
GitHub issue #5868: Is it possible to change a cached file and 're-cache' it instead of re-generating?
State: closed | Created: 2023-05-16T03:45:42Z

### Feature request

Hi,
I have a huge cached file using `map`(over 500GB), and I want to change an attribution of each element, is there possible to do it using some method instead of re-generating, because `map` takes over 24 hours

### Motivation

For large datasets, I think it is very important because we always face the problem which is changing something in the original cache without re-generating it.

### Your contribution

For now, I can't help, sorry.
GitHub issue #5866: Issue with Sequence features
State: closed | Created: 2023-05-15T17:13:29Z

### Describe the bug

Sequences features  sometimes causes errors when the specified length is not -1

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Features, ClassLabel, Sequence, Value, Dataset
feats = Features(**{'target': ClassLabel(names=[0, 1]),'x': Sequence(feature=Value(dtype='float64',id=None), length=2, id=None)})
Dataset.from_dict({"target": np.ones(2000).astype(int), "x": np.random.rand(2000,2)},features = feats).flatten_indices()
```
Throws:
```
  TypeError: Couldn't cast array of type
  fixed_size_list<item: double>[2]
  to
  Sequence(feature=Value(dtype='float64', id=None), length=2, id=None)
```
The same code works without any issues when `length = -1`

EDIT: The error seems to happen only when the length of the dataset is bigger than 1000 for some reason
### Expected behavior

No exception

### Environment info

- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1
GitHub issue #5865: Deprecate task api
State: closed | Created: 2023-05-15T16:48:24Z

The task API is not well adopted in the ecosystem, so this PR deprecates it. The `train_eval_index` is a newer, more flexible solution that should be used instead (I think?).

These are the projects that still use the task API :
* the image classification example in Transformers: [here](https://github.com/huggingface/transformers/blob/8f76dc8e5aaad58f2df7748b6d6970376f315a9a/examples/pytorch/image-classification/run_image_classification_no_trainer.py#L262) and [here](https://github.com/huggingface/transformers/blob/8f76dc8e5aaad58f2df7748b6d6970376f315a9a/examples/tensorflow/image-classification/run_image_classification.py#L277)
* autotrain: [here](https://github.com/huggingface/autotrain-backend/blob/455e274004b56f9377d64db4ab03671508fcc4cd/zeus/zeus/run/utils.py#L666)
* api-inference-community: [here](https://github.com/huggingface/api-inference-community/blob/fb8fb29d577a5bf01c82944db745489a6d6ed3d4/manage.py#L64) (but the rest of the code does not call the `resolve_dataset` function)

So we need to update these files after the merge.

cc @lewtun
GitHub issue #5864: Slow iteration over Torch tensors
State: open | Created: 2023-05-15T16:43:58Z

### Describe the bug

I have a problem related to this [issue](https://github.com/huggingface/datasets/issues/5841): I get a way slower iteration when using a Torch dataloader if I use vanilla Numpy tensors or if I first apply a ToTensor transform to the input. In particular, it takes 5 seconds to iterate over the vanilla input and ~30s after the transformation.

### Steps to reproduce the bug

Here is the minimum code to reproduce the problem

```python
import numpy as np
from datasets import Dataset, DatasetDict, load_dataset, Array3D, Image, Features
from torch.utils.data import DataLoader
from tqdm import tqdm
import torchvision 
from torchvision.transforms import ToTensor, Normalize


#################################
# Without transform
#################################
    
train_dataset = load_dataset(
    'cifar100',
    split='train',
    use_auth_token=True,
)

train_dataset.set_format(type="numpy", columns=["img", "fine_label"])

train_loader= DataLoader(
    train_dataset,
    batch_size=100,
    pin_memory=False,
    shuffle=True,
    num_workers=8,
)

for batch in tqdm(train_loader, desc="Loading data, no transform"):
    pass


#################################
# With transform
#################################

transform_func = torchvision.transforms.Compose([
    ToTensor(), 
    Normalize(mean=[0.485, 0.456, 0.406], std= [0.229, 0.224, 0.225]),]          
)
    
train_dataset = train_dataset.map(
    desc=f"Preprocessing samples",
    function=lambda x: {"img": transform_func(x["img"])},
)

train_dataset.set_format(type="numpy", columns=["img", "fine_label"])


train_loader= DataLoader(
    train_dataset,
    batch_size=100,
    pin_memory=False,
    shuffle=True,
    num_workers=8,
)


for batch in tqdm(train_loader, desc="Loading data after transform"):
    pass 
```

I have also tried converting the Image column to an Array3D
```python
img_shape = train_dataset[0]["img"].shape

features = train_dataset.features.copy()
features["x"] = Array3D(shape=img_shape, dtype="float32")

train_dataset = train_dataset.map(
    desc=f"Preprocessing samples",
    function=lambda x: {"x": np.array(x["img"], dtype=np.uint8)},
    features=features,
)
train_dataset.cast_column("x", Array3D(shape=img_shape, dtype="float32"))
train_dataset.set_format(type="numpy", columns=["x", "fine_label"])
```
but to no avail. Any clue?

### Expected behavior

The iteration should take approximately the same time with or without the transformation, as it doesn't change the shape of the input. What may be the issue here?

### Environment info

```
- `datasets` version: 2.12.0
- Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
```
GitHub issue #5862: IndexError: list index out of range with data hosted on Zenodo
State: open | Created: 2023-05-15T13:47:19Z

The dataset viewer sometimes raises an `IndexError`:
```
IndexError: list index out of range
```
See:
- huggingface/datasets-server#1151
  - https://huggingface.co/datasets/reddit/discussions/5
- huggingface/datasets-server#1118
  - https://huggingface.co/datasets/krr-oxford/OntoLAMA/discussions/1
- https://huggingface.co/datasets/hyperpartisan_news_detection/discussions/3
- https://huggingface.co/datasets/um005/discussions/2
- https://huggingface.co/datasets/tapaco/discussions/2
- https://huggingface.co/datasets/common_language/discussions/3
- https://huggingface.co/datasets/pass/discussions/1

After investigation:
- This happens with data files hosted on Zenodo
- Indeed, there is an underlying 429 HTTP error: Too Many Requests

Note that some time ago, it also happened with data files hosted on Google Drive. See:
  - #4581
  - #4580 

The reason then was that there was a 403 HTTP error: Forbidden
GitHub issue #5858: Throw an error when dataset improperly indexed
State: closed | Created: 2023-05-15T05:15:53Z

### Describe the bug

Pandas-style subset indexing on dataset does not throw an error, when maybe it should. Instead returns the first instance of the dataset regardless of index condition.

### Steps to reproduce the bug

Steps to reproduce the behavior:

1. `squad = datasets.load_dataset("squad_v2", split="validation")`
2. `item = squad[squad['question'] ==  "Who was the Norse leader?"]`
or `it = squad[squad['id'] == '56ddde6b9a695914005b962b']`
3.  returns the first item in the dataset, which does not satisfy the above conditions:

`{'id': '56ddde6b9a695914005b9628', 'title': 'Normans', 'context': 'The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse ("Norman" comes from "Norseman") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.', 'question': 'In what country is Normandy located?', 'answers': {'text': ['France', 'France', 'France', 'France'], 'answer_start': [159, 159, 159, 159]}}`

### Expected behavior

Should either throw an error message, or return the dataset item that satisfies the condition.

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
GitHub issue #5857: Adding chemistry dataset/models in huggingface
State: closed | Created: 2023-05-15T05:09:49Z

### Feature request

Huggingface is really amazing platform for open science.

In addition to computer vision, video and NLP, would it be of interest to add chemistry/materials science dataset/models in Huggingface? Or, if its already done, can you provide some pointers.

We have been working on a comprehensive benchmark on this topic: [JARVIS-Leaderboard](https://pages.nist.gov/jarvis_leaderboard/) and I am wondering if we could contribute/integrate this project as a part of huggingface. 

### Motivation

Similar to the main stream AI field, there is need of large scale benchmarks/models/infrastructure for chemistry/materials data.

### Your contribution

We can start adding datasets as our [benchmarks](https://github.com/usnistgov/jarvis_leaderboard/tree/main/jarvis_leaderboard/benchmarks) should be easily convertible to the dataset format.
GitHub issue #5856: Error loading natural_questions
State: closed | Created: 2023-05-15T02:46:04Z

### Describe the bug

When try to load natural_questions through datasets == 2.12.0 with python == 3.8.9:

```python
import datasets
datasets.load_dataset('natural_questions',beam_runner='DirectRunner')
```

It failed with following info:

`pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs`

### Steps to reproduce the bug

In python console:

```python
import datasets
datasets.load_dataset('natural_questions',beam_runner='DirectRunner')
```

Then the trace is:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 2019, in _download_and_prepare
    num_examples, num_bytes = beam_writer.finalize(metrics.query(m_filter))
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/arrow_writer.py", line 694, in finalize
    shard_num_bytes, _ = parquet_to_arrow(source, destination)
  File "/home/nlp/.cache/pypoetry/virtualenvs/drg-W3LF4Ol9-py3.8/lib/python3.8/site-packages/datasets/arrow_writer.py", line 737, in parquet_to_arrow
    for record_batch in parquet_file.iter_batches():
  File "pyarrow/_parquet.pyx", line 1323, in iter_batches
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs
```

### Expected behavior

load natural_question questions

### Environment info

```
- `datasets` version: 2.12.0
- Platform: Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.9
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.1
```
GitHub issue #5855: `to_tf_dataset` consumes too much memory
State: closed | Created: 2023-05-14T01:22:29Z

### Describe the bug

Hi, I'm using `to_tf_dataset` to convert a _large_ dataset to `tf.data.Dataset`. I observed that the data loading *before* training took a lot of time and memory, even with `batch_size=1`.

After some digging, i believe the reason lies in the shuffle behavior. The [source code](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/tf_utils.py#L185) uses `len(dataset)` as the `buffer_size`, which may load all the data into the memory, and the [tf.data doc](https://www.tensorflow.org/guide/data#randomly_shuffling_input_data) also states that "While large buffer_sizes shuffle more thoroughly, they can take a lot of memory, and significant time to fill".

### Steps to reproduce the bug

```python
from datasets import Dataset

def gen():  # some large data
    for i in range(50000000):
        yield {"data": i}

ds = Dataset.from_generator(gen, cache_dir="./huggingface")

tf_ds = ds.to_tf_dataset(
    batch_size=64,
    shuffle=False,  # no shuffle
    drop_remainder=False,
    prefetch=True,
)

# fast and memory friendly 🤗
for batch in tf_ds: 
    ...

tf_ds_shuffle = ds.to_tf_dataset(
    batch_size=64,
    shuffle=True,
    drop_remainder=False,
    prefetch=True,
)

# slow and memory hungry for simple iteration 😱
for batch in tf_ds_shuffle: 
    ...

```

### Expected behavior

Shuffling should not load all the data into the memory. Would adding a `buffer_size` parameter in the `to_tf_dataset` API alleviate the problem?

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.17.1-051701-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 1.4.3
GitHub issue #5854: Can not load audiofolder dataset on kaggle
State: closed | Created: 2023-05-14T00:50:47Z

### Describe the bug
It's crash log:
FileNotFoundError: Couldn't find a dataset script at /kaggle/working/audiofolder/audiofolder.py or any data file in the same directory. Couldn't find 'audiofolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/audiofolder/audiofolder.py


### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/93691919/a2829d27-d15c-4acc-86fb-d1987c760468)
common_voice = load_dataset("audiofolder", data_dir="/kaggle/working/data")

### Expected behavior

load dataset without error. It works ok on colab, but on kaggle it happends.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.31
- Python version: 3.10.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
GitHub issue #5852: Iterable torch formatting
State: closed | Created: 2023-05-12T16:48:49Z

Used the TorchFormatter to get torch tensors in iterable dataset with format set to "torch".

It uses the data from Arrow if possible, otherwise applies recursive_tensorize.

When set back to format_type=None, cast_to_python_objects is used.

requires https://github.com/huggingface/datasets/pull/5821

close https://github.com/huggingface/datasets/issues/5793
GitHub issue #5849: CSV datasets should only read the CSV data files in the repo
State: closed | Created: 2023-05-12T12:29:53Z

When a no-script dataset has many CSV files and a JPG file, the library infers to use the Csv builder, but tries to read as CSV all files in the repo, also the JPG file.

I think the Csv builder should filter out non-CSV files when reading.

An analogue solution should be implemented for other packaged builders.

Related to:
- https://huggingface.co/datasets/abidlabs/img2text/discussions/1
- https://github.com/gradio-app/gradio/pull/3973#issuecomment-1545409061

CC: @abidlabs @severo
GitHub issue #5848: Add `accelerate` as metric's test dependency to fix CI error
State: closed | Created: 2023-05-12T12:01:01Z

The `frugalscore` metric uses Transformers' Trainer, which requires `accelerate` (as of recently).

Fixes the following [CI error](https://github.com/huggingface/datasets/actions/runs/4950900048/jobs/8855148703?pr=5845).
GitHub issue #5851: Error message not clear in interleaving datasets
State: closed | Created: 2023-05-11T20:52:13Z

### System Info

standard env

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm trying to interleave 'sciq', 'wiki' and the 'pile-enron' dataset. I think the error I made was that I loaded the train split of one, but for the other but the error is not too helpful- 

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/home/suryahari/Vornoi/save_model_ops.py](https://vscode-remote+ssh-002dremote-002bthomsonlab-002d2-002ejamesgornet-002ecom.vscode-resource.vscode-cdn.net/home/suryahari/Vornoi/save_model_ops.py) in line 3
      [41](file:///home/suryahari/Vornoi/save_model_ops.py?line=40) # %%
----> [43](file:///home/suryahari/Vornoi/save_model_ops.py?line=42) dataset = interleave_datasets(datasets, stopping_strategy="all_exhausted")

File [~/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py:124](https://vscode-remote+ssh-002dremote-002bthomsonlab-002d2-002ejamesgornet-002ecom.vscode-resource.vscode-cdn.net/home/suryahari/~/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py:124), in interleave_datasets(datasets, probabilities, seed, info, split, stopping_strategy)
    [122](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=121) for dataset in datasets[1:]:
    [123](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=122)     if (map_style and not isinstance(dataset, Dataset)) or (iterable and not isinstance(dataset, IterableDataset)):
--> [124](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=123)         raise ValueError(
    [125](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=124)             f"Unable to interleave a {type(datasets[0])} with a {type(dataset)}. Expected a list of Dataset objects or a list of IterableDataset objects."
    [126](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=125)         )
    [127](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=126) if stopping_strategy not in ["first_exhausted", "all_exhausted"]:
    [128](file:///home/suryahari/miniconda3/envs/vornoi/lib/python3.10/site-packages/datasets/combine.py?line=127)     raise ValueError(f"{stopping_strategy} is not supported. Please enter a valid stopping_strategy.")

ValueError: Unable to interleave a  with a . Expected a list of Dataset objects or a list of IterableDataset objects.
```

### Expected behavior

the error message should hopefully be more clear
GitHub issue #5846: load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!
State: closed | Created: 2023-05-11T17:58:57Z

### Describe the bug

Running

```
import datasets
ds = datasets.load_dataset('bigcode/the-stack-dedup', streaming=True)
```

takes about 2.5 minutes! 

I would expect this to be near instantaneous. With other datasets, the runtime is one or two seconds.


### Environment info

- `datasets` version: 2.11.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0