MarouaneSanhaji/github-issues huggingface dataset:
[WIP] WebDataset export
State: closed

I added a first draft of the `IterableDataset.to_wds` method.

You can use it to savea dataset loaded in streamign mode as a webdataset locally.
The API can be further improved to allow to export to a cloud storage like the HF Hub.

I also included sharding with a default max shard size of 500MB (uncompressed), and it is single-processed fo rnow.

Choosing the number of shards is not implemented yet - though if we know the size of the `IterableDataset` this is probably doable`.

For example
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
>>> ds.to_wds("output_dir", compress=True)
>>> import webdataset as wds
>>> ds = wds.WebDataset("output_dir/rotten_tomatoes-train-000000.tar.gz").decode()
>>> next(iter(ds))
{'__key__': '0',
'__url__': 'output_dir/rotten_tomatoes-train-000000.tar.gz',
'label.cls': 1,
'text.txt': 'the rock is destined to be the 21st century\'s new ..., jean-claud van damme or steven segal .'}
```

### Implementation details

The WebDataset format is made of TAR archives containing a series of files per example. For example one pair of `image.jpg` and `label.cls` for image classification.

WebDataset automatically decodes serialized data based on the extension of the files, and output a dictionary. For example `{"image.png": np.array(...), "label.cls": 0}` if you choose the numpy decoding.

To use the automatic decoding, I store each field of each example as a file with its corresponding extension (jpg, json, cls, etc.)

While this is useful to end up with a dictionary with one key per column and appropriate decoding, it can create huge TAR archives if the dataset is made of small samples of text - probably because of useless TAR metadata for each file. This also makes loading super slow: iterating on SQuAD takes 50sec vs 7sec using `datasets` in streaming mode.

I haven't taken a look at alternatives for text datasets made out of small samples, but for image datasets this can already be used to run some benchmarks.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5127). All of your documentation changes will be reflected on that endpoint.

Should we close this PR?
Fix class name of symbolic link
State: closed

Fix #5098 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5126). All of your documentation changes will be reflected on that endpoint.

I have removed the reference to the Issue in the PR title, so that we avoid to have both references (to the issue and to the PR) in the merge commit to the main branch.

Instead, it should be commented in the PR description, so that the PR is appropriately linked by GitHub to its corresponding Issue:

> Fix #5098.

@albertvillanova What should I test in your opinion? Also, where should I save the test file and how should I name it? Thanks for your support

The regression test to be implemented should test what your PR fixes: that is, that `_resolve_single_pattern_locally` function does not resolve any symbolic link when passed a directory that does contain any.

As you are testing a function in `data_files.py`, the corresponding test should be in `tests/test_data_files.py`.

You could name the test something lilke: `test_resolve_single_pattern_locally_does_not_resolve_symbolic_links`

You could take inspiration from other tests there in that file.
Add warning
State: closed

Fixes: #5105 

I think removing the directory with warning is a better solution for this issue. Because if we decide to keep existing files in directory, then we should deal with the case providing same directory for several datasets! Which we know is not possible since `dataset_info.json` exists in that directory.

As mentioned in https://github.com/huggingface/datasets/issues/5105 I think we just need to keep the existing files instead of deleting them.
The `dataset_info.json` file contains the split names anyway, so we know which files belong to the dataset, and which ones don't.
Bugfix ignore function when creating new_fingerprint for caching
State: closed

maybe fixes: #5109

Adding "function" to the kwargs to ignore when computing the fingerprint will break `map` caching. Indeed passing two different function would result in two different datasets that have the same fingerprint - and the cache wouldn't be able to distinguish them.

E.g this code would reload ds1 from the cache insetad of computing the dataset for ds2
```python
ds = Dataset.from_dict({"a": [1, 2, 3]})
ds1 = ds.map(lambda x: {"b": 1})
ds2 = ds.map(lambda x: {"b": 2})
```
Installing `datasets` on M1 computers
State: closed

## Describe the bug
I wanted to install `datasets` dependencies on my M1 (in order to start contributing to the project). However, I got an error regarding `tensorflow`.

On M1, `tensorflow-macos` needs to be installed instead. Can we add a conditional requirement, so that `tensorflow-macos` would be installed on M1?

## Steps to reproduce the bug
Fresh clone this project (on m1), create a virtualenv and run this:
```python
pip install -e ".[dev]"
```

## Expected results
Installation should be smooth, and all the dependencies should be installed on M1.

## Actual results
You should receive an error, saying pip couldn't find a version that matches this pattern:
```
tensorflow>=2.3,!=2.6.0,!=2.6.1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.2.dev0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



Thanks for reporting, @david1542.
Progress bars have color red and never completed to 100%
State: closed

## Describe the bug
Progress bars after transformative operations turn in red and never be completed to 100%

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset('rotten_tomatoes', split='test').filter(lambda o: True)
```

## Expected results
Progress bar should be 100% and green

## Actual results
Progress  bar turn in red and never completed to 100%

## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Hi @echatzikyriakidis, thanks for submitting the issue.
Which shell are you using exactly? I tried to run the command you sent, but I don't see colors at all 🧐

I tried from bash and zsh as well.

Hi @david1542 ,

I use Google Colab.


Got it. I [created a PR](https://github.com/huggingface/datasets/pull/5120) that fixes this issue. Turns out that the wrapping logic for the inner loop was slightly incorrect.

Thank you!

Hello @mariosasko 

I am still facing this issue. Was this problem fixed?

![Screen Shot 2024-06-19 at 11 35 57 AM](https://github.com/huggingface/datasets/assets/66180831/cd9c6ac4-178c-4297-821c-a56411d34c89)

I cleared the hugging face cache before running, and no error message was given. Let me know if you need a minimal repro of my code.
Fix iter_batches
State: closed

The `pa.Table.to_reader()` method available in `pyarrow>=8.0.0` may return chunks of size < `max_chunksize`, therefore `iter_batches` can return batches smaller than the `batch_size` specified by the user

Therefore batched `map` couldn't always use batches of the right size, e.g. this fails because it runs only on one batch of one element:
```python

from datasets import Dataset, concatenate_datasets

ds = concatenate_datasets([Dataset.from_dict({"a": [i]}) for i in range(10)])

ds2 = ds.map(lambda _: {}, batched=True)
assert list(ds2) == list(ds)
```

This was introduced in https://github.com/huggingface/datasets/pull/5030

Close https://github.com/huggingface/datasets/issues/5111

This will require a patch release along with https://github.com/huggingface/datasets/pull/5113

TODO:
- [x] fix tests
- [x] add more tests

_The documentation is not available anymore as the PR was closed or merged._

I also ran the code in https://github.com/huggingface/datasets/issues/5111 and it works fine now :)

This is ready for review :)
load_from_disk with remote filesystem fails due to a wrong temporary local folder path
State: open

## Describe the bug

The function load_from_disk fails when using a remote filesystem because of a wrong temporary path generation in the load_from_disk method of arrow_dataset.py:

```python
if is_remote_filesystem(fs):
    src_dataset_path = extract_path_from_uri(dataset_path)
    dataset_path = Dataset._build_local_temp_path(src_dataset_path)
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
```
If _dataset_path_ is `gs://speech/mydataset/train`, then _src_dataset_path_ will be `speech/mydataset/train` and _dataset_path_ will be something like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train`

Then, after downloading the **folder** _src_dataset_path_, you will get a path like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train/train/state.json` (notice we have train twice)

Instead of downloading the remote folder we should be downloading all the files in the folder for the path to be right:

```python
fs.download(os.path.join(src_dataset_path,*), dataset_path.as_posix(), recursive=True)
```

## Steps to reproduce the bug
```python
fs = gcsfs.GCSFileSystem(**storage_options)
dataset = load_from_disk("common_voice_processed") # loading local dataset previously saved locally, works fine
dataset.save_to_disk(output_dir, fs=fs) #works fine
dataset = load_from_disk(output_dir, fs=fs) # crashes
```

## Expected results
The dataset is loaded

## Actual results

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/9s/gf0b9jz15d517yrf7m3nvlxr0000gn/T/tmp6t5e221_/speech/datasets/tests/common_voice_processed/train/state.json'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-2.6.1.dev0
- Platform: mac os monterey 12.5.1
- Python version: 3.8.13
- PyArrow version:pyarrow==9.0.0


Hi Hubert! Could you please probably create a publicly available `gs://` dataset link? I think this would be easier for others to directly start to debug.

What seems to work is to change the line to:
```
fs.download(src_dataset_path, dataset_path.parent.as_posix(), recursive=True)
```
Bug with filtered indices
State: closed

## Describe the bug
As reported by @PartiallyTyped (and by @Muennighoff):
- https://github.com/huggingface/datasets/issues/5111#issuecomment-1278652524

There is an issue with the indices of a filtered dataset.

## Steps to reproduce the bug
```python
ds = Dataset.from_dict({"num": [0, 1, 2, 3]})
ds = ds.filter(lambda num: num % 2 == 0, input_columns="num", batch_size=2)
assert all(item["num"] % 2 == 0 for item in ds)
```

## Expected results
The indices of the filtered dataset should correspond to the examples with "language" equals to "english".

## Actual results
Indices to items with other languages are included in the filtered dataset indices

## Preliminar investigation
It seems a bug introduced by:
- #5030


The issue is here:
https://github.com/huggingface/datasets/blob/3ad9644b9a2e4558dd1d0f1e43c67658674e6228/src/datasets/arrow_dataset.py#L2964

@PartiallyTyped, @Muennighoff: the issue is fixed.

We are planning to make a patch release today.

Thanks a lot for the swift response! For a brief moment yesterday I thought I had gone insane 🤣On 14 Oct 2022, at 15:44, Albert Villanova del Moral ***@***.***> wrote:﻿
@PartiallyTyped, @Muennighoff: the issue is fixed.
We are planning to make a patch release today.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Map caching not working for some class methods
State: closed

## Describe the bug
The cache loading is not working as expected for some class methods with a model stored in an attribute.
The new fingerprint for `_map_single` is not the same at each run. The hasher generate a different hash for the class method.
This comes from `dumps` function in `datasets.utils.py_utils` which generates a different dump at each run.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import AutoConfig, AutoModel, AutoTokenizer

dataset = load_dataset("ethos", "binary")
BASE_MODELNAME = "sentence-transformers/all-MiniLM-L6-v2"

class Object:
    def __init__(self):
        config = AutoConfig.from_pretrained(BASE_MODELNAME)
        self.bert = AutoModel.from_config(config=config, add_pooling_layer=False)
        self.tok = AutoTokenizer.from_pretrained(BASE_MODELNAME)

    def tokenize(self, examples):

        tokenized_texts = self.tok(
            examples["text"],
            padding="max_length",
            truncation=True,
            max_length=256,
        )
        return tokenized_texts

instance = Object()
result = dict()
for phase in ["train"]:
    result[phase] = dataset[phase].map(instance.tokenize, batched=True, load_from_cache_file=True, num_proc=2)

```

## Expected results
Load cache instead of recompute result.

## Actual results
Result recomputed from scratch at each run.
The cache works fine when deleting `bert` attribute.

## Environment info
- `datasets` version: 2.5.3.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



The hash used for caching is computed by pickling recursively the function passed to `map`. Maybe some objects don't have the same hash across sessions.  In particular you can check the hash of your model using
```python
from datasets.fingerprint import Hasher
obj = AutoModel.from_config(config=config, add_pooling_layer=False)
print(Hasher.hash(obj))
```

You can find mode info here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument

Indeed, the hash is changing. The `dumps` function serialize the model object in different ways because the model object is not deterministic
```python
from datasets.utils.py_utils import dumps
obj1 = AutoModel.from_config(config=config, add_pooling_layer=False)
obj2 = AutoModel.from_config(config=config, add_pooling_layer=False)

dumps(bert) == dumps(bert2). # False
```

> You can find mode info here: https://huggingface.co/docs/datasets/about_cache
> 
> You can also provide your own unique hash in map if you want, with the new_fingerprint argument


Thanks, the doc is so helpful. Indeed, we can fix the hash and get cache hit using `new_fingerprint`. Closing the issue.
Fix task template reload from dict
State: closed

Since #4926 the JSON dumps are simplified and it made task template dicts empty by default.

I fixed this by always including the task name which is needed to reload a task from a dict

_The documentation is not available anymore as the PR was closed or merged._

> Just wondering if there might be other data classes default values missed that could cause an issue... Apart from feature-like classes and tasks, I don't see any others though...

I think we're good ! `asdict` is used on the DatasetInfo attributes like features, tasks etc. and they all support dict conversion properly now

> And a question: but this information about the tasks is no longer being saved as YAML tags in the dataset card; won't be a problem with current datasets using task templates (with this information in their metadata JSON) once we replace the JSON by the YAML tags (which do not have this information about the task templates)?

In the long run we'll use the train_eval_index YAML tags instead, but I agree when removing the JSON files we should try to not break existing code that may rely on this
Error in create a dataset from a Python generator
State: closed

## Describe the bug
In HOW-TO-GUIDES > Load > [Python generator](https://huggingface.co/docs/datasets/v2.5.2/en/loading#python-generator), the code example defines the `my_gen` function, but when creating the dataset, an undefined `my_dict` is passed in.  

```Python
>>> from datasets import Dataset
>>> def my_gen():
...    for i in range(1, 4):
...        yield {"a": i}
>>> dataset = Dataset.from_generator(my_dict)
```

Hi, thanks for reporting! The last line should be `dataset = Dataset.from_generator(my_gen)`.

Can I work on this one?
Classes label error when loading symbolic links using imagefolder
State: closed

**Is your feature request related to a problem? Please describe.**
Like this: #4015
When there are **symbolic links** to pictures in the data folder, the parent folder name of the **real file** will be used as the class name instead of the parent folder of the symbolic link itself. Can you give an option to decide whether to enable symbolic link tracking?

This is inconsistent with the `torchvision.datasets.ImageFolder` behavior.

For example:
![image](https://user-images.githubusercontent.com/49552732/195008591-3cce644e-aabe-4f39-90b9-832861cadb3d.png)
![image](https://user-images.githubusercontent.com/49552732/195008841-0b0c2289-eb7f-411a-977b-37426f23a277.png)

It use `others` in green circle as class label but not `abnormal`, I wish `load_dataset` not use the real file parent as label.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.


It can be solved temporarily  by remove `resolve` in 
https://github.com/huggingface/datasets/blob/bef23be3d9543b1ca2da87ab2f05070201044ddc/src/datasets/data_files.py#L278

Hi, thanks for reporting and suggesting a fix! We still need to account for `.`/`..` in the file path, so a more robust fix would be `Path(os.path.abspath(filepath))`.

> Hi, thanks for reporting and suggesting a fix! We still need to account for `.`/`..` in the file path, so a more robust fix would be `Path(os.path.abspath(filepath))`.

Thanks for your reply!
Mismatch between tutoriel and doc
State: closed

## Describe the bug
In the "Process text data" tutorial, [`map` has `return_tensors` as kwarg](https://huggingface.co/docs/datasets/main/en/nlp_process#map). It does not seem to appear in the [function documentation](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.map), nor to work.

## Steps to reproduce the bug
MWE:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

from datasets import load_dataset
dataset  = load_dataset("lhoestq/demo1", split="train")

dataset = dataset.map(lambda examples: tokenizer(examples["review"]), batched=True, return_tensors="pt")

```

## Expected results
return_tensors to be a valid kwarg :smiley: 

## Actual results
```python
>> TypeError: map() got an unexpected keyword argument 'return_tensors'
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1052-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Hi, thanks for reporting! This line should be replaced with 
```python
dataset = dataset.map(lambda examples: tokenizer(examples["text"], return_tensors="np"), batched=True)
```
for it to work (the `return_tensors` part inside the `tokenizer` call).

Can I work on this?

Fixed in https://github.com/huggingface/datasets/pull/5095
Use HTML relative paths for tiles in the docs
State: closed

This PR replaces the absolute paths in the landing page tiles with relative ones so that one can test navigation both locally in and in future PRs (see [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5084/en/index) for an example PR where the links don't work).

I encountered this while working on the `optimum` docs and figured I'd fix it elsewhere too :)

Internal Slack thread: https://huggingface.slack.com/archives/C02GLJ5S0E9/p1665129710176619

_The documentation is not available anymore as the PR was closed or merged._

> Good catch, @lewtun. Thanks for the fix.
> 
> Do you know if there are other absolute paths in the docs that should be fixed as well?

I found a few more in [0d4796b](https://github.com/huggingface/datasets/pull/5092/commits/0d4796b747e6620d9fcc17a8f74acc5cf4bba7be).

However, I noticed that none of the cross-references (e.g. to API classes / methods) work locally, but that is probably just a limitation of the local build

Thanks.
Allow connection objects in `from_sql` + small doc improvement
State: closed

Allow connection objects in `from_sql` (emit a warning that they are cachable) and add a tip that explains the format of the con parameter when provided as a URI string.

PS: ~~This PR contains a parameter link, so https://github.com/huggingface/doc-builder/pull/311 needs to be merged before it's "ready for review".~~ Done!

_The documentation is not available anymore as the PR was closed or merged._
Review sync issues from GitHub to Hub
State: closed

## Describe the bug
We have discovered that sometimes there were sync issues between GitHub and Hub datasets, after a merge commit to main branch.

For example:
- this merge commit: https://github.com/huggingface/datasets/commit/d74a9e8e4bfff1fed03a4cab99180a841d7caf4b
- was not properly synced with the Hub: https://github.com/huggingface/datasets/actions/runs/3002495269/jobs/4819769684
```
[main 9e641de]  Add Papers with Code ID to scifact dataset (#4941)
 Author: Albert Villanova del Moral <albertvillanova@users.noreply.huggingface.co>
 1 file changed, 42 insertions(+), 14 deletions(-)
push failed !
GitCommandError(['git', 'push'], 1, b'remote: ----------------------------------------------------------        \nremote: Sorry, your push was rejected during YAML metadata verification:        \nremote: - Error: "license" does not match any of the allowed types        \nremote: ----------------------------------------------------------        \nremote: Please find the documentation at:        \nremote: https://huggingface.co/docs/hub/models-cards#model-card-metadata        \nremote: ----------------------------------------------------------        \nTo [https://huggingface.co/datasets/scifact.git\n](https://huggingface.co/datasets/scifact.git/n) ! [remote rejected] main -> main (pre-receive hook declined)\nerror: failed to push some refs to \'[https://huggingface.co/datasets/scifact.git\](https://huggingface.co/datasets/scifact.git/)'', b'')
```
We are reviewing sync issues in previous commits to recover them and repushing to the Hub.

TODO: Review
- [x] #4941
  - scifact
- [x] #4931
  - scifact
- [x] #4753
  - wikipedia 
- [x] #4554
  - wmt17, wmt19, wmt_t2t
  - Fixed with "Release 2.4.0" commit: https://github.com/huggingface/datasets/commit/401d4c4f9b9594cb6527c599c0e7a72ce1a0ea49
    - https://huggingface.co/datasets/wmt17/commit/5c0afa83fbbd3508ff7627c07f1b27756d1379ea
    - https://huggingface.co/datasets/wmt19/commit/b8ad5bf1960208a376a0ab20bc8eac9638f7b400
    - https://huggingface.co/datasets/wmt_t2t/commit/b6d67191804dd0933476fede36754a436b48d1fc
- [x] #4607
- [x] #4416
  - lccc
  - Fixed with "Release 2.3.0" commit: https://huggingface.co/datasets/lccc/commit/8b1f8cf425b5653a0a4357a53205aac82ce038d1
- [x] #4367



Nice!!
Resume failed process
State: open

**Is your feature request related to a problem? Please describe.**
When a process (`map`, `filter`, etc.) crashes part-way through, you lose all progress.

**Describe the solution you'd like**
It would be good if the cache reflected the partial progress, so that after we restart the script, the process can restart where it left off.

**Describe alternatives you've considered**
Doing processing outside of `datasets`, by writing the dataset to json files and building a restart mechanism myself.

**Additional context**
N/A
Filtering on an empty dataset returns a corrupted dataset.
State: closed

## Describe the bug
When filtering a dataset twice, where the first result is an empty dataset, the second dataset seems corrupted.

## Steps to reproduce the bug
```python
  datasets = load_dataset("glue", "sst2")
  dataset_split = datasets['validation']
  ds_filter_1 = dataset_split.filter(lambda x: False)  # Some filtering condition that leads to an empty dataset
  assert ds_filter_1.num_rows == 0
  sentences = ds_filter_1['sentence']
  assert len(sentences) == 0
  ds_filter_2 = ds_filter_1.filter(lambda x: False)  # Some other filtering condition
  assert ds_filter_2.num_rows == 0
  assert 'sentence' in ds_filter_2.column_names
  sentences = ds_filter_2['sentence']
```

## Expected results
The last line should be returning an empty list, same as 4 lines above.

## Actual results
The last line currently raises `IndexError: index out of bounds`.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.5.2
- Platform: macOS-11.6.6-x86_64-i386-64bit
- Python version: 3.9.11
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


~~It seems like #5043 fix (merged recently) is the root cause of such behaviour. When we empty indices mapping (because the dataset length equals to zero), we can no longer get column item like: `ds_filter_2['sentence']` which uses
`ds_filter_1._indices.column(0)`~~

**UPDATE:**
Empty datasets are returned without going through partial function on `map` method, which will not work to get indices for `filter`: we need to run `get_indices_from_mask_function` partial function on the dataset to get output = `{"indices": []}`. But this is complicated since functions used in args, in particular `get_indices_from_mask_function`, do not support empty datasets.
We can just handle empty datasets aside on filter method.

#self-assign

Thank you for solving this amazingly quickly!
IterableDataset formatting in numpy/torch/tf/jax
State: closed

This code now returns a numpy array:
```python
from datasets import load_dataset

ds = load_dataset("imagenet-1k", split="train", streaming=True).with_format("np")
print(next(iter(ds))["image"])
```

It also works with "arrow", "pandas", "torch", "tf" and "jax"

### Implementation details:

I'm using the existing code to format an Arrow Table to the right output format for simplicity.
Therefore it's probbaly not the most optimized approach.

For example to output PyTorch tensors it does this for every example:

python data -> arrow table -> numpy extracted data -> pytorch formatted data

### Releasing this feature

Even though I consider this as a bug/inconsistency, this change is a breaking change.
And I'm sure some users were relying on the torch iterable dataset to return PIL Image and used data collators to convert to pytorch.

So I guess this is `datasets` 3.0 ?

### TODO

- [x] merge https://github.com/huggingface/datasets/pull/5072
- [ ] docs
- [ ] tests

Close https://github.com/huggingface/datasets/issues/5083

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5084). All of your documentation changes will be reflected on that endpoint.

Actually I'm not happy with this implementation. It always require the iterable dataset to have definite `features`, which removes a lot of flexibility. So I think we need an actual formatting from python objects, not from arrow data.

Closing this one since it has too many conflicts and still require some work - it will be easier to open a new PR
Support numpy/torch/tf/jax formatting for IterableDataset
State: closed

Right now `IterableDataset` doesn't do any formatting.

In particular this code should return a numpy array:
```python
from datasets import load_dataset

ds = load_dataset("imagenet-1k", split="train", streaming=True).with_format("np")
print(next(iter(ds))["image"])
```

Right now it returns a PIL.Image.

Setting `streaming=False` does return a numpy array after #5072 

hii @lhoestq, can you assign this issue to me? Though i am new to open source still I would love to put my best foot forward. I can see there isn't anyone right now assigned to this issue.

Hi @zutarich ! This issue was fixed by #5852 - sorry I forgot to close it

Feel free to look for other issues and ping me or @mariosasko if you have questions :)
Also let us know if we can help find an issue that can correspond to what you're looking for
adding keep in memory
State: closed

Fixing #514 .

Hello @mariosasko 👋, I have implemented what you have recommanded to fix the keep in memory problem for shuffle on the issue #514 . 

_The documentation is not available anymore as the PR was closed or merged._

Hi @mariosasko , I have added a test for the `keep_in_memory` version. I have also removed the `Compatible with temp_seed` part in the scope of `dset_shuffled`, please verify if that makes sense.
Use hfh for caching
State: open

## Is your feature request related to a problem?

As previously discussed in our meeting with @Wauplin and agreed on our last datasets team sync meeting, I'm investigating how `datasets` can use `hfh` for caching.

## Describe the solution you'd like

Due to the peculiarities of the `datasets` cache, I would propose adopting `hfh` caching system in stages.

First, we could easily start using `hfh` caching for:
- dataset Python scripts
- dataset READMEs
- dataset infos JSON files (now deprecated)

Second, we could also use `hfh` caching for data files downloaded from the Hub.

Further investigation is needed for:
- files downloaded from non-Hub hosts
- extracted files from downloaded archive/compressed files
- generated Arrow files

## Additional context

Docs about the `hfh` caching system:
- [Manage huggingface_hub cache-system](https://huggingface.co/docs/huggingface_hub/main/en/how-to-cache)
- [Cache-system reference](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/cache)

The `transformers` library has already adopted `hfh` for caching. See:
- huggingface/transformers#18438
- huggingface/transformers#18857
- huggingface/transformers#18966


There is some discussion in https://github.com/huggingface/huggingface_hub/pull/1088 if it can help :)
refactor: replace AssertionError with more meaningful exceptions (#5074)
State: closed

Closes #5074

Replaces `AssertionError` in the following files with more descriptive exceptions:

- `src/datasets/arrow_reader.py`
- `src/datasets/builder.py`
- `src/datasets/utils/version.py`

The issue listed more files that needed to be fixed, but the rest of them were contained in the top-level `datasets` directory, which was removed when #4974 was merged

_The documentation is not available anymore as the PR was closed or merged._
Fix header level in Audio docs
State: closed

Fixes header level so `Dataset features` is the doc title instead of `The Audio type`:

![Screen Shot 2022-10-05 at 1 22 02 PM](https://user-images.githubusercontent.com/59462357/194155840-eeb5d62f-f4eb-411e-b281-8494c5fffdce.png)

_The documentation is not available anymore as the PR was closed or merged._
Throw EnvironmentError when token is not present
State: closed

Throw EnvironmentError instead of OSError ([link](https://github.com/huggingface/datasets/blob/6ad430ba0cdeeb601170f732d4bd977f5c04594d/src/datasets/arrow_dataset.py#L4306) to the line) in `push_to_hub` when the Hub token is not present.

@mariosasko I've raised a PR #5076 against this issue. Please help to review. Thanks.
Replace AssertionErrors with more meaningful errors
State: closed

Replace the AssertionErrors with more meaningful errors such as ValueError, TypeError, etc.

The files with AssertionErrors that need to be replaced:
```
src/datasets/arrow_reader.py
src/datasets/builder.py
src/datasets/utils/version.py
```

Hi, can I pick up this issue?

#self-assign

Looks like the top-level `datasource` directory was removed when https://github.com/huggingface/datasets/pull/4974 was merged, so there are 3 source files to fix.
Restore saved format state in `load_from_disk`
State: closed

Hello! @mariosasko 

This pull request relates to issue #5050 and intends to add the format to datasets loaded from disk. 
All I did was add a set_format in the Dataset.load_from_disk, as DatasetDict.load_from_disk relies on the first.

I don't know if I should add a test and where, so let me know if I should and I can work on that as well!


_The documentation is not available anymore as the PR was closed or merged._
Image & Audio formatting for numpy/torch/tf/jax
State: closed

Added support for image and audio formatting for numpy, torch, tf and jax.

For images, the dtype used is the one of the image (the one returned by PIL.Image), e.g. uint8

I also added support for string, binary and None types. In particular for torch and jax, strings are kept unchanged (previously it was returning an error because you can't create a tensor of strings)

_The documentation is not available anymore as the PR was closed or merged._

I just added a consolidation step so that numpy arrays or tensors of images are stacked together if the shapes match, instead of having lists of tensors

Feel free to review @mariosasko :)

I added a few lines in the docs and reverted the ragged numpy array change :)

ready for another review @mariosasko !
Support default config name when no builder configs
State: closed

**Is your feature request related to a problem? Please describe.**
As discussed with @stas00, we could support defining a default config name, even if no predefined allowed config names are set. That is, support `DEFAULT_CONFIG_NAME`, even when `BUILDER_CONFIGS` is not defined. 

**Additional context**
In order to support creating configs on the fly **by name** (not using kwargs), the list of allowed builder configs `BUILDER_CONFIGS` must not be set.

However, if so, then `DEFAULT_CONFIG_NAME` is not supported.


Thank you for creating this feature request, Albert.

For context this is the datatest where Albert has been helping me to switch to on-the-fly split config https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing

and the attempt to switch on-the-fly splits was here: https://huggingface.co/datasets/HuggingFaceM4/cm4-synthetic-testing/discussions/2/files

but which I had to revert since providing no split breaks at run time.
Fix CONTRIBUTING once dataset scripts transferred to Hub
State: closed

This PR updates the `CONTRIBUTING.md` guide, once the all dataset scripts have been removed from the GitHub repo and transferred to the HF Hub:
- #4974

See diff here: https://github.com/huggingface/datasets/commit/e3291ecff9e54f09fcee3f313f051a03fdc3d94b

Additionally, this PR fixes the line separator that by some previous mistake was CRLF instead of LF.

_The documentation is not available anymore as the PR was closed or merged._
Ci py3.10
State: closed

Added a CI job for python 3.10

Some dependencies don't work on 3.10 like apache beam, so I remove them from the extras in this case.
I also removed some s3 fixtures that we don't use anymore (and that don't work on 3.10 anyway)

_The documentation is not available anymore as the PR was closed or merged._

Does it sound good to you @albertvillanova ?
Fix CI hfh token warning
State: closed

In our CI, we get warnings from `hfh` about using deprecated `token`: https://github.com/huggingface/datasets/actions/runs/3174626525/jobs/5171672431
```
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_private
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files_with_max_shard_size
tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_overwrite_files
  C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\huggingface_hub\utils\_deprecation.py:97: FutureWarning: Deprecated argument(s) used in 'dataset_info': token. Will not be supported from version '0.12'.
    warnings.warn(message, FutureWarning)
```

This PR fixes the tests in `TestPushToHub` so that we fix these warnings.

Continuation of:
- #5031

CC: @lhoestq 

_The documentation is not available anymore as the PR was closed or merged._

good catch !
Mark CI tests as xfail when 502 error
State: closed

To make CI more robust, we could mark as xfail when the Hub raises a 502 error (besides 500 error):

- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_to_hub_skip_identical_files
  - https://github.com/huggingface/datasets/actions/runs/3174626525/jobs/5171672431
  ```
  >           raise HTTPError(http_error_msg, response=self)
  E           requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16648055339047.git/info/lfs/objects/batch
  ```

- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_overwrite_files
  - https://github.com/huggingface/datasets/actions/runs/3145587033/jobs/5113074889
  ```
  >           raise HTTPError(http_error_msg, response=self)
  E           requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16643866807322.git/info/lfs/objects/verify
  ```

Currently, we mark as xfail when 500 error:
- #4845

_The documentation is not available anymore as the PR was closed or merged._
Fix broken URL's (GEM)
State: closed

This PR fixes the broken URL's in GEM. cc. @lhoestq, @albertvillanova 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5056). All of your documentation changes will be reflected on that endpoint.

Thanks, @manandey. We have removed all dataset scripts from this repo. Subsequent PRs should be opened directly on the Hugging Face Hub.
Fix backward compatibility for dataset_infos.json
State: closed

While working on https://github.com/huggingface/datasets/pull/5018 I noticed a small bug introduced in #4926 regarding backward compatibility for dataset_infos.json

Indeed, when a dataset repo had both dataset_infos.json and README.md, the JSON file was ignored. This is unexpected: in practice it should be ignored only if the README.md has a dataset_info field, which has precedence over the data in the JSON file.

_The documentation is not available anymore as the PR was closed or merged._
Fix license/citation information of squadshifts dataset card
State: closed

This PR fixes the license/citation information of squadshifts dataset card, once the dataset owners have responded to our request for information:
- https://github.com/modestyachts/squadshifts-website/issues/1

Additionally, we have updated the mention in their website to our `datasets` library (they were referring old name `nlp`):
- https://github.com/modestyachts/squadshifts-website/pull/2#event-7500953009

_The documentation is not available anymore as the PR was closed or merged._
added from_generator method to IterableDataset class.
State: closed

Hello, 

This resolves issues #4988.

I added a method `from_generator` to class `IterableDataset`.

I modified the `read` method of input stream generator to also return Iterable_dataset.


_The documentation is not available anymore as the PR was closed or merged._

I added a test and moved the `streaming` param from `read` to `__init_`. Then, I also decided to update the `read` method of the rest of the packaged modules to account for this param. 

@hamid-vakilzadeh Are you OK with these changes? 

@mariosasko these all look great! Thanks for the updates.
Revert task removal in folder-based builders
State: closed

Reverts the removal of `task_templates` in the folder-based builders. I also added the `AudioClassifaction` task for consistency.

This is needed to fix https://github.com/huggingface/transformers/issues/19177.

I think we should soon deprecate and remove the current task API (and investigate if it's possible to integrate the `train eval index` API), but we need to update the Transformers examples before that so we don't break them.

cc @NielsRogge 

_The documentation is not available anymore as the PR was closed or merged._
Restore saved format state in `load_from_disk`
State: closed

Even though we save the `format` state in `save_to_disk`, we don't restore it in `load_from_disk`. We should fix that.
 
Reported here: https://discuss.huggingface.co/t/save-to-disk-loses-formatting-information/23815

Hi, can I work on this?

Hi, sure! Let us know if you need some pointers/help.
Fix bug with labels of eurlex config of lex_glue dataset
State: closed

Fix for a critical bug in the EURLEX dataset label list to make LexGLUE EURLEX results replicable. 

In LexGLUE (Chalkidis et al., 2022), the following is mentioned w.r.t. EUR-LEX: _"It supports four different label granularities, comprising 21, 127, 567, 7390 EuroVoc concepts, respectively. We use the 100 most frequent concepts from level 2 [...]”._ The current label list has all 127 labels, which leads to different (lower) results, as communicated by users.

Thanks!

_The documentation is not available anymore as the PR was closed or merged._

@JamesLYC88 here is the fix! Thanks again!

Thanks, @albertvillanova. When do you expect that this change will take effect when someone downloads the dataset?

The change is immediately available now, since this change we made to our library:
- #4059
Audiofolder creates empty Dataset if files same level as metadata
State: closed

## Describe the bug
When audio files are at the same level as the metadata (`metadata.csv` or `metadata.jsonl` ), the `load_dataset` returns a `DatasetDict` with no rows but the correct columns.

https://github.com/huggingface/datasets/blob/1ea4d091b7a4b83a85b2eeb8df65115d39af3766/docs/source/audio_dataset.mdx?plain=1#L88

## Steps to reproduce the bug
`metadata.csv`:
```csv
file_name,duration,transcription
./2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav,10.768,hello
```


```python
>>> audio_dataset = load_dataset("audiofolder", data_dir="/audio-data/")
>>> audio_dataset
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

I've tried, with no success,:
- setting `split` to something else so I don't get a `DatasetDict`,
- removing the `./`,
- using `.jsonl`.


## Expected results
```
 Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 1
})
```

## Actual results
```
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.13.0-1025-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.0



Hi! Unfortunately, I can't reproduce this behavior. Instead, I get `ValueError: audio at 2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav doesn't have metadata in /audio-data/metadata.csv`, which can be fixed by removing the `./` from the file name.

(Link to a Colab that tries to reproduce this behavior: https://colab.research.google.com/drive/1IhQzULYi0Van1xLrN_SddBX1JF7mLZZK?usp=sharing)

I think we can make the file name matching part more robust by replacing `file_name` with `os.path.normpath(file_name)`, to ignore "./" among other things, in these two places:
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L319
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L388

@mariosasko Some tests failed (see my PR). Any thoughts on that?

Yes, I mentioned the solution in my review.

I realized what I was doing wrong.

The documentation puts the files in a subfolder.
Once I have done that, it worked.

But l agree that this should be handled better if possible.
Hendrycks Checksum
State: closed

Hi, 
The checksum for [hendrycks_test](https://huggingface.co/datasets/hendrycks_test) does not compare correctly, I guess it has been updated on the remote.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://people.eecs.berkeley.edu/~hendrycks/data.tar']
```


Thanks for reporting, @DanielHesslow. We are fixing it. 

@albertvillanova thanks for taking care of this so quickly!

The dataset metadata is fixed. You can download it normally.
`Dataset.unique` showing wrong output after filtering
State: closed

## Describe the bug
After filtering a dataset, and if no samples remain, `Dataset.unique` will return the unique values of the unfiltered dataset.

## Steps to reproduce the bug
```python
from datasets import Dataset

dataset = Dataset.from_dict({'id': [0]})
dataset = dataset.filter(lambda _: False)
print(dataset.unique('id'))
```

## Expected results
The above code should return an empty list since the dataset is empty.

## Actual results
```bash
[0]
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.18.19-100.fc35.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.14
- PyArrow version: 7.0.0
- Pandas version: 1.3.5

Hi! It seems like `flatten_indices` (called in `unique`) doesn't know how to handle empty indices mappings. I'm working on the fix.

Thanks, that was fast!
Add oversampling strategy iterable datasets interleave
State: closed

Hello everyone,
Following the issue #4893 and the PR #4831, I propose here an oversampling strategy for a `IterableDataset` list.  
The `all_exhausted` strategy stops building the new dataset as soon as all samples in each dataset have been added at least once. 
It follows roughly the same logic behind #4831, namely:
- if ``probabilities`` is `None` and the strategy is `all_exhausted`, it simply performs a round robin interleaving that stops when the longest dataset is out of samples. Here the new dataset length will be $maxLengthDataset*nbDataset$.
- if ``probabilities`` is not `None` and the strategy is `all_exhausted`, it keeps trace of the datasets which were out of samples but continues to add them to the new dataset, and stops as soons as every dataset runs out of samples at least once.

In order to be consistent and also to align with the `Dataset` behavior, please note that the behavior of the default strategy (`first_exhausted`) has been changed. Namely, it really stops when a dataset is out of samples whereas it used to stop when receiving the `StopIteration` error.  
To give an example of the last note, with the following snippet:
```
>>> from tests.test_iterable_dataset import *
>>> d1 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [0, 1, 2]])), {}))
>>> d2 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [10, 11, 12, 13]])), {}))
>>> d3 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [20, 21, 22, 23, 24]])), {}))
>>> dataset = interleave_datasets([d1, d2, d3])
>>> [x["a"] for x in dataset]
```
The result here will then be `[10, 0, 11, 1, 2]` instead of `[10, 0, 11, 1, 2, 20, 12, 13]`.

I modified the behavior because I found it to be consistent with the under/oversampling approach and because it unified the undersampling and oversampling code, but I stay open to any suggestions.


_The documentation is not available anymore as the PR was closed or merged._