noamaanMulla-03/datasets-issues huggingface dataset:
Support torch dataloader without torch formatting

In https://github.com/huggingface/datasets/pull/5084 we make the torch formatting consistent with the map-style datasets formatting: a torch formatted iterable dataset will yield torch tensors.

The previous behavior of the torch formatting for iterable dataset was simply to make the iterable dataset inherit from `torch.utils.data.Dataset` to make it work in a torch DataLoader. However ideally an unformatted dataset should also work with a DataLoader. To fix that, `datasets.IterableDataset` should inherit from `torch.utils.data.IterableDataset`.

Since we don't want to import torch on startup, I created this PR to dynamically make the `datasets.IterableDataset` class inherit form the torch one when a `datasets.IterableDataset` is instantiated and if PyTorch is available.

```python
>>> from datasets import load_dataset
>>> ds = load_dataset("c4", "en", streaming=True, split="train")
>>> import torch.utils.data
>>> isinstance(ds, torch.utils.data.IterableDataset)
True
>>> dataloader = torch.utils.data.DataLoader(ds, batch_size=32, num_workers=4)
>>> for example in dataloader:
...:     ...
```


_The documentation is not available anymore as the PR was closed or merged._

Need some more time to fix the tests, especially with pickle

> And I actually don't quite understand the idea - what's the motivation behind making only IterableDataset compatible with torch DataLoader without setting the format explicitly?

Setting the format to pytorch = set the output types of the dataset to be pytorch tensors. However sometimes your dataset is not made of tensors but you still want to be able to use a pytorch DataLoader

A bit more context. 

The arrow-backed `Dataset` supports `DataLoader(ds)` (even if the format is not "torch"), and we want to be able to do the same with `IterableDataset` for consistency. However, this is when the PyTorch internals come into play - an iterable dataset needs to be an instance of `torch.utils.data.IterableDataset` due to [this](https://github.com/pytorch/pytorch/blob/abc54f93145830b502400faa92bec86e05422fbd/torch/utils/data/dataloader.py#L276) check (notice there is no check for the map-style version). Hence the explicit subclassing in this PR.

Exactly :) Btw I just took your comments into account @polinaeterna , so feel free to review again

@lhoestq just checking, does this change still preserve the fix to the "data duplicate when setting num_works > 1 with streaming data" issue from before?

https://github.com/huggingface/datasets/issues/3423

Yes :)
Support Dataset conversion from/to Spark

This PR implements Spark integration by supporting `Dataset` conversion from/to Spark `DataFrame`.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5401). All of your documentation changes will be reflected on that endpoint.

Cool thanks !

Spark DataFrame are usually quite big, and I believe here `from_spark` would load everything in the driver node's RAM, which is quite limiting. Same for `to_spark` which would load everything in the driver node's RAM before sending the data to the executor. Maybe we can mention this in the docstring ?

To transfer big datasets from/into the HF ecosystem using Spark maybe we can just make sure that `pyspark` can read/write to the HF Hub, and that `datasets` can read from HDFS/S3/etc.

Yes @lhoestq , consider this as a first integration of the Datasets library with Spark.
- This PR implements the basic conversion between both.
  - And yes, we are using the Spark's `pandas` API (that uses `pyarrow` under the hood): everything is transferred to the driver.
  - Note that we are converting from/to a Datasets dataset: this is not distributed

The next step is to support the integration of the HF Hub with Spark, that I think should be done using `hffs`.

Thinking more about it I don't really see how those two methods help in practice, since one can already do `datasets` <-> pandas <-> spark and those two methods don't add value over this.

However I think it can be good documentation to explain that it's possible to do it and it's super simple
Datasets.from_sql() generates deprecation warning

### Describe the bug

Calling `Datasets.from_sql()` generates a warning:

`.../site-packages/datasets/builder.py:712: FutureWarning: 'use_auth_token' was deprecated in version 2.7.1 and will be removed in 3.0.0. Pass 'use_auth_token' to the initializer/'load_dataset_builder' instead.`

### Steps to reproduce the bug

Any valid call to `Datasets.from_sql()` will produce the deprecation warning.

### Expected behavior

No warning.

The fix should be simply to remove the parameter `use_auth_token` from the call to `builder.download_and_prepare()` at line 43 of `io/sql.py` (it is set to `None` anyway, and is not needed).

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-4.15.0-169-generic-x86_64-with-glibc2.27
- Python version: 3.9.15
- PyArrow version: 10.0.1
- Pandas version: 1.5.2



Thanks for reporting @msummerfield. We are fixing it.
Type error when calling `map` over dataset containing 0-d tensors

### Describe the bug

0-dimensional tensors in a dataset lead to `TypeError: iteration over a 0-d array` when calling `map`.  It is easy to generate such tensors by using `.with_format("...")` on the whole dataset.

### Steps to reproduce the bug

```
ds = datasets.Dataset.from_list([{"a": 1}, {"a": 1}]).with_format("torch")
ds.map(None)
```

### Expected behavior

Getting back `ds` without errors.

### Environment info

Python 3.10.8
datasets 2.6.
torch 1.13.0


Hi! 

We could address this by calling `.item()` on such tensors to extract the value, but this would lose us the type, which could lead to storing the generated dataset in a suboptimal format. Considering this, I think the only proper fix would be implementing support for 0-D tensors on Apache Arrow's side (Arrow is the underlying format we use to store datasets on disk/in memory). WDYT @lhoestq?

I think we can just convert the item to a numpy typed scalar using `.numpy()` ?

For example this works:
```python
import numpy as np
import pyarrow as pa

assert pa.array([np.float64(1.0)]).type == pa.float64()
assert pa.array([np.float32(1.0)]).type == pa.float32()
assert pa.array([np.int32(1)]).type == pa.int32()
assert pa.array([np.int64(1)]).type == pa.int64()
```

And therefore it would work the same as for PyTorch N-D Tensors: convert to Numpy Array to keep the type in `_cast_to_python_objects`, then convert to Arrow
Support case-insensitive Hub dataset name in load_dataset

### Feature request

The dataset name on the Hub is case-insensitive (see https://github.com/huggingface/moon-landing/pull/2399, internal issue), i.e., https://huggingface.co/datasets/GLUE redirects to https://huggingface.co/datasets/glue.

Ideally, we could load the glue dataset using the following:

```
from datasets import load_dataset
load_dataset('GLUE', 'cola')
```

It breaks because the loading script `GLUE.py` does not exist (`glue.py` should be selected instead).

Minor additional comment: in other cases without a loading script, we can load the dataset, but the automatically generated config name depends on the casing:
- `load_dataset('severo/danish-wit')` generates the config name `severo--danish-wit-e6fda5b070deb133`, while
- `load_dataset('severo/danish-WIT')` generates the config name `severo--danish-WIT-e6fda5b070deb133`

### Motivation

To follow the same UX on the Hub and in the datasets library.

### Your contribution

...

Labels: name: enhancement

Closing as case-insensitivity should be only for URL redirection on the Hub. In the APIs, we will only support the canonical name (https://github.com/huggingface/moon-landing/pull/2399#issuecomment-1382085611)
feat(dataset): multiprocessing _generate_examples

forking this out of #741, this issue is only regarding multiprocessing

I'd love if there was a dataset configuration parameter `workers`, where when it is `1` it behaves as it does right now, and when its `>1` maybe `_generate_examples` can also get the `pool` and return an iterable using the pool.

In my use case, I would instead of:
```python
for datum in data:
     yield self.load_datum(datum)
```
do:
```python
return pool.map(self.load_datum, data)
```

As the dataset in question, as an example, has **only** 7000 rows, and takes 10 seconds to load each row on average, it takes almost 20 hours to load the entire dataset.
If this was a larger dataset (and many such datasets exist), it would take multiple days to complete.

Using multiprocessing, for example, 40 cores, could speed it up dramatically. For this dataset, hopefully to fully load in under an hour.


I agree that would be cool :)
Right now the only distributed dataset builder is based on Apache Beam so you can use distributed processing frameworks like Dataflow, Spark, Flink etc. to build your dataset but it's not really well suited for single-worker parallel processing afaik

`_generate_examples` can now be run in parallel thanks to https://github.com/huggingface/datasets/pull/5107. You can find more info [here](https://huggingface.co/docs/datasets/dataset_script#sharding).
Force soundfile to return float32 instead of the default float64 

(Fixes issue #5345)


cc @polinaeterna

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5347). All of your documentation changes will be reflected on that endpoint.

Cool ! Feel free to add a comment in the code to explain that and we can merge :)

I'm not sure if this is a good change since we plan to get rid of `torchaudio` in the next couple of months...

What do you think @polinaeterna @patrickvonplaten ? Models are usually using float32 (e.g. Wev2vec2 in `transformers`) IIRC

IMO we can safely assume that float32 is always good enough when using audio models in inference or training. Nevertheless there might be use cases for audio datasets in the future where float64 is needed. 

=> I would by default always cast to float32, but then possible allow the user to cast to float64 

> I'm not sure if this is a good change since we plan to get rid of torchaudio in the next couple of months...

@mariosasko I agree but who knows how long we will have to wait until we are really able to do so (https://github.com/bastibe/libsndfile-binaries/pull/17 is a draft. so as @patrickvonplaten is okay with float32, I'd merge.




@polinaeterna Can you comment on the linked PR to see why it's still a draft? Maybe we can help somehow to get this merged finally.

I think it's weird to align `soundfile` with `torchaudio` when the latter is only used for MP3 (and prob for not much longer).
Support latest Docker image in CI benchmarks

Once we find out the root cause of:
- #5431 

we should revert the temporary pin on the Docker image version introduced by:
- #5432

Labels: name: enhancement

Sorry, it was us:[^1] https://github.com/iterative/cml/pull/1317 & https://github.com/iterative/cml/issues/1319#issuecomment-1385599559; should be fixed with [v0.18.17](https://github.com/iterative/cml/releases/tag/v0.18.17).

[^1]: More or less, see https://github.com/yargs/yargs/issues/873.

Opened https://github.com/huggingface/datasets/pull/5436 unpinning again the container image.

Hi @0x2b3bfa0, thanks a lot for the investigation, the context about the the root cause and for fixing it!!

We are reviewing your PR to unpin the container image.
CI benchmarks are broken: Unknown arguments: runnerPath, path

Our CI benchmarks are broken, raising `Unknown arguments` error: https://github.com/huggingface/datasets/actions/runs/3932397079/jobs/6724905161
```
Unknown arguments: runnerPath, path
```

Stack trace:
```
100%|██████████| 500/500 [00:01<00:00, 338.98ba/s]
Updating lock file 'dvc.lock'

To track the changes with git, run:

	git add dvc.lock

To enable auto staging, run:

	dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.
cml send-comment <markdown file>

Global Options:
  --log                    Logging verbosity
          [string] [choices: "error", "warn", "info", "debug"] [default: "info"]
  --driver                 Git provider where the repository is hosted
    [string] [choices: "github", "gitlab", "bitbucket"] [default: infer from the
                                                                    environment]
  --repo                   Repository URL or slug
                                  [string] [default: infer from the environment]
  --driver-token, --token  CI driver personal/project access token (PAT)
                                  [string] [default: infer from the environment]
  --help                   Show help                                   [boolean]

Options:
  --target                    Comment type (`commit`, `pr`, `commit/f00bar`,
                              `pr/42`, `issue/1337`),default is automatic (`pr`
                              but fallback to `commit`).                [string]
  --watch                     Watch for changes and automatically update the
                              comment                                  [boolean]
  --publish                   Upload any local images found in the Markdown
                              report                   [boolean] [default: true]
  --publish-url               Self-hosted image server URL
                                     [string] [default: "https://asset.cml.dev/"]
  --publish-native, --native  Uses driver's native capabilities to upload assets
                              instead of CML's storage; not available on GitHub
                                                                       [boolean]
  --watermark-title           Hidden comment marker (used for targeting in
                              subsequent `cml comment update`); "{workflow}" &
                              "{run}" are auto-replaced   [string] [default: ""]

Unknown arguments: runnerPath, path
Error: Process completed with exit code 1.
```

Issue reported to iterative/cml:
- iterative/cml#1319

Labels: name: maintenance
Unable to download dataset id_clickbait

### Describe the bug

I tried to download dataset `id_clickbait`, but receive this error message.

```
FileNotFoundError: Couldn't find file at https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/k42j7x2kpn-1.zip
```

When i open the link using browser, i got this XML data.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>md-datasets-cache-zipfiles-prod</BucketName><RequestId>NVRM6VEEQD69SD00</RequestId><HostId>W/SPDxLGvlCGi0OD6d7mSDvfOAUqLAfvs9nTX50BkJrjMny+X9Jnqp/Li2lG9eTUuT4MUkAA2jjTfCrCiUmu7A==</HostId></Error>
```

### Steps to reproduce the bug

Code snippet:

```
from datasets import load_dataset

load_dataset('id_clickbait', 'annotated')
load_dataset('id_clickbait', 'raw')
```

Link to Kaggle notebook: https://www.kaggle.com/code/ilosvigil/bug-check-on-id-clickbait-dataset

### Expected behavior

Successfully download and load `id_newspaper` dataset.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 8.0.0
- Pandas version: 1.3.5


Thanks for reporting, @ilos-vigil.

We have transferred this issue to the corresponding dataset on the Hugging Face Hub: https://huggingface.co/datasets/id_clickbait/discussions/1
RuntimeError: Sharding is ambiguous for this dataset

### Describe the bug

When loading some datasets, a RuntimeError is raised.

For example, for "ami" dataset: https://huggingface.co/datasets/ami/discussions/3
```
.../huggingface/datasets/src/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
   1415         fpath = path_join(self._output_dir, fname)
   1416 
-> 1417         num_input_shards = _number_of_shards_in_gen_kwargs(split_generator.gen_kwargs)
   1418         if num_input_shards <= 1 and num_proc is not None:
   1419             logger.warning(

.../huggingface/datasets/src/datasets/utils/sharding.py in _number_of_shards_in_gen_kwargs(gen_kwargs)
     10     lists_lengths = {key: len(value) for key, value in gen_kwargs.items() if isinstance(value, list)}
     11     if len(set(lists_lengths.values())) > 1:
---> 12         raise RuntimeError(
     13             (
     14                 "Sharding is ambiguous for this dataset: "

RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
    - key samples_paths has length 6
    - key ids has length 7
    - key verification_ids has length 6
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.
```

This behavior was introduced when implementing multiprocessing by PR:
- #5107

### Steps to reproduce the bug

```python
ds = load_dataset("ami", "microphone-single", split="train", revision="2d7620bb7c3f1aab9f329615c3bdb598069d907a")
```

### Expected behavior

No error raised.

### Environment info

Since datasets 2.7.0
Can't load png dataset with 4 channel (RGBA)

I try to create dataset which contains about 9000 png images 64x64 in size, and they are all 4-channel (RGBA). When trying to use load_dataset() then a dataset is created from only 2 images. What exactly interferes I can not understand.![Screenshot_20230117_212213.jpg](https://user-images.githubusercontent.com/41611046/212980147-9aa68e30-76e9-4b61-a937-c2fdabd56564.jpg)


Hi! Can you please share the directory structure of your image folder and the `load_dataset` call? We decode images with Pillow, and Pillow supports RGBA PNGs, so this shouldn't be a problem.



> Hi! Can you please share the directory structure of your image folder and the `load_dataset` call? We decode images with Pillow, and Pillow supports RGBA PNGs, so this shouldn't be a problem.
> 
> 

I have only 1 folder that I use in the load_dataset function with the name "IMGDATA" and all my 9000 images are located in this folder.
`
from datasets import load_dataset

dataset = load_dataset("IMGDATA")
`
At the same time, using another data set with images consisting of 3 RGB channels, everything works

Okay, I figured out what was wrong. When uploading my dataset via Google Drive, the images broke and Pillow couldn't open them. As a result, I solved the problem by downloading the ZIP archive
Wrong statement in "Load a Dataset in Streaming mode" leads to data leakage

### Describe the bug

In the [Split your dataset with take and skip](https://huggingface.co/docs/datasets/v1.10.2/dataset_streaming.html#split-your-dataset-with-take-and-skip), it states:

> Using take (or skip) prevents future calls to shuffle from shuffling the dataset shards order, otherwise the taken examples could come from other shards. In this case it only uses the shuffle buffer. Therefore it is advised to shuffle the dataset before splitting using take or skip. See more details in the [Shuffling the dataset: shuffle](https://huggingface.co/docs/datasets/v1.10.2/dataset_streaming.html#iterable-dataset-shuffling) section.`
>> \# You can also create splits from a shuffled dataset
>> train_dataset = shuffled_dataset.skip(1000)
>> eval_dataset = shuffled_dataset.take(1000)


Where the shuffled dataset comes from:
`shuffled_dataset = dataset.shuffle(buffer_size=10_000, seed=42)`

At least in Tensorflow 2.9/2.10/2.11, [docs](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle) states the `reshuffle_each_iteration` argument is `True` by default. This means the dataset would be shuffled after each epoch, and as a result **the validation data would leak into training test**.






### Steps to reproduce the bug

N/A

### Expected behavior

The `reshuffle_each_iteration` argument should be set to `False`.

### Environment info

Tensorflow 2.9/2.10/2.11


Just for your information, Tensorflow confirmed this issue [here.](https://github.com/tensorflow/tensorflow/issues/59279)

Thanks for reporting, @HaoyuYang59.

Please note that these are different "dataset" objects: our docs refer to Hugging Face `datasets.Dataset` and not to TensorFlow `tf.data.Dataset`.

Our `datasets.Dataset.shuffle` method does not have a `reshuffle_each_iteration` argument. Therefore, I would say the statement in our docs is True because they refer to `datasets.Dataset.shuffle`, `datasets.Dataset.skip` and `datasets.Dataset.take`.

I think this issue is restricted to TensorFlow dataset, and this would be addressed by them in the issue you opened in their repo: https://github.com/tensorflow/tensorflow/issues/59279

Also note that you are referring to an outdated documentation page: datasets 1.10.2 version

Current datasets version is 2.8.0 and the corresponding documentation page is: https://huggingface.co/docs/datasets/stream#split-dataset

Hi @albertvillanova thanks for your reply and your explaination here. 

Sorry for the confusion as I'm not actually a user of your repo and I just happen to find the thread by Google (and didn't read carefully).

Great to know that and you made everything very clear now.

Thanks for your time and sorry for the consusion.

Wishing you a wonderful time. 

Regards
sample_dataset module not found



Hi! Can you describe what the actual error is?

working on the setfit example script

   from setfit import SetFitModel, SetFitTrainer, sample_dataset

ImportError: cannot import name 'sample_dataset' from 'setfit' (C:\Python\Python38\lib\site-packages\setfit\__init__.py)

 apart from that, I also had to hack these loads to import thses modules:
    from datasets.load import load_dataset                        
    from datasets.arrow_dataset import Dataset
    from datasets.dataset_dict import DatasetDict

Hi! This issue is related to the [SetFit](https://github.com/huggingface/setfit) project, so can you please open it there?
CI tests are broken: AttributeError: 'mappingproxy' object has no attribute 'target'

CI tests are broken, raising `AttributeError: 'mappingproxy' object has no attribute 'target'`. See: https://github.com/huggingface/datasets/actions/runs/3966497597/jobs/6797384185
```
...
ERROR tests/test_streaming_download_manager.py::TestxPath::test_xpath_rglob[mock://top_level-date=2019-10-0[1-4]/*-expected_paths4] - AttributeError: 'mappingproxy' object has no attribute 'target'
===== 2076 passed, 19 skipped, 15 warnings, 47 errors in 115.54s (0:01:55) =====
```

Labels: name: bug
test v0.12.0.rc0

DO NOT MERGE.
Only to test the CI.

cc @lhoestq @albertvillanova 


_The documentation is not available anymore as the PR was closed or merged._

@Wauplin I was testing it in a dedicated branch without opening a PR: https://github.com/huggingface/datasets/commits/test-hfh-0.12.0rc0

Oops, sorry @albertvillanova. I thought for next time I'll start the CIs before pinging everyone.
I'm closing this one.

@Wauplin in your Slack message, you asked people from every major dependent library to check that our CI work. That is why I am checking it... :)

Also, I think for this purpose it is better to test it in a dedicated branch, rather than opening and closing a PR.

Yes, yes I know. Completely my fault on this one