noamaanMulla-03/datasets-issues huggingface dataset:
using torchrun, save_to_disk suddenly shows SIGTERM

### Describe the bug

When I run my code using the "torchrun" command, when the code reaches the "save_to_disk" part, suddenly I get the following warning and error messages:
Because the dataset is too large, the "save_to_disk" function splits it into 70 parts for saving. However, an error occurs suddenly when it reaches the 14th shard.
WARNING: torch.distributed.elastic.multiprocessing.api: Sending process 2224968 closing signal SIGTERM
ERROR: torch.distributed.elastic.multiprocessing.api: failed (exitcode: -7). traceback: Signal 7 (SIGBUS) received by PID 2224967.

### Steps to reproduce the bug

ds_shard = ds_shard.map(map_fn, *args, **kwargs)
ds_shard.save_to_disk(ds_shard_filepaths[rank])

Saving the dataset (14/70 shards):  20%|██        | 875350/4376702 [00:19<01:53, 30863.15 examples/s]
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2224968 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 2224967) of binary: /home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/python
Traceback (most recent call last):
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
==========================================================
run.py FAILED
----------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
----------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-12-08_20:09:04
  rank      : 0 (local_rank: 0)
  exitcode  : -7 (pid: 2224967)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2224967



### Expected behavior

I hope it can save successfully without any issues, but it seems there is a problem.

### Environment info

`datasets` version: 2.14.6
- Platform: Linux-4.19.90-24.4.v2101.ky10.aarch64-aarch64-with-glibc2.28
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 14.0.0
- Pandas version: 2.1.2
load_dataset imageflder for aws s3 path 

### Feature request

I would like to load a dataset from S3 using the imagefolder option 
something like 
`dataset = datasets.load_dataset('imagefolder', data_dir='s3://.../lsun/train/bedroom', fs=S3FileSystem(), streaming=True)  `

### Motivation

no need of data_files

### Your contribution

no experience with this

Labels: name: enhancement
New Preprocessing Feature  - Deduplication [Request]

**Is your feature request related to a problem? Please describe.**
Many large datasets are full of duplications and it has been shown that deduplicating datasets can lead to better performance while training, and more truthful evaluation at test-time.

A feature that allows one to easily deduplicate a dataset can be cool!

**Describe the solution you'd like**
We can define a function and keep only the first/last data-point that yields the value according to this function.

**Describe alternatives you've considered**
The clear alternative is to repeat a clear boilerplate every time someone want to deduplicate a dataset.


Labels: name: duplicate

name: enhancement

Hi! The [datasets_sql](https://github.com/mariosasko/datasets_sql) package lets you easily find distinct rows in a dataset (an example with `SELECT DISTINCT` is in the readme). Deduplication is (still) not part of the official API because it's hard to implement for datasets bigger than RAM while only using the native PyArrow ops.

(Btw, this is a duplicate of https://github.com/huggingface/datasets/issues/2514)

Here is an example using the [datasets_sql](https://github.com/mariosasko/datasets_sql) mentioned 

```python 
from datasets_sql import query

dataset = load_dataset("imdb", split="train")

# If you dont have an id column just add one by enumerating
dataset=dataset.add_column("id", range(len(dataset)))

id_column='id'
unique_column='text'

# always selects min id
unique_dataset = query(f"SELECT dataset.* FROM dataset JOIN (SELECT MIN({id_column}) as unique_id FROM dataset group by {unique_column}) ON unique_id=dataset.{id_column}")
```
Not ideal for large datasets but good enough for basic cases.
Sure would be nice to have in the library 🤗
`load_dataset(...,save_infos=True)` not working without loading script

### Describe the bug

It seems that saving a dataset infos back into the card file is not working for datasets without a loading script.

After tracking the problem a bit it looks like saving the infos uses `Builder.get_imported_module_dir()` as its destination directory.
Internally this is a call to `inspect.getfile()` but since the actual builder class used is dynamically created (cf. `datasets.load.configure_builder_class`) this method actually return te path to the parent builder class (e.g. `datasets.packaged_modules.json.JSON`). 



### Steps to reproduce the bug

1. Have a local dataset without any loading script
2. Make sure there are no dataset infos in the README.md
3. Load with `save_infos=True`
4. No change in the dataset README.md
5. A new README.md file is created in the directory of the parent builder class (e.g. for json in `.../site-packages/datasets/packaged_modules/json/README.md`)

### Expected behavior

The dataset README.md should be updated and no file should be created in the python environment.

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.6.0



Also, once the README.md exists in the python environment it is used when loading another dataset in the same format (e.g. json) since it always resolves the path to the same directory.
The consequence here is any other dataset won't load because of infos mismatch.
To reproduce this aspect:
1. Do a `load_datasets(...,save_infos=True)` with one dataset without a loading script
2. Try to load another dataset without a loading script in the same format (e.g. json) but with a different schema
Trouble Loading a Gated Dataset For User with Granted Permission

### Describe the bug

I have granted permissions to several users to access a gated huggingface dataset. The users accepted the invite and when trying to load the dataset using their access token they get 
`FileNotFoundError: Couldn't find a dataset script at .....` . Also when they try to click the url link for the dataset they get a 404 error.

### Steps to reproduce the bug

1. Grant access to gated dataset for specific users 
2. Users accept invitation
3. Users login to hugging face hub using cli login
4. Users run load_dataset

### Expected behavior

Dataset is loaded normally for users who were granted access to the gated dataset.

### Environment info

datasets==2.15.0



> Also when they try to click the url link for the dataset they get a 404 error.

This seems to be a Hub error then (cc @SBrandeis)

Could you report this to https://discuss.huggingface.co/c/hub/23, providing the URL of the dataset, or at least if the dataset is public or private?

Thanks for the reply! I've created an issue on the hub's board here: https://discuss.huggingface.co/t/trouble-loading-a-gated-dataset-for-user-with-granted-permission/65565
Image Data loaded Twice

### Describe the bug

![1702472610561](https://github.com/huggingface/datasets/assets/28867010/4b7ef5e7-32c3-4b73-84cb-5de059caa0b6)
When I learn from https://huggingface.co/docs/datasets/image_load and try to load image data from a folder. I noticed that the image was read twice in the returned data. As you can see in the attached image, there are only four images in the train folder, but reading brings up eight images

### Steps to reproduce the bug

from datasets import Dataset, load_dataset

dataset = load_dataset("imagefolder", data_dir="data/", drop_labels=False)
# print(dataset["train"][0]["image"] == dataset["train"][1]["image"])

print(dataset)
print(dataset["train"]["image"])
print(len(dataset["train"]["image"]))

### Expected behavior

DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 8
    })
})
[<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D1CA8B0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D2452E0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245310>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2453A0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245460>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245430>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2454F0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245550>]
8

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.17
- Huggingface_hub version: 0.19.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3
FileNotFoundError: [Errno 2] No such file or directory: 'nul'

### Describe the bug

it seems that sth wrong with my terrible "bug body" life, When i run this code, "import datasets"
i meet this error FileNotFoundError: [Errno 2] No such file or directory: 'nul'
![image](https://github.com/huggingface/datasets/assets/73683903/3973c120-ebb1-42b7-bede-b9de053e861d)
![image](https://github.com/huggingface/datasets/assets/73683903/0496adff-a7a7-4dcb-929e-ec11ede71f04)


### Steps to reproduce the bug

1.import datasets

### Expected behavior

i just run a single line code and stuct in this bug 

### Environment info

OS: Windows10
Datasets==2.15.0
python=3.10


Hi! It seems like the problem is your environment. Maybe this issue can help: https://github.com/pytest-dev/pytest/issues/9519.
Error when writing a dataset to HF Hub: A commit has happened since. Please refresh and try again.

**Describe the bug**

Getting a `412 Client Error: Precondition Failed` when trying to write a dataset to the HF hub.

```
huggingface_hub.utils._errors.HfHubHTTPError: 412 Client Error: Precondition Failed for url: https://huggingface.co/api/datasets/GLorr/test-dask/commit/main (Request ID: Root=1-657ae26f-3bd92bf861bb254b2cc0826c;50a09ab7-9347-406a-ba49-69f98abee9cc)

A commit has happened since. Please refresh and try again.
```

**Steps to reproduce the bug**
This is a minimal reproducer:
```
import dask.dataframe as dd
import pandas as pd
import random
import os

import huggingface_hub

import datasets

huggingface_hub.login(token=os.getenv("HF_TOKEN"))

data = {"number": [random.randint(0,10) for _ in range(1000)]}
df = pd.DataFrame.from_dict(data)
dataframe = dd.from_pandas(df, npartitions=1)
dataframe = dataframe.repartition(npartitions=3)

schema = datasets.Features({"number": datasets.Value("int64")}).arrow_schema

repo_id = "GLorr/test-dask"
repo_path = f"hf://datasets/{repo_id}"
huggingface_hub.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True)
dd.to_parquet(dataframe, path=f"{repo_path}/data", schema=schema)
```

**Expected behavior**
Would expect to write to the hub without any problem.

**Environment info**
```
datasets==2.15.0
huggingface-hub==0.19.4
```



I transferred from datasets-server, since the issue is more about `datasets` and the integration with `huggingface_hub`.
Increase speed of data files resolution

Certain datasets like `bigcode/the-stack-dedup` have so many files that loading them takes forever right from the data files resolution step.

`datasets` uses file patterns to check the structure of the repository but it takes too much time to iterate over and over again on all the data files.

This comes from `resolve_patterns_in_dataset_repository` which calls `_resolve_single_pattern_in_dataset_repository`, which iterates on all the files at
```python
glob_iter = [PurePath(filepath) for filepath in fs.glob(PurePath(pattern).as_posix()) if fs.isfile(filepath)]
```
but calling `glob` on such a dataset is too expensive. Indeed it calls `ls()` in `hffilesystem.py` too many times.

Maybe `glob` can be more optimized in `hffilesystem.py`, or the data files resolution can directly be implemented in the filesystem by checking its `dir_cache` ?

Labels: name: enhancement

name: good second issue

#self-assign

You were right, if `self.dir_cache` is not None in glob, it is exactly the same as what is returned by find, at least for all the tests we have, and some extended evaluation I did across a random sample of about 1000 datasets. 

Thanks for the nice hints, and let me know if this is not exactly what we want here!

see PR: https://github.com/huggingface/datasets/pull/5704



I think we can make the data files resolution (significantly) faster in 2 steps:

1. `glob` calls `find` (which in turn calls `ls`), so we need `find` to be fast, and this can be achieved by fetching all the entries in a single API call and avoiding calls to `ls`. Implementing this for `HfFileSystem.find` (the one in `huggingface_hub`) is on my TO-DO list.
2. caching the repeated `find` calls in `_get_data_files_patterns` when the `data_files` patterns are not provided in `load_dataset`. To address this, we can introduce a `_resolve_single_pattern` function that would accept a filesystem object and a list of regex patterns to resolve. Then we can wrap this filesystem object in `_get_data_files_patterns` with an object that would cache the find calls before resolving the patterns with `_resolve_single_pattern`. (Feel free to suggest a cleaner implementation)

WDYT?

Good idea :) 

For 2:

That would work ! It's also possible to have a FileSystem with a cache on `.find` and use it inside the resolver passed to `_get_data_files_patterns`. Right now they're pretty simple:

```python
# for remote repositories
resolver = partial(_resolve_single_pattern_in_dataset_repository, dataset_info, base_path=base_path)
# for local
resolver = partial(_resolve_single_pattern_locally, base_path)
```

something like this maybe (with Quentin's reimplementation of `HfFilesystem.find`)?

 ```
 @lru_cache(max_size=None)
 def _find(self, path, maxdepth=None, withdirs=False, detail=False, **kwargs):
```

In any case please let me know if I can help in any way!
Error Pushing to Hub

### Describe the bug

Error when trying to push a dataset in a special format to hub

### Steps to reproduce the bug

```
import datasets
from datasets import Dataset
dataset_dict = {
    "filename": ["apple", "banana"],
    "token": [[[1,2],[3,4]],[[1,2],[3,4]]],
    "label": [0, 1],
}
dataset = Dataset.from_dict(dataset_dict)
dataset = dataset.cast_column("token", datasets.features.features.Array2D(shape=(2, 2),dtype="int16"))

dataset.push_to_hub("SequenceModel/imagenet_val_256")
```
Error:
```
...

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/tuple'
  in "<unicode string>", line 8, column 16:
            shape: !!python/tuple
                   ^
```

### Expected behavior

Dataset being pushed to hub

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.19.0-1022-gcp-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
Added return methods for URL-references to the pushed dataset

Hi,

I was missing the ability to easily open the pushed dataset and it seemed like a quick fix.
Maybe we also want to log this info somewhere, but let me know if I need to add that too.

Cheers,
David


Hi ! Maybe we'd need to align with `transformers` and other libraries that implement `push_to_hub` to agree on what it should return.

e.g. in `transformers` the typing says it returns a string, but in practice it returns a `CommitInfo`.

Therefore I'd not add an output to `push_to_hub` here unless we had a chance to discuss more broadly.

Anyway in my opinion it should no just return the URL of the repository, but ideally the URL at the revision where the data were pushed

Perhaps a mixin or something similar could be defined on the `hfh` side to ensure the `push_to_hub` API is aligned across our projects. 

PS: this would also mean that the PRs such as https://github.com/huggingface/datasets/pull/5528 would no longer be our responsibility

cc @Wauplin 

I agree, with universability and the idea is more about returning at least something that references where to find the uploaded file/model or otherwise. 

Ideally, the referenced PR would work.

imo this would be a good use case to just use `huggingface_hub` and align to what we do there :)

@mariosasko, can you give me some pointers to where I might help implementing this for the `huggingface-hub`?

> @mariosasko: Perhaps a mixin or something similar could be defined on the hfh side to ensure the push_to_hub API is aligned across our projects.

> @julien-c: imo this would be a good use case to just use huggingface_hub and align to what we do there :)

I (finally) opened a PR to harmonize return types: https://github.com/huggingface/huggingface_hub/pull/1921. It should hopefully be shipped in next release later this week (:crossed_fingers:).
Incorrect test set labels for RTE and CoLA datasets via load_dataset

### Describe the bug

The test set labels for the RTE and CoLA datasets when loading via datasets load_dataset are all -1.
Edit: It appears this is also the case for every other dataset except for MRPC (stsb, sst2, qqp, mnli (both matched and mismatched), qnli, wnli, ax). Is this intended behavior to safeguard the test set for evaluation purposes?

### Steps to reproduce the bug

!pip install datasets
from datasets import load_dataset

rte_data = load_dataset('glue', 'rte')
cola_data = load_dataset('glue', 'cola')
print(rte_data['test'][0:30]['label'])
print(cola_data['test'][0:30]['label'])

Output:
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]

The non-label test data seems to be fine:
e.g. rte_data['test'][1] is:
{'sentence1': "Authorities in Brazil say that more than 200 people are being held hostage in a prison in the country's remote, Amazonian-jungle state of Rondonia.",
 'sentence2': 'Authorities in Brazil hold 200 people as hostage.',
 'label': -1,
 'idx': 1}
Training and validation data are also fine:
e.g. rte_data['train][0] is:
{'sentence1': 'No Weapons of Mass Destruction Found in Iraq Yet.',
 'sentence2': 'Weapons of Mass Destruction Found in Iraq.',
 'label': 1,
 'idx': 0}


### Expected behavior

Expected the labels to be binary 0/1 values; Got all -1s instead

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0


As this is a specific issue of the "glue" dataset, I have transferred it to the dataset Discussion page: https://huggingface.co/datasets/glue/discussions/15

Let's continue the discussion there!
Dataset sizes are in MiB instead of MB in dataset cards

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations

Labels: name: bug

name: dataset-viewer

Example of bulk edit: https://huggingface.co/datasets/aeslc/discussions/5

looks great! 

Do you encode the fact that you've already converted a dataset? (to not convert it twice) or do you base yourself on the info contained in `dataset_info`

I am only looping trough the dataset cards, assuming that all of them were created with MiB.

I agree we should only run the bulk edit once for all canonical datasets: I'm using a for-loop over canonical datasets.

yes, worst case, we have this in structured data:

<img width="337" alt="image" src="https://user-images.githubusercontent.com/326577/230037051-06caddcb-08c8-4953-a710-f3d122917db3.png">


I have just included as well the conversion from MB to GB if necessary. See: 
- https://huggingface.co/datasets/bookcorpus/discussions/2/files
- https://huggingface.co/datasets/asnq/discussions/2/files

Nice. Is it another loop? Because in https://huggingface.co/datasets/amazon_us_reviews/discussions/2/files we have `32377.29 MB` for example

First, I tested some batches to check the changes made. Then I incorporated the MB to GB conversion. Now I'm running the rest.

The bulk edit parsed 751 canonical datasets and updated 166.

Thanks a lot!

The sizes now match as expected!

<img width="1446" alt="Capture d’écran 2023-04-05 à 16 10 15" src="https://user-images.githubusercontent.com/1676121/230107044-ac2a76ea-a4fe-4e81-a925-f464b85f5edd.png">


I made another bulk edit of ancient canonical datasets that were moved to community organization. I have parsed 11 datasets and opened a PR on 3 of them:
- [x] "allenai/scicite": https://huggingface.co/datasets/allenai/scicite/discussions/3
- [x] "allenai/scifact": https://huggingface.co/datasets/allenai/scifact/discussions/2
- [x] "dair-ai/emotion": https://huggingface.co/datasets/dair-ai/emotion/discussions/6

should we force merge the PR and close this issue?

I merged the PRs for "scicite" and "scifact".