ahmedfarazsyk/datasets-github-issues huggingface dataset:
Dataset order is not deterministic with ZIP archives and `iter_files`
Author: fxmarty
State: closed

### Describe the bug

For the `beans` dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

### Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

```python
from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])
```

### Expected behavior

The order of the images is the same on all machines.

### Environment info

On the EC2 instance:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked
```

On my local laptop:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1
```

On github actions:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4
```

Comments: Thanks for reporting ! The issue doesn't come from shuffling, but from `beans` row order not being deterministic:

https://huggingface.co/datasets/beans/blob/main/beans.py uses `dl_manager.iter_files` on ZIP archives and the file order doesn't seen to be deterministic and changes across machines

Thank you for noticing indeed!

This is still a bug, so I'd keep this one open if you don't mind ;)

Besides the linked PR, to make the loading process fully deterministic, I believe we should also sort the data files [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L276) and [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L485) (e.g. fsspec's `LocalFileSystem.glob` relies on `os.scandir`, which yields the contents in arbitrary order). My concern is the overhead of these sorts... Maybe we could introduce a new flag to `load_dataset` similar to TFDS' [`shuffle_files`](https://www.tensorflow.org/datasets/determinism#determinism_when_reading) or sort only if the number of data files is small?

We already return the result sorted at the end of `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository` if I'm not mistaken

@lhoestq Oh, you are right. Feel free to ignore my comment.

I think the corresponding PR is ready to be merged :hugs: 

@albertvillanova Thanks for the fix!
Inconsistent documentation on map remove_columns
Author: zhaowei-wang-nlp
State: closed
Labels: name: documentation

name: duplicate

name: good first issue

name: hacktoberfest

### Describe the bug

The page [process](https://huggingface.co/docs/datasets/process) says this about the parameter `remove_columns` of the function `map`:
When you remove a column, it is only removed after the example has been provided to the mapped function.

So it seems that the `remove_columns` parameter removes after the mapped functions.

However, another page, [the documentation of the function map](https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.Dataset.map.remove_columns) says:
Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in remove_columns, these columns will be kept.

So one page says "after the mapped function" and another says "before the mapped function."
Is there something wrong?


### Steps to reproduce the bug

Not about code.

### Expected behavior

consistent about the descriptions of the behavior of the parameter `remove_columns` in the function `map`.

### Environment info

datasets V2.6.0

Comments: Thanks for reporting, @zhaowei-wang-nlp.

You are right, the documentation is confusing on the behavior of `remove_columns`. We should better explain it. 

This is a duplicate of https://github.com/huggingface/datasets/issues/2343.

I'm closing this issue because as @mariosasko pointed out, it is a duplicate of:
- #2343
DownloadManager Git LFS support
Author: Muennighoff
State: closed
Labels: name: enhancement

### Feature request

Maybe I'm mistaken but the `DownloadManager` does not support extracting git lfs files out of the box right?
Using `dl_manager.download()` or `dl_manager.download_and_extract()` still returns lfs files afaict. 

Is there a good way to write a dataset loading script for a repo with lfs files?

### Motivation

/

### Your contribution

/

Comments: Hey ! Actually it works, just pass the right URL ;)
The URL must be the one with “/resolve/”

e.g. https://huggingface.co/datasets/imagenet-1k/resolve/main/data/test_images.tar.gz

You can even pass a relative path to the dl_manager instead, like `dl_manager.download("data/test_images.tar.gz")`

Amazing it works, thanks!
Update docs once dataset scripts transferred to the Hub
Author: albertvillanova
State: closed
Labels: name: documentation

## Describe the bug
As discussed in:
- https://github.com/huggingface/hub-docs/pull/423#pullrequestreview-1146083701

we should update our docs once dataset scripts have been transferred to the Hub (and removed from GitHub):
- #4974

Concretely:
- [x] Datasets on GitHub (legacy): https://huggingface.co/docs/datasets/main/en/share#datasets-on-github-legacy
- [x] ADD_NEW_DATASET: https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md
- ...

This PR complements the work of:
- #5067

This PR is a follow-up of PRs:
- #3777

CC: @julien-c
Raise ImportError instead of OSError if required extraction library is not installed
Author: mariosasko
State: closed
Labels: name: enhancement

name: good first issue

name: hacktoberfest

According to the official Python docs, `OSError` should be thrown in the following situations:
> This exception is raised when a system function returns a system-related error, including I/O failures such as “file not found” or “disk full” (not for illegal argument types or other incidental errors).

Hence, it makes more sense to raise `ImportError` instead of `OSError` when the required extraction/decompression library is not installed.

Comments: hey ,i would like to work on this issue . Please assign it to me.

hey @mariosasko , i made a pr for this issue. Could you please review it.
Also i found multiple `OSError` in `extract.py` file which i thought could be replaced too but wasn't sure about them.
Please do tell if that also needs to be done.
Tensor operation not functioning in dataset mapping
Author: xinghaow99
State: closed
Labels: name: bug

## Describe the bug
I'm doing a torch.mean() operation in data preprocessing, and it's not working.

## Steps to reproduce the bug
```
from transformers import pipeline
import torch
import numpy as np
from datasets import load_dataset
device = 'cuda:0'
raw_dataset = load_dataset("glue", "sst2")
feature_extraction = pipeline('feature-extraction', 'bert-base-uncased', device=device)
def extracted_data(examples):
    # feature = torch.tensor(feature_extraction(examples['sentence'], batch_size=16), device=device)
    # feature = torch.mean(feature, dim=1)
    feature = np.asarray(feature_extraction(examples['sentence'], batch_size=16)).squeeze().mean(1)
    print(feature.shape)
    return {'feature': feature}

extracted_dataset = raw_dataset.map(extracted_data, batched=True, batch_size=16)
```

## Results
When running with torch.mean(), the shape printed out is [16, seq_len, 768], which is exactly the same before the operation. While numpy works just fine, which gives [16, 768].


## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-4.4.0-142-generic-x86_64-with-glibc2.31
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0


Comments: Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

> Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

Thank you.
Depracate `num_proc` parameter in `DownloadManager.extract`
Author: mariosasko
State: closed
Labels: name: enhancement

name: good first issue

name: hacktoberfest

The `num_proc` parameter is only present in `DownloadManager.extract` but not in `StreamingDownloadManager.extract`, making it impossible to support streaming in the dataset scripts that use it (`openwebtext` and `the_pile_stack_exchange`). We can avoid this situation by deprecating this parameter and passing `DownloadConfig`'s `num_proc` to `map_nested` instead, as it's done in `DownloadManager.download`.

Comments: I can take this! #self-assign

#self-assign

@lazarust i'm already working on this issue :smile: 

#self-assign

hey @mariosasko , i made a pr for this issue. Could you please review it.
WikiText 103 tokenizer hangs
Author: TrentBrick
State: closed
Labels: name: bug

See issue here: https://github.com/huggingface/transformers/issues/19702 

Comments: any updates on this? It happens to me on [OpenWikiText-20%](https://huggingface.co/datasets/Bingsu/openwebtext_20p) dataset, but not on [OpenWebText-10k](https://huggingface.co/datasets/stas/openwebtext-10k). This is really strange because I don't change anything else in my running script.

transformers version 4.18.0.dev0
datasets version 1.18.0
unexpected `cast` or `class_encode_column` result after `rename_column`
Author: quaeast
State: closed
Labels: name: bug

## Describe the bug
When invoke `cast` or `class_encode_column` to a colunm renamed by `rename_column` , it will convert all the variables in this column into one variable. I also run this script in version 2.5.2, this bug does not appear. So I switched to the older version.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi", "en")
data = dataset['train']

data = data.remove_columns(
                [
                    "review_id",
                    "product_id",
                    "reviewer_id",
                    "review_title",
                    "language",
                    "product_category",
                ]
            )
data = data.rename_column("review_body", "text")
data1 = data.class_encode_column("stars")
print(set(data1.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}


data = data.rename_column("stars", "label")
print(set(data.data.columns[0]))
# output: {<pyarrow.Int32Scalar: 5>, <pyarrow.Int32Scalar: 4>, <pyarrow.Int32Scalar: 1>, <pyarrow.Int32Scalar: 3>, <pyarrow.Int32Scalar: 2>}
data2 = data.class_encode_column("label")
print(set(data2.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 0>}
```

## Expected results
the last print should be: 
{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}

## Actual results
but it output:
{<pyarrow.Int64Scalar: 0>}

## Environment info
- `datasets` version: 2.6.1
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0



Comments: Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Hi, 方子东. I tried running the code with exact the same configuration (both datasets 2.5.2 and 2.6.1, python, pyarrow, pandas), but on Linux. The results seem to be the expected `{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}`.
I don't have a Mac device. I can't verify whether this is a M1 chip-specific problem.

I've just tested the code on my M1 Mac, and it behaves as expected.

> Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Thank you for your attention and feel sorry to take your time. Since this is a bug of old version, I think mybe my problem is because `cast` operation directaly used cached data generated by older verion of `datasets`. I tried to deleted the cached data and I got expected result.
Installing `datasets` on M1 computers
Author: david1542
State: closed
Labels: name: bug

## Describe the bug
I wanted to install `datasets` dependencies on my M1 (in order to start contributing to the project). However, I got an error regarding `tensorflow`.

On M1, `tensorflow-macos` needs to be installed instead. Can we add a conditional requirement, so that `tensorflow-macos` would be installed on M1?

## Steps to reproduce the bug
Fresh clone this project (on m1), create a virtualenv and run this:
```python
pip install -e ".[dev]"
```

## Expected results
Installation should be smooth, and all the dependencies should be installed on M1.

## Actual results
You should receive an error, saying pip couldn't find a version that matches this pattern:
```
tensorflow>=2.3,!=2.6.0,!=2.6.1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.2.dev0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



Comments: Thanks for reporting, @david1542.
Progress bars have color red and never completed to 100%
Author: echatzikyriakidis
State: closed
Labels: name: bug

## Describe the bug
Progress bars after transformative operations turn in red and never be completed to 100%

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset('rotten_tomatoes', split='test').filter(lambda o: True)
```

## Expected results
Progress bar should be 100% and green

## Actual results
Progress  bar turn in red and never completed to 100%

## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Comments: Hi @echatzikyriakidis, thanks for submitting the issue.
Which shell are you using exactly? I tried to run the command you sent, but I don't see colors at all 🧐

I tried from bash and zsh as well.

Hi @david1542 ,

I use Google Colab.


Got it. I [created a PR](https://github.com/huggingface/datasets/pull/5120) that fixes this issue. Turns out that the wrapping logic for the inner loop was slightly incorrect.

Thank you!

Hello @mariosasko 

I am still facing this issue. Was this problem fixed?

![Screen Shot 2024-06-19 at 11 35 57 AM](https://github.com/huggingface/datasets/assets/66180831/cd9c6ac4-178c-4297-821c-a56411d34c89)

I cleared the hugging face cache before running, and no error message was given. Let me know if you need a minimal repro of my code.
load_from_disk with remote filesystem fails due to a wrong temporary local folder path
Author: bruno-hays
State: open
Labels: name: bug

## Describe the bug

The function load_from_disk fails when using a remote filesystem because of a wrong temporary path generation in the load_from_disk method of arrow_dataset.py:

```python
if is_remote_filesystem(fs):
    src_dataset_path = extract_path_from_uri(dataset_path)
    dataset_path = Dataset._build_local_temp_path(src_dataset_path)
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
```
If _dataset_path_ is `gs://speech/mydataset/train`, then _src_dataset_path_ will be `speech/mydataset/train` and _dataset_path_ will be something like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train`

Then, after downloading the **folder** _src_dataset_path_, you will get a path like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train/train/state.json` (notice we have train twice)

Instead of downloading the remote folder we should be downloading all the files in the folder for the path to be right:

```python
fs.download(os.path.join(src_dataset_path,*), dataset_path.as_posix(), recursive=True)
```

## Steps to reproduce the bug
```python
fs = gcsfs.GCSFileSystem(**storage_options)
dataset = load_from_disk("common_voice_processed") # loading local dataset previously saved locally, works fine
dataset.save_to_disk(output_dir, fs=fs) #works fine
dataset = load_from_disk(output_dir, fs=fs) # crashes
```

## Expected results
The dataset is loaded

## Actual results

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/9s/gf0b9jz15d517yrf7m3nvlxr0000gn/T/tmp6t5e221_/speech/datasets/tests/common_voice_processed/train/state.json'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-2.6.1.dev0
- Platform: mac os monterey 12.5.1
- Python version: 3.8.13
- PyArrow version:pyarrow==9.0.0


Comments: Hi Hubert! Could you please probably create a publicly available `gs://` dataset link? I think this would be easier for others to directly start to debug.

What seems to work is to change the line to:
```
fs.download(src_dataset_path, dataset_path.parent.as_posix(), recursive=True)
```
Bug with filtered indices
Author: albertvillanova
State: closed
Labels: name: bug

## Describe the bug
As reported by @PartiallyTyped (and by @Muennighoff):
- https://github.com/huggingface/datasets/issues/5111#issuecomment-1278652524

There is an issue with the indices of a filtered dataset.

## Steps to reproduce the bug
```python
ds = Dataset.from_dict({"num": [0, 1, 2, 3]})
ds = ds.filter(lambda num: num % 2 == 0, input_columns="num", batch_size=2)
assert all(item["num"] % 2 == 0 for item in ds)
```

## Expected results
The indices of the filtered dataset should correspond to the examples with "language" equals to "english".

## Actual results
Indices to items with other languages are included in the filtered dataset indices

## Preliminar investigation
It seems a bug introduced by:
- #5030


Comments: The issue is here:
https://github.com/huggingface/datasets/blob/3ad9644b9a2e4558dd1d0f1e43c67658674e6228/src/datasets/arrow_dataset.py#L2964

@PartiallyTyped, @Muennighoff: the issue is fixed.

We are planning to make a patch release today.

Thanks a lot for the swift response! For a brief moment yesterday I thought I had gone insane 🤣On 14 Oct 2022, at 15:44, Albert Villanova del Moral ***@***.***> wrote:﻿
@PartiallyTyped, @Muennighoff: the issue is fixed.
We are planning to make a patch release today.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Map caching not working for some class methods
Author: Mouhanedg56
State: closed
Labels: name: bug

## Describe the bug
The cache loading is not working as expected for some class methods with a model stored in an attribute.
The new fingerprint for `_map_single` is not the same at each run. The hasher generate a different hash for the class method.
This comes from `dumps` function in `datasets.utils.py_utils` which generates a different dump at each run.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import AutoConfig, AutoModel, AutoTokenizer

dataset = load_dataset("ethos", "binary")
BASE_MODELNAME = "sentence-transformers/all-MiniLM-L6-v2"

class Object:
    def __init__(self):
        config = AutoConfig.from_pretrained(BASE_MODELNAME)
        self.bert = AutoModel.from_config(config=config, add_pooling_layer=False)
        self.tok = AutoTokenizer.from_pretrained(BASE_MODELNAME)

    def tokenize(self, examples):

        tokenized_texts = self.tok(
            examples["text"],
            padding="max_length",
            truncation=True,
            max_length=256,
        )
        return tokenized_texts

instance = Object()
result = dict()
for phase in ["train"]:
    result[phase] = dataset[phase].map(instance.tokenize, batched=True, load_from_cache_file=True, num_proc=2)

```

## Expected results
Load cache instead of recompute result.

## Actual results
Result recomputed from scratch at each run.
The cache works fine when deleting `bert` attribute.

## Environment info
- `datasets` version: 2.5.3.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



Comments: The hash used for caching is computed by pickling recursively the function passed to `map`. Maybe some objects don't have the same hash across sessions.  In particular you can check the hash of your model using
```python
from datasets.fingerprint import Hasher
obj = AutoModel.from_config(config=config, add_pooling_layer=False)
print(Hasher.hash(obj))
```

You can find mode info here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument

Indeed, the hash is changing. The `dumps` function serialize the model object in different ways because the model object is not deterministic
```python
from datasets.utils.py_utils import dumps
obj1 = AutoModel.from_config(config=config, add_pooling_layer=False)
obj2 = AutoModel.from_config(config=config, add_pooling_layer=False)

dumps(bert) == dumps(bert2). # False
```

> You can find mode info here: https://huggingface.co/docs/datasets/about_cache
> 
> You can also provide your own unique hash in map if you want, with the new_fingerprint argument


Thanks, the doc is so helpful. Indeed, we can fix the hash and get cache hit using `new_fingerprint`. Closing the issue.
Error in create a dataset from a Python generator
Author: yangxuhui
State: closed
Labels: name: bug

name: good first issue

name: hacktoberfest

## Describe the bug
In HOW-TO-GUIDES > Load > [Python generator](https://huggingface.co/docs/datasets/v2.5.2/en/loading#python-generator), the code example defines the `my_gen` function, but when creating the dataset, an undefined `my_dict` is passed in.  

```Python
>>> from datasets import Dataset
>>> def my_gen():
...    for i in range(1, 4):
...        yield {"a": i}
>>> dataset = Dataset.from_generator(my_dict)
```

Comments: Hi, thanks for reporting! The last line should be `dataset = Dataset.from_generator(my_gen)`.

Can I work on this one?