liuqingwen/github-issues huggingface dataset:
refactor FolderBasedBuilder and Image/AudioFolder tests
Labels: name: refactoring
description: Restructuring existing code without changing its external behavior

Tests for FolderBasedBuilder, ImageFolder and AudioFolder are mostly duplicating each other. They need to be refactored and Audio/ImageFolder should have only tests specific to the loader. 

Comments:
Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)
Labels: name: enhancement
description: New feature or request

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets. 


Comments:
also asked in https://discuss.huggingface.co/t/create-multiple-dataset-configs-with-push-to-hub-method/25480
Inconsistent documentation on map remove_columns
Labels: name: documentation
description: Improvements or additions to documentation

name: duplicate
description: This issue or pull request already exists

name: good first issue
description: Good for newcomers

name: hacktoberfest

### Describe the bug

The page [process](https://huggingface.co/docs/datasets/process) says this about the parameter `remove_columns` of the function `map`:
When you remove a column, it is only removed after the example has been provided to the mapped function.

So it seems that the `remove_columns` parameter removes after the mapped functions.

However, another page, [the documentation of the function map](https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.Dataset.map.remove_columns) says:
Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in remove_columns, these columns will be kept.

So one page says "after the mapped function" and another says "before the mapped function."
Is there something wrong?


### Steps to reproduce the bug

Not about code.

### Expected behavior

consistent about the descriptions of the behavior of the parameter `remove_columns` in the function `map`.

### Environment info

datasets V2.6.0

Comments:
Thanks for reporting, @zhaowei-wang-nlp.

You are right, the documentation is confusing on the behavior of `remove_columns`. We should better explain it. 

This is a duplicate of https://github.com/huggingface/datasets/issues/2343.

I'm closing this issue because as @mariosasko pointed out, it is a duplicate of:
- #2343
DownloadManager Git LFS support
Labels: name: enhancement
description: New feature or request

### Feature request

Maybe I'm mistaken but the `DownloadManager` does not support extracting git lfs files out of the box right?
Using `dl_manager.download()` or `dl_manager.download_and_extract()` still returns lfs files afaict. 

Is there a good way to write a dataset loading script for a repo with lfs files?

### Motivation

/

### Your contribution

/

Comments:
Hey ! Actually it works, just pass the right URL ;)
The URL must be the one with “/resolve/”

e.g. https://huggingface.co/datasets/imagenet-1k/resolve/main/data/test_images.tar.gz

You can even pass a relative path to the dl_manager instead, like `dl_manager.download("data/test_images.tar.gz")`

Amazing it works, thanks!
Update docs once dataset scripts transferred to the Hub
Labels: name: documentation
description: Improvements or additions to documentation

## Describe the bug
As discussed in:
- https://github.com/huggingface/hub-docs/pull/423#pullrequestreview-1146083701

we should update our docs once dataset scripts have been transferred to the Hub (and removed from GitHub):
- #4974

Concretely:
- [x] Datasets on GitHub (legacy): https://huggingface.co/docs/datasets/main/en/share#datasets-on-github-legacy
- [x] ADD_NEW_DATASET: https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md
- ...

This PR complements the work of:
- #5067

This PR is a follow-up of PRs:
- #3777

CC: @julien-c 

Comments:
Raise ImportError instead of OSError if required extraction library is not installed
Labels: name: enhancement
description: New feature or request

name: good first issue
description: Good for newcomers

name: hacktoberfest

According to the official Python docs, `OSError` should be thrown in the following situations:
> This exception is raised when a system function returns a system-related error, including I/O failures such as “file not found” or “disk full” (not for illegal argument types or other incidental errors).

Hence, it makes more sense to raise `ImportError` instead of `OSError` when the required extraction/decompression library is not installed.

Comments:
hey ,i would like to work on this issue . Please assign it to me.

hey @mariosasko , i made a pr for this issue. Could you please review it.
Also i found multiple `OSError` in `extract.py` file which i thought could be replaced too but wasn't sure about them.
Please do tell if that also needs to be done.
Tensor operation not functioning in dataset mapping
Labels: name: bug
description: Something isn't working

## Describe the bug
I'm doing a torch.mean() operation in data preprocessing, and it's not working.

## Steps to reproduce the bug
```
from transformers import pipeline
import torch
import numpy as np
from datasets import load_dataset
device = 'cuda:0'
raw_dataset = load_dataset("glue", "sst2")
feature_extraction = pipeline('feature-extraction', 'bert-base-uncased', device=device)
def extracted_data(examples):
    # feature = torch.tensor(feature_extraction(examples['sentence'], batch_size=16), device=device)
    # feature = torch.mean(feature, dim=1)
    feature = np.asarray(feature_extraction(examples['sentence'], batch_size=16)).squeeze().mean(1)
    print(feature.shape)
    return {'feature': feature}

extracted_dataset = raw_dataset.map(extracted_data, batched=True, batch_size=16)
```

## Results
When running with torch.mean(), the shape printed out is [16, seq_len, 768], which is exactly the same before the operation. While numpy works just fine, which gives [16, 768].


## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-4.4.0-142-generic-x86_64-with-glibc2.31
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0


Comments:
Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

> Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

Thank you.
Depracate `num_proc` parameter in `DownloadManager.extract`
Labels: name: enhancement
description: New feature or request

name: good first issue
description: Good for newcomers

name: hacktoberfest

The `num_proc` parameter is only present in `DownloadManager.extract` but not in `StreamingDownloadManager.extract`, making it impossible to support streaming in the dataset scripts that use it (`openwebtext` and `the_pile_stack_exchange`). We can avoid this situation by deprecating this parameter and passing `DownloadConfig`'s `num_proc` to `map_nested` instead, as it's done in `DownloadManager.download`.

Comments:
I can take this! #self-assign

#self-assign

@lazarust i'm already working on this issue :smile: 

#self-assign

hey @mariosasko , i made a pr for this issue. Could you please review it.
WikiText 103 tokenizer hangs
Labels: name: bug
description: Something isn't working

See issue here: https://github.com/huggingface/transformers/issues/19702 

Comments:
unexpected `cast` or `class_encode_column` result after `rename_column`
Labels: name: bug
description: Something isn't working

## Describe the bug
When invoke `cast` or `class_encode_column` to a colunm renamed by `rename_column` , it will convert all the variables in this column into one variable. I also run this script in version 2.5.2, this bug does not appear. So I switched to the older version.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi", "en")
data = dataset['train']

data = data.remove_columns(
                [
                    "review_id",
                    "product_id",
                    "reviewer_id",
                    "review_title",
                    "language",
                    "product_category",
                ]
            )
data = data.rename_column("review_body", "text")
data1 = data.class_encode_column("stars")
print(set(data1.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}


data = data.rename_column("stars", "label")
print(set(data.data.columns[0]))
# output: {<pyarrow.Int32Scalar: 5>, <pyarrow.Int32Scalar: 4>, <pyarrow.Int32Scalar: 1>, <pyarrow.Int32Scalar: 3>, <pyarrow.Int32Scalar: 2>}
data2 = data.class_encode_column("label")
print(set(data2.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 0>}
```

## Expected results
the last print should be: 
{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}

## Actual results
but it output:
{<pyarrow.Int64Scalar: 0>}

## Environment info
- `datasets` version: 2.6.1
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0



Comments:
Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Hi, 方子东. I tried running the code with exact the same configuration (both datasets 2.5.2 and 2.6.1, python, pyarrow, pandas), but on Linux. The results seem to be the expected `{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}`.
I don't have a Mac device. I can't verify whether this is a M1 chip-specific problem.

I've just tested the code on my M1 Mac, and it behaves as expected.

> Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Thank you for your attention and feel sorry to take your time. Since this is a bug of old version, I think mybe my problem is because `cast` operation directaly used cached data generated by older verion of `datasets`. I tried to deleted the cached data and I got expected result.
[WIP] WebDataset export

I added a first draft of the `IterableDataset.to_wds` method.

You can use it to savea dataset loaded in streamign mode as a webdataset locally.
The API can be further improved to allow to export to a cloud storage like the HF Hub.

I also included sharding with a default max shard size of 500MB (uncompressed), and it is single-processed fo rnow.

Choosing the number of shards is not implemented yet - though if we know the size of the `IterableDataset` this is probably doable`.

For example
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
>>> ds.to_wds("output_dir", compress=True)
>>> import webdataset as wds
>>> ds = wds.WebDataset("output_dir/rotten_tomatoes-train-000000.tar.gz").decode()
>>> next(iter(ds))
{'__key__': '0',
'__url__': 'output_dir/rotten_tomatoes-train-000000.tar.gz',
'label.cls': 1,
'text.txt': 'the rock is destined to be the 21st century\'s new ..., jean-claud van damme or steven segal .'}
```

### Implementation details

The WebDataset format is made of TAR archives containing a series of files per example. For example one pair of `image.jpg` and `label.cls` for image classification.

WebDataset automatically decodes serialized data based on the extension of the files, and output a dictionary. For example `{"image.png": np.array(...), "label.cls": 0}` if you choose the numpy decoding.

To use the automatic decoding, I store each field of each example as a file with its corresponding extension (jpg, json, cls, etc.)

While this is useful to end up with a dictionary with one key per column and appropriate decoding, it can create huge TAR archives if the dataset is made of small samples of text - probably because of useless TAR metadata for each file. This also makes loading super slow: iterating on SQuAD takes 50sec vs 7sec using `datasets` in streaming mode.

I haven't taken a look at alternatives for text datasets made out of small samples, but for image datasets this can already be used to run some benchmarks.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5127). All of your documentation changes will be reflected on that endpoint.
Installing `datasets` on M1 computers
Labels: name: bug
description: Something isn't working

## Describe the bug
I wanted to install `datasets` dependencies on my M1 (in order to start contributing to the project). However, I got an error regarding `tensorflow`.

On M1, `tensorflow-macos` needs to be installed instead. Can we add a conditional requirement, so that `tensorflow-macos` would be installed on M1?

## Steps to reproduce the bug
Fresh clone this project (on m1), create a virtualenv and run this:
```python
pip install -e ".[dev]"
```

## Expected results
Installation should be smooth, and all the dependencies should be installed on M1.

## Actual results
You should receive an error, saying pip couldn't find a version that matches this pattern:
```
tensorflow>=2.3,!=2.6.0,!=2.6.1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.2.dev0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



Comments:
Thanks for reporting, @david1542.
Progress bars have color red and never completed to 100%
Labels: name: bug
description: Something isn't working

## Describe the bug
Progress bars after transformative operations turn in red and never be completed to 100%

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset('rotten_tomatoes', split='test').filter(lambda o: True)
```

## Expected results
Progress bar should be 100% and green

## Actual results
Progress  bar turn in red and never completed to 100%

## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Comments:
Hi @echatzikyriakidis, thanks for submitting the issue.
Which shell are you using exactly? I tried to run the command you sent, but I don't see colors at all 🧐

I tried from bash and zsh as well.

Hi @david1542 ,

I use Google Colab.


Got it. I [created a PR](https://github.com/huggingface/datasets/pull/5120) that fixes this issue. Turns out that the wrapping logic for the inner loop was slightly incorrect.

Thank you!
load_from_disk with remote filesystem fails due to a wrong temporary local folder path
Labels: name: bug
description: Something isn't working

## Describe the bug

The function load_from_disk fails when using a remote filesystem because of a wrong temporary path generation in the load_from_disk method of arrow_dataset.py:

```python
if is_remote_filesystem(fs):
    src_dataset_path = extract_path_from_uri(dataset_path)
    dataset_path = Dataset._build_local_temp_path(src_dataset_path)
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
```
If _dataset_path_ is `gs://speech/mydataset/train`, then _src_dataset_path_ will be `speech/mydataset/train` and _dataset_path_ will be something like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train`

Then, after downloading the **folder** _src_dataset_path_, you will get a path like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train/train/state.json` (notice we have train twice)

Instead of downloading the remote folder we should be downloading all the files in the folder for the path to be right:

```python
fs.download(os.path.join(src_dataset_path,*), dataset_path.as_posix(), recursive=True)
```

## Steps to reproduce the bug
```python
fs = gcsfs.GCSFileSystem(**storage_options)
dataset = load_from_disk("common_voice_processed") # loading local dataset previously saved locally, works fine
dataset.save_to_disk(output_dir, fs=fs) #works fine
dataset = load_from_disk(output_dir, fs=fs) # crashes
```

## Expected results
The dataset is loaded

## Actual results

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/9s/gf0b9jz15d517yrf7m3nvlxr0000gn/T/tmp6t5e221_/speech/datasets/tests/common_voice_processed/train/state.json'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-2.6.1.dev0
- Platform: mac os monterey 12.5.1
- Python version: 3.8.13
- PyArrow version:pyarrow==9.0.0


Comments:
Hi Hubert! Could you please probably create a publicly available `gs://` dataset link? I think this would be easier for others to directly start to debug.

What seems to work is to change the line to:
```
fs.download(src_dataset_path, dataset_path.parent.as_posix(), recursive=True)
```
Bug with filtered indices
Labels: name: bug
description: Something isn't working

## Describe the bug
As reported by @PartiallyTyped (and by @Muennighoff):
- https://github.com/huggingface/datasets/issues/5111#issuecomment-1278652524

There is an issue with the indices of a filtered dataset.

## Steps to reproduce the bug
```python
ds = Dataset.from_dict({"num": [0, 1, 2, 3]})
ds = ds.filter(lambda num: num % 2 == 0, input_columns="num", batch_size=2)
assert all(item["num"] % 2 == 0 for item in ds)
```

## Expected results
The indices of the filtered dataset should correspond to the examples with "language" equals to "english".

## Actual results
Indices to items with other languages are included in the filtered dataset indices

## Preliminar investigation
It seems a bug introduced by:
- #5030


Comments:
The issue is here:
https://github.com/huggingface/datasets/blob/3ad9644b9a2e4558dd1d0f1e43c67658674e6228/src/datasets/arrow_dataset.py#L2964

@PartiallyTyped, @Muennighoff: the issue is fixed.

We are planning to make a patch release today.

Thanks a lot for the swift response! For a brief moment yesterday I thought I had gone insane 🤣On 14 Oct 2022, at 15:44, Albert Villanova del Moral ***@***.***> wrote:﻿
@PartiallyTyped, @Muennighoff: the issue is fixed.
We are planning to make a patch release today.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Map caching not working for some class methods
Labels: name: bug
description: Something isn't working

## Describe the bug
The cache loading is not working as expected for some class methods with a model stored in an attribute.
The new fingerprint for `_map_single` is not the same at each run. The hasher generate a different hash for the class method.
This comes from `dumps` function in `datasets.utils.py_utils` which generates a different dump at each run.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import AutoConfig, AutoModel, AutoTokenizer

dataset = load_dataset("ethos", "binary")
BASE_MODELNAME = "sentence-transformers/all-MiniLM-L6-v2"

class Object:
    def __init__(self):
        config = AutoConfig.from_pretrained(BASE_MODELNAME)
        self.bert = AutoModel.from_config(config=config, add_pooling_layer=False)
        self.tok = AutoTokenizer.from_pretrained(BASE_MODELNAME)

    def tokenize(self, examples):

        tokenized_texts = self.tok(
            examples["text"],
            padding="max_length",
            truncation=True,
            max_length=256,
        )
        return tokenized_texts

instance = Object()
result = dict()
for phase in ["train"]:
    result[phase] = dataset[phase].map(instance.tokenize, batched=True, load_from_cache_file=True, num_proc=2)

```

## Expected results
Load cache instead of recompute result.

## Actual results
Result recomputed from scratch at each run.
The cache works fine when deleting `bert` attribute.

## Environment info
- `datasets` version: 2.5.3.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



Comments:
The hash used for caching is computed by pickling recursively the function passed to `map`. Maybe some objects don't have the same hash across sessions.  In particular you can check the hash of your model using
```python
from datasets.fingerprint import Hasher
obj = AutoModel.from_config(config=config, add_pooling_layer=False)
print(Hasher.hash(obj))
```

You can find mode info here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument

Indeed, the hash is changing. The `dumps` function serialize the model object in different ways because the model object is not deterministic
```python
from datasets.utils.py_utils import dumps
obj1 = AutoModel.from_config(config=config, add_pooling_layer=False)
obj2 = AutoModel.from_config(config=config, add_pooling_layer=False)

dumps(bert) == dumps(bert2). # False
```

> You can find mode info here: https://huggingface.co/docs/datasets/about_cache
> 
> You can also provide your own unique hash in map if you want, with the new_fingerprint argument


Thanks, the doc is so helpful. Indeed, we can fix the hash and get cache hit using `new_fingerprint`. Closing the issue.
Error in create a dataset from a Python generator
Labels: name: bug
description: Something isn't working

name: good first issue
description: Good for newcomers

name: hacktoberfest

## Describe the bug
In HOW-TO-GUIDES > Load > [Python generator](https://huggingface.co/docs/datasets/v2.5.2/en/loading#python-generator), the code example defines the `my_gen` function, but when creating the dataset, an undefined `my_dict` is passed in.  

```Python
>>> from datasets import Dataset
>>> def my_gen():
...    for i in range(1, 4):
...        yield {"a": i}
>>> dataset = Dataset.from_generator(my_dict)
```

Comments:
Hi, thanks for reporting! The last line should be `dataset = Dataset.from_generator(my_gen)`.

Can I work on this one?