NyssaW/github-issues huggingface dataset:
Consistent caching between python and jupyter

### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen

Comments:
Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`

Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!
TextConfig: added "errors"

This patch adds the ability to set the `errors` option of `open` for loading text datasets. I needed it because some data I had scraped had bad bytes in it, so I needed `errors='ignore'`. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for adding this ! You can fix the CI by formatting your code using the `make style` command :)

[**@lhoestq**](https://github.com/lhoestq) commented on [Oct 27, 2022, 4:08 PM GMT+3:30](https://github.com/huggingface/datasets/pull/5155#issuecomment-1293464680 "2022-10-27T12:38:04Z - Replied by Github Reply Comments"):
> Thanks for adding this ! You can fix the CI by formatting your code using the `make  style` command :)

I ran this and force pushed the changes.
default Image/AudioFolder infers labels when there is no metadata files even if there is only one dir

### Describe the bug

By default FolderBasedBuilder infers labels if there is not metadata files, even if it's meaningless (for example, they are in a single directory or in the root folder, see this repo as an example: https://huggingface.co/datasets/patrickvonplaten/audios

As this is a corner case for quick exploration of images or audios on the Hub. 

### Steps to reproduce the bug

If you have directory like this:

```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
or

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
```
doing `ds = load_dataset(repo)` would create `label` feature:
```python
print(ds["train"][0])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 0}
```

Also, if you have the following structure:

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
   image4.jpg
   image5.jpg
   image6.jpg
```
it will infer two labels:

```python
print(ds["train"][0])
print(ds["train"][-1])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 1}
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x415 at 0x7FB5326555B0>, 'label': 0}
```

### Expected behavior

We should have only one base feature (Image/Audio) in such cases.

### Environment info

all versions of `datasets`

Comments:
Makes sense! For the last structure, we could count the path segments (delimited by "/" for URLs and `os.sep`  for local paths) to ensure all inferred labels are on the same level. Otherwise, I think it's safe to assume they are meaningless and ignore them.
Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets. 


Comments:
also asked in https://discuss.huggingface.co/t/create-multiple-dataset-configs-with-push-to-hub-method/25480
Cannot find the rvl_cdip dataset

Hi,
I am trying to use load_dataset to load the official "rvl_cdip" dataset but getting an error.

dataset = load_dataset("rvl_cdip")

Couldn't find 'rvl_cdip' on the Hugging Face Hub either: FileNotFoundError: Couldn't find the file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/rvl_cdip/rvl_cdip.py

Regards,


Comments:
Hi, @santule.

We have transferred all dataset scripts from GitHub to the Hugging Face Hub: https://huggingface.co/datasets
- Concretely, you have "rvl_cdip" here: https://huggingface.co/datasets/rvl_cdip

To be able to load them, you should update your `datasets` library:
```
pip install -U datasets
```

thank you, it worked
Dataset order is not deterministic with ZIP archives and `iter_files`

### Describe the bug

For the `beans` dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

### Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

```python
from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])
```

### Expected behavior

The order of the images is the same on all machines.

### Environment info

On the EC2 instance:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked
```

On my local laptop:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1
```

On github actions:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4
```

Comments:
Thanks for reporting ! The issue doesn't come from shuffling, but from `beans` row order not being deterministic:

https://huggingface.co/datasets/beans/blob/main/beans.py uses `dl_manager.iter_files` on ZIP archives and the file order doesn't seen to be deterministic and changes across machines

Thank you for noticing indeed!

This is still a bug, so I'd keep this one open if you don't mind ;)

Besides the linked PR, to make the loading process fully deterministic, I believe we should also sort the data files [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L276) and [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L485) (e.g. fsspec's `LocalFileSystem.glob` relies on `os.scandir`, which yields the contents in arbitrary order). My concern is the overhead of these sorts... Maybe we could introduce a new flag to `load_dataset` similar to TFDS' [`shuffle_files`](https://www.tensorflow.org/datasets/determinism#determinism_when_reading) or sort only if the number of data files is small?

We already return the result sorted at the end of `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository` if I'm not mistaken

@lhoestq Oh, you are right. Feel free to ignore my comment.

I think the corresponding PR is ready to be merged :hugs: 

@albertvillanova Thanks for the fix!
Inconsistent documentation on map remove_columns

### Describe the bug

The page [process](https://huggingface.co/docs/datasets/process) says this about the parameter `remove_columns` of the function `map`:
When you remove a column, it is only removed after the example has been provided to the mapped function.

So it seems that the `remove_columns` parameter removes after the mapped functions.

However, another page, [the documentation of the function map](https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.Dataset.map.remove_columns) says:
Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in remove_columns, these columns will be kept.

So one page says "after the mapped function" and another says "before the mapped function."
Is there something wrong?


### Steps to reproduce the bug

Not about code.

### Expected behavior

consistent about the descriptions of the behavior of the parameter `remove_columns` in the function `map`.

### Environment info

datasets V2.6.0

Comments:
Thanks for reporting, @zhaowei-wang-nlp.

You are right, the documentation is confusing on the behavior of `remove_columns`. We should better explain it. 

This is a duplicate of https://github.com/huggingface/datasets/issues/2343.

I'm closing this issue because as @mariosasko pointed out, it is a duplicate of:
- #2343
DownloadManager Git LFS support

### Feature request

Maybe I'm mistaken but the `DownloadManager` does not support extracting git lfs files out of the box right?
Using `dl_manager.download()` or `dl_manager.download_and_extract()` still returns lfs files afaict. 

Is there a good way to write a dataset loading script for a repo with lfs files?

### Motivation

/

### Your contribution

/

Comments:
Hey ! Actually it works, just pass the right URL ;)
The URL must be the one with “/resolve/”

e.g. https://huggingface.co/datasets/imagenet-1k/resolve/main/data/test_images.tar.gz

You can even pass a relative path to the dl_manager instead, like `dl_manager.download("data/test_images.tar.gz")`

Amazing it works, thanks!
Deprecate num_proc parameter in  DownloadManager.extract

fixes #5132 : Deprecated the `num_proc` parameter in `DownloadManager.extract` by passing `num_proc` parameter to `map_nested` .

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @mariosasko . Can you please help me with why the tests keep failing.  I have reviewed the code changes multiple times but can't spot any mistakes. 

You can fix this failure by formatting your code with the `make style` command (run it from the root of the cloned repo).

hey @mariosasko ,i cant understand how to use the `make style` command .I searched for it on the internet but cant find any results. 
So i formatted the code using vs-code document formatter. Hope this helps.

`make style` runs the "style" target defined here: https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12

This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile





> `make style` runs the "style" target defined here:
> 
> https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12
> 
> This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile

Thanks! I will look into this :relaxed:
Update docs once dataset scripts transferred to the Hub

Todo:
- [x] Update docs:
  - [x] Datasets on GitHub (legacy)
  - [x] Load: offline
  - [x] About dataset load:
    - [x] Maintaining integrity
    - [x] Security
- [x] Update docstrings:
  - [x] Inspect:
    - [x] get_dataset_config_info
    - [x] get_dataset_split_names
  - [x] Load:
    - [x] dataset_module_factory
    - [x] load_dataset_builder
    - [x] load_dataset
- [x] Remove `ADD_NEW_DATASET.md`
- [x] Update `.github/ISSUE_TEMPLATE/config.yml`

Fix #5135.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update docs once dataset scripts transferred to the Hub

## Describe the bug
As discussed in:
- https://github.com/huggingface/hub-docs/pull/423#pullrequestreview-1146083701

we should update our docs once dataset scripts have been transferred to the Hub (and removed from GitHub):
- #4974

Concretely:
- [x] Datasets on GitHub (legacy): https://huggingface.co/docs/datasets/main/en/share#datasets-on-github-legacy
- [x] ADD_NEW_DATASET: https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md
- ...

This PR complements the work of:
- #5067

This PR is a follow-up of PRs:
- #3777

CC: @julien-c 

Comments:
Raise ImportError instead of OSError if required extraction library is not installed

According to the official Python docs, `OSError` should be thrown in the following situations:
> This exception is raised when a system function returns a system-related error, including I/O failures such as “file not found” or “disk full” (not for illegal argument types or other incidental errors).

Hence, it makes more sense to raise `ImportError` instead of `OSError` when the required extraction/decompression library is not installed.

Comments:
hey ,i would like to work on this issue . Please assign it to me.

hey @mariosasko , i made a pr for this issue. Could you please review it.
Also i found multiple `OSError` in `extract.py` file which i thought could be replaced too but wasn't sure about them.
Please do tell if that also needs to be done.
Tensor operation not functioning in dataset mapping

## Describe the bug
I'm doing a torch.mean() operation in data preprocessing, and it's not working.

## Steps to reproduce the bug
```
from transformers import pipeline
import torch
import numpy as np
from datasets import load_dataset
device = 'cuda:0'
raw_dataset = load_dataset("glue", "sst2")
feature_extraction = pipeline('feature-extraction', 'bert-base-uncased', device=device)
def extracted_data(examples):
    # feature = torch.tensor(feature_extraction(examples['sentence'], batch_size=16), device=device)
    # feature = torch.mean(feature, dim=1)
    feature = np.asarray(feature_extraction(examples['sentence'], batch_size=16)).squeeze().mean(1)
    print(feature.shape)
    return {'feature': feature}

extracted_dataset = raw_dataset.map(extracted_data, batched=True, batch_size=16)
```

## Results
When running with torch.mean(), the shape printed out is [16, seq_len, 768], which is exactly the same before the operation. While numpy works just fine, which gives [16, 768].


## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-4.4.0-142-generic-x86_64-with-glibc2.31
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0


Comments:
Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

> Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

Thank you.
Depracate `num_proc` parameter in `DownloadManager.extract`

The `num_proc` parameter is only present in `DownloadManager.extract` but not in `StreamingDownloadManager.extract`, making it impossible to support streaming in the dataset scripts that use it (`openwebtext` and `the_pile_stack_exchange`). We can avoid this situation by deprecating this parameter and passing `DownloadConfig`'s `num_proc` to `map_nested` instead, as it's done in `DownloadManager.download`.

Comments:
I can take this! #self-assign

#self-assign

@lazarust i'm already working on this issue :smile: 

#self-assign

hey @mariosasko , i made a pr for this issue. Could you please review it.
WikiText 103 tokenizer hangs

See issue here: https://github.com/huggingface/transformers/issues/19702 

Comments:
any updates on this? It happens to me on [OpenWikiText-20%](https://huggingface.co/datasets/Bingsu/openwebtext_20p) dataset, but not on [OpenWebText-10k](https://huggingface.co/datasets/stas/openwebtext-10k). This is really strange because I don't change anything else in my running script.

transformers version 4.18.0.dev0
datasets version 1.18.0
unexpected `cast` or `class_encode_column` result after `rename_column`

## Describe the bug
When invoke `cast` or `class_encode_column` to a colunm renamed by `rename_column` , it will convert all the variables in this column into one variable. I also run this script in version 2.5.2, this bug does not appear. So I switched to the older version.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi", "en")
data = dataset['train']

data = data.remove_columns(
                [
                    "review_id",
                    "product_id",
                    "reviewer_id",
                    "review_title",
                    "language",
                    "product_category",
                ]
            )
data = data.rename_column("review_body", "text")
data1 = data.class_encode_column("stars")
print(set(data1.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}


data = data.rename_column("stars", "label")
print(set(data.data.columns[0]))
# output: {<pyarrow.Int32Scalar: 5>, <pyarrow.Int32Scalar: 4>, <pyarrow.Int32Scalar: 1>, <pyarrow.Int32Scalar: 3>, <pyarrow.Int32Scalar: 2>}
data2 = data.class_encode_column("label")
print(set(data2.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 0>}
```

## Expected results
the last print should be: 
{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}

## Actual results
but it output:
{<pyarrow.Int64Scalar: 0>}

## Environment info
- `datasets` version: 2.6.1
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0



Comments:
Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Hi, 方子东. I tried running the code with exact the same configuration (both datasets 2.5.2 and 2.6.1, python, pyarrow, pandas), but on Linux. The results seem to be the expected `{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}`.
I don't have a Mac device. I can't verify whether this is a M1 chip-specific problem.

I've just tested the code on my M1 Mac, and it behaves as expected.

> Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Thank you for your attention and feel sorry to take your time. Since this is a bug of old version, I think mybe my problem is because `cast` operation directaly used cached data generated by older verion of `datasets`. I tried to deleted the cached data and I got expected result.
Make filename matching more robust

Fix  #5046

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> I think we should also modify one of the metadata files in the `folder_based_builder` tests to make sure "./" is ignored now in the `file_name`

@mariosasko what do you mean here? I'm not sure which metadata file I should modify here

You can modify this line for instance: https://github.com/huggingface/datasets/blob/2699593b33ee63d17aad2a2bfddedd38a8df57b8/tests/packaged_modules/test_folder_based_builder.py#L135
[WIP] WebDataset export

I added a first draft of the `IterableDataset.to_wds` method.

You can use it to savea dataset loaded in streamign mode as a webdataset locally.
The API can be further improved to allow to export to a cloud storage like the HF Hub.

I also included sharding with a default max shard size of 500MB (uncompressed), and it is single-processed fo rnow.

Choosing the number of shards is not implemented yet - though if we know the size of the `IterableDataset` this is probably doable`.

For example
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
>>> ds.to_wds("output_dir", compress=True)
>>> import webdataset as wds
>>> ds = wds.WebDataset("output_dir/rotten_tomatoes-train-000000.tar.gz").decode()
>>> next(iter(ds))
{'__key__': '0',
'__url__': 'output_dir/rotten_tomatoes-train-000000.tar.gz',
'label.cls': 1,
'text.txt': 'the rock is destined to be the 21st century\'s new ..., jean-claud van damme or steven segal .'}
```

### Implementation details

The WebDataset format is made of TAR archives containing a series of files per example. For example one pair of `image.jpg` and `label.cls` for image classification.

WebDataset automatically decodes serialized data based on the extension of the files, and output a dictionary. For example `{"image.png": np.array(...), "label.cls": 0}` if you choose the numpy decoding.

To use the automatic decoding, I store each field of each example as a file with its corresponding extension (jpg, json, cls, etc.)

While this is useful to end up with a dictionary with one key per column and appropriate decoding, it can create huge TAR archives if the dataset is made of small samples of text - probably because of useless TAR metadata for each file. This also makes loading super slow: iterating on SQuAD takes 50sec vs 7sec using `datasets` in streaming mode.

I haven't taken a look at alternatives for text datasets made out of small samples, but for image datasets this can already be used to run some benchmarks.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5127). All of your documentation changes will be reflected on that endpoint.

Should we close this PR?
Fix class name of symbolic link

Fix #5098 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5126). All of your documentation changes will be reflected on that endpoint.

I have removed the reference to the Issue in the PR title, so that we avoid to have both references (to the issue and to the PR) in the merge commit to the main branch.

Instead, it should be commented in the PR description, so that the PR is appropriately linked by GitHub to its corresponding Issue:

> Fix #5098.

@albertvillanova What should I test in your opinion? Also, where should I save the test file and how should I name it? Thanks for your support

The regression test to be implemented should test what your PR fixes: that is, that `_resolve_single_pattern_locally` function does not resolve any symbolic link when passed a directory that does contain any.

As you are testing a function in `data_files.py`, the corresponding test should be in `tests/test_data_files.py`.

You could name the test something lilke: `test_resolve_single_pattern_locally_does_not_resolve_symbolic_links`

You could take inspiration from other tests there in that file.
Add warning

Fixes: #5105 

I think removing the directory with warning is a better solution for this issue. Because if we decide to keep existing files in directory, then we should deal with the case providing same directory for several datasets! Which we know is not possible since `dataset_info.json` exists in that directory.

Comments:
As mentioned in https://github.com/huggingface/datasets/issues/5105 I think we just need to keep the existing files instead of deleting them.
The `dataset_info.json` file contains the split names anyway, so we know which files belong to the dataset, and which ones don't.
Bugfix ignore function when creating new_fingerprint for caching

maybe fixes: #5109

Comments:
Adding "function" to the kwargs to ignore when computing the fingerprint will break `map` caching. Indeed passing two different function would result in two different datasets that have the same fingerprint - and the cache wouldn't be able to distinguish them.

E.g this code would reload ds1 from the cache insetad of computing the dataset for ds2
```python
ds = Dataset.from_dict({"a": [1, 2, 3]})
ds1 = ds.map(lambda x: {"b": 1})
ds2 = ds.map(lambda x: {"b": 2})
```
Installing `datasets` on M1 computers

## Describe the bug
I wanted to install `datasets` dependencies on my M1 (in order to start contributing to the project). However, I got an error regarding `tensorflow`.

On M1, `tensorflow-macos` needs to be installed instead. Can we add a conditional requirement, so that `tensorflow-macos` would be installed on M1?

## Steps to reproduce the bug
Fresh clone this project (on m1), create a virtualenv and run this:
```python
pip install -e ".[dev]"
```

## Expected results
Installation should be smooth, and all the dependencies should be installed on M1.

## Actual results
You should receive an error, saying pip couldn't find a version that matches this pattern:
```
tensorflow>=2.3,!=2.6.0,!=2.6.1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.2.dev0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



Comments:
Thanks for reporting, @david1542.
Progress bars have color red and never completed to 100%

## Describe the bug
Progress bars after transformative operations turn in red and never be completed to 100%

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset('rotten_tomatoes', split='test').filter(lambda o: True)
```

## Expected results
Progress bar should be 100% and green

## Actual results
Progress  bar turn in red and never completed to 100%

## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Comments:
Hi @echatzikyriakidis, thanks for submitting the issue.
Which shell are you using exactly? I tried to run the command you sent, but I don't see colors at all 🧐

I tried from bash and zsh as well.

Hi @david1542 ,

I use Google Colab.


Got it. I [created a PR](https://github.com/huggingface/datasets/pull/5120) that fixes this issue. Turns out that the wrapping logic for the inner loop was slightly incorrect.

Thank you!

Hello @mariosasko 

I am still facing this issue. Was this problem fixed?

![Screen Shot 2024-06-19 at 11 35 57 AM](https://github.com/huggingface/datasets/assets/66180831/cd9c6ac4-178c-4297-821c-a56411d34c89)

I cleared the hugging face cache before running, and no error message was given. Let me know if you need a minimal repro of my code.
Fix iter_batches

The `pa.Table.to_reader()` method available in `pyarrow>=8.0.0` may return chunks of size < `max_chunksize`, therefore `iter_batches` can return batches smaller than the `batch_size` specified by the user

Therefore batched `map` couldn't always use batches of the right size, e.g. this fails because it runs only on one batch of one element:
```python

from datasets import Dataset, concatenate_datasets

ds = concatenate_datasets([Dataset.from_dict({"a": [i]}) for i in range(10)])

ds2 = ds.map(lambda _: {}, batched=True)
assert list(ds2) == list(ds)
```

This was introduced in https://github.com/huggingface/datasets/pull/5030

Close https://github.com/huggingface/datasets/issues/5111

This will require a patch release along with https://github.com/huggingface/datasets/pull/5113

TODO:
- [x] fix tests
- [x] add more tests

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I also ran the code in https://github.com/huggingface/datasets/issues/5111 and it works fine now :)

This is ready for review :)
load_from_disk with remote filesystem fails due to a wrong temporary local folder path

## Describe the bug

The function load_from_disk fails when using a remote filesystem because of a wrong temporary path generation in the load_from_disk method of arrow_dataset.py:

```python
if is_remote_filesystem(fs):
    src_dataset_path = extract_path_from_uri(dataset_path)
    dataset_path = Dataset._build_local_temp_path(src_dataset_path)
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
```
If _dataset_path_ is `gs://speech/mydataset/train`, then _src_dataset_path_ will be `speech/mydataset/train` and _dataset_path_ will be something like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train`

Then, after downloading the **folder** _src_dataset_path_, you will get a path like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train/train/state.json` (notice we have train twice)

Instead of downloading the remote folder we should be downloading all the files in the folder for the path to be right:

```python
fs.download(os.path.join(src_dataset_path,*), dataset_path.as_posix(), recursive=True)
```

## Steps to reproduce the bug
```python
fs = gcsfs.GCSFileSystem(**storage_options)
dataset = load_from_disk("common_voice_processed") # loading local dataset previously saved locally, works fine
dataset.save_to_disk(output_dir, fs=fs) #works fine
dataset = load_from_disk(output_dir, fs=fs) # crashes
```

## Expected results
The dataset is loaded

## Actual results

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/9s/gf0b9jz15d517yrf7m3nvlxr0000gn/T/tmp6t5e221_/speech/datasets/tests/common_voice_processed/train/state.json'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-2.6.1.dev0
- Platform: mac os monterey 12.5.1
- Python version: 3.8.13
- PyArrow version:pyarrow==9.0.0


Comments:
Hi Hubert! Could you please probably create a publicly available `gs://` dataset link? I think this would be easier for others to directly start to debug.

What seems to work is to change the line to:
```
fs.download(src_dataset_path, dataset_path.parent.as_posix(), recursive=True)
```
Bug with filtered indices

## Describe the bug
As reported by @PartiallyTyped (and by @Muennighoff):
- https://github.com/huggingface/datasets/issues/5111#issuecomment-1278652524

There is an issue with the indices of a filtered dataset.

## Steps to reproduce the bug
```python
ds = Dataset.from_dict({"num": [0, 1, 2, 3]})
ds = ds.filter(lambda num: num % 2 == 0, input_columns="num", batch_size=2)
assert all(item["num"] % 2 == 0 for item in ds)
```

## Expected results
The indices of the filtered dataset should correspond to the examples with "language" equals to "english".

## Actual results
Indices to items with other languages are included in the filtered dataset indices

## Preliminar investigation
It seems a bug introduced by:
- #5030


Comments:
The issue is here:
https://github.com/huggingface/datasets/blob/3ad9644b9a2e4558dd1d0f1e43c67658674e6228/src/datasets/arrow_dataset.py#L2964

@PartiallyTyped, @Muennighoff: the issue is fixed.

We are planning to make a patch release today.

Thanks a lot for the swift response! For a brief moment yesterday I thought I had gone insane 🤣On 14 Oct 2022, at 15:44, Albert Villanova del Moral ***@***.***> wrote:﻿
@PartiallyTyped, @Muennighoff: the issue is fixed.
We are planning to make a patch release today.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Map caching not working for some class methods

## Describe the bug
The cache loading is not working as expected for some class methods with a model stored in an attribute.
The new fingerprint for `_map_single` is not the same at each run. The hasher generate a different hash for the class method.
This comes from `dumps` function in `datasets.utils.py_utils` which generates a different dump at each run.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import AutoConfig, AutoModel, AutoTokenizer

dataset = load_dataset("ethos", "binary")
BASE_MODELNAME = "sentence-transformers/all-MiniLM-L6-v2"

class Object:
    def __init__(self):
        config = AutoConfig.from_pretrained(BASE_MODELNAME)
        self.bert = AutoModel.from_config(config=config, add_pooling_layer=False)
        self.tok = AutoTokenizer.from_pretrained(BASE_MODELNAME)

    def tokenize(self, examples):

        tokenized_texts = self.tok(
            examples["text"],
            padding="max_length",
            truncation=True,
            max_length=256,
        )
        return tokenized_texts

instance = Object()
result = dict()
for phase in ["train"]:
    result[phase] = dataset[phase].map(instance.tokenize, batched=True, load_from_cache_file=True, num_proc=2)

```

## Expected results
Load cache instead of recompute result.

## Actual results
Result recomputed from scratch at each run.
The cache works fine when deleting `bert` attribute.

## Environment info
- `datasets` version: 2.5.3.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 7.0.0
- Pandas version: 1.5.0



Comments:
The hash used for caching is computed by pickling recursively the function passed to `map`. Maybe some objects don't have the same hash across sessions.  In particular you can check the hash of your model using
```python
from datasets.fingerprint import Hasher
obj = AutoModel.from_config(config=config, add_pooling_layer=False)
print(Hasher.hash(obj))
```

You can find mode info here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument

Indeed, the hash is changing. The `dumps` function serialize the model object in different ways because the model object is not deterministic
```python
from datasets.utils.py_utils import dumps
obj1 = AutoModel.from_config(config=config, add_pooling_layer=False)
obj2 = AutoModel.from_config(config=config, add_pooling_layer=False)

dumps(bert) == dumps(bert2). # False
```

> You can find mode info here: https://huggingface.co/docs/datasets/about_cache
> 
> You can also provide your own unique hash in map if you want, with the new_fingerprint argument


Thanks, the doc is so helpful. Indeed, we can fix the hash and get cache hit using `new_fingerprint`. Closing the issue.
Fix task template reload from dict

Since #4926 the JSON dumps are simplified and it made task template dicts empty by default.

I fixed this by always including the task name which is needed to reload a task from a dict

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Just wondering if there might be other data classes default values missed that could cause an issue... Apart from feature-like classes and tasks, I don't see any others though...

I think we're good ! `asdict` is used on the DatasetInfo attributes like features, tasks etc. and they all support dict conversion properly now

> And a question: but this information about the tasks is no longer being saved as YAML tags in the dataset card; won't be a problem with current datasets using task templates (with this information in their metadata JSON) once we replace the JSON by the YAML tags (which do not have this information about the task templates)?

In the long run we'll use the train_eval_index YAML tags instead, but I agree when removing the JSON files we should try to not break existing code that may rely on this
Error in create a dataset from a Python generator

## Describe the bug
In HOW-TO-GUIDES > Load > [Python generator](https://huggingface.co/docs/datasets/v2.5.2/en/loading#python-generator), the code example defines the `my_gen` function, but when creating the dataset, an undefined `my_dict` is passed in.  

```Python
>>> from datasets import Dataset
>>> def my_gen():
...    for i in range(1, 4):
...        yield {"a": i}
>>> dataset = Dataset.from_generator(my_dict)
```

Comments:
Hi, thanks for reporting! The last line should be `dataset = Dataset.from_generator(my_gen)`.

Can I work on this one?
Classes label error when loading symbolic links using imagefolder

**Is your feature request related to a problem? Please describe.**
Like this: #4015
When there are **symbolic links** to pictures in the data folder, the parent folder name of the **real file** will be used as the class name instead of the parent folder of the symbolic link itself. Can you give an option to decide whether to enable symbolic link tracking?

This is inconsistent with the `torchvision.datasets.ImageFolder` behavior.

For example:
![image](https://user-images.githubusercontent.com/49552732/195008591-3cce644e-aabe-4f39-90b9-832861cadb3d.png)
![image](https://user-images.githubusercontent.com/49552732/195008841-0b0c2289-eb7f-411a-977b-37426f23a277.png)

It use `others` in green circle as class label but not `abnormal`, I wish `load_dataset` not use the real file parent as label.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.


Comments:
It can be solved temporarily  by remove `resolve` in 
https://github.com/huggingface/datasets/blob/bef23be3d9543b1ca2da87ab2f05070201044ddc/src/datasets/data_files.py#L278

Hi, thanks for reporting and suggesting a fix! We still need to account for `.`/`..` in the file path, so a more robust fix would be `Path(os.path.abspath(filepath))`.

> Hi, thanks for reporting and suggesting a fix! We still need to account for `.`/`..` in the file path, so a more robust fix would be `Path(os.path.abspath(filepath))`.

Thanks for your reply!
Mismatch between tutoriel and doc

## Describe the bug
In the "Process text data" tutorial, [`map` has `return_tensors` as kwarg](https://huggingface.co/docs/datasets/main/en/nlp_process#map). It does not seem to appear in the [function documentation](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.map), nor to work.

## Steps to reproduce the bug
MWE:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

from datasets import load_dataset
dataset  = load_dataset("lhoestq/demo1", split="train")

dataset = dataset.map(lambda examples: tokenizer(examples["review"]), batched=True, return_tensors="pt")

```

## Expected results
return_tensors to be a valid kwarg :smiley: 

## Actual results
```python
>> TypeError: map() got an unexpected keyword argument 'return_tensors'
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1052-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Comments:
Hi, thanks for reporting! This line should be replaced with 
```python
dataset = dataset.map(lambda examples: tokenizer(examples["text"], return_tensors="np"), batched=True)
```
for it to work (the `return_tensors` part inside the `tokenizer` call).

Can I work on this?

Fixed in https://github.com/huggingface/datasets/pull/5095
Use HTML relative paths for tiles in the docs

This PR replaces the absolute paths in the landing page tiles with relative ones so that one can test navigation both locally in and in future PRs (see [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5084/en/index) for an example PR where the links don't work).

I encountered this while working on the `optimum` docs and figured I'd fix it elsewhere too :)

Internal Slack thread: https://huggingface.slack.com/archives/C02GLJ5S0E9/p1665129710176619

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Good catch, @lewtun. Thanks for the fix.
> 
> Do you know if there are other absolute paths in the docs that should be fixed as well?

I found a few more in [0d4796b](https://github.com/huggingface/datasets/pull/5092/commits/0d4796b747e6620d9fcc17a8f74acc5cf4bba7be).

However, I noticed that none of the cross-references (e.g. to API classes / methods) work locally, but that is probably just a limitation of the local build

Thanks.
Allow connection objects in `from_sql` + small doc improvement

Allow connection objects in `from_sql` (emit a warning that they are cachable) and add a tip that explains the format of the con parameter when provided as a URI string.

PS: ~~This PR contains a parameter link, so https://github.com/huggingface/doc-builder/pull/311 needs to be merged before it's "ready for review".~~ Done!

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Review sync issues from GitHub to Hub

## Describe the bug
We have discovered that sometimes there were sync issues between GitHub and Hub datasets, after a merge commit to main branch.

For example:
- this merge commit: https://github.com/huggingface/datasets/commit/d74a9e8e4bfff1fed03a4cab99180a841d7caf4b
- was not properly synced with the Hub: https://github.com/huggingface/datasets/actions/runs/3002495269/jobs/4819769684
```
[main 9e641de]  Add Papers with Code ID to scifact dataset (#4941)
 Author: Albert Villanova del Moral <albertvillanova@users.noreply.huggingface.co>
 1 file changed, 42 insertions(+), 14 deletions(-)
push failed !
GitCommandError(['git', 'push'], 1, b'remote: ----------------------------------------------------------        \nremote: Sorry, your push was rejected during YAML metadata verification:        \nremote: - Error: "license" does not match any of the allowed types        \nremote: ----------------------------------------------------------        \nremote: Please find the documentation at:        \nremote: https://huggingface.co/docs/hub/models-cards#model-card-metadata        \nremote: ----------------------------------------------------------        \nTo [https://huggingface.co/datasets/scifact.git\n](https://huggingface.co/datasets/scifact.git/n) ! [remote rejected] main -> main (pre-receive hook declined)\nerror: failed to push some refs to \'[https://huggingface.co/datasets/scifact.git\](https://huggingface.co/datasets/scifact.git/)'', b'')
```
We are reviewing sync issues in previous commits to recover them and repushing to the Hub.

TODO: Review
- [x] #4941
  - scifact
- [x] #4931
  - scifact
- [x] #4753
  - wikipedia 
- [x] #4554
  - wmt17, wmt19, wmt_t2t
  - Fixed with "Release 2.4.0" commit: https://github.com/huggingface/datasets/commit/401d4c4f9b9594cb6527c599c0e7a72ce1a0ea49
    - https://huggingface.co/datasets/wmt17/commit/5c0afa83fbbd3508ff7627c07f1b27756d1379ea
    - https://huggingface.co/datasets/wmt19/commit/b8ad5bf1960208a376a0ab20bc8eac9638f7b400
    - https://huggingface.co/datasets/wmt_t2t/commit/b6d67191804dd0933476fede36754a436b48d1fc
- [x] #4607
- [x] #4416
  - lccc
  - Fixed with "Release 2.3.0" commit: https://huggingface.co/datasets/lccc/commit/8b1f8cf425b5653a0a4357a53205aac82ce038d1
- [x] #4367



Comments:
Nice!!
Resume failed process

**Is your feature request related to a problem? Please describe.**
When a process (`map`, `filter`, etc.) crashes part-way through, you lose all progress.

**Describe the solution you'd like**
It would be good if the cache reflected the partial progress, so that after we restart the script, the process can restart where it left off.

**Describe alternatives you've considered**
Doing processing outside of `datasets`, by writing the dataset to json files and building a restart mechanism myself.

**Additional context**
N/A


Comments:
Filtering on an empty dataset returns a corrupted dataset.

## Describe the bug
When filtering a dataset twice, where the first result is an empty dataset, the second dataset seems corrupted.

## Steps to reproduce the bug
```python
  datasets = load_dataset("glue", "sst2")
  dataset_split = datasets['validation']
  ds_filter_1 = dataset_split.filter(lambda x: False)  # Some filtering condition that leads to an empty dataset
  assert ds_filter_1.num_rows == 0
  sentences = ds_filter_1['sentence']
  assert len(sentences) == 0
  ds_filter_2 = ds_filter_1.filter(lambda x: False)  # Some other filtering condition
  assert ds_filter_2.num_rows == 0
  assert 'sentence' in ds_filter_2.column_names
  sentences = ds_filter_2['sentence']
```

## Expected results
The last line should be returning an empty list, same as 4 lines above.

## Actual results
The last line currently raises `IndexError: index out of bounds`.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.5.2
- Platform: macOS-11.6.6-x86_64-i386-64bit
- Python version: 3.9.11
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Comments:
~~It seems like #5043 fix (merged recently) is the root cause of such behaviour. When we empty indices mapping (because the dataset length equals to zero), we can no longer get column item like: `ds_filter_2['sentence']` which uses
`ds_filter_1._indices.column(0)`~~

**UPDATE:**
Empty datasets are returned without going through partial function on `map` method, which will not work to get indices for `filter`: we need to run `get_indices_from_mask_function` partial function on the dataset to get output = `{"indices": []}`. But this is complicated since functions used in args, in particular `get_indices_from_mask_function`, do not support empty datasets.
We can just handle empty datasets aside on filter method.

#self-assign

Thank you for solving this amazingly quickly!
IterableDataset formatting in numpy/torch/tf/jax

This code now returns a numpy array:
```python
from datasets import load_dataset

ds = load_dataset("imagenet-1k", split="train", streaming=True).with_format("np")
print(next(iter(ds))["image"])
```

It also works with "arrow", "pandas", "torch", "tf" and "jax"

### Implementation details:

I'm using the existing code to format an Arrow Table to the right output format for simplicity.
Therefore it's probbaly not the most optimized approach.

For example to output PyTorch tensors it does this for every example:

python data -> arrow table -> numpy extracted data -> pytorch formatted data

### Releasing this feature

Even though I consider this as a bug/inconsistency, this change is a breaking change.
And I'm sure some users were relying on the torch iterable dataset to return PIL Image and used data collators to convert to pytorch.

So I guess this is `datasets` 3.0 ?

### TODO

- [x] merge https://github.com/huggingface/datasets/pull/5072
- [ ] docs
- [ ] tests

Close https://github.com/huggingface/datasets/issues/5083

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5084). All of your documentation changes will be reflected on that endpoint.

Actually I'm not happy with this implementation. It always require the iterable dataset to have definite `features`, which removes a lot of flexibility. So I think we need an actual formatting from python objects, not from arrow data.

Closing this one since it has too many conflicts and still require some work - it will be easier to open a new PR