kardwalker/my_dataset huggingface dataset:
GitHub issue #5170: [Caching] Deterministic hashing of torch tensors
State: closed | Created: 2022-10-27T09:15:15Z

Currently this fails
```python
import torch
from datasets.fingerprint import Hasher

t = torch.tensor([1.])

def func(x):
    return t + x

hash1 = Hasher.hash(func)
t = torch.tensor([1.])
hash2 = Hasher.hash(func)
assert hash1 == hash2
```

Also as noticed in https://discuss.huggingface.co/t/dataset-cant-cache-models-outputs/24945, using a model in a `map` function doesn't work well with caching. Indeed the `bert-base-uncased` model has a different hash every time you reload it. Supporting torch tensors may also help in this case.

This can be fixed by registering a custom pickling functions for torch tensors - as we did for other objects such as CodeType, FunctionType and Regex in `py_utils.py`
GitHub issue #5168: Fix CI require beam
State: closed | Created: 2022-10-26T16:49:33Z

This PR:
- Fixes the CI `require_beam`: before it was requiring PyTorch instead
  ```python
  def require_beam(test_case):
      if not config.TORCH_AVAILABLE:
          test_case = unittest.skip("test requires PyTorch")(test_case)
      return test_case
  ```
- Fixes a missing `require_beam` in `test_beam_based_builder_download_and_prepare_as_parquet`
- Refactors `require_beam` to use `pytest` (`skipif`) instead
GitHub issue #5167: Add ffmpeg4 installation instructions in warnings
State: closed | Created: 2022-10-26T14:21:14Z

Adds instructions on how to install `ffmpeg=4` on Linux (relevant for Colab users).

Looks pretty ugly because I didn't find a way to check `ffmpeg` version from python (without `subprocess.call()`; `ctypes.util.find_library` doesn't work`), so the warning is raised on each decoding. Any suggestions on how to make it look nice are welcome!

This is how it looks on Colab:
![image](https://user-images.githubusercontent.com/16348744/198052412-d48018d1-4416-4aa5-9114-f7f9b4af031f.png)
GitHub issue #5166: Support dill 0.3.6
State: closed | Created: 2022-10-26T08:24:59Z

This PR:
- ~~Unpins dill to allow installing dill>=0.3.6~~
- ~~Removes the fix on dill for >=0.3.6 because they implemented a deterministic mode (to be confirmed by @anivegesana)~~
- Pins dill<0.3.7 to allow latest dill 0.3.6
- Implements a fix for dill `save_function` for dill 0.3.6
- Additionally had to implement a fix for dill `save_code` and `_save_regex` for dill 0.3.6
- Fixes the CI so that the latest dill version is tested (besides the minimum 0.3.1.1 required by apache-beam 2.42.0)

Fix #5162.
GitHub issue #5165: Memory explosion when trying to access 4d tensors in datasets cast to torch or np
State: open | Created: 2022-10-26T08:14:47Z

### Describe the bug

When trying to access an item by index, in a datasets.Dataset cast to torch/np using `set_format` or `with_format`, we get a memory explosion if the item contains 4d (or above) tensors. 

### Steps to reproduce the bug

MWE:

```python
from datasets import load_dataset
import numpy as np

def create_4d_tensor(item):  
    i = item["num_nodes"]
    item["x_big"] = np.random.rand(i, 2*i, int(i/2), 1) + 1 # we create a big 4d tensor
    return item

if __name__ == "__main__":
    dataset = load_dataset(path=f"graphs-datasets/PROTEINS")

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset = dataset.map( 
        create_4d_tensor, 
        batched=False, 
        writer_batch_size=100, 
    )

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset.set_format("torch")
    
    print(dataset["train"].format)
    # This gets killed :(
    print(dataset["train"][0].keys())
```

The problem likely comes from `format_table` [here](https://cs.github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/src/datasets/arrow_dataset.py#L2328)

### Expected behavior

No memory explosion when trying to access dataset items after cast.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
GitHub issue #5164: WIP: drop labels in Image and Audio folders by default
State: closed | Created: 2022-10-25T17:21:49Z

will fix https://github.com/huggingface/datasets/issues/5153 and redundant labels displaying for most of the images datasets on the Hub (which are used just to store files)

TODO: discuss adding `drop_labels` (and `drop_metadata`) params to yaml
GitHub issue #5162: Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6
State: closed | Created: 2022-10-25T13:23:50Z

### Describe the bug

When using `pip-compile` (part of `pip-tools`) to generate a pinned requirements file that includes `datasets`, a version conflict of `dill` appears.

It is caused by a transitive dependency conflict between `datasets` and `multiprocess`.

### Steps to reproduce the bug

```bash
$ echo "datasets" > requirements.in
$ pip install pip-tools
$ pip-compile requirements.in 
Could not find a version that matches dill<0.3.6,>=0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
Tried: 0.2, 0.2, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.2.6, 0.2.7, 0.2.7.1, 0.2.8, 0.2.8.1, 0.2.8.2, 0.2.9, 0.3.0, 0.3.1, 0.3.1.1, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.3.5, 0.3.5, 0.3.5.1, 0.3.5.1, 0.3.6, 0.3.6
Skipped pre-versions: 0.1a1, 0.2a1, 0.2a1, 0.2b1, 0.2b1
There are incompatible versions in the resolved dependencies:
  dill<0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
  dill>=0.3.6 (from multiprocess==0.70.14->datasets==2.6.1->-r requirements.in (line 1))
```

### Expected behavior

A correctly generated file `requirements.txt` with pinned dependencies

### Environment info

Tested with versions `2.6.1, 2.6.0, 2.5.2` on Python 3.8 and 3.10 on Ubuntu 20.04LTS and Python 3.10 on MacOS 12.6 (M1).
GitHub issue #5161: Dataset can’t cache model’s outputs
State: closed | Created: 2022-10-25T12:19:00Z

### Describe the bug

Hi,

I try to cache some outputs of teacher model( Knowledge Distillation ) by using map function of Dataset library, while every time I run my code, I still recompute all the sequences. I tested Bert Model like this, I got different hash every single run, so any idea to deal with this?

### Steps to reproduce the bug

1. run below code
2. get different hash
```
from transformers import BertModel
from transformers import AutoTokenizer
import torch
token = ['hello']
model = BertModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
def abcd():
    with torch.no_grad():
        
        out = model(**tok(token,return_tensors='pt'))[0]
        # out = tok(token)
    return out
from datasets.fingerprint import Hasher
my_func = abcd
print(Hasher.hash(my_func))

print(abcd())
```

### Expected behavior

I wanna cache all the model output 

### Environment info

datasets:2.5.0
GitHub issue #5160: Automatically add filename for image/audio folder
State: open | Created: 2022-10-25T09:56:49Z

### Feature request

When creating a custom audio of image dataset, it would be great to automatically have access to the filename. It should be both:
a) Automatically displayed in the viewer
b) Automatically added as a column to the dataset when doing `load_dataset`

In `diffusers` our test rely quite heavily on images and audio files now and it's a bit tedious at the moment to download specific images from a datasets repo. 

E.g. we have a dataset of images for tests in `diffusers`: https://huggingface.co/datasets/hf-internal-testing/diffusers-images
where it would be extremely nice to have direct access to the filename both visually on the datasets page (@severo ) as well as via the `load_datasets` function. We currently have some akward functionality to download images by path name: https://github.com/huggingface/diffusers/blob/2fb8fafa4b761f6fc144cf75a6f6f0ea6af3a1c1/src/diffusers/utils/testing_utils.py#L131 
It would be much nicer to just go over `load_dataset(...)` 

### Motivation

Intuitively the filename is something people understand directly. E.g if you upload a folder of images online, it's nice if you recognize the image as well as the filename next to it directly and that you're able to use it right away. 

The label on the other hand is less intuitive to understand as you haven't added it yourself.

### Your contribution

Not sure if I have the time to add it myself anytime soon, but it would help us a lot for `diffusers`.
GitHub issue #5158: Fix language and license tag names in all Hub datasets
State: closed | Created: 2022-10-25T08:19:29Z

While working on this:
- #5137

we realized there are still many datasets with deprecated "languages" and "licenses" tag names (instead of "language" and "license").

This is a blocking issue: no subsequent PR can be opened to modify their metadata: a ValueError will be thrown.

We should fix the "language" and "license" tag names in all Hub datasets.

TODO:
- [x] Fix language and license tag names in 402 Hub datasets

CC: @julien-c
GitHub issue #5157: Consistent caching between python and jupyter
State: closed | Created: 2022-10-25T01:34:33Z

### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen
GitHub issue #5156: Unable to download dataset using Azure Data Lake Gen 2
State: closed | Created: 2022-10-25T00:43:18Z

### Describe the bug

When using the DatasetBuilder method with the credentials for the cloud storage Azure Data Lake (adl) Gen2, the following error is showed:
```
Traceback (most recent call last):
  File "download_hf_dataset.py", line 143, in <module>
    main()
  File "download_hf_dataset.py", line 102, in main
    builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/datasets/builder.py", line 671, in download_and_prepare
    fs_token_paths = fsspec.get_fs_token_paths(output_dir, storage_options=storage_options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/core.py", line 639, in get_fs_token_paths
    fs = cls(**options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'account_name'
```

If I don't pass the storage_options argument (leave it as None), it requires the credentials used in ADL Gen 1:
`TypeError: __init__() missing 3 required positional arguments: 'tenant_id', 'client_id', and 'client_secret'`

Thus, it is not possible to download a dataset from the cloud using Azure Data Lake (adl) Gen2.

### Steps to reproduce the bug

Assuming that you have an account on Azure and at Storage Account that can be used for reproduce:

1. Create a dict with the format to connect to Azure Data Lake Gen 2
```
storage_options = {"account_name": ACCOUNT_NAME, "account_key": ACCOUNT_KEY)  # gen 2 filesystem
```
2. Create a dataset builder for any HF hosted dataset
```
builder = load_dataset_builder(dataset_name)
```
3. Try to download the dataset passing the storage_options as an argument
```
save_dir = 'adl://my_save_dir'
builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
```

### Expected behavior

Not seeing the error mentioned above and being able to download the dataset to the provided path on ADL

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
GitHub issue #5153: default Image/AudioFolder infers labels when there is no metadata files even if there is only one dir
State: closed | Created: 2022-10-24T13:28:18Z

### Describe the bug

By default FolderBasedBuilder infers labels if there is not metadata files, even if it's meaningless (for example, they are in a single directory or in the root folder, see this repo as an example: https://huggingface.co/datasets/patrickvonplaten/audios

As this is a corner case for quick exploration of images or audios on the Hub. 

### Steps to reproduce the bug

If you have directory like this:

```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
or

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
```
doing `ds = load_dataset(repo)` would create `label` feature:
```python
print(ds["train"][0])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 0}
```

Also, if you have the following structure:

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
   image4.jpg
   image5.jpg
   image6.jpg
```
it will infer two labels:

```python
print(ds["train"][0])
print(ds["train"][-1])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 1}
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x415 at 0x7FB5326555B0>, 'label': 0}
```

### Expected behavior

We should have only one base feature (Image/Audio) in such cases.

### Environment info

all versions of `datasets`
GitHub issue #5151: Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)
State: open | Created: 2022-10-24T12:59:18Z

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets.
GitHub issue #5150: Problems after upgrading to 2.6.1
State: open | Created: 2022-10-24T11:32:36Z

### Describe the bug

Loading a dataset_dict from disk with `load_from_disk` is now creating a `KeyError "length"` that was not occurring in v2.5.2.

Context: 
- Each individual dataset in the dict is created with `Dataset.from_pandas` 
- The dataset_dict is create from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- The pandas dataframe, besides text columns, has a column with a dictionary inside and potentially different keys in each row. Correctly the `Dataset.from_pandas` function adds `key: None` to all dictionaries in each row so that the schema can be correctly inferred.

### Steps to reproduce the bug

Steps to reproduce:

- Upgrade to datasets==2.6.1
- Create a dataset from pandas dataframe with `Dataset.from_pandas`
- Create a dataset_dict from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- Save to disk with the `save` function

### Expected behavior

Same as in v2.5.2, that is load from disk without errors

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.4.209-129.367.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
GitHub issue #5148: Cannot find the rvl_cdip dataset
State: closed | Created: 2022-10-24T04:57:42Z

Hi,
I am trying to use load_dataset to load the official "rvl_cdip" dataset but getting an error.

dataset = load_dataset("rvl_cdip")

Couldn't find 'rvl_cdip' on the Hugging Face Hub either: FileNotFoundError: Couldn't find the file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/rvl_cdip/rvl_cdip.py

Regards,
GitHub issue #5147: Allow ignoring kwargs inside fn_kwargs during dataset.map's fingerprinting
State: open | Created: 2022-10-22T21:46:38Z

### Feature request

`dataset.map` accepts a `fn_kwargs` that is passed to `fn`. Currently, the whole `fn_kwargs` is used by `fingerprint_transform` to calculate the new fingerprint.

I'd like to be able to inform `fingerprint_transform` which `fn_kwargs` shoud/shouldn't be taken into account during hashing.

Of course, users should be aware to properly use this new feature, just like the internal usages of `fingerprint_transform` [does](https://github.com/huggingface/datasets/blob/2699593b33ee63d17aad2a2bfddedd38a8df57b8/src/datasets/arrow_dataset.py#L2700).

### Motivation

This is originally motivated by https://github.com/huggingface/transformers/pull/18351#issuecomment-1263588680. 

Nonetheless, consider a more general processing function that accepts a kwarg that does not influence it's output:
```python
def fn(example, verbose=False):
    ...
```

Then `dataset.map(fn, verbose=True)` would not benefit from dataset caching.

I'm not sure if other methods in the `Dataset` API could benefit from this feature.

### Your contribution

Based on `fingerprint_transform `'s `wrapper` function [here](https://github.com/huggingface/datasets/blob/c59cc34fcd2a369d27b77cc678017f5976a926a9/src/datasets/fingerprint.py#L443), it seems to me that it should be possible to make `.map`/`._map_single` accept something like `fn_use_fingerprint_kwargs`/`fn_ignore_fingerprint_kwargs` (probably another arg name). This would then be used by `fingerprint_transform.wrapper` to better/more flexibly hash the transformation.

I could contribute with a PR if this feature and approach look good to you.
GitHub issue #5145: Dataset order is not deterministic with ZIP archives and `iter_files`
State: closed | Created: 2022-10-21T09:00:03Z

### Describe the bug

For the `beans` dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

### Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

```python
from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])
```

### Expected behavior

The order of the images is the same on all machines.

### Environment info

On the EC2 instance:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked
```

On my local laptop:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1
```

On github actions:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4
```
GitHub issue #5144: Inconsistent documentation on map remove_columns
State: closed | Created: 2022-10-21T08:37:53Z

### Describe the bug

The page [process](https://huggingface.co/docs/datasets/process) says this about the parameter `remove_columns` of the function `map`:
When you remove a column, it is only removed after the example has been provided to the mapped function.

So it seems that the `remove_columns` parameter removes after the mapped functions.

However, another page, [the documentation of the function map](https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.Dataset.map.remove_columns) says:
Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in remove_columns, these columns will be kept.

So one page says "after the mapped function" and another says "before the mapped function."
Is there something wrong?


### Steps to reproduce the bug

Not about code.

### Expected behavior

consistent about the descriptions of the behavior of the parameter `remove_columns` in the function `map`.

### Environment info

datasets V2.6.0
GitHub issue #5143: DownloadManager Git LFS support
State: closed | Created: 2022-10-20T15:29:29Z

### Feature request

Maybe I'm mistaken but the `DownloadManager` does not support extracting git lfs files out of the box right?
Using `dl_manager.download()` or `dl_manager.download_and_extract()` still returns lfs files afaict. 

Is there a good way to write a dataset loading script for a repo with lfs files?

### Motivation

/

### Your contribution

/
GitHub issue #5137: Align task tags in dataset metadata
State: closed | Created: 2022-10-19T09:41:42Z

## Describe
Once we have agreed on a common naming for task tags for all open source projects, we should align on them.

## Steps
- [x] Align task tags in canonical datasets
  - [x] task_categories: 4 datasets
  - [x] task_ids (by @lhoestq)
- [x] Open PRs in community datasets
  - [x] task_categories: 451 datasets
  - [x] task_ids: 556 datasets
GitHub issue #5136: Update docs once dataset scripts transferred to the Hub
State: closed | Created: 2022-10-19T07:58:27Z

Todo:
- [x] Update docs:
  - [x] Datasets on GitHub (legacy)
  - [x] Load: offline
  - [x] About dataset load:
    - [x] Maintaining integrity
    - [x] Security
- [x] Update docstrings:
  - [x] Inspect:
    - [x] get_dataset_config_info
    - [x] get_dataset_split_names
  - [x] Load:
    - [x] dataset_module_factory
    - [x] load_dataset_builder
    - [x] load_dataset
- [x] Remove `ADD_NEW_DATASET.md`
- [x] Update `.github/ISSUE_TEMPLATE/config.yml`

Fix #5135.
GitHub issue #5135: Update docs once dataset scripts transferred to the Hub
State: closed | Created: 2022-10-19T06:58:19Z

## Describe the bug
As discussed in:
- https://github.com/huggingface/hub-docs/pull/423#pullrequestreview-1146083701

we should update our docs once dataset scripts have been transferred to the Hub (and removed from GitHub):
- #4974

Concretely:
- [x] Datasets on GitHub (legacy): https://huggingface.co/docs/datasets/main/en/share#datasets-on-github-legacy
- [x] ADD_NEW_DATASET: https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md
- ...

This PR complements the work of:
- #5067

This PR is a follow-up of PRs:
- #3777

CC: @julien-c
GitHub issue #5134: Raise ImportError instead of OSError if required extraction library is not installed
State: closed | Created: 2022-10-18T17:53:46Z

According to the official Python docs, `OSError` should be thrown in the following situations:
> This exception is raised when a system function returns a system-related error, including I/O failures such as “file not found” or “disk full” (not for illegal argument types or other incidental errors).

Hence, it makes more sense to raise `ImportError` instead of `OSError` when the required extraction/decompression library is not installed.
GitHub issue #5133: Tensor operation not functioning in dataset mapping
State: closed | Created: 2022-10-18T17:53:35Z

## Describe the bug
I'm doing a torch.mean() operation in data preprocessing, and it's not working.

## Steps to reproduce the bug
```
from transformers import pipeline
import torch
import numpy as np
from datasets import load_dataset
device = 'cuda:0'
raw_dataset = load_dataset("glue", "sst2")
feature_extraction = pipeline('feature-extraction', 'bert-base-uncased', device=device)
def extracted_data(examples):
    # feature = torch.tensor(feature_extraction(examples['sentence'], batch_size=16), device=device)
    # feature = torch.mean(feature, dim=1)
    feature = np.asarray(feature_extraction(examples['sentence'], batch_size=16)).squeeze().mean(1)
    print(feature.shape)
    return {'feature': feature}

extracted_dataset = raw_dataset.map(extracted_data, batched=True, batch_size=16)
```

## Results
When running with torch.mean(), the shape printed out is [16, seq_len, 768], which is exactly the same before the operation. While numpy works just fine, which gives [16, 768].


## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-4.4.0-142-generic-x86_64-with-glibc2.31
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0
GitHub issue #5132: Depracate `num_proc` parameter in `DownloadManager.extract`
State: closed | Created: 2022-10-18T17:41:05Z

The `num_proc` parameter is only present in `DownloadManager.extract` but not in `StreamingDownloadManager.extract`, making it impossible to support streaming in the dataset scripts that use it (`openwebtext` and `the_pile_stack_exchange`). We can avoid this situation by deprecating this parameter and passing `DownloadConfig`'s `num_proc` to `map_nested` instead, as it's done in `DownloadManager.download`.
GitHub issue #5129: unexpected `cast` or `class_encode_column` result after `rename_column`
State: closed | Created: 2022-10-18T11:15:24Z

## Describe the bug
When invoke `cast` or `class_encode_column` to a colunm renamed by `rename_column` , it will convert all the variables in this column into one variable. I also run this script in version 2.5.2, this bug does not appear. So I switched to the older version.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi", "en")
data = dataset['train']

data = data.remove_columns(
                [
                    "review_id",
                    "product_id",
                    "reviewer_id",
                    "review_title",
                    "language",
                    "product_category",
                ]
            )
data = data.rename_column("review_body", "text")
data1 = data.class_encode_column("stars")
print(set(data1.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}


data = data.rename_column("stars", "label")
print(set(data.data.columns[0]))
# output: {<pyarrow.Int32Scalar: 5>, <pyarrow.Int32Scalar: 4>, <pyarrow.Int32Scalar: 1>, <pyarrow.Int32Scalar: 3>, <pyarrow.Int32Scalar: 2>}
data2 = data.class_encode_column("label")
print(set(data2.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 0>}
```

## Expected results
the last print should be: 
{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}

## Actual results
but it output:
{<pyarrow.Int64Scalar: 0>}

## Environment info
- `datasets` version: 2.6.1
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0
GitHub issue #5127: [WIP] WebDataset export
State: closed | Created: 2022-10-17T16:50:22Z

I added a first draft of the `IterableDataset.to_wds` method.

You can use it to savea dataset loaded in streamign mode as a webdataset locally.
The API can be further improved to allow to export to a cloud storage like the HF Hub.

I also included sharding with a default max shard size of 500MB (uncompressed), and it is single-processed fo rnow.

Choosing the number of shards is not implemented yet - though if we know the size of the `IterableDataset` this is probably doable`.

For example
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
>>> ds.to_wds("output_dir", compress=True)
>>> import webdataset as wds
>>> ds = wds.WebDataset("output_dir/rotten_tomatoes-train-000000.tar.gz").decode()
>>> next(iter(ds))
{'__key__': '0',
'__url__': 'output_dir/rotten_tomatoes-train-000000.tar.gz',
'label.cls': 1,
'text.txt': 'the rock is destined to be the 21st century\'s new ..., jean-claud van damme or steven segal .'}
```

### Implementation details

The WebDataset format is made of TAR archives containing a series of files per example. For example one pair of `image.jpg` and `label.cls` for image classification.

WebDataset automatically decodes serialized data based on the extension of the files, and output a dictionary. For example `{"image.png": np.array(...), "label.cls": 0}` if you choose the numpy decoding.

To use the automatic decoding, I store each field of each example as a file with its corresponding extension (jpg, json, cls, etc.)

While this is useful to end up with a dictionary with one key per column and appropriate decoding, it can create huge TAR archives if the dataset is made of small samples of text - probably because of useless TAR metadata for each file. This also makes loading super slow: iterating on SQuAD takes 50sec vs 7sec using `datasets` in streaming mode.

I haven't taken a look at alternatives for text datasets made out of small samples, but for image datasets this can already be used to run some benchmarks.
GitHub issue #5120: Fix `tqdm` zip bug
State: closed | Created: 2022-10-16T22:19:18Z

This PR solves #5117, by wrapping the entire `zip` clause in tqdm.

For more information, please checkout this Stack Overflow thread:
https://stackoverflow.com/questions/41171191/tqdm-progressbar-and-zip-built-in-do-not-work-together
GitHub issue #5118: Installing `datasets` on M1 computers
State: closed | Created: 2022-10-16T16:50:08Z

## Describe the bug
I wanted to install `datasets` dependencies on my M1 (in order to start contributing to the project). However, I got an error regarding `tensorflow`.

On M1, `tensorflow-macos` needs to be installed instead. Can we add a conditional requirement, so that `tensorflow-macos` would be installed on M1?

## Steps to reproduce the bug
Fresh clone this project (on m1), create a virtualenv and run this:
```python
pip install -e ".[dev]"
```

## Expected results
Installation should be smooth, and all the dependencies should be installed on M1.

## Actual results
You should receive an error, saying pip couldn't find a version that matches this pattern:
```
tensorflow>=2.3,!=2.6.0,!=2.6.1
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.2.dev0
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.0
GitHub issue #5117: Progress bars have color red and never completed to 100%
State: closed | Created: 2022-10-14T16:12:30Z

## Describe the bug
Progress bars after transformative operations turn in red and never be completed to 100%

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset('rotten_tomatoes', split='test').filter(lambda o: True)
```

## Expected results
Progress bar should be 100% and green

## Actual results
Progress  bar turn in red and never completed to 100%

## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- PyArrow version: 6.0.1
- Pandas version: 1.3.5
GitHub issue #5115: Fix iter_batches
State: closed | Created: 2022-10-14T12:06:14Z

The `pa.Table.to_reader()` method available in `pyarrow>=8.0.0` may return chunks of size < `max_chunksize`, therefore `iter_batches` can return batches smaller than the `batch_size` specified by the user

Therefore batched `map` couldn't always use batches of the right size, e.g. this fails because it runs only on one batch of one element:
```python

from datasets import Dataset, concatenate_datasets

ds = concatenate_datasets([Dataset.from_dict({"a": [i]}) for i in range(10)])

ds2 = ds.map(lambda _: {}, batched=True)
assert list(ds2) == list(ds)
```

This was introduced in https://github.com/huggingface/datasets/pull/5030

Close https://github.com/huggingface/datasets/issues/5111

This will require a patch release along with https://github.com/huggingface/datasets/pull/5113

TODO:
- [x] fix tests
- [x] add more tests
GitHub issue #5114: load_from_disk with remote filesystem fails due to a wrong temporary local folder path
State: open | Created: 2022-10-14T11:54:53Z

## Describe the bug

The function load_from_disk fails when using a remote filesystem because of a wrong temporary path generation in the load_from_disk method of arrow_dataset.py:

```python
if is_remote_filesystem(fs):
    src_dataset_path = extract_path_from_uri(dataset_path)
    dataset_path = Dataset._build_local_temp_path(src_dataset_path)
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
```
If _dataset_path_ is `gs://speech/mydataset/train`, then _src_dataset_path_ will be `speech/mydataset/train` and _dataset_path_ will be something like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train`

Then, after downloading the **folder** _src_dataset_path_, you will get a path like `/var/folders/9s/gf0b/T/tmp6t/speech/mydataset/train/train/state.json` (notice we have train twice)

Instead of downloading the remote folder we should be downloading all the files in the folder for the path to be right:

```python
fs.download(os.path.join(src_dataset_path,*), dataset_path.as_posix(), recursive=True)
```

## Steps to reproduce the bug
```python
fs = gcsfs.GCSFileSystem(**storage_options)
dataset = load_from_disk("common_voice_processed") # loading local dataset previously saved locally, works fine
dataset.save_to_disk(output_dir, fs=fs) #works fine
dataset = load_from_disk(output_dir, fs=fs) # crashes
```

## Expected results
The dataset is loaded

## Actual results

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/9s/gf0b9jz15d517yrf7m3nvlxr0000gn/T/tmp6t5e221_/speech/datasets/tests/common_voice_processed/train/state.json'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-2.6.1.dev0
- Platform: mac os monterey 12.5.1
- Python version: 3.8.13
- PyArrow version:pyarrow==9.0.0
GitHub issue #5112: Bug with filtered indices
State: closed | Created: 2022-10-14T10:35:47Z

## Describe the bug
As reported by @PartiallyTyped (and by @Muennighoff):
- https://github.com/huggingface/datasets/issues/5111#issuecomment-1278652524

There is an issue with the indices of a filtered dataset.

## Steps to reproduce the bug
```python
ds = Dataset.from_dict({"num": [0, 1, 2, 3]})
ds = ds.filter(lambda num: num % 2 == 0, input_columns="num", batch_size=2)
assert all(item["num"] % 2 == 0 for item in ds)
```

## Expected results
The indices of the filtered dataset should correspond to the examples with "language" equals to "english".

## Actual results
Indices to items with other languages are included in the filtered dataset indices

## Preliminar investigation
It seems a bug introduced by:
- #5030
GitHub issue #5111: map and filter not working properly in multiprocessing with the new release 2.6.0
State: closed | Created: 2022-10-13T17:00:55Z

## Describe the bug
When mapping is used on a dataset with more than one process, there is a weird behavior when trying to use `filter` , it's like only the samples from one worker are retrieved, one needs to specify the same `num_proc` in filter for it to work properly. This doesn't happen with `datasets` version 2.5.2

In the code below the data is filtered differently when we increase `num_proc` used in `map` although the datsets before and after mapping have identical elements.

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset

def preprocess(example):
    return example

ds = load_dataset("codeparrot/codeparrot-clean-valid", split="train").select([i for i in range(10)])
ds1 = ds.map(preprocess, num_proc=2)
ds2 = ds.map(preprocess)

# the datasets elements are the same
for i in range(len(ds1)):
    assert ds1[i]==ds2[i]

print(f'Target column before filtering {ds1["autogenerated"]}')
print(f'Target column before filtering {ds2["autogenerated"]}')
print(f"datasets version {datasets.__version__}")

ds_filtered_1 = ds1.filter(lambda x: not x["autogenerated"])
ds_filtered_2 = ds2.filter(lambda x: not x["autogenerated"])

# all elements in Target column are false so they should all be kept, but for ds2 only the first 5=num_samples/num_proc are kept
print(ds_filtered_1)
print(ds_filtered_2)
```
```
Target column before filtering [False, False, False, False, False, False, False, False, False, False]
Target column before filtering [False, False, False, False, False, False, False, False, False, False]

Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 5
})
Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 10
})
```
## Expected results
Increasing `num_proc` in mapping shouldn't alter filtering. With the previous version 2.5.2 this doesn't happen

## Actual results
Filtering doesn't work properly when we increase `num_proc` in mapping but not when calling `filter`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.0
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2
GitHub issue #5109: Map caching not working for some class methods
State: closed | Created: 2022-10-13T09:12:58Z

## Describe the bug
The cache loading is not working as expected for some class methods with a model stored in an attribute.
The new fingerprint for `_map_single` is not the same at each run. The hasher generate a different hash for the class method.
This comes from `dumps` function in `datasets.utils.py_utils` which generates a different dump at each run.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import AutoConfig, AutoModel, AutoTokenizer

dataset = load_dataset("ethos", "binary")
BASE_MODELNAME = "sentence-transformers/all-MiniLM-L6-v2"

class Object:
    def __init__(self):
        config = AutoConfig.from_pretrained(BASE_MODELNAME)
        self.bert = AutoModel.from_config(config=config, add_pooling_layer=False)
        self.tok = AutoTokenizer.from_pretrained(BASE_MODELNAME)

    def tokenize(self, examples):

        tokenized_texts = self.tok(
            examples["text"],
            padding="max_length",
            truncation=True,
            max_length=256,
        )
        return tokenized_texts

instance = Object()
result = dict()
for phase in ["train"]:
    result[phase] = dataset[phase].map(instance.tokenize, batched=True, load_from_cache_file=True, num_proc=2)

```

## Expected results
Load cache instead of recompute result.

## Actual results
Result recomputed from scratch at each run.
The cache works fine when deleting `bert` attribute.

## Environment info
- `datasets` version: 2.5.3.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 7.0.0
- Pandas version: 1.5.0
GitHub issue #5105: Specifying an exisiting folder in download_and_prepare deletes everything in it
State: open | Created: 2022-10-12T11:53:33Z

## Describe the bug
The builder correctly creates the `output_dir` folder if it doesn't exist, but if the folder exists everything within it is deleted. Specifying `"."` as the `output_dir` deletes everything in your current dir but also leads to **another bug** whose traceback is the following:

```
                          Traceback (most recent call last)
Input In [11], in <cell line: 1>()
----> 1 rotten_tomatoes_builder.download_and_prepare(output_dir=".", max_shard_size="200MB", file_format="parquet")

File ~/BIGSCIENCE/env/lib/python3.9/site-packages/datasets/builder.py:818, in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, storage_options, **download_and_prepare_kwargs)

File /usr/lib/python3.9/contextlib.py:124, in _GeneratorContextManager.__exit__(self, type, value, traceback)
    122 if type is None:
    123     try:
--> 124         next(self.gen)
    125     except StopIteration:
    126         return False

File ~/BIGSCIENCE/env/lib/python3.9/site-packages/datasets/builder.py:760, in incomplete_dir(dirname)

File /usr/lib/python3.9/shutil.py:722, in rmtree(path, ignore_errors, onerror)
    720         os.rmdir(path)
    721     except OSError:
--> 722         onerror(os.rmdir, path, sys.exc_info())
    723 else:
    724     try:
    725         # symlinks to directories are forbidden, see bug #1669

File /usr/lib/python3.9/shutil.py:720, in rmtree(path, ignore_errors, onerror)
    718 _rmtree_safe_fd(fd, path, onerror)
    719 try:
--> 720     os.rmdir(path)
    721 except OSError:
    722     onerror(os.rmdir, path, sys.exc_info())

OSError: [Errno 22] Invalid argument: '/home/christopher/BIGSCIENCE/.'
``` 

## Steps to reproduce the bug
```python
rotten_tomatoes_builder = load_dataset_builder("rotten_tomatoes")
rotten_tomatoes_builder.download_and_prepare(output_dir="./test_folder", max_shard_size="200MB", file_format="parquet")
```
If `test_folder` contains any files they will all be deleted
 
## Expected results
Either a warning that all files will be deleted, but preferably that they not be deleted at all.

## Actual results
N/A

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
GitHub issue #5102: Error in create a dataset from a Python generator
State: closed | Created: 2022-10-11T14:28:58Z

## Describe the bug
In HOW-TO-GUIDES > Load > [Python generator](https://huggingface.co/docs/datasets/v2.5.2/en/loading#python-generator), the code example defines the `my_gen` function, but when creating the dataset, an undefined `my_dict` is passed in.  

```Python
>>> from datasets import Dataset
>>> def my_gen():
...    for i in range(1, 4):
...        yield {"a": i}
>>> dataset = Dataset.from_generator(my_dict)
```
GitHub issue #5099: datasets doesn't support # in data paths
State: closed | Created: 2022-10-11T10:05:32Z

## Describe the bug
dataset files with `#` symbol their paths aren't read correctly.

## Steps to reproduce the bug
The data in folder `c#`of this [dataset](https://huggingface.co/datasets/loubnabnl/bigcode_csharp) can't be loaded. While the folder `c_sharp` with the same data is loaded properly
```python
ds = load_dataset('loubnabnl/bigcode_csharp', split="train", data_files=["data/c#/*"])
```
```
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/loubnabnl/bigcode_csharp/resolve/27a3166cff4bb18e11919cafa6f169c0f57483de/data/c#/data_0003.jsonl
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.5.2
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3

cc @lhoestq
GitHub issue #5098: Classes label error when loading symbolic links using imagefolder
State: closed | Created: 2022-10-11T06:10:58Z

**Is your feature request related to a problem? Please describe.**
Like this: #4015
When there are **symbolic links** to pictures in the data folder, the parent folder name of the **real file** will be used as the class name instead of the parent folder of the symbolic link itself. Can you give an option to decide whether to enable symbolic link tracking?

This is inconsistent with the `torchvision.datasets.ImageFolder` behavior.

For example:
![image](https://user-images.githubusercontent.com/49552732/195008591-3cce644e-aabe-4f39-90b9-832861cadb3d.png)
![image](https://user-images.githubusercontent.com/49552732/195008841-0b0c2289-eb7f-411a-977b-37426f23a277.png)

It use `others` in green circle as class label but not `abnormal`, I wish `load_dataset` not use the real file parent as label.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.
GitHub issue #5097: Fatal error with pyarrow/libarrow.so
State: closed | Created: 2022-10-10T20:29:04Z

## Describe the bug
When using datasets, at the very end of my jobs the program crashes (see trace below).
It doesn't seem to affect anything, as it appears to happen as the program is closing down. Just importing `datasets` is enough to cause the error.

## Steps to reproduce the bug
This is sufficient to reproduce the problem:

```bash
python -c "import datasets"
```

## Expected results
Program should run to completion without an error.

## Actual results
```bash
Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io/src/9e6648842a-364b708815.clean/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200af06) [0x150dff547f06]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x20028e5) [0x150dff53f8e5]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1f27e09) [0x150dff464e09]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x150dff548a3d]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1f25948) [0x150dff462948]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x150dff548a3d]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1ee0b46) [0x150dff41db46]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x194546a) [0x150dfee8246a]
/lib64/libc.so.6(+0x39b0c) [0x150e15eadb0c]
/lib64/libc.so.6(on_exit+0) [0x150e15eadc40]
/u/user/miniconda3/envs/env/bin/python(+0x28db18) [0x560ae370eb18]
/u/user/miniconda3/envs/env/bin/python(+0x28db4b) [0x560ae370eb4b]
/u/user/miniconda3/envs/env/bin/python(+0x28db90) [0x560ae370eb90]
/u/user/miniconda3/envs/env/bin/python(_PyRun_SimpleFileObject+0x1e6) [0x560ae37123e6]
/u/user/miniconda3/envs/env/bin/python(_PyRun_AnyFileObject+0x44) [0x560ae37124c4]
/u/user/miniconda3/envs/env/bin/python(Py_RunMain+0x35d) [0x560ae37135bd]
/u/user/miniconda3/envs/env/bin/python(Py_BytesMain+0x39) [0x560ae37137d9]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x150e15e97493]
/u/user/miniconda3/envs/env/bin/python(+0x2125d4) [0x560ae36935d4]
Aborted (core dumped)
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.3