noamaanMulla-03/datasets-issues huggingface dataset:
Support remote file systems for `Audio`

### Feature request

Hi there!

It would be super cool if `Audio()`, and potentially other features, could read files from a remote file system.

### Motivation

Large amounts of data is often stored in buckets. `load_from_disk` is able to retrieve data from cloud storage but to my knowledge actually copies the datasets across first, so if you're working off a system with smaller disk specs (like a VM), you can run out of space very quickly.

### Your contribution

Something like this (for Google Cloud Platform in this instance):



```python
from datasets import Dataset, Audio
import gcsfs

fs = gcsfs.GCSFileSystem()

list_of_audio_fp = {'audio': ['1', '2', '3']}
ds = Dataset.from_dict(list_of_audio_fp)

ds = ds.cast_column("audio", Audio(sampling_rate=16000, fs=fs))
```

Under the hood:
```python

import librosa
from io import BytesIO

def load_audio(fp, sampling_rate=None, fs=None):
    if fs is not None:
        with fs.open(fp, 'rb') as f:
            arr, sr = librosa.load(BytesIO(f), sr=sampling_rate)
    else:
        # Perform existing io operations
```

Written from memory so some things could be wrong.

Labels: name: enhancement

Just seen https://github.com/huggingface/datasets/issues/5281
Origin/fix missing features error

This fixes the problem of when the dataset_load function reads a function with "features" provided but some read batches don't have columns that later show up. For instance, the provided "features" requires columns A,B,C but only columns B,C show. This fixes this by adding the column A with nulls. 


_The documentation is not available anymore as the PR was closed or merged._

please review :) @lhoestq @ola13 thankoo

Thanks :) I just updated the test to make sure it works even when there's a column missing, and did a minor change to json.py to add the missing columns for the other kinds of JSON files as well (I moved the code to`self._cast_table`)

Thanks Unso! If @lhoestq is happy then I'm also happy :D

When I noticed the ping, this PR had already been merged...

Luckily, PyArrow's `read_json` behaves the same when `explicit_schema` is given via `ParseOptions`, so I'm okay with this change (our JSON loader doesn't use `read_json` for decoding JSON in some scenarios, so this manual approach is the right one).
Image column is not decoded in map when using with with_transform

## Describe the bug
Image column is not _decoded_ in **map** when using with `with_transform`

## Steps to reproduce the bug
```python
from datasets import Image, Dataset

def add_C(batch):
    batch["C"] = batch["A"]
    return batch

ds = Dataset.from_dict({"A": ["image.png"]}).cast_column("A",  Image())
ds = ds.with_transform(lambda x: x)  # <= This line causes the problem
ds = ds.map(add_C, batched=True)
print(ds[0])
```

## Expected results
```
{'C': <PIL.PngImagePlugin.PngImageFile>,  ...}
```

## Actual results
```
{'C': {'bytes': None, 'path': 'image.png'}, ...}
```

If we remove the `with_transform` line, we get the expected result.

## Environment info
- `datasets` version: 2.0.0
- Platform: Mac OSX
- Python version: 3.8.12
- PyArrow version: 7.0.0


Labels: name: bug

Hi! This behavior stems from this line: https://github.com/huggingface/datasets/blob/799b817d97590ddc97cbd38d07469403e030de8c/src/datasets/arrow_dataset.py#L1919
Basically, the `Image`/`Audio` columns are decoded only if the `format_type` attribute is `None` (`set_format`/`with_format` and `set_transform`/`with_transform` assign a non-`None` value to it) and the  `input_columns` param is not specified (see https://github.com/huggingface/datasets/issues/3756). We will remove these limitations soon.
Images get decoded when using `map()` with `input_columns` argument on a dataset

## Describe the bug

The `datasets.features.Image` feature class decodes image data by default. Expectedly, when indexing a dataset or using the `map()` method, images are returned as PIL Image instances.

However, when calling `map()` and setting a specific data column with the `input_columns` argument, the image data is passed as raw byte representation to the mapping function.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torchvision import transforms
from PIL.Image import Image

dataset = load_dataset('mnist', split='train')

def transform_all_columns(example):
    # example['image'] is encoded as PIL Image
    assert isinstance(example['image'], Image)
    return example

def transform_image_column(image):
    # image is decoded here and represented as raw bytes
    assert isinstance(image, Image)
    return image

# single-sample dataset for debugging purposes
dev = dataset.select([0])

dev.map(transform_all_columns)
dev.map(transform_image_column, input_columns='image')
```

## Expected results

Image data should be passed in decoded form, i.e. as PIL Image objects to the mapping function unless the `decode` attribute on the image feature is set to `False`.

## Actual results

The mapping function receives images as raw byte data.

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.11.0-49-generic-x86_64-with-glibc2.32
- Python version: 3.8.0b4
- PyArrow version: 7.0.0


Labels: name: bug

Hi! If I'm not mistaken, this behavior is intentional, but I agree it could be more intuitive.

@albertvillanova Do you remember why you decided not to decode columns in the `Audio` feature PR when `input_columns` is not `None`? IMO we should decode those columns, and we don't even have to use lazy structures here because the user explicitly requires them in the map transform. 

cc @lhoestq for visibility

I think I excluded to decorate the function when `input_columns` were passed as a quick fix for some non-passing tests: 
- https://github.com/huggingface/datasets/pull/2324/commits/9d7c3e8fa53e23ec636859b4407eeec904b1b3f9

That PR was quite complex and I decided to focus on the main feature requests, leaving refinements for subsequent PRs.

Note that when `input_columns` are passed, the signature of the function is effectively changed, while the decorated function expects an item (whether an example or a batch) as first arg (which is not the case when passing `input_columns`.

I agree we should consider supporting the case when `input_columns` are passed.
Support for decoding Image/Audio types in map when format type is not default one

Add support for decoding the `Image`/`Audio` types in `map` for the formats (Numpy, TF, Jax, PyTorch) other than the default one (Python).

Additional improvements:
* make `Dataset`'s "iter" API  cleaner by removing `_iter` and replacing `_iter_batches` with `iter(batch_size)` (also implemented for `IterableDataset`)
* iterate over arrow tables in `map` to avoid `_getitem` calls, which are much slower than `__iter__`/`iter(batch_size)`, when the `format_type` is not Python
* fix `_iter_batches` (now named `iter`) when `drop_last_batch=True` and `pyarrow<=8.0.0` is installed
* lazily extract and decode arrow data in the default format 

TODO:
* [x] update the `iter` benchmark in the docs (the `BeamBuilder` cannot load the preprocessed datasets from our bucket, so wait for this to be fixed (cc @lhoestq))

Fix https://github.com/huggingface/datasets/issues/3992, fix https://github.com/huggingface/datasets/issues/3756


_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5252). All of your documentation changes will be reflected on that endpoint.

Yes, if the image column is the first in the batch keys, it will decode the images because it reads the actual values. We could avoid this by checking the batch type, and if it's `LazyDict`, `num_examples` is equal to `len(batch.pa_table)`, which doesn't lead to decoding.

Good idea. This can be done in a subsequent PR btw, since it's out of scope of the original goal of this PR

Just fixed a small bug where it would show the pyarrow 10 warning about None -> empty lists conversions even with an Array2D with no nulls

Fixed another bug when your map function returns a mix of LazyDict or regular dict and added some tests
Raise error for `.tar` archives in the same way as for `.tar.gz` and `.tgz` in `_get_extraction_protocol`

Currently `download_and_extract` doesn't throw an error when it is used with files with `.tar` extension in streaming mode because `_get_extraction_protocol` doesn't do it (like it does for  `tar.gz` and `tgz`). `_get_extraction_protocol` returns formatted url as if we support tar protocol but we don't.
That means that in dataset scripts `.tar` files would be attempted to load and fail during examples generation (after `download_and_extract` execution). So this PR raises error for `tar` files too.



_The documentation is not available anymore as the PR was closed or merged._
Add optional progress bar for .save_to_disk(..) and .load_from_disk(..) when working with remote filesystems

**Is your feature request related to a problem? Please describe.**
When working with large datasets stored on remote filesystems(such as s3), the process of uploading a dataset could take really long time. For instance: I was uploading a re-processed version of wmt17 en-ru to my s3 bucket and it took like 35 minutes(and that's given that I have a fiber optic connection). The only output during that process was a progress bar for flattening indices and then ~35 minutes of complete silence.

**Describe the solution you'd like**
I want to be able to enable a progress bar when calling .save_to_disk(..) and .load_from_disk(..), it would track either amount of bytes sent/received or amount of records written/loaded, and will give some ETA. Basically just tqdm. 

**Describe alternatives you've considered**
- Save dataset to tmp folder at the disk and then upload it using custom wrapper over botocore, which will work with progress bar, like [this](https://alexwlchan.net/2021/04/s3-progress-bars/).

Labels: name: enhancement

Hi! I like this idea. For consistency with `load_dataset`, we can use `fsspec`'s `TqdmCallback` in  `.load_from_disk` to monitor the number of bytes downloaded, and in `.save_to_disk`, we can track the number of saved shards for consistency with `push_to_hub` (after we implement https://github.com/huggingface/datasets/issues/4196).
Embed image and audio files in `save_to_disk`

Following https://github.com/huggingface/datasets/pull/4184, currently a dataset saved using `save_to_disk` doesn't actually contain the bytes of the image or audio files. Instead it stores the path to your local files. 

Adding `embed_external_files` and set it to True by default to save_to_disk would be kind of a breaking change since some users will get bigger Arrow files when updating the lib, but the advantages are nice:

- the resulting dataset is self contained, in case you want to delete your cache for example or share it with someone else
- users also upload these Arrow files to cloud storage via the fs parameter, and in this case they would expect to upload a self-contained dataset
- consistency with push_to_hub

This can be implemented at the same time as sharding for `save_to_disk` for efficiency, and reuse the helpers from `push_to_hub` to embed the external files.

cc @mariosasko
Sharded save_to_disk + multiprocessing

Added `num_shards=` and `num_proc=` to `save_to_disk()`

EDIT: also added `max_shard_size=` to `save_to_disk()`, and also `num_shards=` to `push_to_hub`

I also:
- deprecated the fs parameter in favor of storage_options (for consistency with the rest of the lib) in save_to_disk and load_from_disk
- always embed the image/audio data in arrow when doing `save_to_disk`
- added a tqdm bar in `save_to_disk`
- Use the MockFileSystem in tests for `save_to_disk` and `load_from_disk`
- removed the unused integration tests with S3, since we can now test with `mockfs` instead of `s3fs`

TODO:
- [x] implem save_to_disk for dataset dict
- [x] save_to_disk for dataset dict tests
- [x] deprecate fs in dataset dict load_from_disk as well
- [x] update docs

Close #5263 
Close https://github.com/huggingface/datasets/issues/4196
Close https://github.com/huggingface/datasets/issues/4351


_The documentation is not available anymore as the PR was closed or merged._

Added both num_shards and max_shard_size in push_to_hub/save_to_disk. Will take care of updating the tests later

It's ready for a final review @mariosasko and @albertvillanova, let me know what you think :)

Took your comments into account, and also changed `iflatmap_unordered` to take an iterable of kwargs to make the code more redable :)
Run 'GPT-J' failure due to download dataset fail (' ConnectionError: Couldn't reach http://eaidata.bmk.sh/data/enron_emails.jsonl.zst ' ) 

### Describe the bug

Run model "GPT-J" with dataset "the_pile" fail.
The fail out is as below:
![image](https://user-images.githubusercontent.com/52023469/207750127-118d9896-35f4-4ee9-90d4-d0ab9aae9c74.png)

Looks like which is due to "http://eaidata.bmk.sh/data/enron_emails.jsonl.zst" unreachable .

### Steps to reproduce the bug

Steps to reproduce this issue:

git clone https://github.com/huggingface/transformers
cd transformers
python examples/pytorch/language-modeling/run_clm.py --model_name_or_path EleutherAI/gpt-j-6B --dataset_name the_pile --dataset_config_name enron_emails --do_eval --output_dir /tmp/output --overwrite_output_dir

### Expected behavior

This issue looks like due to "http://eaidata.bmk.sh/data/enron_emails.jsonl.zst " couldn't be reached.
Is there another way to download the dataset "the_pile" ?
Is there another way to cache the dataset "the_pile" but not let the hg to download it when runtime ?

### Environment info

huggingface_hub version: 0.11.1
Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Python version: 3.9.12
Running in iPython ?: No
Running in notebook ?: No
Running in Google Colab ?: No
Token path ?: /home/taosy/.huggingface/token
Has saved token ?: False
Configured git credential helpers:
FastAI: N/A
Tensorflow: N/A
Torch: N/A
Jinja2: N/A
Graphviz: N/A
Pydot: N/A


Thanks for reporting, @shaoyuta.

We have checked and yes, apparently there is an issue with the server hosting the data of the "enron_emails" subset of "the_pile" dataset: http://eaidata.bmk.sh/data/enron_emails.jsonl.zst
It seems to be down: The connection has timed out.

Please note that at the Hugging Face Hub, we are not hosting their data for this dataset, but only a script that downloads the data from their servers. We are updating the data URL to one in another server.

In the meantime, please note that you can train your model in the entire "the_pile" dataset, by passing the "all" config (instead of the "enron_emails" one).

We have transferred this issue to the corresponding dataset Community tab: https://huggingface.co/datasets/the_pile/discussions/2

Please, follow the updates there.
Trying to load FEVER dataset results in NonMatchingChecksumError

## Describe the bug
Trying to load the `fever` dataset fails with `datasets.utils.info_utils.NonMatchingChecksumError`.

I tried with `download_mode="force_redownload"` but that did not fix the error. I also tried with `ignore_verification=True` but then that raised a `json.decoder.JSONDecodeError`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset('fever', 'v1.0') # Fails with NonMatchingChecksumError
dataset = load_dataset('fever', 'v1.0', download_mode="force_redownload") # Fails with NonMatchingChecksumError
dataset = load_dataset('fever', 'v1.0', ignore_verification=True)` # Fails with JSONDecodeError
```

## Expected results
I expect this call to return with no error raised.

## Actual results
With `ignore_verification=False`:
```
*** datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://s3-eu-west-1.amazonaws.com/fever.public/train.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev_public.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_test.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_test.jsonl']
```
With `ignore_verification=True`:
```
*** json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.3.dev0
- Platform: Linux-4.15.0-50-generic-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Labels: name: bug

Thanks for reporting @santhnm2. We are fixing it.

Data owners updated their URLs recently. We have to align with them, otherwise you do not download anything (that is why ignore_verifications does not work).

Hello! Is there any update on this? I am having the same issue 6 months later.
Import error

https://github.com/huggingface/datasets/blob/cd3d8e637cfab62d352a3f4e5e60e96597b5f0e9/src/datasets/__init__.py#L28

Hy, 
I have error at the above line. I have python version 3.8.13, the message says I need python>=3.7, which is True, but I think the if statement not working properly (or the message wrong)


Hi ! Can you 
```python
import platform
print(platform.python_version())
```
to see that it returns ?

Hi,

3.8.13

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:37:02 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Hi ! Can you

import platform
print(platform.python_version())

to see that it returns ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323691385>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F5YGG32W6WABYC25NJTWJTD75ANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


Then it should work as expected if you use the same python when using `datasets`

Please make sure you're running your code in the right environment

It's the right environment. But in if statement I have
"3.8.13" < 3.7
And in the error message is Python>=3.7 which is true in my case (3.8.13 is greater then 3.7), so I don't understand my python should be below the 3.7 which case the if statement is right, but the message is wrong, or above 3.7 which case if statement is wrong, error message is right.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:41:43 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Then it should work as expected if you use the same python when using datasets

Please make sure you're running your code in the right environment

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323697094>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F54JURTAJJWWDO2QGI3WJTERPANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


If you're having an error then you're not running your code in the right environment.
Fix remove columns from lazy dict

This was introduced in https://github.com/huggingface/datasets/pull/5252 and causing the transformers CI to break: https://app.circleci.com/pipelines/github/huggingface/transformers/53886/workflows/522faf2e-a053-454c-94f8-a617fde33393/jobs/648597

Basically this code should return a dataset with only one column:

```python
from datasets import *

ds = Dataset.from_dict({"a": range(5)})

def f(x):
    x["b"] = x["a"]
    return x

ds = ds.map(f, remove_columns=["a"])

assert ds.column_names == ["b"]
```


_The documentation is not available anymore as the PR was closed or merged._
__init__() got an unexpected keyword argument 'input_size'

### Describe the bug

I try to define a custom configuration with a input_size attribute following the instructions by "Specifying several dataset configurations" in https://huggingface.co/docs/datasets/v1.2.1/add_dataset.html

But when I load the dataset, I got an error "__init__() got an unexpected keyword argument 'input_size'"

### Steps to reproduce the bug

Following is the code to define the dataset:
class CsvConfig(datasets.BuilderConfig):
    """BuilderConfig for CSV."""
    input_size: int = 2048

class MRF(datasets.ArrowBasedBuilder):
    """Archival MRF data"""
    BUILDER_CONFIG_CLASS = CsvConfig
    VERSION = datasets.Version("1.0.0")
    BUILDER_CONFIGS = [
        CsvConfig(name="default", version=VERSION, description="MRF data", input_size=2048),
    ]

...

    def _generate_examples(self):
        input_size = self.config.input_size
        if input_size > 1000:
           numin = 10000
        else:
           numin = 15000


Below is the code to load the dataset:
reader = load_dataset("default", input_size=1024)

### Expected behavior

I hope to pass the "input_size" parameter to MRF datasets, and change "input_size" to any value when loading the datasets. 

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-4.18.0-305.3.1.el8.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.12
- PyArrow version: 9.0.0
- Pandas version: 1.5.0


Hi @J-shel, thanks for reporting.

I think the issue comes from your call to `load_dataset`. As first argument, you should pass:
- either the name of your dataset ("mrf") if this is already published on the Hub
- or the path to the loading script of your dataset ("path/to/your/local/mrf.py").

Hi, following your suggestion, I changed my call to load_dataset. Below is the latest:
reader = load_dataset('data/mrf.py',"default", input_size=1024, split=split, streaming=True, keep_in_memory=None)
However, I still got the same error.
I have one question that is if I only define input_size=2048 in BUILDER_CONFIGS, may I specify input_size=1024 when loading the dataset?  Cause I found that I could only specify name="default" since I only define  name="default" in BUILDER_CONFIGS.