SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: map(...batch_size=None) for IterableDataset

Comment:
Hi! I agree it makes sense for `IterableDataset.map` to support the `batch_size=None` case. This should be super easy to fix.
Issue: map(...batch_size=None) for IterableDataset

Comment:
@mariosasko as this is something simple maybe I can include it as part of https://github.com/huggingface/datasets/pull/5311? Let me know :+1:
Issue: map(...batch_size=None) for IterableDataset

Comment:
Thanks again :)

> For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

This is interesting as well, if anyone wants to explore
Issue: Fix docstrings and types in documentation that appears on the website

Comment:
Ok, I believe we've cleaned up most of the old syntax we were using for the user-facing docs! There are still a couple of `:obj:`'s and `:class:` floating around in the docstrings we don't expose that I'll track down :)
Issue: `ImageFolder` performs poorly with large datasets

Comment:
Hi ! ImageFolder is made for small scale datasets indeed. For large scale image datasets you better group your images in TAR archives or Arrow/Parquet files. This is true not just for ImageFolder loading performance, but also because having millions of files is not ideal for your filesystem or when moving the data around.

Option 1. use TAR archives

I'd suggest you to take a look at how we load [Imagenet](https://huggingface.co/datasets/imagenet-1k/tree/main) for example. The dataset is sharded in multiple TAR archives and there is a [script](https://huggingface.co/datasets/imagenet-1k/blob/main/imagenet-1k.py) that iterates over the archives to load the images.

Option 2. use Arrow/Parquet

You can load your images as an Arrow Dataset with
```python
from datasets import Dataset, Image, load_from_disk, load_dataset

ds = Dataset.from_dict({"image": list(glob.glob("path/to/dir/**/*.jpg"))})

def add_metadata(example):
    ...

ds = ds.map(add_metadata, num_proc=16)  # num_proc for multiprocessing
ds = ds.cast_column("image", Image())

# save as Arrow locally
ds.save_to_disk("output_dir")
reloaded = load_from_disk("output_dir")

# OR save as Parquet on the HF Hub
ds.push_to_hub("username/dataset_name")
reloaded = load_dataset("username/dataset_name")
# reloaded = load_dataset("username/dataset_name", num_proc=16)  # to use multiprocessing
```

PS: maybe we can actually have something similar to ImageFolder but for image archives at one point ?
Issue: `ImageFolder` performs poorly with large datasets

Comment:
@lhoestq Thanks!

Perhaps it'd be worth adding a note on the documentation that `ImageFolder` is not intended for large datasets? This limitation is not intuitively obvious to someone who has not used it before, I think.
Issue: `ImageFolder` performs poorly with large datasets

Comment:
Thanks for the feedback @salieri! I opened #5329 to make it clear `ImageFolder` is not intended for large datasets. Please feel free to comment if you have any other feedback! 🙂
Issue: Adding new splits to a dataset script with existing old splits info in metadata's `dataset_info` fails

Comment:
EDIT:
I think in this case, the metadata files (either README or JSON) should not be read (i.e. `self.info.splits` should be None).

One idea: 
- I think ideally we should set this behavior when we pass `--save_info` to the CLI `test`
- However, currently, the builder is unaware of this: `save_info` arg is not passed to it
Issue: Adding new splits to a dataset script with existing old splits info in metadata's `dataset_info` fails

Comment:
> I think in this case

@albertvillanova  You mean in cases when the script was changed? 

I suggest that we:
* add a check on the slice (like 'split_name[n%]) kind of format here: https://github.com/huggingface/datasets/blob/main/src/datasets/splits.py#L523 to catch things like this. 
* Error here happens before splits verification, but in `_prepare_split`, and `_prepare_split` doesn't perform any verification and don't know about it. so we can pass this parameter and take splits from `split_generator`, not from `split.info` in case when `verify_infos` is False
* we can check if split **names** from split_generators and self.info.splits are the same **before** preparing splits (if `verify_info=True`) so that we don't spend time on generating unwanted data. 
* provide some user-friendly warnings about `ignore_verifications` parameter so that users know that if something is not matching they can ignore it

I started it here: https://github.com/huggingface/datasets/pull/5327/files

What do you think @albertvillanova ?
Issue: Adding new splits to a dataset script with existing old splits info in metadata's `dataset_info` fails

Comment:
I edited my previous comment:
- First I proposed setting `self.info.splits` to None when `ignore_verifications=True`
  - I thought it was the easiest implementation because `ignore_verifications` is passed to `DatasetBuilder.download_and_prepare`
  - However, afterwards, I realized this might not be a good idea for this use case:
    - A user wants to optimize the loading of the dataset, and passes `ignore_verifications=False` to avoid all the verifications
      - In this case, we want `self.info.splits` to be read from metadata file
- Then, I thought that it might be better to set `self.info.splits` to None when we pass `--save_info` to the CLI test: if we are going to save the info to the metadata file, it makes no sense to read the info from the metadata file
  - This implementation is not so easy because the Builder knows nothing about `--save_info`

I agree with you there are 2 things to be addressed here:
- One is what I have just commented: `self.info.splits` should be None in this case
- The other, a validation should be implemented when calling `make_file_instructions` and/or `SplitDict.__getitem__`, so that when passing "training" to it, we get a more descriptive error other than `TypeError: expected str, bytes or os.PathLike object, not NoneType`
Issue: Datasets: classification_report() got an unexpected keyword argument 'suffix'

Comment:
@JonathanAlis also note that the metrics are deprecated in our `datasets` library.

Please, use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate
Issue: Dataset joelito/mc4_legal does not work with multiple files

Comment:
Thanks for reporting @JoelNiklaus.

Please note that since we moved all dataset loading scripts to the Hub, the issues and pull requests relative to specific datasets are directly handled on the Hub, in their Community tab. I'm transferring this issue there: https://huggingface.co/datasets/joelito/mc4_legal/discussions

I am also having a look at the bug in your script.
Issue: timit_asr doesn't load the test split.

Comment:
The [timit_asr.py](https://huggingface.co/datasets/timit_asr/blob/main/timit_asr.py) script iterates over the WAV files per split directory using this:
```python
wav_paths = sorted(Path(data_dir).glob(f"**/{split}/**/*.wav"))
wav_paths = wav_paths if wav_paths else sorted(Path(data_dir).glob(f"**/{split.upper()}/**/*.WAV"))
```

Can you check that there is a directory named "test" somewhere in your timit data directory ?
Issue: Extractions failed when .zip file located on read-only path (e.g., SageMaker FastFile mode)

Comment:
Hi ! Thanks for reporting. Indeed the lock file should be placed in a directory with write permission (e.g. in the directory where the archive is extracted).
Issue: Extractions failed when .zip file located on read-only path (e.g., SageMaker FastFile mode)

Comment:
I opened https://github.com/huggingface/datasets/pull/5320 to fix this - it places the lock file in the cache directory instead of trying to put in next to the ZIP where it's read-only
Issue: Missing documentation build for versions 2.7.1 and 2.6.2 

Comment:
- Build docs for 2.6.2:
  - Commit: a6a5a1cf4cdf1e0be65168aed5a327f543001fe8
  - Build docs GH Action: https://github.com/huggingface/datasets/actions/runs/3539470622/jobs/5941404044
- Build docs for 2.7.1:
  - Commit: 5ef1ab1cc06c2b7a574bf2df454cd9fcb071ccb2
  - Build docs GH Action: https://github.com/huggingface/datasets/actions/runs/3539574442/jobs/5941636792
Issue: Lossy json serialization - deserialization of dataset info 

Comment:
Hi ! JSON is a lossy format indeed. If you want to keep the feature types or other metadata I'd encourage you to store them as well. For example you can use `dataset.info.write_to_directory` and `DatasetInfo.from_directory` to store the feature types, split info, description, license etc.
Issue: FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json

Comment:
I found a solution 

If you specifically install datasets==1.18 and then run

import datasets
wiki = datasets.load_dataset('wikipedia', '20200501.en')
then this should work (it worked for me.)
Issue: Features of IterableDataset set to None by remove column

Comment:
Thanks @lhoestq and @alvarobartt!

This would be extremely helpful to have working for the Whisper fine-tuning event - we're **only** training using streaming mode, so it'll be quite important to have this feature working to make training as easy as possible!

_c.f._ https://twitter.com/sanchitgandhi99/status/1592188332171493377
Issue: Features of IterableDataset set to None by remove column

Comment:
> Thanks @lhoestq and @alvarobartt!
> 
> 
> 
> This would be extremely helpful to have working for the Whisper fine-tuning event - we're **only** training using streaming mode, so it'll be quite important to have this feature working to make training as easy as possible!
> 
> 
> 
> _c.f._ https://twitter.com/sanchitgandhi99/status/1592188332171493377

I'm almost done with at least a temporary fix to `rename_column`, `rename_columns`, and `remove_columns`, just trying to figure out how to extend it to the `map` function itself!

I'll probably open the PR for review either tomorrow or Sunday hopefully! Glad I can help you and HuggingFace 🤗
Issue: Features of IterableDataset set to None by remove column

Comment:
@sanchit-gandhi PR is ready and open for review at #5287, but there's still one issue I may need @lhoestq's input :hugs:
Issue: Features of IterableDataset set to None by remove column

Comment:
Let us know @sanchit-gandhi if you need a new release of `datasets` soon with this fix included :)
Issue: Features of IterableDataset set to None by remove column

Comment:
Thanks for the fix guys! We can direct people to install `datasets` from main if that's easier!
Issue: Features of IterableDataset set to None by remove column

Comment:
Hi @asennoussi so this was already fixed and released as part of https://github.com/huggingface/datasets/releases/tag/2.8.0, so you should be able to install it as `pip install datasets==2.8.0` or just to use `pip install datasets --upgrade` to get the latest version, as of now, the https://github.com/huggingface/datasets/releases/tag/2.9.0 released last week! 🤗
Issue: Features of IterableDataset set to None by remove column

Comment:
Still facing the same issue though: 
```
from datasets import IterableDatasetDict, load_dataset

raw_datasets = vectorized_datasets = IterableDatasetDict()


raw_datasets["train"] = load_dataset("asennoussi/private", split="train", use_auth_token=True, streaming=True)
raw_datasets["test"] = load_dataset("asennoussi/private", split="test", use_auth_token=True, streaming=True)

print("Original features: ", raw_datasets['train'].features.keys())

...

def prepare_dataset(batch):

    # load and (possibly) resample audio datato 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array 
    batch["input_features"] = processor.feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]
    # compute input length of audio sample in seconds
    batch["input_length"] = len(audio["array"]) / audio["sampling_rate"]
    
    # optional pre-processing steps
    transcription = batch["sentence"]
    
    # encode target text to label ids
    batch["labels"] = processor.tokenizer(transcription).input_ids
    batch["labels_length"] = len(batch["labels"])
    return batch
...
vectorized_datasets = vectorized_datasets.remove_columns(['input_length', 'labels_length']+list(next(iter(raw_datasets.values())).features))
print("Processed features: ", vectorized_datasets['train'].features)
print("First sample:", next(iter(vectorized_datasets['train'])))

```

Output: 
```
Original features:  dict_keys(['path', 'audio', 'sentence'])
Processed features:  None
```
Issue: Features of IterableDataset set to None by remove column

Comment:
Hmm weird, could you try to print

```python
print("Processed features: ", vectorized_datasets['train'].features)
```

again after iterating over the `vectorized_datasets`? In the code above, should be last line :)
Issue: Features of IterableDataset set to None by remove column

Comment:
Didn't seem to fix it: 
```
Original features:  dict_keys(['path', 'audio', 'sentence'])
Processed features:  None
Processed features:  None
```
Issue: Features of IterableDataset set to None by remove column

Comment:
Actually the culprit looks to be this one: 
`vectorized_datasets = raw_datasets.map(prepare_dataset).with_format("torch")`
When I remove this line: `vectorized_datasets = vectorized_datasets.remove_columns(['input_length', 'labels_length']+list(next(iter(raw_datasets.values())).features))`

I still get 
```
Processed features:  None
```
Issue: Features of IterableDataset set to None by remove column

Comment:
> The culprit is definitely `.map` Just validated it. Any idea please?

Yes, indeed `.map` losses the features, because AFAIK pre-fetching the data to infer the features is expensive and not ideal, that's part of this issue https://github.com/huggingface/datasets/issues/3888

Anyway, now you can pass the `features` as a param to `.map` as follows:

```python
from datasets import Features
vectorized_datasets = raw_datasets.map(
    prepare_dataset,
    features=Features(
        {"path": raw_datasets["train"].info.features["path"], "audio": raw_datasets["train"].info.features["audio"], "sentence": raw_datasets["train"].info.features["sentence"]}
    ),
).with_format("torch")
```

Also, to let you know, when calling `.remove_columns` over an `IterableDataset`, the `features` are not lost, as well as `.rename_column` and `rename_columns` :)

More information about the latter at https://github.com/huggingface/datasets/pull/5287
Issue: Features of IterableDataset set to None by remove column

Comment:
@asennoussi alternatively you can just call `._resolve_features()` from your `IterableDataset` and it will pre-fetch the data to resolve the features, but note that feature-inference is not as accurate as if you manually specify which features and feature-types the `IterableDataset` has, as mentioned in the comment above, the alternative is to provide `features` param to `.map` :hugs:
Issue: Support cloud storage in load_dataset

Comment:
Or for example an archive on GitHub releases!  Before I added support for JXL (locally only, PR still pending) I was considering hosting my files on GitHub instead...
Issue: Support cloud storage in load_dataset

Comment:
+1 to this. I would like to use 'audiofolder' with a data_dir that's on S3, for example. I don't want to upload my dataset to the Hub, but I would find all the fingerprinting/caching features useful.
Issue: Support cloud storage in load_dataset

Comment:
Adding to the conversation, Dask also uses `fsspec` for this feature.

[Dask: How to connect to remote data](https://docs.dask.org/en/stable/how-to/connect-to-remote-data.html)

Happy to help on this feature :D
Issue: Support cloud storage in load_dataset

Comment:
+1 to this feature request since I think it also tackles my use-case. I am collaborating with a team, working with a loading script which takes some time to generate the dataset artifacts. It would be very handy to use this as a cloud cache to avoid duplicating the effort. 

Currently we could use `builder.download_and_prepare(path_to_cloud_storage, storage_options, ...)`  to cache the artifacts to cloud storage, but then `builder.as_dataset()` yields `NotImplementedError: Loading a dataset cached in SomeCloudFileSystem is not supported`
Issue: Support cloud storage in load_dataset

Comment:
Makes sense ! If you want to load locally a dataset that you download_and_prepared on a cloud storage, you would use `load_dataset(path_to_cloud_storage)` indeed. It would download the data from the cloud storage, cache them locally, and return a `Dataset`.
Issue: Support cloud storage in load_dataset

Comment:
It seems currently the `cached_path` function handles all URLs by `get_from_cache` that only supports `ftp` and `http(s)` here:
https://github.com/huggingface/datasets/blob/b5672a956d5de864e6f5550e493527d962d6ae55/src/datasets/utils/file_utils.py#L181

I guess one can add another condition that handles `s3://` or `gs://` URLs via `fsspec` here.
Issue: Support cloud storage in load_dataset

Comment:
I could use this functionality, so I put together a PR using @kyamagu's suggestion to use `fsspec` in `datasets.utils.file_utils`

https://github.com/huggingface/datasets/pull/5580
Issue: Support cloud storage in load_dataset

Comment:
Thanks @dwyatte for adding support for fsspec urls

Let me just reopen this since the original issue is not resolved
Issue: Support cloud storage in load_dataset

Comment:
I'm not yet understanding how to use https://github.com/huggingface/datasets/pull/5580 in order to use `load_dataset(data_files="s3://...")`. Any help/example would be much appreciated :) thanks!
Issue: Support cloud storage in load_dataset

Comment:
It's still not officially supported x) But you can try to update `request_etag` in `file_utils.py` to use `fsspec_head` instead of `http_head`. It is responsible of getting the ETags of the remote files for caching. This change may do the trick for S3 urls
Issue: Support cloud storage in load_dataset

Comment:
Thank you for your guys help on this and merging in #5580. I manually pulled the changes to my local datasets package (datasets.utils.file_utils.py) since it only seemed to be this file that was changed in the PR and I'm getting the error: 
InvalidSchema: No connection adapters were found for 's3://bucket/folder/'. I'm calling load_dataset using the S3 URI. When I use the S3 URL I get HTTPError: 403 Client Error. 
Am I not supposed to use the S3 URI? How do I pull in the changes from this merge? I'm running datasets 2.10.1.
Issue: Support cloud storage in load_dataset

Comment:
The current implementation depends on gcsfs/s3fs being able to authenticate through some other means e.g., environmental variables. For AWS, it looks like you can set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`

Note that while testing this just now, I did note a discrepancy between gcsfs and  s3fs that we might want to address where gcsfs passes the timeout from `storage_options` [here](https://github.com/huggingface/datasets/blob/3e6269979fc80ae8939294d26298897f0db5b84d/src/datasets/utils/file_utils.py#L333) down into the `aiohttp.ClientSession.request`, but s3fs does not handle this (tries to pass to the `aiobotocore.session.AioSession` constructor raising `TypeError: __init__() got an unexpected keyword argument 'requests_timeout'`).

It seems like some work trying to unify kwargs across different fsspec implementations, so if the plan is to pass down `storage_options`, I wonder if we should just let users control the timeout (and other kwargs) using that and if not specified, use the default?
Issue: Support cloud storage in load_dataset

Comment:
> Note that while testing this just now, I did note a discrepancy between gcsfs and s3fs that we might want to address where gcsfs passes the timeout from storage_options [here](https://github.com/huggingface/datasets/blob/3e6269979fc80ae8939294d26298897f0db5b84d/src/datasets/utils/file_utils.py#L333) down into the aiohttp.ClientSession.request, but s3fs does not handle this (tries to pass to the aiobotocore.session.AioSession constructor raising TypeError: __init__() got an unexpected keyword argument 'requests_timeout').

> It seems like some work trying to unify kwargs across different fsspec implementations, so if the plan is to pass down storage_options, I wonder if we should just let users control the timeout (and other kwargs) and if not specified, use the default?

@lhoestq here's a small PR for this: https://github.com/huggingface/datasets/pull/5673
Issue: Support cloud storage in load_dataset

Comment:
@lhoestq 
hello, i still have problem with loading json from S3:


storage_options = {
    "key": xxxx,
    "secret": xxx,
    "endpoint_url": xxxx
}
path = 's3://xxx/xxxxxxx.json'
dataset = load_dataset("json", data_files=path, storage_options=storage_options)

and it throws an error:
TypeError: AioSession.__init__() got an unexpected keyword argument 'hf'
and I use the lastest 2.14.4_dev0 version
Issue: Support cloud storage in load_dataset

Comment:
Hi @lhoestq, thanks for getting back to me :) you have been busy over the summer I see... I was on `2.12.0`. I have updated to `2.14.4`. 

Now `d = load_dataset("new_dataset.py", storage_options=storage_options, split="train", streaming=True)` works for Azure blob storage (with a local data loader script) when I explicitly list all blobs  (I am struggling to make `fs.ls(<path>)` work in the script to make the list available to the download manager).

 Any chance that it could work out-of-the-box by supplying just the image folder, not the full list of image filenames? It seems that `dl_manager.download(_URL)` always wants one or more (possibly archived) files. In my situation, where I don't want to archive or download, it would be great to just supply the folder (seems reasonably doable with fsspec).

Let me know if there is anything I can do to help.

Thanks,
Issue: Support cloud storage in load_dataset

Comment:
> Any chance that it could work out-of-the-box by supplying just the image folder, not the full list of image filenames? It seems that dl_manager.download(_URL) always wants one or more (possibly archived) files. In my situation, where I don't want to archive or download, it would be great to just supply the folder (seems reasonably doable with fsspec).

@mayorblock  This is not supported right now, you have to use archives or implement a way to get the list by yourself

> TypeError: AioSession.init() got an unexpected keyword argument 'hf'

@hjq133 Can you update `fsspec` and try again ?

```python
pip install -U fsspec
```
Issue: Import error

Comment:
Hi,

3.8.13

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:37:02 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Hi ! Can you

import platform
print(platform.python_version())

to see that it returns ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323691385>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F5YGG32W6WABYC25NJTWJTD75ANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
Issue: Import error

Comment:
Then it should work as expected if you use the same python when using `datasets`

Please make sure you're running your code in the right environment