SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Errror when saving to disk a dataset of images

Comment:
Looks like as long as the number of shards makes a batch lower than 1000 images it works. In my training set I have 40K images. If I use `num_shards=40` (batch of 1000 images) I get the error, but if I update it to `num_shards=50` (batch of 800 images) it works.

I will be happy to share my dataset privately if it can help to better debug.
Issue: Errror when saving to disk a dataset of images

Comment:
Hi! I didn't manage to reproduce this behavior, so sharing the dataset with us would help a lot. 

> My dataset is around 50K images, is this error might be due to a bad image?

This shouldn't be the case as we save raw data to disk without decoding it.
Issue: Errror when saving to disk a dataset of images

Comment:
OK, thanks! The dataset is currently hosted on a gcs bucket. How would you like to proceed for sharing the link?
Issue: Errror when saving to disk a dataset of images

Comment:
You could follow [this](https://cloud.google.com/storage/docs/collaboration#browser) procedure or upload the dataset to Google Drive (50K images is not that much unless high-res) and send me an email with the link.
Issue: Errror when saving to disk a dataset of images

Comment:
Thanks @jplu! I managed to reproduce the `TypeError` - it stems from [this](https://github.com/huggingface/datasets/blob/e3f4f124a1b118a5bfff5bae76b25a68aedbebbc/src/datasets/features/image.py#L258-L264) line, which can return a `ChunkedArray` (its mask is also chunked then, which Arrow does not allow) when the embedded data is too big to fit in a standard `Array`.

I'm working on a fix.
Issue: Handle empty audio

Comment:
Hi! Can you share one of the problematic audio files with us?

I tried to reproduce the error with the following code: 
```python
import soundfile as sf
import numpy as np
from datasets import Audio

sf.write("empty.wav",  np.array([]), 16000)
Audio(sampling_rate=24000).decode_example({"path": "empty.wav", "bytes": None})
```
But without success.

Also, what version of `librosa` is installed in your env? (You can get this info with `python -c "import librosa; print(librosa.__version__)`)
Issue: Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List

Comment:
Hi! 

You can use [`.set_format("np")`](https://huggingface.co/docs/datasets/process#format) to get NumPy arrays (or Pytorch tensors with `.set_format("torch")`) in `__getitem__`.

Also, have you been able to reproduce the linked PyTorch issue with a HF dataset?
Issue: ArrowNotImplementedError when loading dataset from the hub

Comment:
Hi Julien ! This sounds related to https://github.com/huggingface/datasets/issues/5695 - TL;DR: you need to have shards smaller than 2GB to avoid this issue

The number of rows per shard is computed using an estimated size of the full dataset, which can sometimes lead to shards bigger than `max_shard_size`. The estimation is currently done using the first samples of the dataset (which can surely be improved). We should probably open an issue to fix this once and for all.

Anyway for your specific dataset I'd suggest you to pass `num_shards` instead of `max_shard_size` for now,  and make sure to have enough shards to end up with shards smaller than 2GB
Issue: ArrowNotImplementedError when loading dataset from the hub

Comment:
Hi Quentin! Thanks a lot! Using `num_shards` instead of `max_shard_size` works as expected.

Indeed the way you describe how the size is computed cannot really work with the dataset I'm building as all the image doesn't have the same resolution and then size. Opening an issue on this might be a good idea.
Issue: load_dataset in v2.11.0 raises "ValueError: seek of closed file" in np.load()

Comment:
> Closing since this is a duplicate of #5711

Sorry @mariosasko , my internet went down went submitting the issue, and somehow it ended up creating a duplicate
Issue: load_dataset in v2.11.0 raises "ValueError: seek of closed file" in np.load()

Comment:
It seems like https://github.com/huggingface/datasets/pull/5626 has introduced this error. 

cc @albertvillanova 

I think replacing:
https://github.com/huggingface/datasets/blob/0803a006db1c395ac715662cc6079651f77c11ea/src/datasets/download/streaming_download_manager.py#L777-L778
with:
```python
return np.load(xopen(filepath_or_buffer, "rb", use_auth_token=use_auth_token), *args, **kwargs)
```
should fix the issue.

(Maybe this is also worth doing a patch release afterward)
Issue: OSError: Memory mapping file failed: Cannot allocate memory

Comment:
Hi! This error means that PyArrow's internal [`mmap`](https://man7.org/linux/man-pages/man2/mmap.2.html) call failed to allocate memory, which can be tricky to debug. Since this error is more related to PyArrow than us, I think it's best to report this issue in their [repo](https://github.com/apache/arrow) (they are more experienced on this matter). Also, googling "mmap cannot allocate memory" returns some approaches to solving this problem.
Issue: Manually dataset info made not taken into account

Comment:
hi @jplu ! Did I understand you correctly that you create the dataset, push it to the Hub with `.push_to_hub` and you see a `dataset_infos.json` file there, then you edit this file, load the dataset with `load_dataset` and you don't see any changes in `.info` attribute of a dataset object? 

This is actually weird that when you push your dataset to the Hub, a `dataset_infos.json` file is created, because this file is deprecated and it should create `README.md` with the `dataset_info` field instead. Some keys are also deprecated, like "supervised_keys" and "task_templates".

Can you please provide a toy reproducible example of how you create and push the dataset? And also why do you want to change this file, especially the number of bytes and examples?
Issue: Manually dataset info made not taken into account

Comment:
Hi @polinaeterna Yes I have created the dataset with `Dataset.from_dict` applied some updates afterward and when I pushed to the hub I had a `dataset_infos.json` file and there was a `README.md` file as well.

I didn't know that the JSON file was deprecated. So I have built my dataset with `ImageBuilder` instead and now it works like a charm without having to touch anything.

I haven't succeed to reproduce the creation of the JSON file with a toy example, hence, I certainly did some mistakes when I have manipulated my dataset manually at first. My bad.
Issue: Dataset sizes are in MiB instead of MB in dataset cards

Comment:
looks great! 

Do you encode the fact that you've already converted a dataset? (to not convert it twice) or do you base yourself on the info contained in `dataset_info`
Issue: Dataset sizes are in MiB instead of MB in dataset cards

Comment:
I am only looping trough the dataset cards, assuming that all of them were created with MiB.

I agree we should only run the bulk edit once for all canonical datasets: I'm using a for-loop over canonical datasets.
Issue: Dataset sizes are in MiB instead of MB in dataset cards

Comment:
I have just included as well the conversion from MB to GB if necessary. See: 
- https://huggingface.co/datasets/bookcorpus/discussions/2/files
- https://huggingface.co/datasets/asnq/discussions/2/files
Issue: Dataset sizes are in MiB instead of MB in dataset cards

Comment:
First, I tested some batches to check the changes made. Then I incorporated the MB to GB conversion. Now I'm running the rest.
Issue: Dataset sizes are in MiB instead of MB in dataset cards

Comment:
Thanks a lot!

The sizes now match as expected!

<img width="1446" alt="Capture d’écran 2023-04-05 à 16 10 15" src="https://user-images.githubusercontent.com/1676121/230107044-ac2a76ea-a4fe-4e81-a925-f464b85f5edd.png">
Issue: Dataset sizes are in MiB instead of MB in dataset cards

Comment:
I made another bulk edit of ancient canonical datasets that were moved to community organization. I have parsed 11 datasets and opened a PR on 3 of them:
- [ ] "allenai/scicite": https://huggingface.co/datasets/allenai/scicite/discussions/3
- [ ] "allenai/scifact": https://huggingface.co/datasets/allenai/scifact/discussions/2
- [x] "dair-ai/emotion": https://huggingface.co/datasets/dair-ai/emotion/discussions/6
Issue: Support categorical data types for Parquet

Comment:
Hi ! We could definitely a type that holds the categories and uses a DictionaryType storage. There's a ClassLabel type that is similar with a 'names' parameter (similar to a id2label in deep learning frameworks) that uses an integer array as storage.

It can be added in `features.py`. Here are some pointers:
- the conversion from HF type to PyArrow type is done in `get_nested_type`
- the conversion from Pyarrow type to HF type is done in `generate_from_arrow_type`
- `encode_nested_example` and `decode_nested_example` are used to do user's value (what users see) <-> storage value (what is in the pyarrow.array) if there's any conversion to do
Issue: Getting next item from IterableDataset took forever.

Comment:
Hi! It can take some time to iterate over Parquet files as big as yours, convert the samples to Python, and find the first one that matches a filter predicate before yielding it...
Issue: Getting next item from IterableDataset took forever.

Comment:
Thanks @mariosasko, I figured it was the filter operation. I'm closing this issue because it is not a bug, it is the expected beheaviour.
Issue: Is it possible or how to define a `datasets.Sequence` that could potentially be either a dict, a str, or None?

Comment:
Hi ! `datasets` uses Apache Arrow as backend to store the data, and it requires each column to have a fixed type. Therefore a column can't have a mix of dicts/lists/strings.

Though it's possible to have one (nullable) field for each type:
```python
features = Features({
    "text_alone": Value("string"),
    "text_with_idxes": {
        "text": Value("string"),
        "idxes": Value("int64")
    }
})
```

but you'd have to reformat your data fiels or define a [dataset loading script](https://huggingface.co/docs/datasets/dataset_script) to apply the appropriate parsing.

Alternatively we could explore supporting the Arrow [Union](https://arrow.apache.org/docs/python/generated/pyarrow.UnionType.html) type which could solve this issue, but I don't know if it's well supported in python and with the rest of the ecosystem like Parquet
Issue: Is it possible or how to define a `datasets.Sequence` that could potentially be either a dict, a str, or None?

Comment:
@lhoestq  Thank you! I further wonder if it's possible to use list subscripts as keys of a feature? Like
```python
features = Features({
    0: Value("string"),
    1: {
        "text": Value("string"),
        "idxes": [Value("int64")]
    },
    2: Value("string"),
    # ...
})
```
Issue: Is it possible or how to define a `datasets.Sequence` that could potentially be either a dict, a str, or None?

Comment:
Column names need to be strings, so you could use "1", "2", etc. or give appropriate column names
Issue: Loading big dataset raises pyarrow.lib.ArrowNotImplementedError

Comment:
Hi ! It looks like an issue with PyArrow: https://issues.apache.org/jira/browse/ARROW-5030

It appears it can happen when you have parquet files with row groups larger than 2GB.
I can see that your parquet files are around 10GB. It is usually advised to keep a value around the default value 500MB to avoid these issues.

Note that currently the row group size is simply defined by the number of rows `datasets.config.DEFAULT_MAX_BATCH_SIZE`, so reducing this value could let you have parquet files bigger than 2GB and with row groups lower than 2GB.

Would it be possible for you to re-upload the dataset with the default shard size 500MB ?
Issue: Loading big dataset raises pyarrow.lib.ArrowNotImplementedError

Comment:
Hey, thanks for the reply! I've since switched to working with the locally-saved dataset (which works).
Maybe it makes sense to show a warning for uploads with large shard sizes? Since the functionality completely breaks (due to the PyArrow bug).
Issue: Loading big dataset raises pyarrow.lib.ArrowNotImplementedError

Comment:
Leaving this other error here for the record, which occurs when I load the +700GB dataset from the hub with shard sizes of 500MB:

```
 Traceback (most recent call last):                                                        
  File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/builder.py", line 1860, in _prepare_split_single
    for _, table in generator:
  File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 69, in _generate_tables
    for batch_idx, record_batch in enumerate(
  File "pyarrow/_parquet.pyx", line 1323, in iter_batches
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Corrupt snappy compressed data.
```
I will probably switch back to the local big dataset or shrink it.
Issue: Dataset configuration

Comment:
Originally we also though about adding it to the YAML part of the README.md:

```yaml
builder_config:
  data_dir: data
  data_files:
  - split: train
    pattern: "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
```

Having it in the README.md could make it easier to modify it in the UI on HF, and for validation on commit
Issue: Dataset configuration

Comment:
From internal discussions we agreed to go with the YAML approach, since it's the one that seems more appropriate to be modified by a human on the Hub or locally (while JSON e.g. for models are usually created programmatically).
Issue: pyarrow.lib.ArrowInvalid: Unable to merge: Field <field> has incompatible types

Comment:
Hi! The link pointing to the code that generated the dataset is broken. Can you please fix it to make debugging easier?
Issue: pyarrow.lib.ArrowInvalid: Unable to merge: Field <field> has incompatible types

Comment:
> Hi! The link pointing to the code that generated the dataset is broken. Can you please fix it to make debugging easier?

Sorry about that, it's fixed now.
Issue: raise AttributeError(f"No {package_name} attribute {name}") AttributeError: No huggingface_hub attribute hf_api

Comment:
Hi @wccccp, thanks for reporting. 
That's weird since `huggingface_hub` _has_ a module called `hf_api` and you are using a recent version of it. 

Which version of `datasets` are you using? And is it a bug that you experienced only recently? (cc @lhoestq can it be somehow related to the recent release of `datasets`?)

~@wccccp what I can suggest you is to uninstall and reinstall completely huggingface_hub and datasets? My first guess is that there is a discrepancy somewhere in your setup 😕~
Issue: raise AttributeError(f"No {package_name} attribute {name}") AttributeError: No huggingface_hub attribute hf_api

Comment:
@wccccp Actually I have also been able to reproduce the error so it's not an issue with your setup.

@huggingface/datasets I found this issue quite weird. Is this a module that is not used very often?
The problematic line is [this one](https://github.com/huggingface/datasets/blame/c33e8ce68b5000988bf6b2e4bca27ffaa469acea/src/datasets/data_files.py#L476) where `huggingface_hub.hf_api.DatasetInfo` is used. `huggingface_hub` is imported [here](https://github.com/huggingface/datasets/blame/c33e8ce68b5000988bf6b2e4bca27ffaa469acea/src/datasets/data_files.py#L6) as `import huggingface_hub`. However since modules are lazy-loaded in `hfh` you need to explicitly import them (i.e. `import huggingface_hub.hf_api`).

What's weird is that nothing has changed for months. Datasets code seems that it didn't change for 2 years when I git-blame this part. And lazy-loading was introduced 1 year ago in `huggingface_hub`. Could it be that `data_files.py` is a file almost never used?
Issue: raise AttributeError(f"No {package_name} attribute {name}") AttributeError: No huggingface_hub attribute hf_api

Comment:
For context, I tried to run `import huggingface_hub; huggingface_hub.hf_api.DatasetInfo` in the terminal with different versions of `hfh` and I need to go back to `huggingface_hub==0.7.0` to make it work (latest is 0.13.3).
Issue: raise AttributeError(f"No {package_name} attribute {name}") AttributeError: No huggingface_hub attribute hf_api

Comment:
Before the error happens at line 120 in `data_files.py`, `datasets.filesystems.hffilesystem` is imported at the top of `data_files.py` and this file does `from huggingface_hub.hf_api import DatasetInfo` - so `huggingface_hub.hf_api` is imported. Not sure how the error could happen, what version of `datasets` are you using @wccccp ?
Issue: Wikipedia download_and_prepare for GCS

Comment:
Hi @adrianfagerland, thanks for reporting.

Please note that "wikipedia" is a special dataset, with an Apache Beam builder: https://beam.apache.org/
You can find more info about Beam datasets in our docs: https://huggingface.co/docs/datasets/beam

It was implemented to be run in parallel processing, using one of the distributed back-ends supported by Apache Beam: https://beam.apache.org/get-started/beam-overview/#apache-beam-pipeline-runners

That is, you are trying to process the source wikipedia data on your machine (not distributed) when passing `beam_runner="DirectRunner"`.

As documented in the wikipedia dataset page (https://huggingface.co/datasets/wikipedia):

    Some subsets of Wikipedia have already been processed by HuggingFace, and you can load them just with:
    
        from datasets import load_dataset
    
        load_dataset("wikipedia", "20220301.en")

    The list of pre-processed subsets is:
    - "20220301.de"
    - "20220301.en"
    - "20220301.fr"
    - "20220301.frr"
    - "20220301.it"
    - "20220301.simple"

To download the available processed data (in Arrow format):
```python
builder = datasets.load_dataset_builder("wikipedia", "20220301.en")
builder.download_and_prepare(your_path)
```
Issue: Wikipedia download_and_prepare for GCS

Comment:
When running this using :
```
import datasets
from apache_beam.options.pipeline_options import PipelineOptions
from gcsfs import GCSFileSystem

storage_options = {"project":"tdt4310", "token":"cloud"}
fs = GCSFileSystem(**storage_options)

output_dir = "gcs://quiz_transformer/"
beam_options = PipelineOptions(
        region="europe-west4",
        project="tdt4310",
        temp_location=output_dir+"tmp/")


builder = datasets.load_dataset_builder("wikipedia", "20220301.en", beam_runner="dataflow", beam_options=beam_options)
builder.download_and_prepare(
        output_dir, storage_options=storage_options, file_format="parquet")
```
I now get this error:
```
raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
Downloading data files:   0%|                                                            | 0/1 [00:00<?, ?it/s]
```

I get the same error for this:
```
import datasets
from gcsfs import GCSFileSystem

storage_options = {"project":"tdt4310", "token":"cloud"}
fs = GCSFileSystem(**storage_options)

output_dir = "gcs://quiz_transformer/"
builder = datasets.load_dataset_builder("wikipedia", "20220301.en")
builder.download_and_prepare(
        output_dir, storage_options=storage_options, file_format="parquet")
```
Issue: Document to compress data files before uploading

Comment:
Great idea!

Should we also take this opportunity to include some audio/image file formats? Currently, it still reads very text heavy. Something like:

> We support many text, audio, and image data extensions such as `.zip`, `.rar`, `.mp3`, and `.jpg` among many others. For data extensions like `.csv`, `.json`, `.jsonl`, and `txt`, we recommend compressing them before uploading to the Hub. These file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of supported file extensions.
Issue: Document to compress data files before uploading

Comment:
Hi @stevhliu, thanks for your suggestion.

I agree it is a good opportunity to mention that audio/image file formats are also supported.

Nit:
I would not mention .zip, .rar after "text, audio, and image data extensions". Those are "compression" extensions and not "text, audio, and image data extensions".

What about something similar to:
> We support many text, audio, and image data extensions such as `.csv`, `.mp3`, and `.jpg` among many others. For text data extensions like `.csv`, `.json`, `.jsonl`, and `.txt`, we recommend compressing them before uploading to the Hub (to `.zip` or `.gz` file extension for example). 
>
> Note that text file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of tracked file extensions by default.

Note that for compressions I have mentioned:
- gz, to compress individual files
- zip, to compress and archive multiple files; zip is preferred rather than tar because it supports streaming out of the box
Issue: Broken Image render on the hub website

Comment:
Hi! 

You can fix the viewer by adding the `dataset_info` YAML field deleted in https://huggingface.co/datasets/Francesco/cell-towers/commit/b95b59ddd91ebe9c12920f0efe0ed415cd0d4298 back to the metadata section of the card. 

To avoid this issue in the feature, you can use `huggingface_hub`'s [RepoCard](https://huggingface.co/docs/huggingface_hub/package_reference/cards) API  to update the dataset card instead of `upload_file`:
```python
from huggingface_hub import DatasetCard
# Load card
card = DatasetCard.load("<namespace>/<repo_id>")
# Modify card content
card.content = ...
# Push card to the Hub
card.push_to_hub("<namespace>/<repo_id>")
```

However, the best solution would be to use the features info stored in the header of the Parquet shards generated with `push_to_hub` on the viewer side to avoid unexpected issues such as this one. This shouldn't be too hard to address.
Issue: Broken Image render on the hub website

Comment:
Thanks for reporting @FrancescoSaverioZuppichini.

For future issues with your specific dataset, you can use its "Community" tab to start a conversation: https://huggingface.co/datasets/Francesco/cell-towers/discussions/new
Issue: Broken Image render on the hub website

Comment:
Thanks @albertvillanova , @mariosasko I was not aware of this requirement from the doc (must have skipped :sweat_smile: )

Confirmed, adding back `dataset_info` fixed the issu
Issue: Add information about patterns search order to the doc about structuring repo

Comment:
Good idea, I think I've seen this a couple of times before too on the forums. I can work on this :)
Issue: Allow load_dataset to take a working dir for intermediate data

Comment:
Hi ! AFAIK a dataset must be present on a local disk to be able to efficiently memory map the datasets Arrow files. What makes you think that it is possible to load from a cloud storage and have good performance ?

Anyway it's already possible to download_and_prepare a dataset as Arrow files in a cloud storage with:
```python
builder = load_dataset_builder(..., cache_dir="/temp/dir")
builder.download_and_prepare("/cloud_dir")
```

but then 
```python
ds = builder.as_dataset()
```
would fail if "/cloud_dir" is not a local directory.
Issue: Allow load_dataset to take a working dir for intermediate data

Comment:
In my use case, I am trying to mount the S3 bucket as local system with S3FS-FUSE / [goofys](https://github.com/kahing/goofys). I want to use S3 to save the download data and save checkpoint for training for persistent. Setting the s3 location as cache directory is not fast enough. That is why I want to set a work directory for temp data for memory map and only save the final result to s3 cache.