SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
The URL is wrong indeed, the right one is the one with "resolve" (the one you get when clicking on "download")- otherwise you try to download an html page ;)
```
https://huggingface.co/datasets/bigcode/the-stack-dedup-pjj/resolve/v1.1.a1/data/java/data_0000.parquet
```
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
Ah thanks! So I tried it with the first parquet file and it works, is there a way to know which parquet file was causing the issue since there are a lot of shards?
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
I think you have to try them all :/

Alternatively you can add a try/catch in `parquet.py` in `datasets` to raise the name of the file that fails at doing `parquet_file = pq.ParquetFile(f)` when you run your initial code
```python
load_dataset("bigcode/the-stack-dedup-pjj", data_dir="data/java", split="train", revision="v1.1.a1", use_auth_token=True)
```
but it will still iterate on all the files until it fails
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
Can you check the JSON file associated to `/home/loubna_huggingface_co/.cache/huggingface/datasets/downloads/93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7` ? In the JSON file we can know from where it was downloaded

You can find it at `/home/loubna_huggingface_co/.cache/huggingface/datasets/downloads/93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7.json`
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
I'm able to load it properly using
```python
ds = load_dataset("parquet", data_files=a_parquet_file_url, use_auth_token=token)
```

My guess is that your download was corrupted. Please delete `93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7` and `93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7.json` locally and try again
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
That worked, thanks! But I thought if something went wrong with a download `datasets` creates new cache for all the files, that's not the case? (at some point I even changed dataset versions so it was still using that cache?)
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
Cool !

> But I thought if something went wrong with a download datasets creates new cache for all the files

We don't perform integrity verifications if we don't know in advance the hash of the file to download.

> at some point I even changed dataset versions so it was still using that cache?

`datasets` caches the files by URL and ETag. If the content of a file changes, then the ETag changes and so it redownloads the file
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
I experience the same error in v 2.12.0. But found out it was due to one column from polars was a categorical dtype (related to the error from #5706. Temporarily resolved it by casting the column to str instead.
Issue: AttributeError: 'Value' object has no attribute 'names'

Comment:
Hi ! It looks like your "isDif" column is a Sequence of Value("string"), not a Sequence of ClassLabel.

You can convert your Value("string") feature type to a ClassLabel feature type this way:
```python
from datasets import ClassLabel, Sequence

# provide the label_names yourself
label_names = [...]
# OR get them from the dataset
label_names = sorted(set(label for labels in raw_datasets["train"]["isDif"] for label in labels))

# Cast to ClassLabel
raw_datasets = raw_datasets.cast_column("isDif", Sequence(ClassLabel(names=label_names)))
```
Issue: Add PubTables-1M

Comment:
cc @albertvillanova the author would like to add this dataset to the hub: https://github.com/microsoft/table-transformer/issues/68#issuecomment-1319114621. Could you help him out?
Issue: consumer-finance-complaints dataset not loading

Comment:
I have opened an issue in that dataset Community tab on the Hub: https://huggingface.co/datasets/consumer-finance-complaints/discussions/1

Please note that in the meantime, you can load the dataset by passing `ignore_verifications=True`:
```python
>>> ds = load_dataset("consumer-finance-complaints", ignore_verifications=True)
>>> ds
DatasetDict({
    train: Dataset({
        features: ['Date Received', 'Product', 'Sub Product', 'Issue', 'Sub Issue', 'Complaint Text', 'Company Public Response', 'Company', 'State', 'Zip Code', 'Tags', 'Consumer Consent Provided', 'Submitted via', 'Date Sent To Company', 'Company Response To Consumer', 'Timely Response', 'Consumer Disputed', 'Complaint ID'],
        num_rows: 3079747
    })
})
```
Issue: datasets 2.7 introduces sharding error

Comment:
I notice a comment in the code says:
`Having lists of different sizes makes sharding ambigious, raise an error in this case until we decide how to define sharding without ambiguity for users` 
    
 ... which suggests this update was pushed knowing that it might break some things. But, it didn't seem to have a useful error message of an argument that could be passed to avoid the error.
Issue: datasets 2.7 introduces sharding error

Comment:
Sorry for the inconvenience, I opened a PR in your repo to fix this: https://huggingface.co/datasets/sil-ai/bloom-speech/discussions/2

Basically we've always considered lists in `gen_kwargs` to be a shard list that we can split and pass into different workers to generate the dataset (e.g. if you pass `num_proc=` in `load_dataset()` to generate the dataset in parallel), but it was documented only recently
Issue: Restore order of split names in dataset_info for canonical datasets

Comment:
The bulk edit is running...

See for example: 
- A single config: https://huggingface.co/datasets/acronym_identification/discussions/2
- Multiple configs: https://huggingface.co/datasets/babi_qa/discussions/1
Issue: Restore order of split names in dataset_info for canonical datasets

Comment:
TODO: Add "dataset_info" YAML metadata to:
- [x] "chr_en" has no metadata JSON file, nor "dataset_info" YAML tag in its card
  - Fixing PR: https://huggingface.co/datasets/chr_en/discussions/1 
- [x] "conll2000" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "crime_and_punish" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "dart" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "iwslt2017" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [ ] "mc4" has no metadata JSON file, nor "dataset_info" YAML tag in its card
- [ ] "the_pile" has no metadata JSON file, nor "dataset_info" YAML tag in its card
- [ ] "timit_asr" has no metadata JSON file, nor "dataset_info" YAML tag in its card
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
Cool ! Let us know if you have questions or if we can help :)

I guess we'll also have to create the NYU CS Department on the Hub ?
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> I guess we'll also have to create the NYU CS Department on the Hub ?

Yes, you're right! Let me add it to my profile first, and then we can transfer. Meanwhile, if it's recommended to loop the dataset author in here, let me know. 

Also, the NYU Depth dataset seems big. Any example scripts for creating image datasets that I could refer?
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
You can check the imagenet-1k one.

PS: If the licenses allows it, it'b be nice to host the dataset as sharded TAR archives (like imagenet-1k) instead of the ZIP format they use:
- it will make streaming much faster
- ZIP compression is not well suited for images
- it will allow parallel processing of the dataset (you can pass a subset of shards to each worker)

> if it's recommended to loop the dataset author in here, let me know.

It's recommended indeed, you can send them an email once you have the dataset ready and invite them to the org on the Hub
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> You can check the imagenet-1k one.

Where can I find the script? Are you referring to https://huggingface.co/docs/datasets/image_process ? Or is there anything more specific?
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
Update: started working on it here: https://huggingface.co/datasets/sayakpaul/nyu_depth_v2. 

I am facing an issue and I have detailed it here: https://huggingface.co/datasets/sayakpaul/nyu_depth_v2/discussions/1

Edit: The issue is gone. 

However, since the dataset is distributed as a single TAR archive (following the [URL used in TensorFlow Datasets](https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/datasets/nyu_depth_v2/nyu_depth_v2_dataset_builder.py)) the loading is taking longer. How would suggest to shard the single TAR archive? 

@lhoestq
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
A Colab Notebook demonstrating the dataset loading part: 

https://colab.research.google.com/gist/sayakpaul/aa0958c8d4ad8518d52a78f28044d871/scratchpad.ipynb

@osanseviero @lhoestq 

I will work on a notebook to work with the dataset including data visualization.
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
@osanseviero @lhoestq things seem to work fine with the current version of the dataset [here](https://huggingface.co/datasets/sayakpaul/nyu_depth_v2). Here's a notebook I developed to help with visualization: https://colab.research.google.com/drive/1K3ZU8XUPRDOYD38MQS9nreQXJYitlKSW?usp=sharing. 

@lhoestq I need your help with the following:

> However, since the dataset is distributed as a single TAR archive (following the [URL used in TensorFlow Datasets](https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/datasets/nyu_depth_v2/nyu_depth_v2_dataset_builder.py)) the loading is taking longer. How would suggest to shard the single TAR archive?

@osanseviero @lhoestq question for you:

Where should we host the dataset? I think hosting it under hf.co/datasets (that is HF is the org) is fine as we have ImageNet-1k hosted similarly. We could then reach out to Diana Wofk (author of [Fast Depth](https://github.com/dwofk/fast-depth) and the owner of the repo on which TFDS NYU Depth V2 is based) for a review. WDYT?
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> However, since the dataset is distributed as a single TAR archive (following the [URL used in TensorFlow Datasets](https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/datasets/nyu_depth_v2/nyu_depth_v2_dataset_builder.py)) the loading is taking longer. How would suggest to shard the single TAR archive?

First you can separate the train data and the validation data.

Then since the dataset is quite big, you can even shard the train split and the validation split in multiple TAR archives. Something around 16 archives for train and 4 for validation would be fine for example.

Also no need to gzip the TAR archives, the images are already compressed in png or jpeg.
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> Then since the dataset is quite big, you can even shard the train split and the validation split in multiple TAR archives. Something around 16 archives for train and 4 for validation would be fine for example.

Yes, I got you. But this process seems to be manual and should be tailored for the given dataset. Do you have any script that you used to create the ImageNet-1k shards? 

> Also no need to gzip the TAR archives, the images are already compressed in png or jpeg.

I was not going to do that. Not sure what brought it up.
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> Yes, I got you. But this process seems to be manual and should be tailored for the given dataset. Do you have any script that you used to create the ImageNet-1k shards?

I don't, but I agree it'd be nice to have a script for that !

> I was not going to do that. Not sure what brought it up.

The original dataset is gzipped for some reason
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> Where should we host the dataset? I think hosting it under hf.co/datasets (that is HF is the org) is fine as we have ImageNet-1k hosted similarly.

Maybe you can create an org for NYU Courant (this is the institute of the lab of the main author of the dataset if I'm not mistaken), and invite the authors to join.

We don't add datasets without namespace anymore
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
Updates: https://huggingface.co/datasets/sayakpaul/nyu_depth_v2/discussions/5

The entire process (preparing multiple archives, preparing data loading script, etc.) was fun and engaging, thanks to the documentation. I believe we could work on a small blog post that would work as a reference for the future contributors following this path. What say?  

Cc: @lhoestq @osanseviero
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> I believe we could work on a small blog post that would work as a reference for the future contributors following this path. What say?

@polinaeterna already mentioned it would be nice to present this process for audio (it's exactly the same), I believe it can be useful to many people
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
Cool. Let's work on that after the NYU Depth Dataset is fully in on Hub (under the appropriate org). 🤗
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
@lhoestq need to discuss something while I am adding the dataset card to https://huggingface.co/datasets/sayakpaul/nyu_depth_v2/. 

As per [Papers With Code](https://paperswithcode.com/dataset/nyuv2), NYU Depth v2 is used for many different tasks:

* Monocular depth estimation
* Depth estimation 
* Semantic segmentation
* Plane instance segmentation 
* ...

So, while writing the supported task part of the dataset card, should we focus on all these? IMO, we could focus on just depth estimation and semantic segmentation for now since we have supported models for these two. WDYT?

Also, I am getting: 


```
remote: Your push was accepted, but with warnings:
remote: - Warning: The task_ids "depth-estimation" is not in the official list: acceptability-classification, entity-linking-classification, fact-checking, intent-classification, multi-class-classification, multi-label-classification, multi-input-text-classification, natural-language-inference, semantic-similarity-classification, sentiment-classification, topic-classification, semantic-similarity-scoring, sentiment-scoring, sentiment-analysis, hate-speech-detection, text-scoring, named-entity-recognition, part-of-speech, parsing, lemmatization, word-sense-disambiguation, coreference-resolution, extractive-qa, open-domain-qa, closed-domain-qa, news-articles-summarization, news-articles-headline-generation, dialogue-generation, dialogue-modeling, language-modeling, text-simplification, explanation-generation, abstractive-qa, open-domain-abstractive-qa, closed-domain-qa, open-book-qa, closed-book-qa, slot-filling, masked-language-modeling, keyword-spotting, speaker-identification, audio-intent-classification, audio-emotion-recognition, audio-language-identification, multi-label-image-classification, multi-class-image-classification, face-detection, vehicle-detection, instance-segmentation, semantic-segmentation, panoptic-segmentation, image-captioning, grasping, task-planning, tabular-multi-class-classification, tabular-multi-label-classification, tabular-single-column-regression, rdf-to-text, multiple-choice-qa, multiple-choice-coreference-resolution, document-retrieval, utterance-retrieval, entity-linking-retrieval, fact-checking-retrieval, univariate-time-series-forecasting, multivariate-time-series-forecasting, visual-question-answering, document-question-answering
remote: ----------------------------------------------------------
remote: Please find the documentation at:
remote: https://huggingface.co/docs/hub/model-cards#model-card-metadata
```

What should be the plan of action for this?

Cc: @osanseviero
Issue: Add a Depth Estimation dataset - DIODE / NYUDepth / KITTI

Comment:
> What should be the plan of action for this?

When you merged https://github.com/huggingface/hub-docs/pull/488, there is a JS Interfaces GitHub Actions workflow that runs https://github.com/huggingface/hub-docs/actions/workflows/js-interfaces-tests.yml. It has a step called [export-task scripts](https://github.com/huggingface/hub-docs/actions/runs/3622479064/jobs/6107238948) which exports an interface you can use in `dataset`. If you look at the logs, it prints out a map. This map can replace https://github.com/huggingface/datasets/blob/main/src/datasets/utils/resources/tasks.json (tasks.json was generated with this script), which should add depth estimation
Issue: Docs are not generated after latest release

Comment:
After a discussion with @mishig25:
- He said that this action should be triggered if we call our release branch according to the regex `v*-release`, as transformers does
- I said that our procedure is different: our release branch is *temporary* and it is deleted just after the release PR is merged to main
  - Indeed the release tag is not yet created when we make the release PR (not event when this is merged to main), but when we make the Release itself.

I was thinking that maybe we could change the triggering event: use `release` instead of `push`.

What do you think, @huggingface/datasets?
Issue: Docs are not generated after latest release

Comment:
He says not; but the branch has no tag yet; does the doc building require the tag? Or just the version number in `__init__.py` or setup.py?
Issue: Docs are not generated after latest release

Comment:
It uses `module.__version__` (i.e. the one defined in `__init__.py`) - no need to have a tag

https://github.com/huggingface/doc-builder/blob/81575cf081964c30ea5fd39450f4820db963f18e/src/doc_builder/commands/build.py#L69
Issue: Docs are not generated after latest release

Comment:
Thanks, @lhoestq.

@mishig25 has manually forced the generation of the docs, that are live for 2.7.0 version: https://huggingface.co/docs/datasets/v2.7.0/en/index
Issue: Docs are not generated after latest release

Comment:
just to confirm, is there anything I need to do from my side ? Or is everything good here ?
Issue: Unable to rename columns in streaming dataset

Comment:
Hi @peregilk this bug is directly related to https://github.com/huggingface/datasets/issues/3888, and still not fixed... But I'll try to have a look!
Issue: Unable to rename columns in streaming dataset

Comment:
Thanks @alvarobartt. It is great if you are able to fix it, but when reading the explanation it seems like it is possible to work around it.

We also tried keeping the 'info.features' and then adding a modified version back after the remove/rename. Unforutunately that leads to a dataset that is not possible to iterate over.
Issue: Unable to rename columns in streaming dataset

Comment:
So if you iterate over the `IterableDataset` as `next(iter(ds))` and then run `rename_columns` when checking that data it will work, but in the end, it's just renaming the column one example/batch at a time, not renaming the column name for all the entries in the dataset, which is the ideal.
Issue: Unable to rename columns in streaming dataset

Comment:
@alvarobartt Thanks. My use case was that I wanted to do multiple things, ie removing all unnecessary columns, renaming some valid columns, and then using cast (in my case checking if the audio is not 16K and casting it). It is just convenient to look into the info.features between each of these operations. Alternatively, I will just plan ahead...;) To me it seems like all the operations are working.

Thanks for the advice. It was very useful.
Issue: Unable to rename columns in streaming dataset

Comment:
If we know the features before renaming, then we know the features after renaming, so we can pass the new features to the returned dataset in `rename_column` indeed ! If anyone is interested in contributing, feel free to open a PR and I'd be happy to help / give some pointers :)
Issue: Allow dataset streaming from private a private source when loading a dataset with a dataset loading script

Comment:
Hi ! What kind of private source ? We're exploring adding support for cloud storage and URIs like s3://, gs:// etc. with authentication in the download manager
Issue: Allow dataset streaming from private a private source when loading a dataset with a dataset loading script

Comment:
Hello! It's a google cloud storage, so gs://, but I'm using it with https.
Being able to provide a file system like [here](https://huggingface.co/docs/datasets/main/filesystems#load-serialized-datasets) would be even more practical indeed.
I've found a quite complicated workaround which consists of monkey patching all of the functions in streaming_download_manager.py to use my own _get_authentication_headers_for_url_ . 

A support for this use case would be greatly appreciated!

For reference my _get_authentication_headers_for_url_ looks like this:
```
import os
from typing import Optional, Union

from datasets import config
from huggingface_hub import HfFolder
from gcsfs.credentials import GoogleCredentials

DEFAULT_PROJECT = os.environ.get("GCSFS_DEFAULT_PROJECT", "")
access = "full_control"
gcs_token = os.environ.get("GCS_TOKEN")


def get_authentication_headers_for_url(url: str, use_auth_token: Optional[Union[str, bool]] = None) -> dict:
    """Handle the HF authentication"""
    headers = {}
    if url.startswith(config.HF_ENDPOINT):
        if use_auth_token is False:
            token = None
        elif isinstance(use_auth_token, str):
            token = use_auth_token
        else:
            token = HfFolder.get_token()
    elif url.startswith("https://storage.googleapis.com"):
        credentials = GoogleCredentials(DEFAULT_PROJECT, access, gcs_token)
        credentials.maybe_refresh()
        token = credentials.credentials.token
    else:
        token = None
    if token:
        headers["authorization"] = f"Bearer {token}"
    return headers
```
Issue: Allow dataset streaming from private a private source when loading a dataset with a dataset loading script

Comment:
I would be a big fan of this feature! @Hubert-Bonisseur if this doesn't become a supported feature, would you mind sharing your code? Thanks!
Issue: Allow dataset streaming from private a private source when loading a dataset with a dataset loading script

Comment:
> I would be a big fan of this feature! @Hubert-Bonisseur if this doesn't become a supported feature, would you mind sharing your code? Thanks!

I published it here:
https://github.com/Hubert-Bonisseur/private-dataset-hub

I modified the names of a lot of functions for privacy and I don't have time to test it again so you may get import errors, but you have the code. The custom_load_dataset is the function you are interested in I think.

It relies a lot on patching, if you find a better way to do this, I'd be interested.
Issue: Allow dataset streaming from private a private source when loading a dataset with a dataset loading script

Comment:
Given the amount of patching it does, this is likely to break at one point. I'd encourage you to wait for a proper support in `datasets` directly if you can wait.