SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Import error

Comment:
It's the right environment. But in if statement I have
"3.8.13" < 3.7
And in the error message is Python>=3.7 which is true in my case (3.8.13 is greater then 3.7), so I don't understand my python should be below the 3.7 which case the if statement is right, but the message is wrong, or above 3.7 which case if statement is wrong, error message is right.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:41:43 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Then it should work as expected if you use the same python when using datasets

Please make sure you're running your code in the right environment

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323697094>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F54JURTAJJWWDO2QGI3WJTERPANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
Issue: load_dataset does not read jsonl metadata file properly

Comment:
Can you try to remove "drop_labels=false" ? It may force the loader to infer the labels instead of reading the metadata
Issue: load_dataset does not read jsonl metadata file properly

Comment:
Okay, now it seems to actually load the metadata and create the train_split, but it still says only returns "image" and "label", which is always 0 since all images are from same folder
Issue: load_dataset does not read jsonl metadata file properly

Comment:
> Can you try updating `datasets` ? Metadata support was added in `datasets` 2.4

Update: This was the issue.
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
Sounds like one of the file is not a valid one, can you make sure you uploaded valid mp3 files ?
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
Well I just sharded the original commonVoice dataset and pushed a small chunk of it in a private rep

What did go wrong?

Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef>
________________________________
Von: Quentin Lhoest ***@***.***>
Gesendet: Tuesday, November 22, 2022 3:03:40 PM
An: huggingface/datasets ***@***.***>
Cc: capsabogdan ***@***.***>; Author ***@***.***>
Betreff: Re: [huggingface/datasets] Bug in downloading common_voice data and snall chunk of it to one's own hub (Issue #5276)


Sounds like one of the file is not a valid one, can you make sure you uploaded valid mp3 files ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5276#issuecomment-1323727434>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALSIFOAPAL2V4TBJTSPMAULWJTHDZANCNFSM6AAAAAASHQJ63U>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
It should be all good then !
Could you share a link to your repository for me to investigate what went wrong ?
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
https://huggingface.co/datasets/DTU54DL/common-voice-test16k

Am Di., 22. Nov. 2022 um 16:43 Uhr schrieb Quentin Lhoest <
***@***.***>:

> It should be all good then !
> Could you share a link to your repository for me to investigate what went
> wrong ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1323876682>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOEUJRZWXAM7DYA5VJDWJTS3NANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
I see ! This is a bug with MP3 files.

When we store audio data in parquet, we store the bytes and the file name. From the file name extension we know if it's a WAV, an MP3 or else. But here it looks like the paths are all None.

It looks like it comes from here:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/audio.py#L212

Cc @polinaeterna maybe we should simply put the file name instead of None values ?
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
@lhoestq I remember we wanted to avoid storing redundant data but maybe it's not that crucial indeed to store one more string value. 
Or we can store paths only for mp3s, considering that for other formats we don't have such a problem with reading from bytes without format specified.
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
thanks for the help!

can I do anything on my side? we are doing a DL project and we need the
data really quick.

thanks
bogdan

> Message ID: ***@***.***>
>
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
I opened a pull requests here: https://github.com/huggingface/datasets/pull/5285, we'll do a new release soon with this fix.

Otherwise if you're really in a hurry you can install `datasets` from this PR
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
any idea on what's going wrong here?

thanks

Am So., 27. Nov. 2022 um 13:53 Uhr schrieb Bogdan Capsa <
***@***.***>:

> [image: image.png]
>
>> Message ID: ***@***.***>
>>
>
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
I have attached this screenshot above . can u pls help? So can not pip from pull request

![image](https://user-images.githubusercontent.com/48530104/204354027-6173e6d1-e3d4-4085-a363-e924cfe1a7f4.png)
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
The pull request has been merged on `main`.
You can install `datasets` from `main` using
```
pip install git+https://github.com/huggingface/datasets.git
```
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
I've tried to load this dataset DTU54DL/common-voice-test16k, but am
getting the same error.

So the bug fix will fix only if I upload a new dataset, or also loading
previously uploaded datasets?

thanks

Am Mo., 28. Nov. 2022 um 19:51 Uhr schrieb Quentin Lhoest <
***@***.***>:

> The pull request has been merged on main.
> You can install datasets from main using
>
> pip install git+https://github.com/huggingface/datasets.git
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1329587334>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOCNYYIGHM2EX3ZIO6DWKT5MXANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
> So the bug fix will fix only if I upload a new dataset, or also loading
previously uploaded datasets?

You have to reupload the dataset, sorry for the inconvenience
Issue: Bug in downloading common_voice data and snall chunk of it to one's own hub

Comment:
thank you so much for the help! works like a charm!

Am Di., 29. Nov. 2022 um 12:15 Uhr schrieb Quentin Lhoest <
***@***.***>:

> So the bug fix will fix only if I upload a new dataset, or also loading
> previously uploaded datasets?
>
> You have to reupload the dataset, sorry for the inconvenience
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1330468393>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOBKEFZO57BAKY4IGW3WKXQUZANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Issue: YAML integer keys are not preserved Hub server-side

Comment:
@huggingface/datasets if you agree, I can make the bulk edit on the Hub to fix integer keys into strings.
Issue: YAML integer keys are not preserved Hub server-side

Comment:
FYI there are still 2k+ weekly users on `datasets` 2.6.1 which doesn't support the string label format for class labels. And among those, some are using datasets with class labels like imdb (60 users), conllpp (40), msra_ner (40), peoples_daily_enr (40), weibo_ner (30), conll2003 (20), etc. And renaming to string would break these users code.
Issue: YAML integer keys are not preserved Hub server-side

Comment:
but isn't `datasets 2.6.1` downloading files from the Hub with the corresponding tag? I thought we had something like this before
Issue: YAML integer keys are not preserved Hub server-side

Comment:
We're using `main` as models do. Some datasets need to be updated from time to time, e.g. when a link to download the data is dead.

But yea a year ago we had those tags, we just ended up not using them
Issue: YAML integer keys are not preserved Hub server-side

Comment:
I opened https://github.com/huggingface/datasets/issues/5406 to communicate on this. Let me know what you think, and if it sounds good to you I can pin this issue
Issue: YAML integer keys are not preserved Hub server-side

Comment:
So, is it OK to make the bulk edit on the Hub now or should we wait longer? If the latter, how long?
Issue: YAML integer keys are not preserved Hub server-side

Comment:
I think we can do it. If you want to be extra cautious you can do it for all datasets except imdb and conllpp for now which are actively used by 2.6.1 users. For those two we can keep the YAML like this for some more time, or alternatively use the old dataset_infos.json file
Issue: YAML integer keys are not preserved Hub server-side

Comment:
The bulk edit of canonical datasets (except imdb and conllpp) is running. 

See e.g.: https://huggingface.co/datasets/acronym_identification/discussions/3

EDITED: 
Done, except for "universal_morphologies", where I get
```
HTTPError: 413 Client Error: Payload Too Large for url: https://huggingface.co/api/validate-yaml
```

Also not done for the datasets missing matadata "dataset_info":
- mc4: https://huggingface.co/datasets/mc4/discussions/3
- the_pile: https://huggingface.co/datasets/the_pile/discussions/6
- timit_asr: https://huggingface.co/datasets/timit_asr/discussions/1
Issue: YAML integer keys are not preserved Hub server-side

Comment:
@lhoestq, there are 6 community datasets with YAML integer keys in their `dataset_info` `class_label`:
- indonlp/indonlu
- rcds/swiss_judgment_prediction
- Jean-Baptiste/wikiner_fr
- Bingsu/Cat_and_Dog
- taskydata/tasky_or_not
- RCC-MSU/collection3

Maybe we could open a PR on them as well?
Issue: YAML integer keys are not preserved Hub server-side

Comment:
Let's do this then:

- [x] [indonlp/indonlu](https://huggingface.co/datasets/indonlp/indonlu/discussions/3)
- [x] rcds/swiss_judgment_prediction
- [x] Jean-Baptiste/wikiner_fr
- [x] Bingsu/Cat_and_Dog -> merged
- [x] taskydata/tasky_or_not (was already using quotes)
- [x] RCC-MSU/collection3

EDIT: all done :)
Issue: YAML integer keys are not preserved Hub server-side

Comment:
@lhoestq I was not asking you to do it, but asking if you agree me to do it... :man_facepalming: 
As I self-assigned this issue... :sweat_smile:
Issue: load_dataset possibly broken for gated datasets?

Comment:
Btw, thanks very much for finding the hub rollback temporary fix and bringing the issue to our attention @KhoomeiK!
Issue: load_dataset possibly broken for gated datasets?

Comment:
I see the same issue when calling `load_dataset('poloclub/diffusiondb', 'large_random_1k')` with `datasets==2.7.1` and `huggingface-hub=0.11.0`. No issue with `datasets=2.6.1` and `huggingface_hub==0.10.1`.

https://github.com/poloclub/diffusiondb/issues/7
Issue: load_dataset possibly broken for gated datasets?

Comment:
I fixed my issue by specifying `repo_type` in `hf_hub_url()`. https://github.com/poloclub/diffusiondb/commit/9eb91c79aaca98b0515a0ce45778b8af65b84652

I opened a PR on the Winoground's repo: https://huggingface.co/datasets/facebook/winoground/discussions/2
Issue: load_dataset possibly broken for gated datasets?

Comment:
This is a bug in the script, indeed. The most robust fix is to use a relative path instead of `hf_hub_url`, which does not depend on `huggingface_hub`'s version 🙂. I've opened a PR here: https://huggingface.co/datasets/facebook/winoground/discussions/3.
Issue: load_dataset possibly broken for gated datasets?

Comment:
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name':

    tokenizer = AutoTokenizer.from_pretrained(ARGS.model_path, trust_remote_code=True)

Please handle automatically for local path and repo name inside, otherwise users always get confused about this
Issue: load_dataset possibly broken for gated datasets?

Comment:
I think I'm also hitting this error, trying to load a model from a local path.
Issue: Use pyarrow Tensor dtype

Comment:
Hi ! We're using the Arrow format for the datasets, and PyArrow tensors are not part of the Arrow format AFAIK:

> There is no direct support in the arrow columnar format to store Tensors as column values.

source: https://github.com/apache/arrow/issues/4802#issuecomment-508494694
Issue: Use pyarrow Tensor dtype

Comment:
@wesm @rok its been around three years. any updates, regarding dataset arrow tensor support? 🙏 I know you must be very busy, would appreciate to learn what is the state of art. I saw the PR is still open [#8510](https://github.com/apache/arrow/pull/8510)
Issue: Use pyarrow Tensor dtype

Comment:
Hey @franz101 & @lhoestq!
There is a plan and a PR to create an [ExtensionArray of Tensors](https://github.com/apache/arrow/pull/8510) of equal sizes as well as a plan to do the same for Tensors of different sizes [ARROW-8714](https://issues.apache.org/jira/browse/ARROW-8714).
Issue: Use pyarrow Tensor dtype

Comment:
The work stalled a little because it was not clear where TensorArray would live. However Arrow community recently agreed to make a [well-known-extension-type document](https://lists.apache.org/thread/sxd5fhc42hb6svs79t3fd79gkqj83pfh) and I would like https://github.com/apache/arrow/pull/8510 to land there and add an implementation to C++/Python + another language. Is that something you would find beneficial to you?
Issue: Use pyarrow Tensor dtype

Comment:
that is a great update, thank you.
it looks like this feature would benefit datasets implementation of [ArrayExtensionArray](https://github.com/huggingface/datasets/blob/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/features/features.py#L585-L641). Is that correct @eladsegal @lhoestq?
Issue: Use pyarrow Tensor dtype

Comment:
TensorArray sounds great ! Looking forward to it :)

We've had our own ExtensionArray for fixed shape tensors for a while now, hoping to see something more standardized by the arrow community.

Also super interested in the extension array for tensors of different sizes cc @mariosasko
Issue: Use pyarrow Tensor dtype

Comment:
@rok Thanks for keeping us updated! I think it's best to introduce a new feature type that would use this extension type under the hood. I'll create an issue to discuss the design with the community in the coming days.

Also, is there a tentative time frame for the variable-shape Tensor extension type?
Issue: Use pyarrow Tensor dtype

Comment:
@mariosasko please tag me in the discussion, perhaps I can contribute.

As for the [variable shape tensor array](https://github.com/apache/arrow/issues/24868) - I'd be interested in working on it but didn't see much interest in community yet. Are you saying `huggingface/datasets` could use it?
Issue: Use pyarrow Tensor dtype

Comment:
pyarrow 12 is out 🎉, will have a look if I can work on it for the ExtensionArray
Issue: Use pyarrow Tensor dtype

Comment:
I think these two issues need to be fixed first on the Arrow side before adding the tensor feature type here: https://github.com/apache/arrow/issues/35573 and https://github.com/apache/arrow/issues/35599.

@rok We've had a couple of requests for supporting variable-shape tensors on the forum/GH, but I did not manage to find the concrete issues using the search. TF/TFDS (and PyTorch with the `nested_tensor` API) support them, so it makes sense for us to do the same eventually (the Ray project has an [extension](https://github.com/ray-project/ray/blob/42a8d1489b37243f203120899a23d919dc85bf2a/python/ray/air/util/tensor_extensions/arrow.py#L634) type to support this case)
Issue: Use pyarrow Tensor dtype

Comment:
> @rok We've had a couple of requests for supporting variable-shape tensors on the forum/GH, but I did not manage to find the concrete issues using the search. TF/TFDS (and PyTorch with the `nested_tensor` API) support them, so it makes sense for us to do the same eventually (the Ray project has an [extension](https://github.com/ray-project/ray/blob/42a8d1489b37243f203120899a23d919dc85bf2a/python/ray/air/util/tensor_extensions/arrow.py#L634) type to support this case)

That does make sense indeed. We should probably also be careful about memory layout to enable zero-copy interface to TF/PyTorch.
Issue: Use pyarrow Tensor dtype

Comment:
Not with with the Arrow format, and therefore not in `datasets`. But they released a new [FixedShapeTensorArray](https://arrow.apache.org/docs/python/extending_types.html#fixed-size-tensor) to store tensors in Arrow format. We plan to support this in `datasets` at one point !
Issue: Use pyarrow Tensor dtype

Comment:
There is also an open issue to enable the conversion of `pyarrow.Tensor` to `pyarrow.FixedShapeTensorType`: https://github.com/apache/arrow/issues/35068. This way one could indirectly use `pyarrow.Tensor` in Arrow format.
Issue: Use pyarrow Tensor dtype

Comment:
We started a [mailing list discussion](https://lists.apache.org/thread/qc9qho0fg5ph1dns4hjq56hp4tj7rk1k) about potential `VariableShapeTensor` extension array, please check it out and give feedback. For more details here's also a PR https://github.com/apache/arrow/pull/37166.
Issue: When len(_URLS) > 16, download will hang

Comment:
Thanks for reporting ! This sounds like an issue with python multiprocessing. If we switch to multithreading for the downloads it should be much more robust - let me know if this is something you'd like to contribute, I'd be happy to help and give you some pointers
Issue: When len(_URLS) > 16, download will hang

Comment:
> an issue with python multiprocessing

If it is an issue with multiprocessing, should we report it to upstream?
Issue: When len(_URLS) > 16, download will hang

Comment:
Debugging this would require quite some work in my opinion, and I've often failed to make reproducible examples, since it's pretty correlated to one's environment + hardware. So I wouldn't spend too much time on this unless we manage to reproduce this on another machine consistently.

Instead I'd encourage a more pragmatic fix that is: not create tons of processes (on regular machines it may slow things down anyway), and instead use multithreading by default.
Issue: When len(_URLS) > 16, download will hang

Comment:
I am not expert of python. I hear about python has GIL, which result in multi processing is worse than multi threading. So I am not sure if this change makes sense?

And if this is a bug of multi processing, why not report to upstream and let them fix? And even if change it to multi threading, how can we make sure it can truly fix this problem?
Issue: When len(_URLS) > 16, download will hang

Comment:
> Just my 2c. No offense.

sure np ^^

> I hear about python has GIL, which result in multi processing is worse than multi threading. So I am not sure if this change makes sense?

Here the bottleneck speed is the bandwidth used to download the files. When downloading, the GIL is released, so multithreading gives the same speed as multiprocessing.

> And if this is a bug of multi processing, why not report to upstream and let them fix?

Usually to fix a bug it's important to be able to reproduce it. This way you can share it, experiment with it, and then make sure it's fixed. Here I'm afraid it's not easy to reproduce. Though I think that spawning too many processes for your machine can lead to this kind of issues.

> And even if change it to multi threading, how can we make sure it can truly fix this problem?

Multithreading is more robust in python because IIRC there are less locks involved which are often the cause of code hanging for no reason.
Issue: Get an IterableDataset from a map-style Dataset

Comment:
I think `stream` could be misleading since the data is not being streamed from remote endpoints (one could think that's the case when they see `load_dataset` followed by `stream`). Hence, I prefer the second option.

PS: When we resolve https://github.com/huggingface/datasets/issues/4542, we could add `as_tf_dataset` to the API for consistency and deprecate `to_tf_dataset`.
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
Could you share the full stack trace please ?


Can you also try running this code ? It can be useful to determine if the issue comes from `datasets` or `fsspec` (streaming) or `pyarrow` (parquet reading):
```python
ds = load_dataset("parquet", data_files=a_parquet_file_url, use_auth_token=True)
```
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
Can you check you used the right token ? You shouldn't get a 401 using your token
Issue: `datasets` can't read a Parquet file in Python 3.9.13

Comment:
I checked it’s the right token, when loading the full dataset I get the error after data extraction so I can access the files. 
```
Downloading and preparing dataset parquet/bigcode--the-stack-dedup-pjj to /home/loubna_huggingface_co/.cache/huggingface/datasets/bigcode___parquet/bigcode--the-stack-dedup-pjj-872ffac7f4bb46ca/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22.38it/s]
Extracting data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 49.91it/s]
Traceback (most recent call last):
  File "/home/loubna_huggingface_co/load_ds.py", line 5, in <module>
    ds = load_dataset("bigcode/the-stack-dedup-pjj", data_dir="data/java", use_auth_token=True,split="train", revision="v1.1.a1")
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py", line 1742, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 814, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 905, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 1502, in _prepare_split
    for key, table in logging.tqdm(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 67, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 286, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1227, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```
Could it be that I'm using a wrong url, I just copied it from the address bar