SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
It seems that migration from dataset-info.json to dataset card YAML has been acted.

Probably it's a good idea, but I didn't find the pros and cons of this decision, so I put some I could think of:

pros:
- only one file to parse, share, sync
- it gives a hint to the users that if you write your dataset card, you should also specify the metadata

cons:
- the metadata header might be very long, before reaching the start of the README/dataset card. It might be surprising when you edit the file because the metadata is not shown on top when the dataset card is rendered (dataset page). It also somewhat prevents including large strings like the checksums
- YAML vs JSON: not sure which one is easier for users to fill and maintain
- two concepts are mixed in the same file (metadata and documentation). This means that if you're interested only in one of them, you still have to know how to parse the whole file.
- [low priority] besides the JSON file, we might want to support yaml or toml file if the user prefers (as [prettier](https://prettier.io/docs/en/configuration.html) and others do for their config files, for example). Inside the md, I understand that only YAML is allowed
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
> the metadata header might be very long, before reaching the start of the README/dataset card. It might be surprising when you edit the file because the metadata is not shown on top when the dataset card is rendered (dataset page). It also somewhat prevents including large strings like the checksums

Note that we could simply not have the checksums in the YAML metadata at all, or maybe at one point have a pointer to another file instead.

We can also choose to hide (collapse) certain sections in the YAML by default when we open the dataset card editor.

> two concepts are mixed in the same file (metadata and documentation). This means that if you're interested only in one of them, you still have to know how to parse the whole file.

I think it's fine for now. Later if we really end up with too many YAML sections we can see if we need to tweak the API endpoints or the `datasets`/`huggingface_hub` tools

> YAML vs JSON: not sure which one is easier for users to fill and maintain

Regarding YAML vs JSON: I think YAML is easier to write by hand, and I also think that it's better for consistency - i.e. we're using more and more YAML to configure models/datasets/spaces
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
> the metadata header might be very long, before reaching the start of the README/dataset card. It might be surprising when you edit the file because the metadata is not shown on top when the dataset card is rendered (dataset page). It also somewhat prevents including large strings like the checksums

We can definitely work on this on the hub side to make the UX better
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
Tensorflow Datasets catalog includes a community catalog where you can find and use HF datasets (see [here](https://www.tensorflow.org/datasets/community_catalog/huggingface)).

FYI I noticed today that they are using the exported dataset_infos.json files from github to get the metadata (see their code [here](https://github.com/tensorflow/datasets/blob/a482f01c036a10496f5e22e69a2ef81b707cc418/tensorflow_datasets/scripts/documentation/build_community_catalog.py#L261))
Issue: Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

Comment:
Metadata is now stored as YAML, and dummy data is deprecated, so I think we can close this issue.
Issue: cast_column function not working with map function in streaming mode for Audio features

Comment:
Hi! This is probably due to the fact that `IterableDataset.map` sets `features` to `None` before mapping examples. We can fix the issue by passing the old `features` dict to the map generator and performing encoding/decoding there (before calling the map transform function).
Issue: Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Comment:
Hi @ToddMorrill, thanks for reporting.

Three weeks ago I contacted the team who created the Pile dataset to report this issue with their data host server: https://the-eye.eu

They told me that unfortunately, the-eye was heavily affected by the recent tornado catastrophe in the US. They hope to have their data back online asap.
Issue: Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Comment:
Hi @ToddMorrill, people from the Pile team have mirrored their data in a new host server: https://mystic.the-eye.eu

See:
- #3627

It should work if you update your URL.

We should also update the URL in our course material.
Issue: Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Comment:
The old URL is still present in the HuggingFace course here: 
https://huggingface.co/course/chapter5/4?fw=pt

I have created a PR for the Notebook here: https://github.com/huggingface/notebooks/pull/148
Not sure if the HTML is in a public repo. I wasn't able to find it.
Issue: Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Comment:
Both URLs are broken now
`HTTPError: 404 Client Error: Not Found for URL: https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst`
And
`ConnectTimeout: HTTPSConnectionPool(host='mystic.the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst (Caused by ConnectTimeoutError(, 'Connection to mystic.the-eye.eu timed out. (connect timeout=10.0)'))`
Issue: Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Comment:
I was able to find a torrent with "The Pile" dataset here: [The Pile An 800GB Dataset of Diverse Text for Language Modeling ](https://academictorrents.com/details/0d366035664fdf51cfbe9f733953ba325776e667)

The complete dataset is huge, so I would suggest you to download only the "PUBMED_title_abstracts_2019_baseline.jsonl.zst" file, which is about 7GB. You can do this by using a torrent client of your choice (I typically utilize Transmission, which is pre-installed in Ubuntu distributions).
Issue: Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Comment:
@albertvillanova another issue:
```
15     experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights()
16   File "/lfs/ampere1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 474, in experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights
17     column_names = next(iter(dataset)).keys()
18   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1353, in __iter__
19     for key, example in ex_iterable:
20   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 207, in __iter__
21     yield from self.generate_examples_fn(**self.kwargs)
22   File "/lfs/ampere1/0/brando9/.cache/huggingface/modules/datasets_modules/datasets/EleutherAI--pile/ebea56d358e91cf4d37b0fde361d563bed1472fbd8221a21b38fc8bb4ba554fb/pile.py", line 236, in _generate_examples
23     with zstd.open(open(files[subset], "rb"), "rt", encoding="utf-8") as f:
24   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/streaming.py", line 74, in wrapper
25     return function(*args, download_config=download_config, **kwargs)
26   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py", line 496, in xopen
27     file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
28   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/core.py", line 134, in open
29     return self.__enter__()
30   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/core.py", line 102, in __enter__
31     f = self.fs.open(self.path, mode=mode)
32   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/spec.py", line 1241, in open
33     f = self._open(
34   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 356, in _open
35     size = size or self.info(path, **kwargs)["size"]
36   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 121, in wrapper
37     return sync(self.loop, func, *args, **kwargs)
38   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 106, in sync
39     raise return_result
40   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 61, in _runner
41     result[0] = await coro
42   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 430, in _info
43     raise FileNotFoundError(url) from exc
44 FileNotFoundError: https://the-eye.eu/public/AI/pile_preliminary_components/NIH_ExPORTER_awarded_grant_text.jsonl.zst
```

any suggestions?
Issue: Adjusting chunk size for streaming datasets

Comment:
Hi ! Data streaming uses `fsspec` to read the data files progressively. IIRC the block size for buffering is 5MiB by default. So every time you finish iterating over a block, it downloads the next one. You can still try to increase the `fsspec` block size for buffering if it can help. To do so you just need to increase `fsspec.spec.AbstractBufferedFile.DEFAULT_BLOCK_SIZE `

Currently this is unfortunately done in a single thread, so it blocks the processing to download and uncompress the next block. At one point it would be nice to be able to do that in parallel !
Issue: Does datasets support load text from HDFS?

Comment:
Hi ! `datasets` currently supports reading local files or files over HTTP. We may add support for other filesystems (cloud storages, hdfs...) at one point though :)
Issue: URL query parameters are set as path in the compression hop for fsspec

Comment:
I think the test passes because it simply ignore what's after `gzip://`.

The returned urlpath is expected to look like `gzip://filename::url`, and the filename is currently considered to be what's after the final `/`, hence the result.

We can decide to change this and simply have `gzip://::url`, this way we don't need to guess the filename, what do you think ?
Issue: skip columns which cannot set to specific format when set_format

Comment:
You can add columns that you wish to set into `torch` format using `dataset.set_format("torch", ['id', 'abc'])` so that input batch of the transform only contains those columns
Issue: skip columns which cannot set to specific format when set_format

Comment:
Sorry, I miss `output_all_columns` args and thought after `dataset.set_format("torch", columns=columns)` I can only get specific columns I assigned.
Issue: make shape verification to use ArrayXD instead of nested lists for map

Comment:
Hi! 

Yes, this makes sense for numeric values, but first I have to finish https://github.com/huggingface/datasets/pull/3336 because currently ArrayXD only allows the first dimension to be dynamic.
Issue: the compression format requested when saving a dataset in json format is not respected

Comment:
Thanks for reporting @SaulLu.

At first sight I think the problem is caused because `pandas` only takes into account the `compression` parameter if called with a non-null file path or buffer. And in our implementation, we call pandas `to_json` with `None` `path_or_buf`.

We should fix this:
- either handling directly the `compression` parameter ourselves
- or refactoring to pass non-null path or buffer to pandas

CC: @lhoestq
Issue: the compression format requested when saving a dataset in json format is not respected

Comment:
I was thinking if we can handle the `compression` parameter by ourselves? Compression types will be similar to what `pandas` offer. Initially, we can try this with 2-3 compression types and see how good/bad it is? Let me know if it sounds good, I can raise a PR for this next week
Issue: the compression format requested when saving a dataset in json format is not respected

Comment:
Hi ! Thanks for your help @bhavitvyamalik :)
Maybe let's start with `gzip` ? I think it's the most common use case, then if we're fine with it we can add other compression methods
Issue: The rotten_tomatoes dataset of movie reviews contains some reviews in Spanish 

Comment:
Hi @puzzler10, thanks for reporting.

Please note this dataset is not hosted on Hugging Face Hub. See: 
https://github.com/huggingface/datasets/blob/c8f914473b041833fd47178fa4373cdcb56ac522/datasets/rotten_tomatoes/rotten_tomatoes.py#L42

If there are issues with the source data of a dataset, you should contact the data owners/creators instead. In the homepage associated with this dataset (http://www.cs.cornell.edu/people/pabo/movie-review-data/), you can find the authors of the dataset and how to contact them:
> If you have any questions or comments regarding this site, please send email to Bo Pang or Lillian Lee.

P.S.: Please also note that the example you gave of non-English review is in Portuguese (not Spanish). ;)
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
As discussed, I remember I set `decoded=False` here to avoid decoding just by iterating over examples of dataset. We wanted to decode only if the "audio" field (for Audio feature) was accessed.
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
> I set decoded=False here to avoid decoding just by iterating over examples of dataset. We wanted to decode only if the "audio" field (for Audio feature) was accessed

https://github.com/huggingface/datasets/pull/3430 will add more control to decoding, so I think it's OK to enable decoding in `__iter__` for now. After we merge the linked PR, the user can easily disable it again.
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
@mariosasko I wonder why there is no issue in `Audio` feature with decoding disabled in `__iter__`, whereas there is in `Image` feature.

Enabling decoding in `__iter__` will make fail Audio regressions tests: https://github.com/huggingface/datasets/runs/4608657230?check_suite_focus=true
```
=========================== short test summary info ============================
FAILED tests/features/test_audio.py::test_dataset_with_audio_feature_map_is_not_decoded
FAILED tests/features/test_audio.py::test_dataset_with_audio_feature_map_is_decoded
========================= 2 failed, 15 passed in 8.37s =========================
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
Please also note that the regression tests were implemented in accordance with the specifications:
- when doing a `map` (wich calls `__iter__`) of a function that doesn't access the audio field, the decoding should be disabled; this is why the decoding is disabled in `__iter__` (and only enabled in `__getitem__`).
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
>  I wonder why there is no issue in Audio feature with decoding disabled in __iter__, whereas there is in Image feature.

@albertvillanova Not sure if I understand this part. Currently, both the Image and the Audio feature don't decode data in `__iter__`, so their behavior is aligned there.
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
Therefore, this is not an issue, neither for Audio nor Image feature.

Could you please elaborate more on the expected use case? @lhoestq @NielsRogge 

The expected use cases (in accordance with the specs: see #2324):
- decoding should be enabled when accessing a specific item (`__getitem__`)
- decoding should be disabled while iterating (`__iter__`) to allow preprocessing of non-audio/image features (like label or text, for example) using `.map`
-  decoding should be enabled in a `.map` only if the `.map` function accesses the audio/image feature (implemented using `LazyDict`)
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
For me it's not an issue, actually. I just (mistakenly) tried to iterate over a PyTorch Dataset instead of a PyTorch DataLoader, 

i.e. I did this:

`batch = next(iter(train_ds)) `

whereas I actually wanted to do

`batch = next(iter(train_dataloader))`

and then it turned out that in the first case, the image was a string of bytes rather than a Pillow image, hence Quentin opened an issue.
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
Thanks @NielsRogge for the context.

So IMO everything is working as expected.

I'm closing this issue. Feel free to reopen it again if further changes of the specs should be addressed.
Issue: Iterating over a vision dataset doesn't decode the images

Comment:
Thanks for the details :)

I still think that it's unexpected to get different results when doing
```python
for i in range(len(dataset)):
    sample = dataset[i]
```
and
```python
for sample in dataset:
    pass
```
even though I understand that if you don't need to decode the data, then decoding image or audio data when iterating is a waste of time and resources.

But in this case users can still drop the column that need decoding to get the full speed back no ?
Issue: Unable to load 'cnn_dailymail' dataset

Comment:
Hi @talha1503, thanks for reporting.

It seems there is an issue with one of the data files hosted at Google Drive:
```
Google Drive - Quota exceeded

Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
```

As you probably know, Hugging Face does not host the data, and in this case the data owner decided to host their data at Google Drive, which has quota limits.

Is there anything we could do, @lhoestq @mariosasko?
Issue: struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Comment:
Hi ! Can you try setting `datasets.config.MAX_TABLE_NBYTES_FOR_PICKLING` to a smaller value than `4 << 30` (4GiB), for example `500 << 20` (500MiB) ? It should reduce the maximum size of the arrow table being pickled during multiprocessing.

If it fixes the issue, we can consider lowering the default value for everyone.
Issue: Add CMU Graphics Lab Motion Capture dataset

Comment:
This dataset has files in ASF/AMC format. [ The skeleton file is the ASF file (Acclaim Skeleton File). The motion file is the AMC file (Acclaim Motion Capture data). ]  

Some questions : 
1. How do we go about representing these features using datasets.Features and generate examples ?
2. The dataset download link for ASF/AMC files does not have metadata information, for eg : category and subcategory information. We will need to crawl the website for this information. The authors mention "Please don't crawl this database for all motions." Can we mail the authors for this information ?
The dataset structure is as follows : 
```
subjects
	- 01
		- 01_01.amc
		- 01_02.amc
		.
		.
		.
		- 01.asf
	- 02
		- 02_01.amc
		- 02_02.amc
		.
		.
		.
		- 02.asf
```
There is no metadata regarding the category, sub-category and motion description.

Need your inputs. @mariosasko / @lhoestq  
Thank you.
Issue: Add CMU Graphics Lab Motion Capture dataset

Comment:
Hi @dnaveenr! Thanks for working on this!

1. We can use the `Sequence(Value("string"))` feature type for the subject's AMC files and `Value("string")` for the subject's ASF file (`Value("string")` represents the file paths) + the types for categories/subcategories and descriptions.
2. We can use this URL to download the motion descriptions: http://mocap.cs.cmu.edu/search.php?subjectnumber=<subject_number>&motion=%%%&maincat=%&subcat=%&subtext=yes where `subject_number` is the number between 1 and 144. And to get categories/subcategories, feel free to contact the authors (they state in the FAQ they are happy to help) and ask them if they can provide the mapping from categories/subcategories to the AMC files to avoid crawling. You can also mention that your goal is to make their dataset more accessible by adding its loading script to the Hub.

The AMC files are also available in the tvd, c3d, mpg and avi formats (the links are in the [FAQ](http://mocap.cs.cmu.edu/faqs.php) section), so it would be nice to have one config for each of these additional formats. 

And additionally, we can add a `Data Preprocessing` section to the card where we explain how to load/process the files. I can help with that.
Issue: Add CMU Graphics Lab Motion Capture dataset

Comment:
Hi @mariosasko ,

1. Thanks for this, so we can add the file paths.
2. Yes, I had already mailed the authors a couple of days back actually, asking for the metadata details[ i.e category, sub-category and motion description] . They are yet to respond though, I will wait for a couple of days and try to follow up with them again. :) Else we can use the workaround solution.

Yes. Supporting all the formats would be helpful. 

> And additionally, we can add a Data Preprocessing section to the card where we explain how to load/process the files. I can help with that.

Okay. Got it.
Issue: Easier information editing

Comment:
Hi ! I guess you are talking about the dataset cards that are in this repository on github ?

I think github allows to submit a PR even for 1 line though the `Edit file` button on the page of the dataset card.

Maybe let's mention this in `CONTRIBUTING.md` ?
Issue: Easier information editing

Comment:
We now host all the datasets on the HF Hub, where you can easily edit them through UI (for single file changes) or Git workflow (for single/multiple file changes)
Issue: why the stratify option is omitted from test_train_split function?

Comment:
Hi ! It's simply not added yet :)

If someone wants to contribute to add the `stratify` parameter I'd be happy to give some pointers.

In the meantime, I guess you can use `sklearn` or other tools to do a stratified train/test split over the **indices** of your dataset and then do
```
train_dataset = dataset.select(train_indices)
test_dataset = dataset.select(test_indices)
```
Issue: why the stratify option is omitted from test_train_split function?

Comment:
Hi @lhoestq I would like to add `stratify` parameter, can you give me some pointers for adding the same ?
Issue: why the stratify option is omitted from test_train_split function?

Comment:
Hi ! Sure :)

The `train_test_split` method is defined here: 

https://github.com/huggingface/datasets/blob/dc62232fa1b3bcfe2fbddcb721f2d141f8908943/src/datasets/arrow_dataset.py#L3253-L3253

and inside `train_test_split ` we need to create the right `train_indices` and `test_indices` that are passed here to `.select()`:

https://github.com/huggingface/datasets/blob/dc62232fa1b3bcfe2fbddcb721f2d141f8908943/src/datasets/arrow_dataset.py#L3450-L3464

For example if your dataset is like
|    |   label |
|---:|--------:|
|  0 |       1 |
|  1 |       1 |
|  2 |       0 |
|  3 |       0 |

and the user passes `stratify=dataset["label"]`, then you should get indices that look like this
```
train_indices = [0, 2]
test_indices = [1, 3]
```

these indices will be passed to `.select` to return the stratified train and test splits :)

Feel free to îng me if you have any question !