SurAyush/github_issues huggingface dataset:
Issue: Very slow between two operations
State: closed
Author: yananchen1989

Hello, in the processing stage, I use two operations. The first one : map + filter, is very fast and it uses the full cores, while the socond step is very slow and did not use full cores. 

Also, there is a significant lag between them.  Am I missing something ?



 ```
raw_datasets = raw_datasets.map(split_func, 
                batched=False,
                num_proc=args.preprocessing_num_workers,
                load_from_cache_file=not args.overwrite_cache, 
                desc = "running split para ==>")\
                .filter(lambda example: example['text1']!='' and example['text2']!='', 
                    num_proc=args.preprocessing_num_workers, desc="filtering ==>")


    processed_datasets = raw_datasets.map(
        preprocess_function,
        batched=True, 
        num_proc=args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not args.overwrite_cache,
        desc="Running tokenizer on dataset===>",
    )
```

Comments:
Issue: Stream private zipped images
State: closed
Author: lhoestq

As mentioned in https://github.com/huggingface/datasets/issues/4139 it's currently not possible to stream private/gated zipped images from the Hub.

This is because `Image.decode_example` does not handle authentication. Indeed decoding requires to access and download the file from the private repository.

In this PR I added authentication to `Image.decode_example` via a `token_per_repo_id` optional argument. I first wanted to just pass `use_auth_token` but a single `Image` instance can be responsible of decoding images from a combination of several datasets together (from `interleave_datasets` for example). Therefore I just used a dictionary `repo_id` -> `token` instead.

I'm getting the `repo_id` from the dataset builder (I replaced the `namepace` attribute with `repo_id`)

I did the same for `Audio.decode_example`

cc @SBrandeis @severo 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

oops looks like some tests are failing sorry, will fix them tomorrow

EDIT: not today but asap hopefully

cc @mariosasko this is ready for review, let me know what you think !
Issue: Add Conceptual 12M
State: closed
Author: thomasw21


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Looks like your dummy_data.zip file is not in the right location ;)
datasets/datasets/conceptual_12m/dummy/default/0.0.0/dummy_data.zip
->
datasets/conceptual_12m/dummy/default/0.0.0/dummy_data.zip
Issue: Add Visual Genome
State: closed
Author: thomasw21


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hum there seems to be some issues with tasks in test:
 - some tasks don't fit anything in `tasks.json`. Do I remove them in `task_categories`?
 - some tasks should exist, typically `visual-question-answering` (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my `master` is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?
 
 cc @mariosasko @lhoestq 

> some tasks don't fit anything in tasks.json. Do I remove them in task_categories?

You can keep them, but add `other-` as a prefix to those tasks to make the CI ignore it

> some tasks should exist, typically visual-question-answering (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my master is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?

Feel free to merge upstream/master into your branch ;)

EDIT: actually I just noticed you've already done this, thanks !

After offline discussions: will keep that image essentially it's necessary as I have a mapping that creates a mapping between url and local path (images are downloaded via a zip file) and dummy data needs to store that dummy image. The issue is when I read an annotation, I get a url, compute the local path, and basically I assume the local path exists since I've extracted all the images ... This isn't true if dummy data doesn't have all the images, so instead I've added a script that "fixes" the dummy data after using the CLI, it essentially adds the dummy image in the zip corresponding to the url.
Issue: RGBA images not showing
State: closed
Author: cceyda

## Dataset viewer issue for ceyda/smithsonian_butterflies_transparent

[**Link:** *link to the dataset viewer page*](https://huggingface.co/datasets/ceyda/smithsonian_butterflies_transparent)

![image](https://user-images.githubusercontent.com/15624271/163117683-e91edb28-41bf-43d9-b371-5c62e14f40c9.png)

Am I the one who added this dataset ? Yes

👉  More of a general issue of 'RGBA' png images not being supported 
(the dataset itself is just for the huggan sprint and not that important, consider it just an example)

Comments:
Thanks for reporting. It's a known issue, and we hope to fix it soon.

Fixed, thanks!
Issue: Adding STSb-TR dataset
State: closed
Author: figenfikri

Semantic Textual Similarity benchmark Turkish (STSb-TR) dataset introduced in our paper [Semantic Similarity Based Evaluation for Abstractive News Summarization](https://aclanthology.org/2021.gem-1.3.pdf) added.

Comments:
Thanks for your contribution, @figenfikri.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Issue: Generate tasks.json taxonomy from `huggingface_hub`
State: closed
Author: julien-c


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ok recomputed the json file, this should be ready to review now! @lhoestq 

Note: the generated JSON from `hf/hub-docs` can be found in the output of a GitHub Action run on that repo, for instance in https://github.com/huggingface/hub-docs/runs/6006686983?check_suite_focus=true

(click on "Run export-tasks script")

Should we not add the tasks with hideInDatasets?

yes, probably true – i'll change that in a PR in `hub-docs`

Yes that's good :) feel free to merge

thanks to the both of you!
Issue: ArrayND error in pyarrow 5
State: closed
Author: lhoestq

As found in https://github.com/huggingface/datasets/pull/3903, The ArrayND features fail on pyarrow 5:
```python
import pyarrow as pa
from datasets import Array2D
from datasets.table import cast_array_to_feature

arr = pa.array([[[0]]])
feature_type = Array2D(shape=(1, 1), dtype="int64")
cast_array_to_feature(arr, feature_type)
```
raises
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-04610f9fa78c> in <module>
----> 1 cast_array_to_feature(pa.array([[[0]]]), Array2D(shape=(1, 1), dtype="int32"))

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in cast_array_to_feature(array, feature, allow_number_to_str)
   1806         return array_cast(array, get_nested_type(feature), allow_number_to_str=allow_number_to_str)
   1807     elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1808         return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
   1809     raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
   1810 

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_number_to_str)
   1705         array = array.storage
   1706     if isinstance(pa_type, pa.ExtensionType):
-> 1707         return pa_type.wrap_array(array)
   1708     elif pa.types.is_struct(array.type):
   1709         if pa.types.is_struct(pa_type) and (

AttributeError: 'Array2DExtensionType' object has no attribute 'wrap_array'
```

The thing is that `cast_array_to_feature` is called when writing an Arrow file, so creating an Arrow dataset using any ArrayND type currently fails.

`wrap_array` has been added in pyarrow 6, so we can either bump the required pyarrow version or fix this for pyarrow 5

Comments:
Where do we bump the required pyarrow version? Any inputs on how I fix this issue? 

We need to bump it in `setup.py` as well as update some CI job to use pyarrow 6 instead of 5 in `.circleci/config.yaml` and `.github/workflows/benchmarks.yaml`
Issue: Inconsistent splits generation for datasets without loading script (packaged dataset puts everything into a single split)
State: closed
Author: polinaeterna

## Describe the bug
Splits for dataset loaders without scripts are prepared inconsistently. I think it might be confusing for users.

## Steps to reproduce the bug
* If you load a packaged datasets from Hub, it infers splits from directory structure / filenames (check out the data [here](https://huggingface.co/datasets/nateraw/test-imagefolder-dataset)):
```python
ds = load_dataset("nateraw/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* If you do the same from locally stored data specifying only directory path you'll get the same:
```python
ds = load_dataset("/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* However, if you explicitely specify package name (like `imagefolder`, `csv`, `json`), all the data is put into a single split:
```python
ds = load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 10
    })
})
```

## Expected results
For `load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")` I expect the same output as of the two first options.

Comments:
Issue: fix confusing bleu metric example
State: closed
Author: aizawa-naoki

**Is your feature request related to a problem? Please describe.**
I would like to see the example in "Metric Card for BLEU" changed.
The 0th element in the predictions list is not closed in square brackets, and the 1st list is missing a comma.
The BLEU score are calculated correctly, but it is difficult to understand, so it would be helpful if you could correct this.
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
>>> references = [
...     [["hello", "there", "general", "kenobi"]],
...     [["foo", "bar", "foobar"]]
... ]
>>> bleu = datasets.load_metric("bleu")
>>> results = bleu.compute(predictions=predictions, references=references)
>>> print(results)
{'bleu': 0.6370964381207871, ...
```

**Describe the solution you'd like**
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
# and
>>> print(results)
{'bleu':1.0, ...
```


Comments:
Issue: Adjust path to datasets tutorial in How-To
State: closed
Author: NimaBoscarino

The link in the How-To overview page to the Datasets tutorials is currently broken. This is just a small adjustment to make it match the format used in https://github.com/huggingface/datasets/blob/master/docs/source/tutorial.md.

(Edit to add: The link in the PR deployment (https://moon-ci-docs.huggingface.co/docs/datasets/pr_4147/en/how_to) is also broken since it's actually hardcoded to `master` and not dynamic to the branch name, but other links seem to behave similarly.)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: SAMSum dataset viewer not working
State: closed
Author: aakashnegi10

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Comments:
https://huggingface.co/datasets/samsum

```
Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file
```

Currently, only the datasets that can be streamed support the dataset viewer. Maybe @lhoestq @albertvillanova or @mariosasko could give more details about why the dataset cannot be streamed.

It looks like the host (https://arxiv.org) doesn't allow HTTP Range requests, which is what we use to stream data.

This can be fix if we host the data ourselves, which is ok since the dataset is under CC BY-NC-ND 4.0
Issue: Redirect TIMIT download from LDC
State: closed
Author: lhoestq

LDC data is protected under US copyright laws and under various legal agreements between the Linguistic Data Consortium/the University of Pennsylvania and data providers which prohibit redistribution of that data by anyone other than LDC. Similarly, LDC's membership agreements, non-member user agreement and various corpus-specific license agreements specifically state that users cannot publish, retransmit, disclose, copy, reproduce or redistribute LDC databases to others outside their organizations.

LDC explicitly asked us to remove the download script for the TIMIT dataset. In this PR I remove all means to download the dataset, and redirect users to download the data from https://catalog.ldc.upenn.edu/LDC93S1 

Comments:
CI is failing because some tags are outdated, but they're fixed in #4067 

_The documentation is not available anymore as the PR was closed or merged._

We may do a release pretty soon (today ?), let me know if it's fine to include it in the new release

Fine to include this change!
Issue: Add ObjectFolder 2.0 dataset
State: open
Author: osanseviero

## Adding a Dataset
- **Name:** ObjectFolder 2.0
- **Description:** ObjectFolder 2.0 is a dataset of 1,000 objects in the form of implicit representations. It contains 1,000 Object Files each containing the complete multisensory profile for an object instance.
- **Paper:** [*link to the dataset paper if available*](https://arxiv.org/abs/2204.02389)
- **Data:** https://github.com/rhgao/ObjectFolder

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Datasets are not tracked in this repository anymore.
Issue: Add single dataset citations for TweetEval
State: closed
Author: gchhablani

This PR adds single data citations as per request of the original creators of the TweetEval dataset.

This is a recent email from the creator:

> Could I ask you a favor? Would you be able to add at the end of the README the citations of the single datasets as well? You can just copy our readme maybe? https://github.com/cardiffnlp/tweeteval#citing-tweeteval
(just to be sure that the creator of the single datasets also get credits when tweeteval is used)

Please let me know if this looks okay or if any changes are needed.

Thanks,
Gunjan


Comments:
_The documentation is not available anymore as the PR was closed or merged._

The `test_dataset_cards` method is failing with the error:

```
if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           The following typing errors are found: {'annotations_creators': "(Expected `typing.List` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)\nOR\n(Expected `typing.Dict` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)"}
```

Adding `found` as annotation creators.
Issue: dataset metadata for reproducibility
State: open
Author: nbroad1881

When pulling a dataset from the hub, it would be useful to have some metadata about the specific dataset and version that is used. The metadata could then be passed to the `Trainer` which could then be saved to a model card. This is useful for people who run many experiments on different versions (commits/branches) of the same dataset. 

The dataset could have a list of “source datasets” metadata and ignore what happens to them before arriving in the Trainer (i.e. ignore mapping, filtering, etc.).

Here is a basic representation (made by @lhoestq )
```python
>>> from datasets import load_dataset
>>> 
>>> my_dataset = load_dataset(...)["train"]
>>> my_dataset = my_dataset.map(...)
>>> 
>>> my_dataset.sources
[HFHubDataset(repo_id=..., revision=..., arguments={...})]
```

Comments:
+1 on this idea. This could be powerful for helping better track datasets used for model training and help with automatic model card creation. 

One possible way of doing this would be to store some/most/all the arguments passed to `load_dataset` if a hub id is passed. i.e. store the Hub ID, configuration, etc. 

cc @tomaarsen
Issue: More robust `cast_to_python_objects` in `TypedSequence`
State: closed
Author: mariosasko

Adds a fallback to run an expensive version of `cast_to_python_objects` which exhaustively checks entire lists to avoid the `ArrowInvalid: Could not convert` error in `TypedSequence`. Currently, this error can happen in situations where only some images are decoded in `map`, in which case `cast_to_python_objects` fails to recognize that it needs to cast `PIL.Image` objects if they are not at the beginning of the sequence and stops after the first image dictionary (e.g., if `data` is `[{"bytes": None, "path": "some path"}, PIL.Image(), ...]`)

Fix #4124

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Issue: dataset viewer issue for common_voice
State: closed
Author: laphang

## Dataset viewer issue for 'common_voice'

**Link:** https://huggingface.co/datasets/common_voice

Server Error
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'

Am I the one who added this dataset ? No


Comments:
Yes, it's a known issue, and we expect to fix it soon.

Fixed.

<img width="1393" alt="Capture d’écran 2022-04-25 à 15 42 05" src="https://user-images.githubusercontent.com/1676121/165101176-d729d85b-efff-45a8-bad1-b69223edba5f.png">
Issue: Building C4 takes forever
State: closed
Author: StellaAthena

## Describe the bug
C4-en is a 300 GB dataset. However, when I try to download it through the hub it takes over _six hours_ to generate the train/test split from the downloaded files. This is an absurd amount of time and an unnecessary waste of resources.

## Steps to reproduce the bug
```python
c4 = datasets.load("c4", "en")
```

## Expected results
I would like to be able to download pre-split data.

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.13.0-35-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Comments:
Hi @StellaAthena, thanks for reporting.

Please note, that our `datasets` library performs several operations in order to load a dataset, among them:
- it downloads all the required files: for C4 "en", 378.69 GB of JSON GZIPped files
- it parses their content to generate the dataset
- it caches the dataset in an Arrow file: for C4 "en", this file size is 1.87 TB
- it memory-maps the Arrow file

If it suits your use case, you might load this dataset in streaming mode:
- no Arrow file is generated
- you can iterate over elements immediately (no need to wait to download all the entire files)

```python
In [45]: from datasets import load_dataset
    ...: ds = load_dataset("c4", "en", split="train", streaming=True)
    ...: for item in ds:
    ...:     print(item)
    ...:     break
    ...: 
{'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': '2019-04-25T12:57:54Z', 'url': 'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
```
I hope this is useful for your use case.
Issue: medical_dialog zh has very slow _generate_examples
State: closed
Author: nbroad1881

## Describe the bug
After downloading the files from Google Drive, `load_dataset("medical_dialog", "zh", data_dir="./")` takes an unreasonable amount of time. Generating the train/test split for 33% of the dataset takes over 4.5 hours.

## Steps to reproduce the bug
The easiest way I've found to download files from Google Drive is to use `gdown` and use Google Colab because the download speeds will be very high due to the fact that they are both in Google Cloud.

```python
file_ids = [
        "1AnKxGEuzjeQsDHHqL3NqI_aplq2hVL_E",
        "1tt7weAT1SZknzRFyLXOT2fizceUUVRXX",
        "1A64VBbsQ_z8wZ2LDox586JIyyO6mIwWc",
        "1AKntx-ECnrxjB07B6BlVZcFRS4YPTB-J",
        "1xUk8AAua_x27bHUr-vNoAuhEAjTxOvsu",
        "1ezKTfe7BgqVN5o-8Vdtr9iAF0IueCSjP",
        "1tA7bSOxR1RRNqZst8cShzhuNHnayUf7c",
        "1pA3bCFA5nZDhsQutqsJcH3d712giFb0S",
        "1pTLFMdN1A3ro-KYghk4w4sMz6aGaMOdU",
        "1dUSnG0nUPq9TEQyHd6ZWvaxO0OpxVjXD",
        "1UfCH05nuWiIPbDZxQzHHGAHyMh8dmPQH",
]
for i in file_ids:
    url = f"https://drive.google.com/uc?id={i}"
    !gdown $url


from datasets import load_dataset

ds = load_dataset("medical_dialog", "zh", data_dir="./")
```

## Expected results
Faster load time

## Actual results
`Generating train split: 33%: 625519/1921127 [4:31:03<31:39:20, 11.37 examples/s]`

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

@vrindaprabhu , could you take a look at this since you implemented it? I think the `_generate_examples` function might need to be rewritten

Comments:
Hi @nbroad1881, thanks for reporting.

Let me have a look to try to improve its performance. 

Thanks @nbroad1881  for reporting! I don't recall it taking so long. I will also have a look at this. 
@albertvillanova please let me know if I am doing something unnecessary or time consuming.

Hi @nbroad1881 and @vrindaprabhu,

As a workaround for the performance of the parsing of the raw data files (this could be addressed in a subsequent PR), I have found that there are also processed data files, that do not require parsing. I have added these as new configurations `processed.en` and `processed.zh`:
```python
ds = load_dataset("medical_dialog", "processed.zh")
```