AniketTec/github-issues huggingface dataset:
Title: Metrics documentation is not accessible in the datasets doc UI
State: closed
Labels: description: New feature or request
name: enhancement

description: Discussions on the metrics
name: Metric discussion

**Is your feature request related to a problem? Please describe.**
Search for a metric name like "seqeval" yields no results on https://huggingface.co/docs/datasets/master/en/index . One needs to go look in `datasets/metrics/README.md` to find the doc. Even in the `README.md`, it can be hard to understand what the metric expects as an input, for example for `squad` there is a [key `id`](https://github.com/huggingface/datasets/blob/1a4c185663a6958f48ec69624473fdc154a36a9d/metrics/squad/squad.py#L42) documented only in the function doc but not in the `README.md`, and one needs to go look into the code to understand what the metric expects.

**Describe the solution you'd like**
Have the documentation for metrics appear as well in the doc UI, e.g. this https://github.com/huggingface/datasets/blob/1a4c185663a6958f48ec69624473fdc154a36a9d/metrics/squad/squad.py#L21-L63

I know there are plans to migrate metrics to the evaluate library, but just pointing this out.


Comments:
Hey @fxmarty :) Yes we are working on showing the docs of all the metrics on the Hugging face website. If you want to follow the advancements you can check the [evaluate](https://github.com/huggingface/evaluate) repository cc @lvwerra @sashavor
Title: Failing CI on Windows for sari and wiki_split metrics
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Our CI is failing from yesterday on Windows for metrics: sari and wiki_split
```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_sari - ...
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_wiki_split
```

See: https://app.circleci.com/pipelines/github/huggingface/datasets/11928/workflows/79daa5e7-65c9-4e85-829b-00d2bfbd076a/jobs/71594

Comments:
Title: Eval metadata Batch 4: Tweet Eval, Tweets Hate Speech Detection, VCTK, Weibo NER, Wisesight Sentiment, XSum, Yahoo Answers Topics, Yelp Polarity, Yelp Review Full
State: closed

Adding evaluation metadata for:

- Tweet Eval
- Tweets Hate Speech Detection
- VCTK
- Weibo NER
- Wisesight Sentiment
- XSum
- Yahoo Answers Topics
- Yelp Polarity
- Yelp Review Full

Comments:
Summary of CircleCI errors:

- **XSum**: missing 6 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', and 'source_datasets'
- **Yelp_polarity**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'

_The documentation is not available anymore as the PR was closed or merged._
Title: Eval metadata batch 3: Reddit, Rotten Tomatoes, SemEval 2010, Sentiment 140, SMS Spam, Snips, SQuAD, SQuAD v2, Timit ASR
State: closed

Adding evaluation metadata for:
- Reddit
- Rotten Tomatoes
- SemEval 2010
- Sentiment 140
- SMS Spam
- Snips
- SQuAD
- SQuAD v2
- Timit ASR

Comments:
Summary of CircleCI errors:

- **sem_eval_2010_task_8**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **sms_spam**: `Data Instances` and`Data Splits` are empty.
- **Quora** : Expected some content in section `Citation Information` but it is empty,  missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **sentiment140**:  missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'

There are also some timeout errors, I don't really understand the source though :confused: 

_The documentation is not available anymore as the PR was closed or merged._
Title: Eval metadata batch 2 : Health Fact, Jigsaw Toxicity, LIAR, LJ Speech, MSRA NER, Multi News, NCBI Disease, Poem Sentiment
State: closed

Adding evaluation metadata for : 
- Health Fact
- Jigsaw Toxicity
- LIAR
- LJ Speech
- MSRA NER
- Multi News
- NCBI Diseas
- Poem Sentiment

Comments:
Summary of CircleCI errors:
- **Jjigsaw_toxicity_pred**:  `Citation Information` but it is empty.
- **LIAR** :  `Data Instances`,`Data Fields`,  `Data Splits`,  `Citation Information` are empty.
- **MSRA NER** : Dataset Summary`, `Data Instances`, `Data Fields`,  `Data Splits`, `Citation Information` are empty.


The CI errors about missing content in the dataset cards can be ignored in this PR btw

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4336). All of your documentation changes will be reflected on that endpoint.
Title: Eval metadata batch 1: BillSum, CoNLL2003, CoNLLPP, CUAD, Emotion, GigaWord, GLUE, Hate Speech 18,  Hate Speech
State: closed

Adding evaluation metadata for:
- BillSum
- CoNLL2003
- CoNLLPP
- CUAD
- Emotion
- GigaWord
- GLUE
- Hate Speech 18 
- Hate Speech Offensive

Comments:
Summary of CircleCI errors:
- **BoolQ**: missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
- **Conllpp**: expected some content in section `Citation Information` but it is empty.
- **GLUE**:   'annotations_creators', 'language_creators', 'source_datasets' :['unknown'] are not registered tags
- **ConLL2003**:  field 'task_ids':  ['part-of-speech-tagging'] are not registered tags for 'task_ids'
-  **Hate_speech18:**   Expected some content in section `Data Instances` but it is empty, Expected some content in section `Data Splits` but it is empty

And yes we can ignore all the CI errors related to missing content in the dataset cards, these issues can be fixed in other PRs

_The documentation is not available anymore as the PR was closed or merged._
Title: `wikipedia` pre-processed datasets
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
[Wikipedia](https://huggingface.co/datasets/wikipedia) dataset readme says that certain subsets are preprocessed. However it seems like they are not available. When I try to load them it takes a really long time, and it seems like it's processing them.

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset("wikipedia", "20220301.en")
```

## Expected results
To load the dataset

## Actual results
Takes a very long time to load (after downloading)

After `Downloading data files: 100%`. It takes hours and gets killed.
Tried `wikipedia.simple` and it got processed after ~30mins.

Comments:
Hi @vpj, thanks for reporting.

I'm sorry, but I can't reproduce your bug: I load "20220301.simple"in 9 seconds:
```shell
time python -c "from datasets import load_dataset; load_dataset('wikipedia', '20220301.simple')"

Downloading and preparing dataset wikipedia/20220301.simple (download: 228.58 MiB, generated: 224.18 MiB, post-processed: Unknown size, total: 452.76 MiB) to .../.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.66k/1.66k [00:00<00:00, 1.02MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 235M/235M [00:02<00:00, 82.8MB/s]
Dataset wikipedia downloaded and prepared to .../.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 290.75it/s]

real	0m9.693s
user	0m6.002s
sys	0m3.260s
```

Could you please check your environment info, as requested when opening this issue?
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```
Maybe you are using an old version of `datasets`...

Downloading and processing `wikipedia simple` dataset completed in under 11sec on M1 Mac. Could you please check `dataset` version as mentioned by @albertvillanova? Also check system specs, if system is under load processing could take some time I guess.
Title: Fix type hint and documentation for `new_fingerprint`
State: closed

Currently, there are no type hints nor `Optional` for the argument `new_fingerprint` in several methods of `datasets.arrow_dataset.Dataset`.

There was some documentation missing as well.

Note that pylance is happy with the type hints, but pyright does not detect that `new_fingerprint` is set within the decorator.

The modifications in this PR are fine since here https://github.com/huggingface/datasets/blob/aa743886221d76afb409d263e1b136e7a71fe2b4/src/datasets/fingerprint.py#L446-L454

for the non-inplace case we make sure to auto-generate a new fingerprint (as indicated in the doc).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Dataset Viewer issue for strombergnlp/offenseval_2020, strombergnlp/polstance
State: closed
Labels: description: Related to the dataset viewer on huggingface.co
name: dataset-viewer

### Link

https://huggingface.co/datasets/strombergnlp/offenseval_2020/viewer/ar/train

### Description

The viewer isn't running for these two datasets. I left it overnight because a wait sometimes helps things get loaded, and the error messages have all gone, but the datasets are still turning up blank in viewer. Maybe it needs a bit more time.

* https://huggingface.co/datasets/strombergnlp/polstance/viewer/PolStance/train

* https://huggingface.co/datasets/strombergnlp/offenseval_2020/viewer/ar/train

While offenseval_2020 is gated w. prompt, the other gated previews I have run fine in Viewer, e.g. https://huggingface.co/datasets/strombergnlp/shaj , so I'm a bit stumped!

### Owner

Yes

Comments:
Not sure if it's related... I was going to raise an issue for https://huggingface.co/datasets/domenicrosati/TruthfulQA which also has the same issue... https://huggingface.co/datasets/domenicrosati/TruthfulQA/viewer/domenicrosati--TruthfulQA/train 



Yes, it's related. The backend behind the dataset viewer is currently under too much load, and these datasets are still in the jobs queue. We're actively working on this issue, and we expect to fix the issue permanently soon. Thanks for your patience 🙏  

Thanks @severo and no worries! - a suggestion for a UI usability thing maybe is to indicate that the dataset processing is in the job queue (rather than no data?)

Thanks, these are working great now (including @domenicrosati 's, afaics!)
Title: Support >1 PWC dataset per dataset card
State: open
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**
Some datasets cover more than one dataset on PapersWithCode. For example, the OffensEval 2020 challenge involved five languages, and there's one dataset to cover all five datasets, [`strombergnlp/offenseval_2020`](https://huggingface.co/datasets/strombergnlp/offenseval_2020). However, the yaml `paperswithcode_id:` dataset card entry only supports one value; when multiple are added, the PWC link disappears from the dataset page.

Because the link from a PapersWithCode dataset to a Hugging Face Hub entry can't be entered manually and seems to be scraped, this means end users don't have a way of getting a dataset reader link to appear on all the PWC datasets supported by one HF Hub Dataset reader.

It's not super unusual to have papers introduce multiple parallel variants of a dataset and would be handy to reflect this, so e.g. dataset maintainers can DRY, and so dataset users can keep what they're doing simple.

**Describe the solution you'd like**
I'd like `paperswithcode_id:` to support lists and be able to connect with multiple PWC datasets.

**Describe alternatives you've considered**
De-normalising the datasets on HF Hub to create multiple readers for each variation on a task, i.e. instead of a single `offenseval_2020`, having `offenseval_2020_ar`, `offenseval_2020_da`, `offenseval_2020_gr`, ...

**Additional context**
Hope that's enough

**Priority**
Low

Comments:
Hi @leondz, I agree it would be nice. We'll see what we can do ;)
Title: Multi-news dataset loader attempts to strip wrong character from beginning of summaries
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug

The `multi_news.py` data loader has [a line which attempts to strip `"- "` from the beginning of summaries](https://github.com/huggingface/datasets/blob/aa743886221d76afb409d263e1b136e7a71fe2b4/datasets/multi_news/multi_news.py#L97). The actual character in the multi-news dataset, however, is `"– "`, which is different, e.g. `"– " != "- "`.

I would have just opened a PR to fix the mistake, but I am wondering what the motivation for stripping this character is? AFAICT most approaches just leave it in, e.g. the current SOTA on this dataset, [PRIMERA](https://huggingface.co/allenai/PRIMERA-multinews) (you can see its in the generated summaries of the model in their [example notebook](https://github.com/allenai/PRIMER/blob/main/Evaluation_Example.ipynb)).

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Comments:
Hi ! Thanks for reporting :)

This dataset was simply converted from [tensorflow datasets](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/multi_news.py)

I think we can just remove the `.strip("- ")` and keep this character

Cool! I made a PR.
Title: Don't check f.loc in _get_extraction_protocol_with_magic_number
State: closed

`f.loc` doesn't always exist for file-like objects in python. I removed it since it was not necessary anyway (we always seek the file to 0 after reading the magic number)

Fix https://github.com/huggingface/datasets/issues/4310

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix cnn_dailymail (dm stories were ignored)
State: closed

https://github.com/huggingface/datasets/pull/4188 introduced a bug in `datasets` 2.2.0: DailyMail stories are ignored when generating the dataset.

I fixed that, and removed the google drive link (it has annoying quota limitations issues)

We can do a patch release after this is merged

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: [Imagefolder] Docs + Don't infer labels from file names when there are metadata + Error messages when metadata and images aren't linked correctly
State: closed

I updated the `docs/source/image_process.mdx` documentation and added an example for image captioning and object detection using `ImageFolder`.

While doing so I also improved a few aspects:
- we don't need to infer labels from file names when there are metadata - they can just be in the metadata if necessary
- raise informative error messages when metadata and images aren't linked correctly:
  - when an image is missing a metadata file
  - when a metadata file is missing an image

I added some tests for these changes as well

cc @mariosasko 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging this one since mario is off, I took care of adding some tests to make sure everything is fine. Will do the release after it
Title: Fixes FrugalScore
State: open
Labels: name: transfer-to-evaluate

There are two minor modifications in this PR:
1) `predictions` and `references` are swapped. Basically Frugalscore is commutative, however some tiny differences can occur if we swap the references and the predictions. I decided to swap them just to obtain the exact results as reported in the paper.
2) I switched to dynamic padding that was was used in the training, forcing the padding to `max_length`  introduces errors for some reason that I ignore.

@lhoestq 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4305). All of your documentation changes will be reflected on that endpoint.

> predictions and references are swapped. Basically Frugalscore is commutative, however some tiny differences can occur if we swap the references and the predictions. I decided to swap them just to obtain the exact results as reported in the paper.

What is the order of magnitude of the difference ? Do you know what causes this ?

> I switched to dynamic padding that was was used in the training, forcing the padding to max_length introduces errors for some reason that I ignore.

What error ?
Title: Language code search does direct matches
State: open
Labels: description: Something isn't working
name: bug

## Describe the bug

Hi. Searching for bcp47 tags that are just the language prefix (e.g. `sq` or `da`) excludes datasets that have added extra information in their language metadata (e.g. `sq-AL` or `da-bornholm`). The example codes given in the [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging) encourages addition of the additional codes ("_expected format is BCP47 tags separated for ';' e.g. 'en-US;fr-FR'_") but this would lead to those datasets being hidden in datasets search.

## Steps to reproduce the bug
1. Add a dataset using a variant tag (e.g. [`sq-AL`](https://huggingface.co/datasets?languages=languages:sq-AL))
2. Look for datasets using the full code 
3. Note that they're missing when just the language is searched for (e.g. [`sq`](https://huggingface.co/datasets?languages=languages:sq))

Some datasets are already affected by this - e.g. `AmazonScience/massive` is listed under `sq-AL` but not `sq`.

One workaround is for dataset creators to add an additional root language tag to dataset YAML metadata, but it's unclear how to communicate this. It might be possible to index the search on `languagecode.split('-')[0]` but I wanted to float this issue before trying to write any code :)

## Expected results
Datasets using longer bcp47 tags also appear under searches for just the language code; e.g. Quebecois datasets (`fr-CA`) would come up when looking for French datasets with no region specification (`fr`), or US English (`en-US`) datasets would come up when searching for English datasets (`en`).

## Actual results
The language codes seem to be directly string matched, excluding datasets with specific language tags from non-specific searches.

## Environment info
(web app)

Comments:
Thanks for reporting ! I forwarded the issue to the front-end team :)

Will keep you posted !

I also changed the tagging app to suggest two letters code for now.
Title: Remove hacking license tags when mirroring datasets on the Hub
State: closed

Currently, when mirroring datasets on the Hub, the license tags are hacked: removed of characters "." and "$". On the contrary, this hacking is not applied to community datasets on the Hub. This generates multiple variants of the same tag on the Hub. 

I guess this hacking is no longer necessary:
- it is not applied to community datasets
- all canonical datasets are validated by maintainers before being merged: CI + maintainers make sure license tags are the right ones

Fix #4298.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The Hub doesn't allow these characters in the YAML tags, and git push fails if you want to push a dataset card containing these characters.

Ok, let me rename the bad config names :) I think I can also keep backward compatibility with a warning

Almost done with it btw, will submit a PR that shows all the configuration name changes (from a bit more than 20 datasets)

Please, let me know when the renaming of configs is done. If not enough bandwidth, I can take care of it...

Will focus on this this afternoon ;)

I realized when renaming all the configurations with dots in https://github.com/huggingface/datasets/pull/4365 that it's not ideal for certain cases. For example:
- many configurations have a version like "1.0.0" in their names
- to avoid breaking changes we need to replace dots with underscores in the user input and show a warning, which hurts the experience
- our second most downloaded dataset at the moment is affected: `newsgroup`
- if we disallow dots, then we'll never be able to make the [allenai/c4](https://huggingface.co/datasets/allenai/c4) work with its different configurations since they contain dots, and we can't rename them because they are the official download links

I was thinking of other alternatives:
1. just stop separating tags per config name completely, and have a single flat YAML for all configurations. Dataset search doesn't use this info anyway
2. use another YAML structure to avoid having config names as keys, such as
```yaml
languages:
- config: 20220301_en
  values:
  - en
```

I'm down for 1, to keep things simple

@lhoestq I agree:
- better not changing config names (so that we do not introduce any braking change)
- therefore, we should not use them as keys

In relation with the proposed solutions, I have no strong opinion:
- option 1 is simpler and aligns better with current usage on the Hub (configs are ignored)
- however:
  - we will lose all the information per config we already have (for those datasets containing config keys; contributors made an effort to put that information per config)
  - and this information might be useful on the Hub in the future, in case we would like to enrich the search feature with more granularity; this is only applicable if this feature could eventually make sense

So, no strong opinion...

Closing in favor of https://github.com/huggingface/datasets/pull/4367
Title: Add ImageNet-Sketch dataset
State: closed

This PR adds the ImageNet-Sketch dataset and resolves #3953 .

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think you can go ahead with uploading the data, and also ping the author in parallel. I think the images may subject to copyright anyway (scrapped from google image) so the dataset author is not allowed to set a license to the data.

I think it's fine to upload the dataset as soon as we mention explicitly that the images may be subject to copyright.
Title: Add API code examples for loading methods
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

This PR adds API code examples for loading methods, let me know if I've missed any important parameters we should showcase :)

I was a bit confused about `inspect_dataset` and `inspect_metric`. The `path` parameter says it will accept a dataset identifier from the Hub. But when I try the identifier `rotten_tomatoes`, it gives me:

```py
from datasets import inspect_dataset
inspect_dataset('rotten_tomatoes', local_path='/content/rotten_tomatoes')

FileNotFoundError: Couldn't find a dataset script at /content/rotten_tomatoes/rotten_tomatoes.py or any data file in the same directory.
```

Does the user need to have an existing copy of `rotten_tomatoes.py` on their local drive (in which case, it seems like the same option as the first option in `path`)?

Comments:
_The documentation is not available anymore as the PR was closed or merged._