noamaanMulla-03/datasets-issues huggingface dataset:
[WER] Better error message for wer

Currently we have the following problem when using the WER. When the input format to the WER metric is wrong, instead of throwing an error message a word-error-rate is computed which is incorrect. E.g. when doing the following: 


```python
from datasets import load_metric

wer = load_metric("wer")

target_str = ["hello this is nice", "hello the weather is bloomy"]
pred_str = [["hello it's nice"], ["hello it's the weather"]]

print("Wrong:", wer.compute(predictions=pred_str, references=target_str))
print("Correct", wer.compute(predictions=[x[0] for x in pred_str], references=target_str))
```
We get:
```
Wrong: 1.0
Correct 0.5555555555555556
```

meaning that we get a word-error rate for incorrectly passed input formats. We should raise an error here instead so that people don't spent hours fixing a model while it's their incorrect evaluation metric is the problem for a low WER.


Hi ! I don't think this would solve this issue.
Currently it looks like there's a bug that converts the list `["hello it's nice"]` to a string `'["hello it's nice"]'` since this is what the metric expects as input. The conversion is done before the data are passed to `_compute()`.

This is `Value("string").encode_example` that is called to do the conversion. Since `str()` encoding is too permissive we should consider raising an error if the example is not a string (even though it can be converted to string). 

> called

Ah yeah you're right

I just opened https://github.com/huggingface/datasets/pull/3460 to fix that. It now raises an error instead of computing the wrong WER

Thank you - that should be good enough!
Update swahili_news dataset

Please note also: the HuggingFace version at https://huggingface.co/datasets/swahili_news is outdated. An updated version, with deduplicated text and official splits, can be found at https://zenodo.org/record/5514203.

## Adding a Dataset
- **Name:** swahili_news

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Related to:
- bigscience-workshop/data_tooling#107


Labels: name: dataset request
Scifi_TV_Shows: Having trouble getting viewer to find appropriate files

## Dataset viewer issue for '*Science Fiction TV Show Plots Corpus  (Scifi_TV_Shows)*'

**Link:** [link](https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows)

I tried adding both a script (https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/blob/main/Scifi_TV_Shows.py) and some dummy examples (https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/tree/main/dummy), but the viewer still has a 404 error ("Not found. Maybe the cache is missing, or maybe the ressource does not exist."). I'm not sure what to try next. Thanks in advance!

Am I the one who added this dataset? Yes


Labels: name: dataset-viewer

Hi ! I think this is because `iter_archive` doesn't support ZIP files yet. See https://github.com/huggingface/datasets/issues/3272

You can navigate into the archive this way instead:
```python
# in split_generators
data_dir = dl_manager.download_and_extract(url)
train_filepath = os.path.join(data_dir, "all-sci-fi-data-train.txt")
return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={
                "filepath": train_filepath,
            },
        ),
...
])

# in generate_examples
with open(filepath, encoding="utf-8") as f:
    ...
```

It's working: https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/viewer/Scifi_TV_Shows/test

<img width="1494" alt="Capture d’écran 2021-12-21 à 11 23 51" src="https://user-images.githubusercontent.com/1676121/146914068-f4b7225f-42c5-471d-9c73-2adac722162f.png">
Dataset viewer issue for 'liweili/c4_200m'

## Dataset viewer issue for '*liweili/c4_200m*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/liweili/c4_200m)*

*Server Error*
```
Status code:   404
Exception:     Status404Error
Message:       Not found. Maybe the cache is missing, or maybe the ressource does not exist.
```

Am I the one who added this dataset ? Yes


Labels: name: dataset-viewer

Hi ! I think the issue comes from this [line](https://huggingface.co/datasets/liweili/c4_200m/blob/main/c4_200m.py#L87):
```python
path = filepath + "/*.tsv*"
```

You can fix this by doing this instead:
```python
path = os.path.join(filepath, "/*.tsv*")
```

Here is why:

Locally you can append `"/*.tsv*"` to your local path, however it doesn't work in streaming mode, and the dataset viewer does use the streaming mode.
In streaming mode, the download and extract part is done lazily. It means that instead of using local paths, it's still passing around URLs and [chained URLs](https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining)

Therefore in streaming mode, `filepath` is not a local path, but instead is equal to
```python
zip://::https://huggingface.co/datasets/liweili/c4_200m/resolve/main/data.zip
```
The `zip://` part means that we navigate inside the remote ZIP file.

You must use `os.path.join` to navigate inside it and get your TSV files:
```python
>>> os.path.join(filepath, "/*.tsv*")
zip://*.tsv*::https://huggingface.co/datasets/liweili/c4_200m/resolve/main/data.zip
```

`datasets` extends `os.path.join`, `glob.glob`, etc. in your dataset scripts to work with remote files.

hi @lhoestq ! thanks for the tip! i've updated the line of code but it's still not working. am i doing something else wrong? thank you!

Hi ! Your dataset code is all good now :)
```python
In [1]: from datasets import load_dataset

In [2]: d = load_dataset("liweili/c4_200m", streaming=True)
Downloading: 100%|█████████████████████████████████████████████| 2.79k/2.79k [00:00<00:00, 4.83MB/s]
Using custom data configuration default

In [3]: next(iter(d["train"]))
Out[3]: 
{'input': 'Bitcoin is for $7,094 this morning, which CoinDesk says.',
 'output': 'Bitcoin goes for $7,094 this morning, according to CoinDesk.'}
```
Though the viewer doesn't seem to be updated, I'll take a look at what's wrong

thank you @lhoestq! 😄 

It's working

<img width="1424" alt="Capture d’écran 2021-12-21 à 11 24 29" src="https://user-images.githubusercontent.com/1676121/146914238-24bf87c0-c68d-4699-8d6c-fa3065656d1d.png">
[Staging] Update dataset repos automatically on the Hub

Let's have a script that updates the dataset repositories on staging for now. This way we can make sure it works fine before going in prod.

Related to https://github.com/huggingface/datasets/issues/3341

The script runs on each commit on `master`. It checks the datasets that were changed, and it pushes the changes to the corresponding repositories on the Hub.

If there's a new dataset, then a new repository is created.

If the commit is a new release of `datasets`, it also pushes the tag to all the repositories.


do keep us updated on how it's going in staging! cc @SBrandeis 

Sure ! For now it works smoothly. We'll also do a new release today.

I can send you some repos to explore on staging, in case you want to see how they look like after being updated.
For example [swahili_news](https://moon-staging.huggingface.co/datasets/swahili_news/tree/main)
Agnews dataset viewer is not working

## Dataset viewer issue for '*name of the dataset*'

**Link:** https://huggingface.co/datasets/ag_news

Hi there, the `ag_news` dataset viewer is not working.

Am I the one who added this dataset? No


Labels: name: dataset-viewer

Hi ! Thanks for reporting
We've already fixed the code that generates the preview for this dataset, we'll release the fix soon :)

Hi @lhoestq, thanks for your feedback!

Fixed in the viewer.

https://huggingface.co/datasets/ag_news
Preserve all feature types when saving a dataset on the Hub with `push_to_hub`

Currently, if one of the dataset features is of type `ClassLabel`, saving the dataset with `push_to_hub` and reloading the dataset with `load_dataset` will return the feature of type `Value`. To fix this, we should do something similar to `save_to_disk` (which correctly preserves the types) and not only push the parquet files in `push_to_hub`, but also the dataset `info` (stored in a JSON file).

Labels: name: bug

According to this [comment in the forum](https://discuss.huggingface.co/t/save-datasetdict-to-huggingface-hub/12075/8?u=lhoestq), using `push_to_hub` on a dataset with `ClassLabel` can also make the feature simply disappear when it's reloaded !

Maybe we can also fix https://github.com/huggingface/datasets/issues/3035 while working on this because, as pointed out in my initial post, `save_to_disk` also saves the `dataset_info.json`  file.
Push dataset infos.json to Hub

When doing `push_to_hub`, the feature types are lost (see issue https://github.com/huggingface/datasets/issues/3394).
This PR fixes this by also pushing a `dataset_infos.json` file to the Hub, that stores the feature types.

Other minor changes:
- renamed the `___` separator to `--`, since `--` is now disallowed in a name in the back-end.

I tested this feature with datasets like conll2003 that has feature types like `ClassLabel` that were previously lost.

Close https://github.com/huggingface/datasets/issues/3394

I would like to include this in today's release (though not mandatory), so feel free to comment/suggest changes


The change from `___` to `--` was allowed by https://github.com/huggingface/moon-landing/pull/1657
[when Image type will exist] provide a way to get the data as binary + filename

**Is your feature request related to a problem? Please describe.**

When a dataset cell contains a value of type Image (be it from a remote URL, an Array2D/3D, or any other way to represent images), I want to be able to write the image to the disk, with the correct filename, and optionally to know its mimetype, in order to serve it on the web.

Note: this issue would apply exactly the same for the `Audio` type.

**Describe the solution you'd like**

If a "cell" has the type `Image`, provide a way to get the binary content of the file, and the filename, eg as:

```python
  filename: str
  data: bytes
```

**Describe alternatives you've considered**

A way to write the cell to the disk (passing a local directory), and then return the pathname, filename, and mimetype.


Labels: name: enhancement

name: dataset-viewer

@severo, maybe somehow related to this PR ?
- #3129

@severo I'll keep that in mind.

You can track progress on the Image feature in #3163 (still in the early stage).  

Hi ! As discussed with @severo offline it looks like the dataset viewer already supports reading PIL images, so maybe the dataset viewer doesn't need to disable decoding after all

Fixed with https://github.com/huggingface/datasets/pull/3163
Extend support for streaming datasets that use ET.parse

This PR extends the support in streaming mode for datasets that use `ET.parse`, by patching the function.

This PR adds support for streaming mode to datasets:
1. ami
1. assin
1. assin2
1. counter
1. enriched_web_nlg
1. europarl_bilingual
1. hyperpartisan_news_detection
1. polsum
1. qa4mre
1. quail
1. ted_talks_iwslt
1. udhr
1. web_nlg
1. winograd_wsc

CC: @severo
Raise exceptions instead of using assertions for control flow

Motivated by https://github.com/huggingface/transformers/issues/12789 in Transformers, one welcoming change would be replacing assertions with proper exceptions. The only type of assertions we should keep are those used as sanity checks.

Currently, there is a total of 87 files with the `assert` statements (located under `datasets` and `src/datasets`), so when working on this, to manage the PR size, only modify 4-5 files at most before submitting a PR.

Labels: name: good first issue

Adding the remaining tasks for this issue to help new code contributors.  
$ cd src/datasets && ack assert -lc 
- [x] commands/convert.py:1
- [x] arrow_reader.py:3
- [x] load.py:7
- [x] utils/py_utils.py:2
- [x] features/features.py:9
- [x] arrow_writer.py:7
- [x] search.py:6
- [x] table.py:1
- [x] metric.py:3
- [x] tasks/image_classification.py:1
- [x] arrow_dataset.py:17
- [x] fingerprint.py:6
- [x] io/json.py:1
- [x] io/csv.py:1

Hi all,
I am interested in taking up `fingerprint.py`, `search.py`, `arrow_writer.py` and `metric.py`. Will raise a PR soon!

Let me look into `arrow_dataset.py`, `table.py`, `data_files.py` & `features.py` 

All the tasks are completed for this issue. This can be closed.
The rotten_tomatoes dataset of movie reviews contains some reviews in Spanish 

## Describe the bug
See title. I don't think this is intentional and they probably should be removed. If they stay the dataset description should be at least updated to make it clear to the user. 

## Steps to reproduce the bug
Go to the [dataset viewer](https://huggingface.co/datasets/viewer/?dataset=rotten_tomatoes) for the dataset, set the offset to 4160 for the train dataset, and scroll through the results. I found ones at index 4166 and 4173. There's others too (e.g. index 2888) but those two are easy to find like that. 

## Expected results
English movie reviews only. 

## Actual results
Example of a Spanish movie review (4173): 
> "É uma pena que , mais tarde , o próprio filme abandone o tom de paródia e passe a utilizar os mesmos clichês que havia satirizado "



Labels: name: bug

Hi @puzzler10, thanks for reporting.

Please note this dataset is not hosted on Hugging Face Hub. See: 
https://github.com/huggingface/datasets/blob/c8f914473b041833fd47178fa4373cdcb56ac522/datasets/rotten_tomatoes/rotten_tomatoes.py#L42

If there are issues with the source data of a dataset, you should contact the data owners/creators instead. In the homepage associated with this dataset (http://www.cs.cornell.edu/people/pabo/movie-review-data/), you can find the authors of the dataset and how to contact them:
> If you have any questions or comments regarding this site, please send email to Bo Pang or Lillian Lee.

P.S.: Please also note that the example you gave of non-English review is in Portuguese (not Spanish). ;)

Maybe best to just put a quick sentence in the dataset description that highlights this?
Extend support for streaming datasets that use os.walk

This PR extends the support in streaming mode for datasets that use `os.walk`, by patching that function.

This PR adds support for streaming mode to datasets:
1. autshumato
1. code_x_glue_cd_code_to_text
1. code_x_glue_tc_nl_code_search_adv
1. nchlt

CC: @severo 


Nice. I'll update the dataset viewer once merged, and test on these four datasets
Fix overriding of filesystem info

Previously, `BaseCompressedFileFileSystem.info` was overridden and transformed from function to dict.

This generated a bug for filesystem methods that use `self.info()`, like e.g. `fs.isfile()`.

This PR:
- Adds tests for `fs.isfile` (that use `fs.info`).
- Fixes custom `BaseCompressedFileFileSystem.info` by removing its overriding.
Dataset preview is not available (I think for all Hugging Face datasets)

## Dataset viewer issue for '*french_book_reviews*'

**Link:** https://huggingface.co/datasets/Abirate/french_book_reviews

**short description of the issue**
For my dataset, the dataset preview is no longer functional (it used to work: The dataset had been added the day before and it was fine...)  
And, after looking over the datasets, I discovered that this issue affects all Hugging Face datasets (as of yesterday, December 23, 2021, around 10 p.m. (CET)).  
**Am I the one who added this dataset** :  Yes   

**Note**: here a screenshot showing the issue
![Dataset preview is not available for my dataset](https://user-images.githubusercontent.com/66887439/147333078-60734578-420d-4e91-8691-a90afeaa8948.jpg)  

**And here for glue dataset :**  

![Dataset preview is not available for other Hugging Face datasets(glue)](https://user-images.githubusercontent.com/66887439/147333492-26fa530c-befd-4992-8361-70c51397a25a.jpg)



Labels: name: bug

name: dataset-viewer

You're right, we have an issue today with the datasets preview. We're investigating.

It should be fixed now. Thanks for reporting.

Down again. 

Fixed for good.
skip columns which cannot set to specific format when set_format

**Is your feature request related to a problem? Please describe.**
When using `dataset.set_format("torch")`, I must make sure every columns in datasets can convert to `torch`, however, sometimes I want to keep some string columns.

**Describe the solution you'd like**
skip columns which cannot set to specific format when set_format instead of raise an error.


Labels: name: enhancement

You can add columns that you wish to set into `torch` format using `dataset.set_format("torch", ['id', 'abc'])` so that input batch of the transform only contains those columns

Sorry, I miss `output_all_columns` args and thought after `dataset.set_format("torch", columns=columns)` I can only get specific columns I assigned.
Add VCTK

## Adding a Dataset
- **Name:** *VCTK*
- **Description:** *This CSTR VCTK Corpus includes speech data uttered by 110 English speakers with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.*
- **Paper:** Homepage: https://datashare.ed.ac.uk/handle/10283/3443
- **Data:** https://datashare.ed.ac.uk/handle/10283/3443
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/vctk

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request

name: speech

@patrickvonplaten I'd like to take this, if nobody has already done it. I have added datasets before through the datasets sprint, but I feel rusty on the details, so I'll look at the guide as well as similar audio PRs (#1878 in particular comes to mind). If there is any detail I should be aware of please, let me know! Otherwise, I'll try to write up a PR in the coming days.

That sounds great @jaketae - let me know if you need any help i.e. feel free to ping me on a first PR :-)
Use `iter_files` instead of `str(Path(...)` in image dataset

Use `iter_files` in the `beans` and the `cats_vs_dogs` dataset scripts as suggested by @albertvillanova.

Additional changes:
* Fix `iter_files` in `MockDownloadManager` (see this [CI error](https://app.circleci.com/pipelines/github/huggingface/datasets/9247/workflows/2657ff8a-b531-4fd9-a9fc-6541a72e8d83/jobs/57028))
* Add support for `os.path.isdir` and `os.path.isfile` in streaming (`os.path.isfile` is needed in `StreamingDownloadManager`'s `iter_files` to make `cats_vs_dogs` streamable)

TODO:
- [ ] add tests for `xisdir` and `xisfile`


`iter_archive` is about to support ZIP archives. I think we should use this no ?

see #3347 https://github.com/huggingface/datasets/pull/3379

I was interested in the support for isfile/dir in remote.

Anyway, `iter_files` will be available for community users.

I'm not a big fan of having two functions that do the same thing. What do you think ?

They do not do the same thing:
- One iterates over files in a directory
- The other I guess will iterate over the members of an archive

Makes sense ! Sounds good then - sorry for my misunderstanding

Note that `iter_archive` will be more performant for data streaming that `iter_files` thanks to the buffering so maybe in the future we can `iter_archive` for some of these datasets

Yes, @lhoestq I agree with you: once `iter_archive` supports zip files, it will be more suitable than `iter_files` for these 2 datasets.

Anyway, this PR also implements `isfile`/`isdir` in streaming mode, besides fixing `iter_files`. And I'm interested in having those in master.

Maybe, could we merge this PR into master and take note to refactor the datasets to use `iter_archive` once zip is supported?
Other option could be to split this PR into 2...
Add QuALITY dataset

## Adding a Dataset
- **Name:** QuALITY
- **Description:** A challenging question answering with very long contexts (Twitter [thread](https://twitter.com/sleepinyourhat/status/1471225421794529281?s=20))
- **Paper:** No ArXiv link yet, but draft is [here](https://github.com/nyu-mll/quality/blob/main/quality_preprint.pdf)
- **Data:** GitHub repo [here](https://github.com/nyu-mll/quality)
- **Motivation:** This dataset would serve as a nice way to benchmark long-range Transformer models like BigBird, Longformer and their descendants. In particular, it would be very interesting to see how the S4 model fares on this given it's impressive performance on the Long Range Arena

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request

I'll take this one if no one hasn't yet!
Add VoxLingua107

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models. 
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** 107 languages, totaling 6628 hours for the train split.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request
Add KorQPair dataset

This PR adds a [Korean paired question dataset](https://github.com/songys/Question_pair) containing labels indicating whether two questions in a given pair are semantically identical. This dataset was used to evaluate the performance of [KoGPT2](https://github.com/SKT-AI/KoGPT2#subtask-evaluations) on a phrase detection downstream task. 


dummy data is missing

Hey @cceyda, thanks for pointing that out. I thought I'd added it, but seems like that wasn't the case. Just pushed a new commit with the dummy data.
Add Korean NER dataset

Supersedes #1177 

> This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).
Update version of pib dataset and make it streamable

This PR:
- Updates version of pib dataset: from 0.0.0 to 1.3.0
- Makes the dataset streamable

Fix #3491.

CC: @severo 


It seems like there is still an error: `Message:       'TarContainedFile' object has no attribute 'readable'`

https://huggingface.co/datasets/pib/viewer

@severo I was wondering about that...

It works fine when I run it in streaming mode in my terminal:
```python
In [3]: from datasets import load_dataset; ds = load_dataset("pib", "gu-pa", split="train", streaming=True); item = next(iter(ds))

In [4]: item
Out[4]: 
{'translation': {'gu': 'એવો નિર્ણય લેવાયો હતો કે ખંતપૂર્વકની કામગીરી હાથ ધરવા, કાયદેસર અને ટેકનિકલ મૂલ્યાંકન કરવા, વેન્ચર કેપિટલ ઇન્વેસ્ટમેન્ટ સમિતિની બેઠક યોજવા વગેરે એઆઇએફને કરવામાં આવેલ પ્રતિબદ્ધતાના 0.50 ટકા સુધી અને બાકીની રકમ એફએફએસને પૂર્ણ કરવામાં આવશે.',
  'pa': 'ਇਹ ਵੀ ਫੈਸਲਾ ਕੀਤਾ ਗਿਆ ਕਿ ਐੱਫਆਈਆਈ ਅਤੇ ਬਕਾਏ ਲਈ ਕੀਤੀਆਂ ਗਈਆਂ ਵਚਨਬੱਧਤਾਵਾਂ ਦੇ 0.50 % ਦੀ ਸੀਮਾ ਤੱਕ ਐੱਫਈਐੱਸ ਨੂੰ ਮਿਲਿਆ ਜਾਏਗਾ, ਇਸ ਨਾਲ ਉੱਦਮ ਪੂੰਜੀ ਨਿਵੇਸ਼ ਕਮੇਟੀ ਦੀ ਬੈਠਕ ਦਾ ਆਯੋਜਨ ਉਚਿਤ ਸਾਵਧਾਨੀ, ਕਾਨੂੰਨੀ ਅਤੇ ਤਕਨੀਕੀ ਮੁੱਲਾਂਕਣ ਲਈ ਸੰਚਾਲਨ ਖਰਚ ਆਦਿ ਦੀ ਪੂਰਤੀ ਹੋਵੇਗੀ।'}}
```

OK, it works now!

<img width="794" alt="Capture d’écran 2022-01-03 à 15 41 44" src="https://user-images.githubusercontent.com/1676121/147943676-6199d1a9-f288-4350-af96-a7c297ebb743.png">