josteece/github-issues huggingface dataset:
Add CodeContests dataset
Opened by mariosasko (closed)

## Adding a Dataset
- **Name:** CodeContests
- **Description:** CodeContests is a competitive programming dataset for machine-learning.
- **Paper:**
- **Data:** https://github.com/deepmind/code_contests
- **Motivation:** This dataset was used when training [AlphaCode](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
@mariosasko Can I take this up?

This dataset is now available here: https://huggingface.co/datasets/deepmind/code_contests.
Add FrugalScore metric
Opened by moussaKam (closed)

This pull request add FrugalScore metric for NLG systems evaluation.

FrugalScore is a reference-based metric for NLG models evaluation. It is based on a distillation approach that allows to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance.

Paper: https://arxiv.org/abs/2110.08559?context=cs
Github: https://github.com/moussaKam/FrugalScore

@lhoestq 

Comments:
@lhoestq 

The model used by default (`moussaKam/frugalscore_tiny_bert-base_bert-score`) is a tiny model.

I still want to make one modification before merging.
I would like to load the model checkpoint once. Do you think it's a good idea if I load it in `_download_and_prepare`? In this case should the model name be the `self.config_name` or another variable say `self.model_name` ? 

OK, I added a commit that loads the checkpoint in `_download_and_prepare`. Please let me know if it looks good. 

@lhoestq is everything OK to merge? 

I triggered the CI and it's failing, can you merge the `master` branch into yours ? It should fix the issues.

Also the doctest apparently raises an error because it outputs `{'scores': [0.6307542, 0.6449357]}` instead of `{'scores': [0.631, 0.645]}` - feel free to edit the code example in the docstring to round the scores, that should fix it

@lhoestq hope it's OK now
`load_dataset("snli")` is different from dataset viewer
Opened by pietrolesci (closed)

## Describe the bug
The dataset that is downloaded from the Hub via `load_dataset("snli")` is different from what is available in the dataset viewer. In the viewer the labels are not encoded (i.e., "neutral", "entailment", "contradiction"), while the downloaded dataset shows the encoded labels (i.e., 0, 1, 2).

Is this expected? 


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Ubuntu 20.4
- Python version: 3.7


Comments:
Yes, we decided to replace the encoded label with the corresponding label when possible in the dataset viewer. But
1. maybe it's the wrong default
2. we could find a way to show both (with a switch, or showing both ie. `0 (neutral)`).


Hi @severo,

Thanks for clarifying. 

I think this default is a bit counterintuitive for the user. However, this is a personal opinion that might not be general. I think it is nice to have the actual (non-encoded) labels in the viewer. On the other hand, it would be nice to match what the user sees with what they get when they download a dataset. I don't know - I can see the difficulty of choosing a default :)
Maybe having non-encoded labels as a default can be useful?

Anyway, I think the issue has been addressed. Thanks a lot for your super-quick answer!

 

Thanks for the 👍  in https://github.com/huggingface/datasets/issues/3673#issuecomment-1029008349 @mariosasko @gary149 @pietrolesci, but as I proposed various solutions, it's not clear to me which you prefer. Could you write your preferences as a comment?

_(note for myself: one idea per comment in the future)_

As I am working with seq2seq, I prefer having the label in string form rather than numeric. So the viewer is fine and the underlying dataset should be "decoded" (from int to str). In this way, the user does not have to search for a mapping `int -> original name` (even though is trivial to find, I reckon). Also, encoding labels is rather easy.

I hope this is useful

I like the idea of "0 (neutral)". The label name can even be greyed to make it clear that it's not part of the actual item in the dataset, it's just the meaning.

I like @lhoestq's idea of having grayed-out labels.

Proposals by @gary149. Which one do you prefer? Please vote with the thumbs

- 👍 

    ![image](https://user-images.githubusercontent.com/1676121/152387949-883c7d7e-a9f3-48aa-bff9-11a691555e6e.png)

- 👎 

    ![image (1)](https://user-images.githubusercontent.com/1676121/152388061-32d95e42-cade-4ae4-9a77-7365e7b72b8f.png)



I like Option 1 better as it shows clearly what the user is downloading

Thanks! 

It's [live](https://huggingface.co/datasets/glue/viewer/cola/train):

<img width="1126" alt="Capture d’écran 2022-02-14 à 10 26 03" src="https://user-images.githubusercontent.com/1676121/153836716-25f6205b-96af-42d8-880a-7c09cb24c420.png">

Thanks all for the help to improve the UI!

Love it ! thanks :)
Prioritize `module.builder_kwargs` over defaults in `TestCommand`
Opened by lvwerra (closed)

This fixes a bug in the `TestCommand` where multiple kwargs for `name` were passed if it was set in both default and `module.builder_kwargs`. Example error:

```Python
Traceback (most recent call last):
  File "create_metadata.py", line 96, in <module>
    main(**vars(args))
  File "create_metadata.py", line 86, in main
    metadata_command.run()
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 144, in run
    for j, builder in enumerate(get_builders()):
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 141, in get_builders
    name=name, cache_dir=self._cache_dir, data_dir=self._data_dir, **module.builder_kwargs
TypeError: type object got multiple values for keyword argument 'name'
```

Let me know what you think.

Comments:
Give an estimate of the dataset size in DatasetInfo
Opened by severo (open)

**Is your feature request related to a problem? Please describe.**

Currently, only part of the datasets provide `dataset_size`, `download_size`, `size_in_bytes` (and `num_bytes` and `num_examples` inside `splits`). I would want to get this information, or an estimation, for all the datasets.

**Describe the solution you'd like**

- get access to the git information for the dataset files hosted on the hub
- look at the [`Content-Length`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length) for the files served by HTTP


Comments:
feat: 🎸 generate info if dataset_infos.json does not exist
Opened by severo (closed)

in get_dataset_infos(). Also: add the `use_auth_token` parameter, and create get_dataset_config_info()

✅ Closes: #3013

Comments:
It's a first attempt at solving https://github.com/huggingface/datasets/issues/3013.

I only kept these ones:
```
    path: str,
    data_files: Optional[Union[Dict, List, str]] = None,
    download_config: Optional[DownloadConfig] = None,
    download_mode: Optional[GenerateMode] = None,
    revision: Optional[Union[str, Version]] = None,
    use_auth_token: Optional[Union[bool, str]] = None,
    **config_kwargs,
```

Let me know if it's better for you now !

(note that there's no breaking change since the ones that are removed can be passed as config_kwargs if you really want)

(https://github.com/huggingface/datasets/pull/3670/commits/5636911880ea4306c27c7f5825fa3f9427ccc2b6 and https://github.com/huggingface/datasets/pull/3670/commits/07c3f0800dd34dfebb9674ad46c67a907b08ded8 -> I has forgotten to update black in my venv)
Common voice validated partition
Opened by shalymin-amzn (closed)

This patch adds access to the 'validated' partitions of CommonVoice datasets (provided by the dataset creators but not available in the HuggingFace interface yet).
As 'validated' contains significantly more data than 'train' (although it contains both test and validation, so one needs to be careful there), it can be useful to train better models where no strict comparison with the previous work is intended.

Comments:
Hi @patrickvonplaten - could you please advise whether this would be a welcomed change, and if so, who I consult regarding the unit-tests?

I'd be happy with adding this change. @anton-l @lhoestq - what do you think?

Cool ! I just fixed the tests by adding a dummy `validated.tsv` file in the dummy data archive of common_voice

I wonder if you should separate the train/valid/test configuration from the validated/invalidated configuration of the splits ? 
In particular having `validated` along with the train/valid/test splits could be a bit weird since it comprises them. We can do that if you think it makes more sense. Otherwise it's also good as it is right now :)


Thanks! I think that there are 2 cases for using the validated partition: 1) trainset = {validated - dev - test}, dev and test as they come; 2) train, dev, and test sampled from validated manually with the desired ratios.
In either case, I think that it's quite a big change on the HF interface part, so could as well be taken care of in the client code. Or is it not? (In which case, what's the most compact way to implement this?)

What's important IMO is to let the users as much flexibility as they need - so we try to not do too much regarding splits to not constrain users. So I guess the way it is right now is ok. Can you confirm that it's ok @patrickvonplaten and that it won't break some speech training script out there ?

@lhoestq all split names are explicit in our example scripts, so this shouldn't break anything, feel free to merge :)
I'll go ahead and add this to the official `mozilla-foundation` datasets as well 

Good for me! This has no real down-sides IMO and surely won't break any training scripts.
Process .opus files with torchaudio
Opened by polinaeterna (closed)

@anton-l suggested to proccess .opus files with `torchaudio` instead of `soundfile` as it's faster: 
![opus](https://user-images.githubusercontent.com/16348744/152177816-2df6076c-f28b-4aef-a08d-b499b921414d.png)

(moreover, I didn't manage to load .opus files with `soundfile` / `librosa` locally on any my machine anyway for some reason, even with `ffmpeg` installed).

For now my current changes work with locally stored file:
```python
# download sample opus file (from MultilingualSpokenWords dataset)
!wget https://huggingface.co/datasets/polinaeterna/test_opus/resolve/main/common_voice_tt_17737010.opus 

from datasets import Dataset, Audio

audio_path = "common_voice_tt_17737010.opus"
dataset = Dataset.from_dict({"audio": [audio_path]}).cast_column("audio", Audio(48000))
dataset[0]
# {'audio': {'path': 'common_voice_tt_17737010.opus',
#   'array': array([ 0.0000000e+00,  0.0000000e+00,  3.0517578e-05, ...,
#          -6.1035156e-05,  6.1035156e-05,  0.0000000e+00], dtype=float32),
#   'sampling_rate': 48000}}
```
But it doesn't work when loading inside s dataset from bytes (I checked on [MultilingualSpokenWords](https://github.com/huggingface/datasets/pull/3666), the PR is a draft now, maybe the bug is somewhere there  )

```python
import torchaudio
with open(audio_path, "rb") as b:
    print(torchaudio.load(b))
# RuntimeError: Error loading audio file: failed to open file <in memory buffer>
```

Comments:
Note that torchaudio is maybe less practical to use for TF or JAX users.
This is not in the scope of this PR, but in the future if we manage to find a way to let the user control the decoding it would be nice

> Note that torchaudio is maybe less practical to use for TF or JAX users. This is not in the scope of this PR, but in the future if we manage to find a way to let the user control the decoding it would be nice

@lhoestq so maybe don't do this PR? :) if it doesn't work anyway with an opened file, only with path

Yes as discussed offline there seems to be issues with torchaudio on opened files. Feel free to close this PR if it's better to stick with soundfile because of that

We should be able to remove torchaudio, which has torch as a hard dependency, soon and use only soundfile for decoding: https://github.com/bastibe/python-soundfile/issues/252#issuecomment-1000246773 (opus + mp3 support is on the way).
process .opus files (for Multilingual Spoken Words)
Opened by polinaeterna (closed)

Opus files requires `libsndfile>=1.0.30`. Add check for this version and tests.


**outdated:**
Add [Multillingual Spoken Words dataset](https://mlcommons.org/en/multilingual-spoken-words/)

You can specify multiple languages for downloading 😌:
```python
ds = load_dataset("datasets/ml_spoken_words", languages=["ar", "tt"])
```

1. I didn't take into account that each time you pass a set of languages the data for a specific language is downloaded even if it was downloaded before (since these are custom configs like `ar+tt` and `ar+tt+br`. Maybe that wasn't a good idea?
2. The script will have to be slightly changed after merge of https://github.com/huggingface/datasets/pull/3664 
2. Just can't figure out what wrong with dummy files... 😞 Maybe we should get rid of them at some point 😁

Comments:
@lhoestq  I still have problems with processing `.opus` files with `soundfile` so I actually cannot fully check that it works but it should... Maybe this should be investigated in case of someone else would also have problems with that.

Also, as the data is in a private repo on the hub (before we come to a decision about audio data privacy), the needed checks cannot be done right now.

@lhoestq I check the data redownloading for configs sharing the same languages, you were right: the data is downloaded once for each language. But samples are generated from scratch each time. Is it a supposed behavior?  

> But samples are generated from scratch each time. Is it a supposed behavior?

Yea that's the way it works right now, because we generate one arrow file per configuration. Since changing the languages creates a new configuration, then it generates a new arrow file.
push_to_hub but preview not working
Opened by thomas-happify (closed)

## Dataset viewer issue for '*happifyhealth/twitter_pnn*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/happifyhealth/twitter_pnn)*

I used 
```
dataset.push_to_hub("happifyhealth/twitter_pnn")
```
but the preview is not working.

Am I the one who added this dataset ? Yes




Comments:
Hi @thomas-happify, please note that the preview may take some time before rendering the data.

I've seen it is already working.

I close this issue. Please feel free to reopen it if the problem arises again.
Extend dataset builder for streaming in `get_dataset_split_names`
Opened by mariosasko (closed)

Currently, `get_dataset_split_names` doesn't extend a builder module to support streaming, even though it uses `StreamingDownloadManager` to download data. This PR fixes that.

To test the change, run the following:
```bash
pip install git+https://github.com/huggingface/datasets.git@fix-get_dataset_split_names-streaming
python -c "from datasets import get_dataset_split_names; print(get_dataset_split_names('facebook/multilingual_librispeech', 'german', download_mode='force_redownload', revision='137923f945552c6afdd8b60e4a7b43e3088972c1'))"
```

Comments:
I'm impatient to see if it has an impact on the number of valid datasets for the dataset viewer. For the record, today:

<img width="660" alt="Capture d’écran 2022-02-01 à 14 32 19" src="https://user-images.githubusercontent.com/1676121/151977579-b5a239d9-6662-4aeb-bfd1-eef6b8249991.png">


This is now available in `datasets` 1.18.3 :)

I'm on it https://github.com/huggingface/datasets-preview-backend/issues/130


The result:
<img width="671" alt="Capture d’écran 2022-02-03 à 23 45 55" src="https://user-images.githubusercontent.com/1676121/152442169-bfdac643-9a00-4901-bfa7-1d60a1679d4b.png">

Not very different. Maybe it fixed issues in the community datasets... But I'm not 100% the two states are comparable (datasets have been created, or updated, meanwhile)
Pubmed dataset not reachable
Opened by abhi-mosaic (closed)

## Describe the bug
Trying to use the `pubmed` dataset fails to reach / download the source files.

## Steps to reproduce the bug
```python
pubmed_train = datasets.load_dataset('pubmed', split='train')
```

## Expected results
Should begin downloading the pubmed dataset.

## Actual results
```
ConnectionError: Couldn't reach ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0865.xml.gz (InvalidSchema("No connection adapters were found for 'ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0865.xml.gz'"))
```

## Environment info
- `datasets` version: 1.18.2
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.2
- PyArrow version: 6.0.0


Comments:
Hi @abhi-mosaic, thanks for reporting.

I'm looking at it... 

also hitting this issue

Hey @albertvillanova, sorry to reopen this... I can confirm that on `master` branch the dataset is downloadable now but it is still broken in streaming mode:

```python
 >>> import datasets
 >>> pubmed_train = datasets.load_dataset('pubmed', split='train', streaming=True)
 >>> next(iter(pubmed_train))
```
```
 No such file or directory: 'gzip://pubmed22n0001.xml::ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n0001.xml.gz'
```


Hi @abhi-mosaic, would you mind opening another issue for this new problem?

First issue (already solved) was a ConnectionError due to the yearly update release of PubMed: we fixed it by updating the URLs from year 2021 to year 2022.

However this is another problem: to make pubmed streamable. Please note that NOT all our datastes are streamable: we are making streamable more and more of them... but this is an on-going process...

Thanks.

@albertvillanova 
When I tried below codes, I got the similar error

```

dataset=load_dataset("pubmed",split="train")

Couldn't reach ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0601.xml.gz
```

@y-rok you need to update `datasets`:
```shell
pip install -U datasets
```
Better TQDM output
Opened by mariosasko (closed)

This PR does the following:
* if `dataset_infos.json` exists for a dataset, uses `num_examples` to print the total number of examples that needs to be generated (in `builder.py`)
* fixes `tqdm` + multiprocessing in Jupyter Notebook/Colab (the issue stems from this commit in the `tqdm` repo: https://github.com/tqdm/tqdm/commit/f7722edecc3010cb35cc1c923ac4850a76336f82) 
* adds the missing `drop_last_batch` and `with_ranks` params to `DatasetDict.map` 
* correctly computes the number of iterations in `map` and the CSV/JSON loader when `batched=True` to fix `tqdm` progress bars
* removes the `bool(logging.get_verbosity() == logging.NOTSET)` (or simplifies `bool(logging.get_verbosity() == logging.NOTSET) or not utils.is_progress_bar_enabled()` to `not utils.is_progress_bar_enabled()`)  condition and uses `utils.is_progress_bar_enabled` to check if `tqdm` output is enabled (this comment from @stas00 explains why  the  `bool(logging.get_verbosity() == logging.NOTSET)` check is problematic: https://github.com/huggingface/transformers/issues/14889#issue-1087318463)

Fix #2630

Comments:
@lhoestq I've created a notebook for you to see the difference: https://colab.research.google.com/drive/1by3EqnoKvC2p-yKW4lPDGOFOZHyGVyeQ?usp=sharing.

Feel free to suggest better descriptions for the progress bars. 

If everything looks good, think we can merge.
`to_json` in multiprocessing fashion sometimes deadlock
Opened by thomasw21 (open)

## Describe the bug

`to_json` in multiprocessing fashion sometimes deadlock, instead of raising exceptions. Temporary solution is to see that it deadlocks, and then reduce the number of processes or batch size in order to reduce the memory footprint.

As @lhoestq pointed out, this might be related to https://bugs.python.org/issue22393#msg315684 where `multiprocessing` fails to raise the OOM exception. One suggested alternative is not use `concurrent.futures` instead.

## Steps to reproduce the bug

## Expected results

Script fails when one worker hits OOM, and raise appropriate error.

## Actual results

Deadlock

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.1
- Platform: Linux
- Python version: 3.8
- PyArrow version: 6.0.1


Comments:
Update link in wiki_bio dataset
Opened by jxmorris12 (closed)

Fixes #3580 and makes the wiki_bio dataset work again. I changed the link and some documentation, and all the tests pass. Thanks @lhoestq for uploading the dataset to the HuggingFace data bucket.

@lhoestq -- all the tests pass, but I'm still not able to import the dataset, as the old Google Drive link is cached somewhere:
```python
>>> from datasets import load_dataset
load_dataset("wiki_bio>>> load_dataset("wiki_bio")
Using custom data configuration default
Downloading and preparing dataset wiki_bio/default (download: 318.53 MiB, generated: 736.94 MiB, post-processed: Unknown size, total: 1.03 GiB) to /home/jxm3/.cache/huggingface/datasets/wiki_bio/default/1.1.0/5293ce565954ba965dada626f1e79684e98172d950371d266bf3caaf87e911c9...
Traceback (most recent call last):
  ...
  File "/home/jxm3/random/datasets/src/datasets/utils/file_utils.py", line 612, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://drive.google.com/uc?export=download&id=1L7aoUXzHPzyzQ0ns4ApBbYepsjFOtXil
```

what do I have to do to invalidate the cache and actually import the dataset? It's clearly set up correctly, since the data is downloaded and processed by the tests.

As an aside, this caching-loading-scripts behavior makes for a really bad developer experience. I just wasted an hour trying to figure out where the caching was happening and how to disable it, and I don't know. All I wanted to do was update the link and submit a pull request! I recommend that you all either change this behavior (i.e. updating the link to a dataset should "just work") or document it, since I couldn't find any information about this in the contributing.md or readme or anywhere else! Thanks!

Comments:
> all the tests pass, but I'm still not able to import the dataset

Since it's not merged on `master` yet, you have to provide the path to your local `wiki_bio.py` to use it.
Indeed the library downloads the dataset files from `master` if you have a dev installation of the library.

I agree it would be nice to change that, and use the local dataset scripts from the `datasets` directory - it feels definitely more natural.

Cool, thanks for your help and I agree!