helmo/github-issues huggingface dataset:
Add support for `Audio` and `Image` feature in `push_to_hub`
State: closed | Author: mariosasko (COLLABORATOR)

Add support for the `Audio` and the `Image` feature in `push_to_hub`. 

The idea is to remove local path information and store file content under "bytes" in the Arrow table before the push.

My initial approach (https://github.com/huggingface/datasets/commit/34c652afeff9686b6b8bf4e703c84d2205d670aa) was to use a map transform similar to [`decode_nested_example`](https://github.com/huggingface/datasets/blob/5e0f6068741464f833ff1802e24ecc2064aaea9f/src/datasets/features/features.py#L1023-L1056) while having decoding turned off, but I wasn't satisfied with the code quality, so I ended up using the `temporary_assignment` decorator to override `cast_storage`, which allows me to directly modify the underlying storage (the final op is similar to `Dataset.cast`) and results in a much simpler code. 

Additionally, I added the `allow_cast` flag that can disable this behavior in the situations where it's not needed (e.g. the dataset is already in the correct format for the Hub, etc.)

EDIT:
`allow_cast` renamed to `embed_external_files`

Comments: > Cool thanks !
> 
> Also cc @patrickvonplaten @anton-l it means that when calling push_to_hub, the audio bytes are embedded in the parquet files (we don't upload the audio files themselves)

Just to verify quickly the size of the dataset doesn't change in this case no? E.g. if a dataset has say 20GB in size when stored in `.mp3` format it could have up to 100GB when stored in WAV. But since we are just taking the bytes here a 20GB .mp3 dataset would also have 20GB when stored in parquet no?

@lhoestq  I've addressed your comments. Additionally, I've modified `cast_storage` to account for possible null (`None`) values.

@patrickvonplaten Yes, the dataset size stays the same (at least because Parquet files are compressed).

Feel free to merge if it's all good to you :)
[fix]: iwslt2017 download urls
State: closed | Author: msarmi9 (NONE)
Labels: name: dataset contribution

Fixes #2076.

Comments: Hi ! Thanks for the fix ! Do you know where this new URL comes from ?

Also we try to not use Google Drive if possible, since it has download quota limitations. Do you know if the data is available from another host than Google Drive ?

Oh, I found it just by following the link from the [IWSLT2017 homepage](https://wit3.fbk.eu/2017-01). Not sure if it's available from another host.

Ok cool ! I guess it's ok to use this URL for now, and we can see later if we need to change it.

Before merging, could you update the `dataset_infos.json` file by running this command please ?
```
datasets-cli test ./datasets/iwslt2017 --save_infos --all_configs
```

sure thing. lmk if there's anything else i can do to help.

just checking in. is there anything i can do to help on my end to get this merged? (the dummy data tests are failing due an incorrect path, i think)

Thanks ! I also fixed the dummy data :)

To fix the CI, feel free to merge the `master` branch into your PR.

If you have some time, feel free to also take a look at the missing YAML tags at the top of the README.md file of this dataset:
```
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           missing 9 required tags: 'annotations_creators', 'language_creators', 'languages', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
```
you can use the dataset tagging app here: https://huggingface.co/spaces/huggingface/datasets-tagging

I guess this PR was superseded by this other:
- #4481

Thanks for your contribution anyway, @msarmi9.
adding told-br for toxic/abusive hatespeech detection
State: closed | Author: joaoaleite (CONTRIBUTOR)

Hey, 

I'm adding our dataset from our paper published at AACL 2020. Feel free to ask for modifications.

Thanks!

Comments: Sorry for using multiple github accounts, I didn't notice I was using my professional account to commit/push. Please consider this @JAugusto97 account as the correct one.

Will remake the PR with the correct github account.
Fix TestCommand to move dataset_infos instead of copying
State: closed | Author: albertvillanova (MEMBER)

Why do we copy instead of moving the file?

CC: @lhoestq @lvwerra 

Comments: All the datasets that are loaded normally with `load_dataset`, if `dataset_infos.json` exists, have this file in the importable directory. So it's fine if we copy the file instead of moving it but it's not a big deal.

Any reason to prefer moving it rather than copying it ?

@lvwerra reported than when generating the `dataset_infos.json` for multiple dataset directories containing only JSONL files, subsequent `dataset_infos.json` files contained all previous directories as configs:
- First generate metadata for dataset in dir `dir1`: dataset_infos.json contains one config for `dir1`
- Then generate metadata for dataset in dir `dir2`: dataset_infos.json contains 2 configs, for `dir1` and `dir2`

The reason is that all dataset_infos.json files are first created in the same dir (the one containing the json builder) and then **copied** to the user dir.

Subsequent calls of TestCommand don't replace the dataset_infos.json already present in the dir of the json builder, but append to it.

MAYBE: we should just move for this use case, and copy for the other use cases? See this use case here:
- #3680

@lhoestq aren't you mentioning the case in the else clause?
```python
else:  # in case of a remote dataset
    dataset_dir = None
```

In that case `dataset_infos.json` is not copied: `dataset_dir = None`

When using the JSON loader, calling `get_imported_module_dir()` returns a path inside the pip installed packages, so we shouldn't write files in it anyway, and the dataset_infos.json file should be written directly in the user's directory instead (some users don't have write access to the pip installed packages for example).

Maybe the packaged modules like `json` should override `_save_infos` to save them in the user's directory instead of next to the builder's script. What do you think ?

Anyway as a hotfix we can just add an exception for the `json` builder for now, if the issue has to be fixed soon

I'm closing this PR.
Download datasets from a private hub
State: closed | Author: juliensimon (NONE)
Labels: name: enhancement

name: private-hub

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

The same issue exists with the transformers library and the CLI. I'm going to create issues there as well, and I'll reference them below.

Comments: For reference:
https://github.com/huggingface/transformers/issues/15514
https://github.com/huggingface/huggingface_hub/issues/650

Hi ! For information one can set the environment variable `HF_ENDPOINT` (default is `https://huggingface.co`) if they want to use a private hub.

We may need to coordinate with the other libraries to have a consistent way of changing the hub endpoint

Yes, I tested it successfully this morning. Thanks.
Add CodeContests dataset
State: closed | Author: mariosasko (COLLABORATOR)
Labels: name: dataset request

## Adding a Dataset
- **Name:** CodeContests
- **Description:** CodeContests is a competitive programming dataset for machine-learning.
- **Paper:**
- **Data:** https://github.com/deepmind/code_contests
- **Motivation:** This dataset was used when training [AlphaCode](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: @mariosasko Can I take this up?

This dataset is now available here: https://huggingface.co/datasets/deepmind/code_contests.
Add FrugalScore metric
State: closed | Author: moussaKam (CONTRIBUTOR)

This pull request add FrugalScore metric for NLG systems evaluation.

FrugalScore is a reference-based metric for NLG models evaluation. It is based on a distillation approach that allows to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance.

Paper: https://arxiv.org/abs/2110.08559?context=cs
Github: https://github.com/moussaKam/FrugalScore

@lhoestq 

Comments: @lhoestq 

The model used by default (`moussaKam/frugalscore_tiny_bert-base_bert-score`) is a tiny model.

I still want to make one modification before merging.
I would like to load the model checkpoint once. Do you think it's a good idea if I load it in `_download_and_prepare`? In this case should the model name be the `self.config_name` or another variable say `self.model_name` ? 

OK, I added a commit that loads the checkpoint in `_download_and_prepare`. Please let me know if it looks good. 

@lhoestq is everything OK to merge? 

I triggered the CI and it's failing, can you merge the `master` branch into yours ? It should fix the issues.

Also the doctest apparently raises an error because it outputs `{'scores': [0.6307542, 0.6449357]}` instead of `{'scores': [0.631, 0.645]}` - feel free to edit the code example in the docstring to round the scores, that should fix it

@lhoestq hope it's OK now
`load_dataset("snli")` is different from dataset viewer
State: closed | Author: pietrolesci (NONE)
Labels: name: bug

name: dataset-viewer

## Describe the bug
The dataset that is downloaded from the Hub via `load_dataset("snli")` is different from what is available in the dataset viewer. In the viewer the labels are not encoded (i.e., "neutral", "entailment", "contradiction"), while the downloaded dataset shows the encoded labels (i.e., 0, 1, 2).

Is this expected? 


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Ubuntu 20.4
- Python version: 3.7


Comments: Yes, we decided to replace the encoded label with the corresponding label when possible in the dataset viewer. But
1. maybe it's the wrong default
2. we could find a way to show both (with a switch, or showing both ie. `0 (neutral)`).


Hi @severo,

Thanks for clarifying. 

I think this default is a bit counterintuitive for the user. However, this is a personal opinion that might not be general. I think it is nice to have the actual (non-encoded) labels in the viewer. On the other hand, it would be nice to match what the user sees with what they get when they download a dataset. I don't know - I can see the difficulty of choosing a default :)
Maybe having non-encoded labels as a default can be useful?

Anyway, I think the issue has been addressed. Thanks a lot for your super-quick answer!

 

Thanks for the 👍  in https://github.com/huggingface/datasets/issues/3673#issuecomment-1029008349 @mariosasko @gary149 @pietrolesci, but as I proposed various solutions, it's not clear to me which you prefer. Could you write your preferences as a comment?

_(note for myself: one idea per comment in the future)_

As I am working with seq2seq, I prefer having the label in string form rather than numeric. So the viewer is fine and the underlying dataset should be "decoded" (from int to str). In this way, the user does not have to search for a mapping `int -> original name` (even though is trivial to find, I reckon). Also, encoding labels is rather easy.

I hope this is useful

I like the idea of "0 (neutral)". The label name can even be greyed to make it clear that it's not part of the actual item in the dataset, it's just the meaning.

I like @lhoestq's idea of having grayed-out labels.

Proposals by @gary149. Which one do you prefer? Please vote with the thumbs

- 👍 

    ![image](https://user-images.githubusercontent.com/1676121/152387949-883c7d7e-a9f3-48aa-bff9-11a691555e6e.png)

- 👎 

    ![image (1)](https://user-images.githubusercontent.com/1676121/152388061-32d95e42-cade-4ae4-9a77-7365e7b72b8f.png)



I like Option 1 better as it shows clearly what the user is downloading

Thanks! 

It's [live](https://huggingface.co/datasets/glue/viewer/cola/train):

<img width="1126" alt="Capture d’écran 2022-02-14 à 10 26 03" src="https://user-images.githubusercontent.com/1676121/153836716-25f6205b-96af-42d8-880a-7c09cb24c420.png">

Thanks all for the help to improve the UI!

Love it ! thanks :)
Prioritize `module.builder_kwargs` over defaults in `TestCommand`
State: closed | Author: lvwerra (MEMBER)

This fixes a bug in the `TestCommand` where multiple kwargs for `name` were passed if it was set in both default and `module.builder_kwargs`. Example error:

```Python
Traceback (most recent call last):
  File "create_metadata.py", line 96, in <module>
    main(**vars(args))
  File "create_metadata.py", line 86, in main
    metadata_command.run()
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 144, in run
    for j, builder in enumerate(get_builders()):
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 141, in get_builders
    name=name, cache_dir=self._cache_dir, data_dir=self._data_dir, **module.builder_kwargs
TypeError: type object got multiple values for keyword argument 'name'
```

Let me know what you think.
Give an estimate of the dataset size in DatasetInfo
State: open | Author: severo (COLLABORATOR)
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

Currently, only part of the datasets provide `dataset_size`, `download_size`, `size_in_bytes` (and `num_bytes` and `num_examples` inside `splits`). I would want to get this information, or an estimation, for all the datasets.

**Describe the solution you'd like**

- get access to the git information for the dataset files hosted on the hub
- look at the [`Content-Length`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length) for the files served by HTTP
feat: 🎸 generate info if dataset_infos.json does not exist
State: closed | Author: severo (COLLABORATOR)

in get_dataset_infos(). Also: add the `use_auth_token` parameter, and create get_dataset_config_info()

✅ Closes: #3013

Comments: It's a first attempt at solving https://github.com/huggingface/datasets/issues/3013.

I only kept these ones:
```
    path: str,
    data_files: Optional[Union[Dict, List, str]] = None,
    download_config: Optional[DownloadConfig] = None,
    download_mode: Optional[GenerateMode] = None,
    revision: Optional[Union[str, Version]] = None,
    use_auth_token: Optional[Union[bool, str]] = None,
    **config_kwargs,
```

Let me know if it's better for you now !

(note that there's no breaking change since the ones that are removed can be passed as config_kwargs if you really want)

(https://github.com/huggingface/datasets/pull/3670/commits/5636911880ea4306c27c7f5825fa3f9427ccc2b6 and https://github.com/huggingface/datasets/pull/3670/commits/07c3f0800dd34dfebb9674ad46c67a907b08ded8 -> I has forgotten to update black in my venv)
Common voice validated partition
State: closed | Author: shalymin-amzn (CONTRIBUTOR)

This patch adds access to the 'validated' partitions of CommonVoice datasets (provided by the dataset creators but not available in the HuggingFace interface yet).
As 'validated' contains significantly more data than 'train' (although it contains both test and validation, so one needs to be careful there), it can be useful to train better models where no strict comparison with the previous work is intended.

Comments: Hi @patrickvonplaten - could you please advise whether this would be a welcomed change, and if so, who I consult regarding the unit-tests?

I'd be happy with adding this change. @anton-l @lhoestq - what do you think?

Cool ! I just fixed the tests by adding a dummy `validated.tsv` file in the dummy data archive of common_voice

I wonder if you should separate the train/valid/test configuration from the validated/invalidated configuration of the splits ? 
In particular having `validated` along with the train/valid/test splits could be a bit weird since it comprises them. We can do that if you think it makes more sense. Otherwise it's also good as it is right now :)


Thanks! I think that there are 2 cases for using the validated partition: 1) trainset = {validated - dev - test}, dev and test as they come; 2) train, dev, and test sampled from validated manually with the desired ratios.
In either case, I think that it's quite a big change on the HF interface part, so could as well be taken care of in the client code. Or is it not? (In which case, what's the most compact way to implement this?)

What's important IMO is to let the users as much flexibility as they need - so we try to not do too much regarding splits to not constrain users. So I guess the way it is right now is ok. Can you confirm that it's ok @patrickvonplaten and that it won't break some speech training script out there ?

@lhoestq all split names are explicit in our example scripts, so this shouldn't break anything, feel free to merge :)
I'll go ahead and add this to the official `mozilla-foundation` datasets as well 

Good for me! This has no real down-sides IMO and surely won't break any training scripts.
Process .opus files with torchaudio
State: closed | Author: polinaeterna (CONTRIBUTOR)

@anton-l suggested to proccess .opus files with `torchaudio` instead of `soundfile` as it's faster: 
![opus](https://user-images.githubusercontent.com/16348744/152177816-2df6076c-f28b-4aef-a08d-b499b921414d.png)

(moreover, I didn't manage to load .opus files with `soundfile` / `librosa` locally on any my machine anyway for some reason, even with `ffmpeg` installed).

For now my current changes work with locally stored file:
```python
# download sample opus file (from MultilingualSpokenWords dataset)
!wget https://huggingface.co/datasets/polinaeterna/test_opus/resolve/main/common_voice_tt_17737010.opus 

from datasets import Dataset, Audio

audio_path = "common_voice_tt_17737010.opus"
dataset = Dataset.from_dict({"audio": [audio_path]}).cast_column("audio", Audio(48000))
dataset[0]
# {'audio': {'path': 'common_voice_tt_17737010.opus',
#   'array': array([ 0.0000000e+00,  0.0000000e+00,  3.0517578e-05, ...,
#          -6.1035156e-05,  6.1035156e-05,  0.0000000e+00], dtype=float32),
#   'sampling_rate': 48000}}
```
But it doesn't work when loading inside s dataset from bytes (I checked on [MultilingualSpokenWords](https://github.com/huggingface/datasets/pull/3666), the PR is a draft now, maybe the bug is somewhere there  )

```python
import torchaudio
with open(audio_path, "rb") as b:
    print(torchaudio.load(b))
# RuntimeError: Error loading audio file: failed to open file <in memory buffer>
```

Comments: Note that torchaudio is maybe less practical to use for TF or JAX users.
This is not in the scope of this PR, but in the future if we manage to find a way to let the user control the decoding it would be nice

> Note that torchaudio is maybe less practical to use for TF or JAX users. This is not in the scope of this PR, but in the future if we manage to find a way to let the user control the decoding it would be nice

@lhoestq so maybe don't do this PR? :) if it doesn't work anyway with an opened file, only with path

Yes as discussed offline there seems to be issues with torchaudio on opened files. Feel free to close this PR if it's better to stick with soundfile because of that

We should be able to remove torchaudio, which has torch as a hard dependency, soon and use only soundfile for decoding: https://github.com/bastibe/python-soundfile/issues/252#issuecomment-1000246773 (opus + mp3 support is on the way).
process .opus files (for Multilingual Spoken Words)
State: closed | Author: polinaeterna (CONTRIBUTOR)

Opus files requires `libsndfile>=1.0.30`. Add check for this version and tests.


**outdated:**
Add [Multillingual Spoken Words dataset](https://mlcommons.org/en/multilingual-spoken-words/)

You can specify multiple languages for downloading 😌:
```python
ds = load_dataset("datasets/ml_spoken_words", languages=["ar", "tt"])
```

1. I didn't take into account that each time you pass a set of languages the data for a specific language is downloaded even if it was downloaded before (since these are custom configs like `ar+tt` and `ar+tt+br`. Maybe that wasn't a good idea?
2. The script will have to be slightly changed after merge of https://github.com/huggingface/datasets/pull/3664 
2. Just can't figure out what wrong with dummy files... 😞 Maybe we should get rid of them at some point 😁

Comments: @lhoestq  I still have problems with processing `.opus` files with `soundfile` so I actually cannot fully check that it works but it should... Maybe this should be investigated in case of someone else would also have problems with that.

Also, as the data is in a private repo on the hub (before we come to a decision about audio data privacy), the needed checks cannot be done right now.

@lhoestq I check the data redownloading for configs sharing the same languages, you were right: the data is downloaded once for each language. But samples are generated from scratch each time. Is it a supposed behavior?  

> But samples are generated from scratch each time. Is it a supposed behavior?

Yea that's the way it works right now, because we generate one arrow file per configuration. Since changing the languages creates a new configuration, then it generates a new arrow file.
push_to_hub but preview not working
State: closed | Author: thomas-happify (NONE)
Labels: name: dataset-viewer

## Dataset viewer issue for '*happifyhealth/twitter_pnn*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/happifyhealth/twitter_pnn)*

I used 
```
dataset.push_to_hub("happifyhealth/twitter_pnn")
```
but the preview is not working.

Am I the one who added this dataset ? Yes




Comments: Hi @thomas-happify, please note that the preview may take some time before rendering the data.

I've seen it is already working.

I close this issue. Please feel free to reopen it if the problem arises again.
Extend dataset builder for streaming in `get_dataset_split_names`
State: closed | Author: mariosasko (COLLABORATOR)

Currently, `get_dataset_split_names` doesn't extend a builder module to support streaming, even though it uses `StreamingDownloadManager` to download data. This PR fixes that.

To test the change, run the following:
```bash
pip install git+https://github.com/huggingface/datasets.git@fix-get_dataset_split_names-streaming
python -c "from datasets import get_dataset_split_names; print(get_dataset_split_names('facebook/multilingual_librispeech', 'german', download_mode='force_redownload', revision='137923f945552c6afdd8b60e4a7b43e3088972c1'))"
```

Comments: I'm impatient to see if it has an impact on the number of valid datasets for the dataset viewer. For the record, today:

<img width="660" alt="Capture d’écran 2022-02-01 à 14 32 19" src="https://user-images.githubusercontent.com/1676121/151977579-b5a239d9-6662-4aeb-bfd1-eef6b8249991.png">


This is now available in `datasets` 1.18.3 :)

I'm on it https://github.com/huggingface/datasets-preview-backend/issues/130


The result:
<img width="671" alt="Capture d’écran 2022-02-03 à 23 45 55" src="https://user-images.githubusercontent.com/1676121/152442169-bfdac643-9a00-4901-bfa7-1d60a1679d4b.png">

Not very different. Maybe it fixed issues in the community datasets... But I'm not 100% the two states are comparable (datasets have been created, or updated, meanwhile)