Minimartzz/github-issues huggingface dataset:
Update: GooAQ - add train/val/test splits
State: closed

[GooAQ](https://github.com/allenai/gooaq) dataset was recently updated after splits were added for the same. This PR contains new updated GooAQ with train/val/test splits and updated README as well.

Comments:
@albertvillanova my tests are failing here:
```
dataset_name = 'gooaq'

    def test_load_dataset(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True, use_local_dummy_data=True)

tests/test_dataset_common.py:234: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_dataset_common.py:187: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
```
When I try loading dataset on local machine it works fine. Any suggestions on how can I avoid this error?

Thanks for the help, @albertvillanova! All tests are passing now.
How to sample every file in a list of files making up a split in a dataset when loading?
State: closed

I am loading a dataset with multiple train, test, and validation files like this:

```
data_files_dict = {
    "train": [train_file1, train_file2],
    "test": [test_file1, test_file2],
    "val": [val_file1, val_file2]
}
dataset = datasets.load_dataset(
    "csv",
    data_files=data_files_dict,
    split=['train[:8]', 'test[:8]', 'val[:8]']
)

```

However, this only selects the first 8 rows from train_file1, test_file1, val_file1, since they are the first files in the lists.

I'm trying to formulate a split argument that can sample from each file specified in my list of files that make up each split.

Is this type of splitting supported? If so, how can I do it?

Comments:
Hi ! This is not possible just with `load_dataset`.

You can do something like this instead:
```python
seed=42
data_files_dict = {
    "train": [train_file1, train_file2],
    "test": [test_file1, test_file2],
    "val": [val_file1, val_file2]
}
dataset = datasets.load_dataset(
    "csv",
    data_files=data_files_dict,
).shuffle(seed=seed)

sample_dataset = {splitname: split.select(range(8)) for splitname, split in dataset.items()}
```

Another alternative is loading each file separately with `split="train[:8]"` and then use `concatenate_datasets` to merge the sample of each file.
Add KS task to SUPERB
State: closed

Add the KS (keyword spotting) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051).

- [s3prl instructions](https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/README.md#ks-keyword-spotting)
- [s3prl implementation](https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/speech_commands/dataset.py)
- [TFDS implementation](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/speech_commands.py)

Some notable quirks:
- The dataset is originally single-archive (train+val+test all in one), but the test set has a "canonical" distribution in a separate archive, which is also used here (see `_split_ks_files()`). 
- The `_background_noise_`/`_silence_` audio files are much longer than others, so they require some sort of slicing for downstream training. I decided to leave the implementation of that up to the users, since TFDS and s3prl take different approaches (either slicing wavs deterministically, or subsampling randomly at runtime)

Related to #2619.

Comments:
thanks a lot for implementing this @anton-l !!

i won't have time to review this while i'm away, so happy for @albertvillanova and @patrickvonplaten to decide when to merge :)

@albertvillanova thanks! Everything should be ready now :)

> The _background_noise_/_silence_ audio files are much longer than others, so they require some sort of slicing for downstream training. I decided to leave the implementation of that up to the users, since TFDS and s3prl take different approaches (either slicing wavs deterministically, or subsampling randomly at runtime)

@anton-l I was thinking that maybe we could give some hints in the dataset card (in a Usage section); something similar as for diarization: https://github.com/huggingface/datasets/blob/master/datasets/superb/README.md#example-of-usage
Note that for diarization it is not yet finished: we have to test it and then provide an end-to-end example: https://github.com/huggingface/datasets/pull/2661/files#r680224909 

@albertvillanova yeah, I'm not sure how to best implement it in pure `datasets` yet. It's something like this, where `sample_noise()` needs to be called from a pytorch batch collator or other framework-specific variant:

```python
def map_to_array(example):
    import soundfile as sf

    speech_array, sample_rate = sf.read(example["file"])
    example["speech"] = speech_array
    example["sample_rate"] = sample_rate
    return example


def sample_noise(example):
    # Use a version of this function in a stateless way to extract random 1 sec slices of background noise
    # on each epoch
    from random import randint

    # _silence_ audios are longer than 1 sec
    if example["label"] == "_silence_":
        random_offset = randint(0, len(example["speech"]) - example["sample_rate"] - 1)
        example["speech"] = example["speech"][random_offset : random_offset + example["sample_rate"]]

    return example
```

I see... Yes, not trivial indeed. Maybe for the moment you could add those functions above to the README (as it is the case for now in diarization)? What do you think?
Fix renaming of corpus_bleu args
State: closed

Last `sacrebleu` release (v2.0.0) has renamed `sacrebleu.corpus_bleu` args from `(sys_stream, ref_streams)` to `(hipotheses, references)`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR passes the args without parameter names, so that it is valid for all versions of `sacrebleu`.

This is a partial hotfix of #2781.

Close #2781.

Comments:
Latest v2.0.0 release of sacrebleu has broken some metrics
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
After `sacrebleu` v2.0.0 release (see changes here: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15), some of `datasets` metrics are broken:
- Default tokenizer `sacrebleu.DEFAULT_TOKENIZER` no longer exists:
  - #2739
  - #2778
- Bleu tokenizers are no longer accessible with `sacrebleu.TOKENIZERS`:
  - #2779
- `corpus_bleu` args have been renamed from `(sys_stream, ref_streams)` to `(hipotheses, references)`: 
  - #2782 

Comments:
Fix sacrebleu tokenizers
State: closed

Last `sacrebleu` release (v2.0.0) has removed `sacrebleu.TOKENIZERS`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR makes a hot fix of the bug by using a private function in `sacrebleu`: `sacrebleu.metrics.bleu._get_tokenizer()`.

Eventually, this should be further fixed in order to use only public functions.

This is a partial hotfix of #2781.

Comments:
Do not pass tokenize to sacrebleu
State: closed

Last `sacrebleu` release (v2.0.0) has removed `sacrebleu.DEFAULT_TOKENIZER`: https://github.com/mjpost/sacrebleu/pull/152/files#diff-2553a315bb1f7e68c9c1b00d56eaeb74f5205aeb3a189bc3e527b122c6078795L17-R15

This PR does not pass `tokenize` to `sacrebleu` (note that the user cannot pass it anyway) and `sacrebleu` will use its default, no matter where it is and how it is called.

Related to #2739.

This is a partial hotfix of #2781.

Comments:
document `config.HF_DATASETS_OFFLINE` and precedence
State: open
Labels: description: New feature or request
name: enhancement

https://github.com/huggingface/datasets/pull/1976 implemented `HF_DATASETS_OFFLINE`, but:
1. `config.HF_DATASETS_OFFLINE` is not documented
2. the precedence is not documented (env, config)

I'm thinking it probably should be similar to what it says https://huggingface.co/docs/datasets/loading_datasets.html#from-the-huggingface-hub about `datasets.config.IN_MEMORY_MAX_SIZE`:

Quote:
> The default in 🤗 Datasets is to memory-map the dataset on disk unless you set datasets.config.IN_MEMORY_MAX_SIZE different from 0 bytes (default). In that case, the dataset will be copied in-memory if its size is smaller than datasets.config.IN_MEMORY_MAX_SIZE bytes, and memory-mapped otherwise. This behavior can be enabled by setting either the configuration option datasets.config.IN_MEMORY_MAX_SIZE (higher precedence) or the environment variable HF_DATASETS_IN_MEMORY_MAX_SIZE (lower precedence) to nonzero.

Context: trying to use `config.HF_DATASETS_OFFLINE` here:
https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/48
but are uncertain if it's safe, since it's not documented as a public API.

Thank you!

@lhoestq, @albertvillanova 

Comments:
Remove dataset_infos.json
State: closed
Labels: description: New feature or request
name: enhancement

description: Generic discussion on the library
name: generic discussion

**Is your feature request related to a problem? Please describe.**
As discussed, there are infos in the `dataset_infos.json` which are redundant and we could have them only in the README file.

Others could be migrated to the README, like: "dataset_size", "size_in_bytes", "download_size", "splits.split_name.[num_bytes, num_examples]",...

However, there are others that do not seem too meaningful in the README, like the checksums.

**Describe the solution you'd like**
Open a discussion to decide what to do with the `dataset_infos.json` files: which information to be migrated and/or which information to be kept.

cc: @julien-c @lhoestq 

Comments:
This was closed by:
- #4926
Remove returned feature constrain
State: open
Labels: description: New feature or request
name: enhancement

In the current version, the returned value of the map function has to be list or ndarray. However, this makes it unsuitable for many tasks. In NLP, many features are sparse like verb words, noun chunks, if we want to assign different values to different words, which will result in a large sparse matrix if we only score useful words like verb words.  

Mostly, when using it on large scale, saving it as a whole takes a lot of disk storage and making it hard to read,  the normal method is saving it in sparse form. However, the NumPy does not support sparse, therefore I have to use PyTorch or scipy to transform a matrix into special sparse form, which is not a form that can be transformed into list or ndarry. This violates the feature constraints of the map function. 

I do appreciate the convenience of Datasets package, but I do not think the compulsory datatype constrain is necessary, in some cases, we just cannot transform it into a list or ndarray due to some reasons.  Any way to fix this? Or what I can do to disable the compulsory datatype constrain?


Comments: