helmo/github-issues huggingface dataset:
Support Zstandard compressed files
State: closed | Author: albertvillanova (MEMBER)
Labels: name: enhancement

Add support for Zstandard compressed files: https://facebook.github.io/zstd/

Comments: I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

```
!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)

WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported
```

I just tried on google colab and this works:
```python
!pip install zstandard
!pip install datasets
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(lds))
```

Can you check that you have a correct installation of `zstandard` ?

@lhoestq please note [this](https://github.com/huggingface/datasets/issues/2572#issuecomment-1363718916) is a duplicate of:
- #5388

Oh thanks I missed that one !

> I just tried on google colab and this works:
> 
> ```python
> !pip install zstandard
> !pip install datasets
> from datasets import load_dataset
> 
> lds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(lds))
> ```
> 
> Can you check that you have a correct installation of `zstandard` ?

I was downloading datasets first then was doing zstandard installation and that was causing the issue. This was highlighted by the Hugging Face staff and that helped. Now the issue is resolved. Thank you.
Weights of model checkpoint not initialized for RobertaModel for Bertscore
State: closed | Author: suzyahyah (NONE)
Labels: name: bug

When applying bertscore out of the box, 

```Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight']```

Following the typical usage from https://huggingface.co/docs/datasets/loading_metrics.html

```
from datasets import load_metric
metric = load_metric('bertscore')

# Example of typical usage
for batch in dataset:
    inputs, references = batch
    predictions = model(inputs)
    metric.add_batch(predictions=predictions, references=references)
score = metric.compute(lang="en")
#score = metric.compute(model_type="roberta-large") # gives the same error
```

I am concerned about this because my usage shouldn't require any further fine-tuning and most people would expect to use BertScore out of the box? I realised the huggingface code is a wrapper around https://github.com/Tiiiger/bert_score, but I think this repo is anyway relying on the model code and weights from huggingface repo.... 

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1041-aws-x86_64-with-glibc2.27
- Python version:  3.9.5
- PyArrow version: 3.0.0


Comments: Hi @suzyahyah, thanks for reporting.

The message you get is indeed not an error message, but a warning coming from Hugging Face `transformers`. The complete warning message is:
```
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

In this case, this behavior IS expected and you can safely ignore the warning message.

The reason is that you are just using RoBERTa to get the contextual embeddings of the input sentences/tokens, thus leaving away its head layer, whose weights are ignored.

Feel free to reopen this issue if you need further explanations.

Hi @suzyahyah, I have created a Pull Request to filter out that warning message in this specific case, since the behavior is as expected and the warning message can only cause confusion for users (as in your case).
Add interleave_datasets for map-style datasets
State: closed | Author: lhoestq (MEMBER)

### Add interleave_datasets for map-style datasets

Add support for map-style datasets (i.e. `Dataset` objects) in `interleave_datasets`.
It was only supporting iterable datasets (i.e. `IterableDataset` objects).

### Implementation details

It works by concatenating the datasets and then re-order the indices to make the new dataset.

### TODO
- [x] tests
- [x] docs

Close #2563
Add ASR task and new languages to resources
State: closed | Author: lewtun (MEMBER)

This PR adds a new `automatic-speech-recognition` task to the list of supported tasks in `tasks.json` and also includes a few new languages missing from `common_voice`.

Note: I used the [Papers with Code list](https://www.paperswithcode.com/area/speech/speech-recognition) as inspiration for the ASR subtasks
fix Dataset.map when num_procs > num rows
State: closed | Author: connor-mccarthy (CONTRIBUTOR)

closes #2470

## Testing notes
To run updated tests:
```sh
pytest tests/test_arrow_dataset.py -k "BaseDatasetTest and test_map_multiprocessing" -s
```
With Python code (to view warning):
```python
from datasets import Dataset


dataset = Dataset.from_dict({"x": ["sample"]})
print(len(dataset))
dataset.map(lambda x: x, num_proc=10)
```
Inject templates for ASR datasets
State: closed | Author: lewtun (MEMBER)

This PR adds ASR templates for 5 of the most common speech datasets on the Hub, where "common" is defined by the number of models trained on them.

I also fixed a bunch of the tags in the READMEs 😎 

Comments: Wait until #2567 is merged so we can benefit from the tagger :)

thanks for the feedback @lhoestq! i've added the new language codes and this PR should be ready for a merge :)
concatenate_datasets for iterable datasets
State: closed | Author: lhoestq (MEMBER)

Currently `concatenate_datasets` only works for map-style `Dataset`.

It would be nice to have it work for `IterableDataset` objects as well.

It would simply chain the iterables of the iterable datasets.

Comments: It is probably worth noting here that the [documentation](https://huggingface.co/docs/datasets/process#concatenate) is misleading (indicating that it does work for IterableDatasets):

> You can also mix several datasets together by taking alternating examples from each one to create a new dataset. This is known as interleaving, and you can use it with [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets). **Both [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets) and [concatenate_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.concatenate_datasets) will work with regular [Dataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset) and [IterableDataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.IterableDataset) objects**. Refer to the [Stream](https://huggingface.co/docs/datasets/stream#interleave) section for an example of how it’s used. 

Thanks for the heads up, I'll fix that
Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`
State: closed | Author: apsdehal (CONTRIBUTOR)
Labels: name: bug

## Describe the bug
If i have local file defining a dataset builder class and I load it using `load_dataset` functionality, the existing cache is ignored whenever the file is update even with `ignore_verifications=True`. This slows down debugging and cache generator for very large datasets.

## Steps to reproduce the bug

- Create a local dataset builder class
- load the local builder class file using `load_dataset` and let the cache build
- update the file's content
- The cache should rebuilt.

## Expected results

With `ignore_verifications=True`, `load_dataset` should pick up existing cache.

## Actual results

Creates new cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0


Comments: Hi ! I just tried to reproduce what you said:
- create a local builder class
- use `load_dataset`
- update the builder class code
- use `load_dataset` again (with or without `ignore_verifications=True`)
And it creates a new cache, as expected.

What modifications did you do to your builder's code ?

Hi @lhoestq. Thanks for your reply. I just did minor modifications for which it should not regenerate cache (for e.g. Adding a print statement). Overall, regardless of cache miss, there should be an explicit option to allow reuse of existing cache if author knows cache shouldn't be affected.

The cache is based on the hash of the dataset builder's code, so changing the code makes it recompute the cache.

You could still rename the cache directory of your previous computation to the new expected cache directory if you want to avoid having to recompute it and if you're sure that it would generate the exact same result.

The verifications are data integrity verifications: it checks the checksums of the downloaded files, as well as the size of the generated splits.

Hi @apsdehal,

If you decide to follow @lhoestq's suggestion to rename the cache directory of your previous computation to the new expected cache directory, you can do the following to get the name of the new expected cache directory once #2500 is merged:
```python
from datasets import load_dataset_builder
dataset_builder = load_dataset_builder("path/to/your/dataset")
print(dataset_builder.cache_dir)
```

This way, you don't have to recompute the hash of the dataset script yourself each time you modify the script.
fix Dataset.map when num_procs > num rows
State: closed | Author: connor-mccarthy (CONTRIBUTOR)

closes #2470

## Testing notes
To run updated tests:
```sh
pytest tests/test_arrow_dataset.py -k "BaseDatasetTest and test_map_multiprocessing" -s
```
With Python code (to view warning):
```python
from datasets import Dataset


dataset = Dataset.from_dict({"x": ["sample"]})
print(len(dataset))
dataset.map(lambda x: x, num_proc=10)
```

Comments: Hi ! Thanks for fixing this :)

Looks like you have tons of changes due to code formatting.
We're using `black` for this, with a custom line length. To run our code formatting, you just need to run
```
make style
```

Then for the windows error in the CI, I'm looking into it. It's probably just a file that isn't properly closed

CI is all green now ! Thanks :)

There are still many code formatting changes in your PR - probably due to the first commit you did.
To avoid conflicts with future PRs it would be nice to only have the changes related to the `num_proc` warning, and not have all those code formatting changes,

Could you try remove those code formatting changes ?

If it's easier for you, you can make a new branch from `master` if needed

Thanks, @lhoestq! Apologies for the half-baked commits yesterday! I wasn’t able to step back in to resolve those CI issues until this morning.

Also, I’m surprised that `make style` isn’t resolving the formatting changes. I’m a bit stumped on that, so I’m going to re-apply on a new branch and open a PR as you suggested.
Memory usage consistently increases when processing a dataset with `.map`
State: closed | Author: apsdehal (CONTRIBUTOR)
Labels: name: bug

## Describe the bug

I have a HF dataset with image paths stored in it and I am trying to load those image paths using `.map` with `num_proc=80`. I am noticing that the memory usage consistently keeps on increasing with time. I tried using `DEFAULT_WRITER_BATCH_SIZE=10` in the builder to decrease arrow writer's batch size but that doesn't seem to help.

## Steps to reproduce the bug

Providing code as it is would be hard. I can provide a MVP if that helps.

## Expected results

Memory usage should become consistent after some time following the launch of processing.

## Actual results

Memory usage keeps on increasing.

## Environment info

- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0

Comments: Hi ! Can you share the function you pass to `map` ?
I know you mentioned it would be hard to share some code but this would really help to understand what happened

This is the same behavior as in #4883, so I'm closing this issue as a duplicate.
Better DuplicateKeysError error to help the user debug the issue
State: closed | Author: lhoestq (MEMBER)
Labels: name: enhancement

name: good first issue

As mentioned in https://github.com/huggingface/datasets/issues/2552 it would be nice to improve the error message when a dataset fails to build because there are duplicate example keys.

The current one is
```python
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

and we could have something that guides the user to debugging the issue:
```python
DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>
```

Comments: excuse me, my `datasets` version is `2.2.2`, but I also just see the error info like 
```
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

Hi ! for which dataset do you have this error ?

Also note that this issue is just about improving the error message, which is not very friendly x)

@lhoestq  I would like to take a hit at improving the error message. Will open a draft PR and will reach out to you for review


> DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.

@lhoestq  when you mention 42th and 1337th in the above case , are these values the examples' "id" or are they the examples' index ? 

Hi ! Thanks @VijayKalmath :)

In the general case, examples don't have an "id" field, so I think it should correspond to the index

@lhoestq  , I have opened a draft PR for this Issue. 

I wanted to check with you if there is a way to get `<path/to/the/dataset/script>` currently or do I need to add extra code to find that. 

If I need to find the script , I can assume that the generator function will always be in `datasets/{dataset_name}/{dataset_name}.py`. 

Thanks !

> I wanted to check with you if there is a way to get <path/to/the/dataset/script> currently or do I need to add extra code to find that.

You don't have access to this info inside the ArrowWriter unfortunately. This info is available in builder.py in the DatasetBuilder code that uses the ArrowWriter though, maybe a try-catch there can do the job
Multilabel metrics not supported
State: closed | Author: GuillemGSubies (NONE)
Labels: name: bug

When I try to use a metric like F1 macro I get the following error:

```
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
```
There is an explicit casting here:

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/src/datasets/features.py#L274

And looks like this is because here

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/metrics/f1/f1.py#L88

the features can only be integers, so we cannot use that F1 for multilabel. Instead, if I create the following F1 (ints replaced with sequence of ints), it will work:

```python
class F1(datasets.Metric):
    def _info(self):
        return datasets.MetricInfo(
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            features=datasets.Features(
                {
                    "predictions": datasets.Sequence(datasets.Value("int32")),
                    "references": datasets.Sequence(datasets.Value("int32")),
                }
            ),
            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html"],
        )

    def _compute(self, predictions, references, labels=None, pos_label=1, average="binary", sample_weight=None):
        return {
            "f1": f1_score(
                references,
                predictions,
                labels=labels,
                pos_label=pos_label,
                average=average,
                sample_weight=sample_weight,
            ),
        }
```


Comments: Hi @GuillemGSubies, thanks for reporting.

I have made a PR to fix this issue and allow metrics to be computed also for multilabel classification problems.

Looks nice, thank you very much! 🚀 

Sorry for reopening but I just noticed that the `_compute` method for the F1 metric is still not good enough for multilabel problems:

https://github.com/huggingface/datasets/blob/92a3ee549705aa0a107c9fa5caf463b3b3da2616/metrics/f1/f1.py#L115

Somehow we should be able to change the parameter `average` at least

@GuillemGSubies, the parameter `average` passed to `_compute` is then passed to `f1_score`. This is right.
load_dataset("web_nlg") NonMatchingChecksumError
State: closed | Author: alxthm (NONE)
Labels: name: bug

Hi! It seems the WebNLG dataset gives a NonMatchingChecksumError.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset('web_nlg', name="release_v3.0_en", split="dev")
```

Gives

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://gitlab.com/shimorina/webnlg-dataset/-/archive/master/webnlg-dataset-master.zip']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: macOS-11.3.1-x86_64-i386-64bit
- Python version: 3.9.4
- PyArrow version: 3.0.0

Also tested on Linux, with python 3.6.8

Comments: Hi ! Thanks for reporting. This is due to the WebNLG repository that got updated today.
I just pushed a fix at #2558 - this shouldn't happen anymore in the future.

This is fixed on `master` now :)
We'll do a new release soon !
Fix FileSystems documentation
State: closed | Author: connor-mccarthy (CONTRIBUTOR)

### What this fixes:
This PR resolves several issues I discovered in the documentation on the `datasets.filesystems` module ([this page](https://huggingface.co/docs/datasets/filesystems.html)).

### What were the issues?
When I originally tried implementing the code examples I faced several bugs attributed to:

- out of date [botocore](https://github.com/boto/botocore) call signatures
- capitalization errors in the `S3FileSystem` class name (written as `S3Filesystem` in one place)
- call signature errors for the `S3FileSystem` class constructor (uses parameter `sessions` instead of `session` in some places) (see [`s3fs`](https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem) for where this constructor signature is defined)

### Testing/reviewing notes
Instructions for generating the documentation locally: [here](https://github.com/huggingface/datasets/tree/master/docs#generating-the-documentation).
Allow for incremental cumulative metric updates in a distributed setup
State: closed | Author: eladsegal (CONTRIBUTOR)
Labels: name: enhancement

Currently, using a metric allows for one of the following:
- Per example/batch metrics
- Cumulative metrics over the whole data

What I'd like is to have an efficient way to get cumulative metrics over the examples/batches added so far, in order to display it as part of the progress bar during training/evaluation.

Since most metrics are just an average of per-example metrics (which aren't?), an efficient calculation can be done as follows:
`((score_cumulative * n_cumulative) + (score_new * n_new)) / (n_cumulative+ n_new)`
where `n` and `score`  refer to number of examples and metric score, `cumulative` refers to the cumulative metric and `new` refers to the addition of new examples.

If you don't want to add this capability in the library, a simple solution exists so users can do it themselves:
It is easy to implement for a single process setup, but in a distributed one there is no way to get the correct `n_new`.
The solution for this is to return the number of examples that was used to compute the metrics in `.compute()` by adding the following line here:
https://github.com/huggingface/datasets/blob/5a3221785311d0ce86c2785b765e86bd6997d516/src/datasets/metric.py#L402-L403
```
output["number_of_examples"] = len(predictions)
```
and also remove the log message here so it won't spam:
https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/src/datasets/metric.py#L411

If this change is ok with you, I'll open a pull request.