konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Support multi-worker with streaming dataset (IterableDataset).

**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**


Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step

Any update on this feature request?

Not yet, I'm happy to provide some guidance if someone wants to give it a shot though.

The code that applies the `map` function is in `iterable_dataset.py`, in `MappedExamplesIterable.__iter__`
Streaming for the Json loader

It was not using `open` in the builder. Therefore `pyarrow.json.read_json` was downloading the full file to start yielding rows.

Moreover, it appeared that `pyarrow.json.read_json` was not really suited for streaming as it was downloading too much data and failing if `block_size` was not properly configured (related to #2573).

So I switched to using `open` which is extended to support reading from remote file progressively, and I removed the pyarrow json reader which was not practical.
Instead, I'm using the classical `json.loads` from the standard library.

A note is that I think we should add a few indicator of status (as mentioned by @stas00 in #2649), probably at the (1) downloading, (2) extracting and (3) reading steps. In particular when loading many very large files it's interesting to know a bit where we are in the process.

I tested locally, and the builtin `json` loader is 4x slower than `pyarrow.json`. Thanks for the comment @albertvillanova !

Therefore I switched back to using `pyarrow.json`, but only on the batch that is read. This way we don't have to deal with its `block_size`, and it only loads in memory one batch at a time.
add image-classification task template

Snippet below is the tl;dr, but you can try it out directly here:

[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/005c025d41f0e48ae3d4ee61c0f20b70/image-classification-task-template-demo.ipynb)

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'labels'],
#         num_rows: 23410
#     })
# })

ds = ds.prepare_for_task('image-classification')
# DatasetDict({
#     train: Dataset({
#         features: ['image_file_path', 'labels'],
#         num_rows: 23410
#     })
# })
```

Awesome!

Thanks for adding a new task template - great work @nateraw  🚀 !
Progress bars are not properly rendered in Jupyter notebook

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf 

To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.

Hi @mludv, thanks for the hint!!! :) 

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...
Load datasets from the Hub without requiring a dataset script

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)

This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉
can't set verbosity for `metric.py`

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places. 
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected 
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation. 

I had to do 
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
``` 
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS 
- Python version: 3.8.10
- PyArrow version: 3.0.0


Thanks @thomas-happify for reporting and thanks @mariosasko for the fix.
Integration with AugLy

**Is your feature request related to a problem? Please describe.**
Facebook recently launched a library, [AugLy](https://github.com/facebookresearch/AugLy) , that has a unified API for augmentations for image, video and text.

It would be pretty exciting to have it hooked up to HF libraries so that we can make NLP models robust to misspellings or to punctuation, or emojis etc. Plus, with Transformers supporting more CV use cases, having augmentations support becomes crucial.

**Describe the solution you'd like**
The biggest difference between augmentations and preprocessing is that preprocessing happens only once, but you are running augmentations once per epoch. AugLy operates on text directly, so this breaks the typical workflow where we would run the tokenizer once, set format to pt tensors and be ready for the Dataloader.

**Describe alternatives you've considered**

One possible way of implementing these is to make a custom Dataset class where getitem(i) runs the augmentation and the tokenizer every time, though this would slow training down considerably given we wouldn't even run the tokenizer in batches.


Hi,

you can define your own custom formatting with `Dataset.set_transform()` and then run the tokenizer with the batches of augmented data as follows:
```python
dset = load_dataset("imdb", split="train")  # Let's say we are working with the IMDB dataset
dset.set_transform(lambda ex: {"text": augly_text_augmentation(ex["text"])}, columns="text", output_all_columns=True)
dataloader = torch.utils.data.DataLoader(dset, batch_size=32)
for epoch in range(5):
    for batch in dataloader:
       tokenizer_output = tokenizer(batch.pop("text"), padding=True, truncation=True, return_tensors="pt")
       batch.update(tokenizer_output)
       output = model(**batch)
       ...
```

Preprocessing functions/augmentations, unless super generic, should be defined in separate libraries, so I'm closing this issue.
Use prefix to allow exceed Windows MAX_PATH

By using this prefix, you can exceed the Windows MAX_PATH limit.

See: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#win32-file-namespaces

Related to #2524, #2220.

Does this mean the `FileNotFoundError` that avoids infinite loop can be removed?

Yes, I think so...

Or maybe we could leave it in case a relative path exceeds the MAX_PATH limit?

 > Or maybe we could leave it in case a relative path exceeds the MAX_PATH limit?

What about converting relative paths to absolute?

Nice ! Have you had a chance to test it on a windows machine with the max path limit enabled ? Afaik the CI doesn't have the path limit

Sure @lhoestq: I've tested on my machine... And this fixes most of the tests... 😅
Add speech processing tasks

This PR replaces the `automatic-speech-recognition` task category with a broader `speech-processing` category. 

The tasks associated with this category are derived from the [SUPERB benchmark](https://arxiv.org/abs/2105.01051), and ASR is included in this set.

Are there any `task_categories:automatic-speech-recognition` dataset for which we should update the tags ?

> Are there any `task_categories:automatic-speech-recognition` dataset for which we should update the tags ?

Yes there's a few - I'll fix them tomorrow :)
Add ASR task for SUPERB

This PR starts building up the SUPERB benchmark by including the ASR task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/v0.2.0/downstream#asr-automatic-speech-recognition).

Usage:

```python
from datasets import load_dataset 

asr = load_dataset("superb", "asr")
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 28539
#     })
#     validation: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2703
#     })
#     test: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2620
#     })
# })
```

I've used the GLUE benchmark as a guide for filling out the README.

To move fast during the evaluation PoC I propose to merge one task at a time, so we can continue building the training / evaluation framework in parallel.

Note: codewise this PR is ready for review - I'll add the missing YAML tags once #2620 is merged :)

Wait until #2620 is merged before pushing the README tags in this PR

> Thanks!
> 
> One question: aren't you adding `task_templates` to the `_info` method (and to the `dataset_infos.json`?

great catch! i've now added the asr task template (along with a mapping from superb task -> template) and updated the `dataset_infos.json` :) 

> Good!
> 
> I have a suggested refactoring... Tell me what you think! :)

your approach is much more elegant - i've included your suggestions 🙏
Support remote data files

Add support for (streaming) remote data files:

```python
data_files = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{relative_file_path}"
ds = load_dataset("json", split="train", data_files=data_files, streaming=True)
```

cc: @thomwolf 

@lhoestq maybe we could also use (if available) the ETag of the remote file in `create_config_id`?

> @lhoestq maybe we could also use (if available) the ETag of the remote file in `create_config_id`?

Sure ! We can get the ETag with
```python
headers = get_authentication_headers_for_url(url, use_auth_token=use_auth_token)  # auth for private repos
etag = http_head(url, headers=headers).headers.get("ETag")
```

Since the computation of the `config_id` is done in the `DatasetBuilder.__init__`, then this means that we need to add a new parameter `use_auth_token` in `DatasetBuilder.__init__`

Does that sound good ? We can add this in a following PR
Jsonlines export error

## Describe the bug
When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

## Steps to reproduce the bug
This what I'm running:

in python:

```
from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")
```

then out of python:

```
head -10000 ptb.jsonl
```

## Expected results
Properly separated lines

## Actual results
The last line is a concatenation of two lines

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 4.0.1

Thanks for reporting @TevenLeScao! I'm having a look...

(not sure what just happened on the assignations sorry)

For some reason this happens (both `datasets` version are on master) only on Python 3.6 and not Python 3.8.

@TevenLeScao we are using `pandas` to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?

@TevenLeScao I have just checked it: this was a bug in `pandas` and it was fixed in version 1.2: https://github.com/pandas-dev/pandas/pull/36898

Thanks ! I'm creating a PR

Well I though it was me who has taken on this issue... 😅 

Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)

I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄 

The fix is available on `master` @TevenLeScao , thanks for reporting
Return Python float instead of numpy.float64 in sklearn metrics

This PR converts the return type of all `sklearn` metrics to be Python `float` instead of `numpy.float64`.

The reason behind this is that our Hub evaluation framework relies on converting benchmark-specific metrics to YAML ([example](https://huggingface.co/datasets/autonlp/autonlp-benchmark-raft-neelalex__raft-test-neelalex__raft-predictions-3/blob/main/README.md#L11)) and the `numpy.float64` format produces garbage like:

```python
import yaml
from datasets import load_metric

metric = load_metric("accuracy")
score = metric.compute(predictions=[0,1], references=[0,1])
print(yaml.dump(score["accuracy"])) # output below
# !!python/object/apply:numpy.core.multiarray.scalar
# - !!python/object/apply:numpy.dtype
#   args:
#   - f8
#   - false
#   - true
#   state: !!python/tuple
#   - 3
#   - <
#   - null
#   - null
#   - null
#   - -1
#   - -1
#   - 0
# - !!binary |
#   AAAAAAAA8D8=
```

I opened an issue on the `sklearn` repo to understand why `numpy.float64` is the default: https://github.com/scikit-learn/scikit-learn/discussions/20490

It could be surprising at first to use `tolist()` on numpy scalars but it works ^^

did the same for Pearsonr here: https://github.com/huggingface/datasets/pull/2614
[Metrics] addition of wiki_split metrics

**Is your feature request related to a problem? Please describe.**
While training the model on sentence split the task in English we require to evaluate the trained model on `Exact Match`, `SARI` and `BLEU` score
like this 
![image](https://user-images.githubusercontent.com/26653468/124746876-ff5a3380-df3e-11eb-9a01-4b48db7a6694.png)
While training we require metrics which can give all the output

Currently, we don't have an exact match for text normalized data

**Describe the solution you'd like**
A custom metrics for wiki_split that can calculate these three values and provide it in the form of a single dictionary
For exact match, we can refer to [this](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py) 

**Describe alternatives you've considered**
Two metrics are already present one more can be added for an exact match then we can run all three metrics in training script

#self-assign

#take
Crash when using multiprocessing (`num_proc` > 1) on `filter` and all samples are discarded

## Describe the bug
If `filter` is applied to a dataset using multiprocessing (`num_proc` > 1) and all sharded datasets are empty afterwards (due to all samples being discarded), the program crashes.

## Steps to reproduce the bug
```python
from datasets import Dataset
data = Dataset.from_dict({'id': [0,1]})
data.filter(lambda x: False, num_proc=2)
```

## Expected results
An empty table should be returned without crashing.

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 185, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2143, in filter
    return self.map(
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1738, in map
    result = concatenate_datasets(transformed_shards)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3267, in concatenate_datasets
    table = concat_tables(tables_to_concat, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 853, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 713, in from_tables
    blocks = to_blocks(tables[0])
IndexError: list index out of range
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.12.11-300.fc34.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.10
- PyArrow version: 3.0.0
ModuleNotFoundError: No module named 'datasets.tasks' while importing common voice datasets

Error traceback:
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-8-a7b592d3bca0> in <module>()
      1 from datasets import load_dataset, load_metric
      2 
----> 3 common_voice_train = load_dataset("common_voice", "pa-IN", split="train+validation")
      4 common_voice_test = load_dataset("common_voice", "pa-IN", split="test")

9 frames
/root/.cache/huggingface/modules/datasets_modules/datasets/common_voice/078d412587e9efeb0ae2e574da99c31e18844c496008d53dc5c60f4159ed639b/common_voice.py in <module>()
     19 
     20 import datasets
---> 21 from datasets.tasks import AutomaticSpeechRecognition
     22 
     23 

ModuleNotFoundError: No module named 'datasets.tasks'

Hi @profsatwinder.

It looks like you are using an old version of `datasets`. Please update it with `pip install -U datasets` and indicate if the problem persists.

@albertvillanova Thanks for the information. I updated it to 1.9.0 and the issue is resolved. Thanks again.
Support multilabel metrics

Currently, multilabel metrics are not supported because `predictions` and `references` are defined as `Value("int32")`.

This PR creates a new feature type `OptionalSequence` which can act as either `Value("int32")` or `Sequence(Value("int32"))`, depending on the data passed.


Close #2554.

Hi ! Thanks for the fix :)

If I understand correctly, `OptionalSequence` doesn't have an associated arrow type that we know in advance unlike the other feature types, because it depends on the type of the examples.

For example, I tested this and it raises an error:
```python
import datasets as ds
import pyarrow as pa

features = ds.Features({"a": ds.features.OptionalSequence(ds.Value("int32"))})
batch = {"a": [[0]]}

writer = ds.ArrowWriter(features=features, stream=pa.BufferOutputStream())
writer.write_batch(batch)
# ArrowInvalid: Could not convert [0] with type list: tried to convert to int
```
This error happens because `features.type` is `StructType(struct<a: int32>)`.

Another way to add support for multilabel would be to have several configurations for these metrics. By default it would set the features without sequences, and for the multi label configuration it would use features with sequences. Let me know what you think

Hi @lhoestq, thanks for your feedback :)

Definitely, your suggested approach is simpler. I am going to refactor all my PR unless we could envision some other use cases where an OptionalSequence might be convenient, but for now I can't think of any...

@albertvillanova @lhoestq I couldnt find the related docs in F1 card: https://huggingface.co/spaces/evaluate-metric/f1

How do I perform multilabel F1 evaluation using evaluate package?

I was going to transfer your question to the `evaluate` GitHub repository, but I saw you have already done it (and even opened a PR):
- https://github.com/huggingface/evaluate/issues/219
- https://github.com/huggingface/evaluate/pull/221

Thanks, @fcakyon. 

Sorry to bomb you on multiple channels :sweat_smile: @albertvillanova, I have solved my problems, and opened a PR so that others also don't get confused :+1:
Add skip and take

As discussed in https://github.com/huggingface/datasets/pull/2375#discussion_r657084544 I added the `IterableDataset.skip` and `IterableDataset.take` methods that allows to do basic splitting of iterable datasets.

You can create new dataset with the first `n` examples using `IterableDataset.take()`, or you can get a dataset with the rest of the examples by skipping the first `n` examples with `IterableDataset.skip()`

One implementation detail:

Using `take` (or `skip`) prevents future dataset shuffling from shuffling the dataset shards, otherwise the taken examples could come from other shards. In this case it only uses the shuffle buffer.
I would have loved to allow the shards of the taken examples to be shuffled anyway, but since we don't know in advance the length of each shard we don't know what shards to take or skip.
I think this is ok though since users can shuffle before doing take or skip. I mentioned this in the documentation

cc @vblagoje @lewtun 

@lhoestq looks good. I tried with https://huggingface.co/datasets/vblagoje/wikipedia_snippets_streamed and it worked nicely. I would add more unit tests for edge cases. What happens if the n is larger than the total number of samples? Just to make sure these cases are handled properly. 

Yup I'll add the tests thanks ;)

Moreover, I just noticed something in your wiki snippets code. FYI you're using `++passage_counter ` at https://huggingface.co/datasets/vblagoje/wikipedia_snippets_streamed/blob/main/wikipedia_snippets_streamed.py#L102 but in python this doesn't increment the value @vblagoje 

Thanks @lhoestq - not easy to convert after 10+ years of Java
Add C4

The old code for the C4 dataset was to generate the C4 with Apache Beam, as in Tensorflow Datasets.
However AllenAI is now hosting the processed C4 dataset in this repo: https://huggingface.co/datasets/allenai/c4
Thanks a lot to them for their amazing work !

In this PR I changed the script to download and prepare the data directly from this repo.
It has 4 variants: en, en.noblocklist, en.noclean, realnewslike

You can load it with
```python
from datasets import load_dataset

c4 = load_dataset("c4", "en")
```

It also supports streaming, if you don't want to download hundreds of GB of data:
```python
c4 = load_dataset("c4", "en", streaming=True)
```

Regarding the dataset_infos.json, I haven't added the infos for en.noclean. I will add them once I have them.

Also we can work on the dataset card at https://huggingface.co/datasets/c4
For now I just added a link to https://huggingface.co/datasets/allenai/c4 as well as a few sections
Support Zstandard compressed files

Add support for Zstandard compressed files: https://facebook.github.io/zstd/

I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

```
!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)

WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported
```

I just tried on google colab and this works:
```python
!pip install zstandard
!pip install datasets
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(lds))
```

Can you check that you have a correct installation of `zstandard` ?

@lhoestq please note [this](https://github.com/huggingface/datasets/issues/2572#issuecomment-1363718916) is a duplicate of:
- #5388

Oh thanks I missed that one !

> I just tried on google colab and this works:
> 
> ```python
> !pip install zstandard
> !pip install datasets
> from datasets import load_dataset
> 
> lds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(lds))
> ```
> 
> Can you check that you have a correct installation of `zstandard` ?

I was downloading datasets first then was doing zstandard installation and that was causing the issue. This was highlighted by the Hugging Face staff and that helped. Now the issue is resolved. Thank you.
Weights of model checkpoint not initialized for RobertaModel for Bertscore

When applying bertscore out of the box, 

```Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight']```

Following the typical usage from https://huggingface.co/docs/datasets/loading_metrics.html

```
from datasets import load_metric
metric = load_metric('bertscore')

# Example of typical usage
for batch in dataset:
    inputs, references = batch
    predictions = model(inputs)
    metric.add_batch(predictions=predictions, references=references)
score = metric.compute(lang="en")
#score = metric.compute(model_type="roberta-large") # gives the same error
```

I am concerned about this because my usage shouldn't require any further fine-tuning and most people would expect to use BertScore out of the box? I realised the huggingface code is a wrapper around https://github.com/Tiiiger/bert_score, but I think this repo is anyway relying on the model code and weights from huggingface repo.... 

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1041-aws-x86_64-with-glibc2.27
- Python version:  3.9.5
- PyArrow version: 3.0.0


Hi @suzyahyah, thanks for reporting.

The message you get is indeed not an error message, but a warning coming from Hugging Face `transformers`. The complete warning message is:
```
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

In this case, this behavior IS expected and you can safely ignore the warning message.

The reason is that you are just using RoBERTa to get the contextual embeddings of the input sentences/tokens, thus leaving away its head layer, whose weights are ignored.

Feel free to reopen this issue if you need further explanations.

Hi @suzyahyah, I have created a Pull Request to filter out that warning message in this specific case, since the behavior is as expected and the warning message can only cause confusion for users (as in your case).
Inject templates for ASR datasets

This PR adds ASR templates for 5 of the most common speech datasets on the Hub, where "common" is defined by the number of models trained on them.

I also fixed a bunch of the tags in the READMEs 😎 

Wait until #2567 is merged so we can benefit from the tagger :)

thanks for the feedback @lhoestq! i've added the new language codes and this PR should be ready for a merge :)
concatenate_datasets for iterable datasets

Currently `concatenate_datasets` only works for map-style `Dataset`.

It would be nice to have it work for `IterableDataset` objects as well.

It would simply chain the iterables of the iterable datasets.

It is probably worth noting here that the [documentation](https://huggingface.co/docs/datasets/process#concatenate) is misleading (indicating that it does work for IterableDatasets):

> You can also mix several datasets together by taking alternating examples from each one to create a new dataset. This is known as interleaving, and you can use it with [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets). **Both [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets) and [concatenate_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.concatenate_datasets) will work with regular [Dataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset) and [IterableDataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.IterableDataset) objects**. Refer to the [Stream](https://huggingface.co/docs/datasets/stream#interleave) section for an example of how it’s used. 

Thanks for the heads up, I'll fix that
Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`

## Describe the bug
If i have local file defining a dataset builder class and I load it using `load_dataset` functionality, the existing cache is ignored whenever the file is update even with `ignore_verifications=True`. This slows down debugging and cache generator for very large datasets.

## Steps to reproduce the bug

- Create a local dataset builder class
- load the local builder class file using `load_dataset` and let the cache build
- update the file's content
- The cache should rebuilt.

## Expected results

With `ignore_verifications=True`, `load_dataset` should pick up existing cache.

## Actual results

Creates new cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0


Hi ! I just tried to reproduce what you said:
- create a local builder class
- use `load_dataset`
- update the builder class code
- use `load_dataset` again (with or without `ignore_verifications=True`)
And it creates a new cache, as expected.

What modifications did you do to your builder's code ?

Hi @lhoestq. Thanks for your reply. I just did minor modifications for which it should not regenerate cache (for e.g. Adding a print statement). Overall, regardless of cache miss, there should be an explicit option to allow reuse of existing cache if author knows cache shouldn't be affected.

The cache is based on the hash of the dataset builder's code, so changing the code makes it recompute the cache.

You could still rename the cache directory of your previous computation to the new expected cache directory if you want to avoid having to recompute it and if you're sure that it would generate the exact same result.

The verifications are data integrity verifications: it checks the checksums of the downloaded files, as well as the size of the generated splits.

Hi @apsdehal,

If you decide to follow @lhoestq's suggestion to rename the cache directory of your previous computation to the new expected cache directory, you can do the following to get the name of the new expected cache directory once #2500 is merged:
```python
from datasets import load_dataset_builder
dataset_builder = load_dataset_builder("path/to/your/dataset")
print(dataset_builder.cache_dir)
```

This way, you don't have to recompute the hash of the dataset script yourself each time you modify the script.
fix Dataset.map when num_procs > num rows

closes #2470

## Testing notes
To run updated tests:
```sh
pytest tests/test_arrow_dataset.py -k "BaseDatasetTest and test_map_multiprocessing" -s
```
With Python code (to view warning):
```python
from datasets import Dataset


dataset = Dataset.from_dict({"x": ["sample"]})
print(len(dataset))
dataset.map(lambda x: x, num_proc=10)
```

Hi ! Thanks for fixing this :)

Looks like you have tons of changes due to code formatting.
We're using `black` for this, with a custom line length. To run our code formatting, you just need to run
```
make style
```

Then for the windows error in the CI, I'm looking into it. It's probably just a file that isn't properly closed

CI is all green now ! Thanks :)

There are still many code formatting changes in your PR - probably due to the first commit you did.
To avoid conflicts with future PRs it would be nice to only have the changes related to the `num_proc` warning, and not have all those code formatting changes,

Could you try remove those code formatting changes ?

If it's easier for you, you can make a new branch from `master` if needed

Thanks, @lhoestq! Apologies for the half-baked commits yesterday! I wasn’t able to step back in to resolve those CI issues until this morning.

Also, I’m surprised that `make style` isn’t resolving the formatting changes. I’m a bit stumped on that, so I’m going to re-apply on a new branch and open a PR as you suggested.
Memory usage consistently increases when processing a dataset with `.map`

## Describe the bug

I have a HF dataset with image paths stored in it and I am trying to load those image paths using `.map` with `num_proc=80`. I am noticing that the memory usage consistently keeps on increasing with time. I tried using `DEFAULT_WRITER_BATCH_SIZE=10` in the builder to decrease arrow writer's batch size but that doesn't seem to help.

## Steps to reproduce the bug

Providing code as it is would be hard. I can provide a MVP if that helps.

## Expected results

Memory usage should become consistent after some time following the launch of processing.

## Actual results

Memory usage keeps on increasing.

## Environment info

- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0

Hi ! Can you share the function you pass to `map` ?
I know you mentioned it would be hard to share some code but this would really help to understand what happened

This is the same behavior as in #4883, so I'm closing this issue as a duplicate.
Better DuplicateKeysError error to help the user debug the issue

As mentioned in https://github.com/huggingface/datasets/issues/2552 it would be nice to improve the error message when a dataset fails to build because there are duplicate example keys.

The current one is
```python
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

and we could have something that guides the user to debugging the issue:
```python
DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>
```

excuse me, my `datasets` version is `2.2.2`, but I also just see the error info like 
```
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

Hi ! for which dataset do you have this error ?

Also note that this issue is just about improving the error message, which is not very friendly x)

@lhoestq  I would like to take a hit at improving the error message. Will open a draft PR and will reach out to you for review


> DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.

@lhoestq  when you mention 42th and 1337th in the above case , are these values the examples' "id" or are they the examples' index ? 

Hi ! Thanks @VijayKalmath :)

In the general case, examples don't have an "id" field, so I think it should correspond to the index

@lhoestq  , I have opened a draft PR for this Issue. 

I wanted to check with you if there is a way to get `<path/to/the/dataset/script>` currently or do I need to add extra code to find that. 

If I need to find the script , I can assume that the generator function will always be in `datasets/{dataset_name}/{dataset_name}.py`. 

Thanks !

> I wanted to check with you if there is a way to get <path/to/the/dataset/script> currently or do I need to add extra code to find that.

You don't have access to this info inside the ArrowWriter unfortunately. This info is available in builder.py in the DatasetBuilder code that uses the ArrowWriter though, maybe a try-catch there can do the job
Multilabel metrics not supported

When I try to use a metric like F1 macro I get the following error:

```
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
```
There is an explicit casting here:

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/src/datasets/features.py#L274

And looks like this is because here

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/metrics/f1/f1.py#L88

the features can only be integers, so we cannot use that F1 for multilabel. Instead, if I create the following F1 (ints replaced with sequence of ints), it will work:

```python
class F1(datasets.Metric):
    def _info(self):
        return datasets.MetricInfo(
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            features=datasets.Features(
                {
                    "predictions": datasets.Sequence(datasets.Value("int32")),
                    "references": datasets.Sequence(datasets.Value("int32")),
                }
            ),
            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html"],
        )

    def _compute(self, predictions, references, labels=None, pos_label=1, average="binary", sample_weight=None):
        return {
            "f1": f1_score(
                references,
                predictions,
                labels=labels,
                pos_label=pos_label,
                average=average,
                sample_weight=sample_weight,
            ),
        }
```


Hi @GuillemGSubies, thanks for reporting.

I have made a PR to fix this issue and allow metrics to be computed also for multilabel classification problems.

Looks nice, thank you very much! 🚀 

Sorry for reopening but I just noticed that the `_compute` method for the F1 metric is still not good enough for multilabel problems:

https://github.com/huggingface/datasets/blob/92a3ee549705aa0a107c9fa5caf463b3b3da2616/metrics/f1/f1.py#L115

Somehow we should be able to change the parameter `average` at least

@GuillemGSubies, the parameter `average` passed to `_compute` is then passed to `f1_score`. This is right.
load_dataset("web_nlg") NonMatchingChecksumError

Hi! It seems the WebNLG dataset gives a NonMatchingChecksumError.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset('web_nlg', name="release_v3.0_en", split="dev")
```

Gives

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://gitlab.com/shimorina/webnlg-dataset/-/archive/master/webnlg-dataset-master.zip']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: macOS-11.3.1-x86_64-i386-64bit
- Python version: 3.9.4
- PyArrow version: 3.0.0

Also tested on Linux, with python 3.6.8

Hi ! Thanks for reporting. This is due to the WebNLG repository that got updated today.
I just pushed a fix at #2558 - this shouldn't happen anymore in the future.

This is fixed on `master` now :)
We'll do a new release soon !
Allow for incremental cumulative metric updates in a distributed setup

Currently, using a metric allows for one of the following:
- Per example/batch metrics
- Cumulative metrics over the whole data

What I'd like is to have an efficient way to get cumulative metrics over the examples/batches added so far, in order to display it as part of the progress bar during training/evaluation.

Since most metrics are just an average of per-example metrics (which aren't?), an efficient calculation can be done as follows:
`((score_cumulative * n_cumulative) + (score_new * n_new)) / (n_cumulative+ n_new)`
where `n` and `score`  refer to number of examples and metric score, `cumulative` refers to the cumulative metric and `new` refers to the addition of new examples.

If you don't want to add this capability in the library, a simple solution exists so users can do it themselves:
It is easy to implement for a single process setup, but in a distributed one there is no way to get the correct `n_new`.
The solution for this is to return the number of examples that was used to compute the metrics in `.compute()` by adding the following line here:
https://github.com/huggingface/datasets/blob/5a3221785311d0ce86c2785b765e86bd6997d516/src/datasets/metric.py#L402-L403
```
output["number_of_examples"] = len(predictions)
```
and also remove the log message here so it won't spam:
https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/src/datasets/metric.py#L411

If this change is ok with you, I'll open a pull request.
Handling unlabeled datasets

Hi!

Is there a way for datasets to produce unlabeled instances (e.g., the `ClassLabel` can be nullable).

For example, I want to use the MNLI dataset reader ( https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py ) on a file that doesn't have the `gold_label` field. I tried setting `"label": data.get("gold_label")`, but got the following error:

```
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 953, in encode_example
    return encode_nested_example(self, example)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in encode_nested_example
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in <dictcomp>
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 875, in encode_nested_example
    return schema.encode_example(obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 653, in encode_example
    if not -1 <= example_data < self.num_classes:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
```

What's the proper way to handle reading unlabeled datasets, especially for downstream usage with Transformers?

Hi @nelson-liu,

You can pass the parameter `features` to `load_dataset`: https://huggingface.co/docs/datasets/_modules/datasets/load.html#load_dataset

If you look at the code of the MNLI script you referred in your question (https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py#L62-L77), you can see how the Features were originally specified. 

Feel free to use it as a template, customize it and pass it to `load_dataset` using the parameter `features`.

ah got it, thanks!
Field order issue in loading json

## Describe the bug
The `load_dataset` function expects columns in alphabetical order when loading json files.

Similar bug was previously reported for csv in #623 and fixed in #684.
## Steps to reproduce the bug

For a json file `j.json`,
```
{"c":321, "a": 1, "b": 2}
```
Running the following,
```
f= datasets.Features({'a': Value('int32'), 'b': Value('int32'), 'c': Value('int32')})
json_data = datasets.load_dataset('json', data_files='j.json', features=f)
```


## Expected results
A successful load.
## Actual results
```
File "pyarrow/table.pxi", line 1409, in pyarrow.lib.Table.cast
ValueError: Target schema's field names are not matching the table's field names: ['c', 'a', 'b'], ['a', 'b', 'c']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.10.0-957.1.3.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 3.0.0



Hi @luyug, thanks for reporting.

The good news is that we fixed this issue only 9 days ago: #2507.

The patch is already in the master branch of our repository and it will be included in our next `datasets` release version 1.9.0.

Feel free to reopen the issue if the problem persists.