Kokolipa/github-issues huggingface dataset:
Issue: Support Zstandard compressed files
State: closed
Labels: name: enhancement

Add support for Zstandard compressed files: https://facebook.github.io/zstd/

I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

```
!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)

WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported
```

I just tried on google colab and this works:
```python
!pip install zstandard
!pip install datasets
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(lds))
```

Can you check that you have a correct installation of `zstandard` ?

@lhoestq please note [this](https://github.com/huggingface/datasets/issues/2572#issuecomment-1363718916) is a duplicate of:
- #5388

Oh thanks I missed that one !

> I just tried on google colab and this works:
> 
> ```python
> !pip install zstandard
> !pip install datasets
> from datasets import load_dataset
> 
> lds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(lds))
> ```
> 
> Can you check that you have a correct installation of `zstandard` ?

I was downloading datasets first then was doing zstandard installation and that was causing the issue. This was highlighted by the Hugging Face staff and that helped. Now the issue is resolved. Thank you.
Issue: Weights of model checkpoint not initialized for RobertaModel for Bertscore
State: closed
Labels: name: bug

When applying bertscore out of the box, 

```Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight']```

Following the typical usage from https://huggingface.co/docs/datasets/loading_metrics.html

```
from datasets import load_metric
metric = load_metric('bertscore')

# Example of typical usage
for batch in dataset:
    inputs, references = batch
    predictions = model(inputs)
    metric.add_batch(predictions=predictions, references=references)
score = metric.compute(lang="en")
#score = metric.compute(model_type="roberta-large") # gives the same error
```

I am concerned about this because my usage shouldn't require any further fine-tuning and most people would expect to use BertScore out of the box? I realised the huggingface code is a wrapper around https://github.com/Tiiiger/bert_score, but I think this repo is anyway relying on the model code and weights from huggingface repo.... 

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1041-aws-x86_64-with-glibc2.27
- Python version:  3.9.5
- PyArrow version: 3.0.0


Hi @suzyahyah, thanks for reporting.

The message you get is indeed not an error message, but a warning coming from Hugging Face `transformers`. The complete warning message is:
```
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

In this case, this behavior IS expected and you can safely ignore the warning message.

The reason is that you are just using RoBERTa to get the contextual embeddings of the input sentences/tokens, thus leaving away its head layer, whose weights are ignored.

Feel free to reopen this issue if you need further explanations.

Hi @suzyahyah, I have created a Pull Request to filter out that warning message in this specific case, since the behavior is as expected and the warning message can only cause confusion for users (as in your case).
Issue: concatenate_datasets for iterable datasets
State: closed

Currently `concatenate_datasets` only works for map-style `Dataset`.

It would be nice to have it work for `IterableDataset` objects as well.

It would simply chain the iterables of the iterable datasets.

It is probably worth noting here that the [documentation](https://huggingface.co/docs/datasets/process#concatenate) is misleading (indicating that it does work for IterableDatasets):

> You can also mix several datasets together by taking alternating examples from each one to create a new dataset. This is known as interleaving, and you can use it with [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets). **Both [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets) and [concatenate_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.concatenate_datasets) will work with regular [Dataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset) and [IterableDataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.IterableDataset) objects**. Refer to the [Stream](https://huggingface.co/docs/datasets/stream#interleave) section for an example of how it’s used. 

Thanks for the heads up, I'll fix that
Issue: Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`
State: closed
Labels: name: bug

## Describe the bug
If i have local file defining a dataset builder class and I load it using `load_dataset` functionality, the existing cache is ignored whenever the file is update even with `ignore_verifications=True`. This slows down debugging and cache generator for very large datasets.

## Steps to reproduce the bug

- Create a local dataset builder class
- load the local builder class file using `load_dataset` and let the cache build
- update the file's content
- The cache should rebuilt.

## Expected results

With `ignore_verifications=True`, `load_dataset` should pick up existing cache.

## Actual results

Creates new cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0


Hi ! I just tried to reproduce what you said:
- create a local builder class
- use `load_dataset`
- update the builder class code
- use `load_dataset` again (with or without `ignore_verifications=True`)
And it creates a new cache, as expected.

What modifications did you do to your builder's code ?

Hi @lhoestq. Thanks for your reply. I just did minor modifications for which it should not regenerate cache (for e.g. Adding a print statement). Overall, regardless of cache miss, there should be an explicit option to allow reuse of existing cache if author knows cache shouldn't be affected.

The cache is based on the hash of the dataset builder's code, so changing the code makes it recompute the cache.

You could still rename the cache directory of your previous computation to the new expected cache directory if you want to avoid having to recompute it and if you're sure that it would generate the exact same result.

The verifications are data integrity verifications: it checks the checksums of the downloaded files, as well as the size of the generated splits.

Hi @apsdehal,

If you decide to follow @lhoestq's suggestion to rename the cache directory of your previous computation to the new expected cache directory, you can do the following to get the name of the new expected cache directory once #2500 is merged:
```python
from datasets import load_dataset_builder
dataset_builder = load_dataset_builder("path/to/your/dataset")
print(dataset_builder.cache_dir)
```

This way, you don't have to recompute the hash of the dataset script yourself each time you modify the script.
Issue: Memory usage consistently increases when processing a dataset with `.map`
State: closed
Labels: name: bug

## Describe the bug

I have a HF dataset with image paths stored in it and I am trying to load those image paths using `.map` with `num_proc=80`. I am noticing that the memory usage consistently keeps on increasing with time. I tried using `DEFAULT_WRITER_BATCH_SIZE=10` in the builder to decrease arrow writer's batch size but that doesn't seem to help.

## Steps to reproduce the bug

Providing code as it is would be hard. I can provide a MVP if that helps.

## Expected results

Memory usage should become consistent after some time following the launch of processing.

## Actual results

Memory usage keeps on increasing.

## Environment info

- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyArrow version: 3.0.0

Hi ! Can you share the function you pass to `map` ?
I know you mentioned it would be hard to share some code but this would really help to understand what happened

This is the same behavior as in #4883, so I'm closing this issue as a duplicate.
Issue: Better DuplicateKeysError error to help the user debug the issue
State: closed
Labels: name: enhancement

name: good first issue

As mentioned in https://github.com/huggingface/datasets/issues/2552 it would be nice to improve the error message when a dataset fails to build because there are duplicate example keys.

The current one is
```python
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

and we could have something that guides the user to debugging the issue:
```python
DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>
```

excuse me, my `datasets` version is `2.2.2`, but I also just see the error info like 
```
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

Hi ! for which dataset do you have this error ?

Also note that this issue is just about improving the error message, which is not very friendly x)

@lhoestq  I would like to take a hit at improving the error message. Will open a draft PR and will reach out to you for review


> DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.

@lhoestq  when you mention 42th and 1337th in the above case , are these values the examples' "id" or are they the examples' index ? 

Hi ! Thanks @VijayKalmath :)

In the general case, examples don't have an "id" field, so I think it should correspond to the index

@lhoestq  , I have opened a draft PR for this Issue. 

I wanted to check with you if there is a way to get `<path/to/the/dataset/script>` currently or do I need to add extra code to find that. 

If I need to find the script , I can assume that the generator function will always be in `datasets/{dataset_name}/{dataset_name}.py`. 

Thanks !

> I wanted to check with you if there is a way to get <path/to/the/dataset/script> currently or do I need to add extra code to find that.

You don't have access to this info inside the ArrowWriter unfortunately. This info is available in builder.py in the DatasetBuilder code that uses the ArrowWriter though, maybe a try-catch there can do the job
Issue: Multilabel metrics not supported
State: closed
Labels: name: bug

When I try to use a metric like F1 macro I get the following error:

```
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
```
There is an explicit casting here:

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/src/datasets/features.py#L274

And looks like this is because here

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/metrics/f1/f1.py#L88

the features can only be integers, so we cannot use that F1 for multilabel. Instead, if I create the following F1 (ints replaced with sequence of ints), it will work:

```python
class F1(datasets.Metric):
    def _info(self):
        return datasets.MetricInfo(
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            features=datasets.Features(
                {
                    "predictions": datasets.Sequence(datasets.Value("int32")),
                    "references": datasets.Sequence(datasets.Value("int32")),
                }
            ),
            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html"],
        )

    def _compute(self, predictions, references, labels=None, pos_label=1, average="binary", sample_weight=None):
        return {
            "f1": f1_score(
                references,
                predictions,
                labels=labels,
                pos_label=pos_label,
                average=average,
                sample_weight=sample_weight,
            ),
        }
```


Hi @GuillemGSubies, thanks for reporting.

I have made a PR to fix this issue and allow metrics to be computed also for multilabel classification problems.

Looks nice, thank you very much! 🚀 

Sorry for reopening but I just noticed that the `_compute` method for the F1 metric is still not good enough for multilabel problems:

https://github.com/huggingface/datasets/blob/92a3ee549705aa0a107c9fa5caf463b3b3da2616/metrics/f1/f1.py#L115

Somehow we should be able to change the parameter `average` at least

@GuillemGSubies, the parameter `average` passed to `_compute` is then passed to `f1_score`. This is right.
Issue: load_dataset("web_nlg") NonMatchingChecksumError
State: closed
Labels: name: bug

Hi! It seems the WebNLG dataset gives a NonMatchingChecksumError.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset('web_nlg', name="release_v3.0_en", split="dev")
```

Gives

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://gitlab.com/shimorina/webnlg-dataset/-/archive/master/webnlg-dataset-master.zip']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: macOS-11.3.1-x86_64-i386-64bit
- Python version: 3.9.4
- PyArrow version: 3.0.0

Also tested on Linux, with python 3.6.8

Hi ! Thanks for reporting. This is due to the WebNLG repository that got updated today.
I just pushed a fix at #2558 - this shouldn't happen anymore in the future.

This is fixed on `master` now :)
We'll do a new release soon !
Issue: Allow for incremental cumulative metric updates in a distributed setup
State: closed
Labels: name: enhancement

Currently, using a metric allows for one of the following:
- Per example/batch metrics
- Cumulative metrics over the whole data

What I'd like is to have an efficient way to get cumulative metrics over the examples/batches added so far, in order to display it as part of the progress bar during training/evaluation.

Since most metrics are just an average of per-example metrics (which aren't?), an efficient calculation can be done as follows:
`((score_cumulative * n_cumulative) + (score_new * n_new)) / (n_cumulative+ n_new)`
where `n` and `score`  refer to number of examples and metric score, `cumulative` refers to the cumulative metric and `new` refers to the addition of new examples.

If you don't want to add this capability in the library, a simple solution exists so users can do it themselves:
It is easy to implement for a single process setup, but in a distributed one there is no way to get the correct `n_new`.
The solution for this is to return the number of examples that was used to compute the metrics in `.compute()` by adding the following line here:
https://github.com/huggingface/datasets/blob/5a3221785311d0ce86c2785b765e86bd6997d516/src/datasets/metric.py#L402-L403
```
output["number_of_examples"] = len(predictions)
```
and also remove the log message here so it won't spam:
https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/src/datasets/metric.py#L411

If this change is ok with you, I'll open a pull request.
Issue: Handling unlabeled datasets
State: closed
Labels: name: enhancement

Hi!

Is there a way for datasets to produce unlabeled instances (e.g., the `ClassLabel` can be nullable).

For example, I want to use the MNLI dataset reader ( https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py ) on a file that doesn't have the `gold_label` field. I tried setting `"label": data.get("gold_label")`, but got the following error:

```
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 953, in encode_example
    return encode_nested_example(self, example)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in encode_nested_example
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in <dictcomp>
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 875, in encode_nested_example
    return schema.encode_example(obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 653, in encode_example
    if not -1 <= example_data < self.num_classes:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
```

What's the proper way to handle reading unlabeled datasets, especially for downstream usage with Transformers?

Hi @nelson-liu,

You can pass the parameter `features` to `load_dataset`: https://huggingface.co/docs/datasets/_modules/datasets/load.html#load_dataset

If you look at the code of the MNLI script you referred in your question (https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py#L62-L77), you can see how the Features were originally specified. 

Feel free to use it as a template, customize it and pass it to `load_dataset` using the parameter `features`.

ah got it, thanks!
Issue: Field order issue in loading json
State: closed
Labels: name: bug

## Describe the bug
The `load_dataset` function expects columns in alphabetical order when loading json files.

Similar bug was previously reported for csv in #623 and fixed in #684.
## Steps to reproduce the bug

For a json file `j.json`,
```
{"c":321, "a": 1, "b": 2}
```
Running the following,
```
f= datasets.Features({'a': Value('int32'), 'b': Value('int32'), 'c': Value('int32')})
json_data = datasets.load_dataset('json', data_files='j.json', features=f)
```


## Expected results
A successful load.
## Actual results
```
File "pyarrow/table.pxi", line 1409, in pyarrow.lib.Table.cast
ValueError: Target schema's field names are not matching the table's field names: ['c', 'a', 'b'], ['a', 'b', 'c']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.10.0-957.1.3.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 3.0.0



Hi @luyug, thanks for reporting.

The good news is that we fixed this issue only 9 days ago: #2507.

The patch is already in the master branch of our repository and it will be included in our next `datasets` release version 1.9.0.

Feel free to reopen the issue if the problem persists.
Issue: Dataset load_from_disk is too slow
State: open
Labels: name: bug

@lhoestq 
## Describe the bug
It's not normal that I have to wait 7-8 hours for a dataset to be loaded from disk, as there are no preprocessing steps, it's only loading it with load_from_disk. I have 96 cpus, however only 1 is used for this, which is inefficient. Moreover, its usage is at 1%... This is happening in the context of a language model training, therefore I'm wasting 100$ each time I have to load the dataset from disk again (because the spot instance was stopped by aws and I need to relaunch it for example). 

## Steps to reproduce the bug
Just get the oscar in  spanish (around 150GGB) and try to first save in disk and then load the processed dataset. It's not dependent on the task you're doing, it just depends on the size of the text dataset.

## Expected results
I expect the dataset to be loaded in a normal time, by using the whole machine for loading it, I mean if you store the dataset in multiple files (.arrow) and then load it from multiple files, you can use multiprocessing for that and therefore don't waste so much time. 


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Ubuntu 18
- Python version: 3.8


I've seen you're planning to include a streaming mode for load_dataset, but that only saves the downloading and processing time, that's not being a problem for me, you cannot save the pure loading from disk time, therefore that's not a solution for my use case or for anyone who wants to use your library for training a language model. 

Hi ! It looks like an issue with the virtual disk you are using.

We load datasets using memory mapping. In general it makes it possible to load very big files instantaneously since it doesn't have to read the file (it just assigns virtual memory to the file on disk).
However there happens to be issues with virtual disks (for example on spot instances), for which memory mapping does a pass over the entire file, and this takes a while. We are discussing about this issue here: #2252 

Memory mapping is something handled by the OS so we can't do much about it, though we're still trying to figure out what's causing this behavior exactly to see what we can do.

Okay, that's exactly my case, with spot instances... Therefore this isn't something we can change in any way to be able to load the dataset faster? I mean, what do you do internally at huggingface for being able to use spot instances with datasets efficiently?

There are no solutions yet unfortunately.
We're still trying to figure out a way to make the loading instantaneous on such disks, I'll keep you posted
Issue: `datasets.keyhash.DuplicatedKeysError` for `drop` and `adversarial_qa/adversarialQA`
State: closed
Labels: name: bug

## Describe the bug
Failure to generate the datasets (`drop` and subset `adversarialQA` from `adversarial_qa`) because of duplicate keys.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("drop")
load_dataset("adversarial_qa", "adversarialQA")
```

## Expected results
The examples keys should be unique.

## Actual results
```bash
>>> load_dataset("drop")
Using custom data configuration default
Downloading and preparing dataset drop/default (download: 7.92 MiB, generated: 111.88 MiB, post-processed: Unknown size, total: 119.80 MiB) to /home/hf/.cache/huggingface/datasets/drop/default/0.1.0/7a94f1e2bb26c4b5c75f89857c06982967d7416e5af935a9374b9bccf5068026...
Traceback (most recent call last):         
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 992, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/arrow_writer.py", line 409, in finalize
    self.check_duplicate_keys()
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 28553293-d719-441b-8f00-ce3dc6df5398
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1044-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0


very much related: https://github.com/huggingface/datasets/pull/2333

Hi @VictorSanh, thank you for reporting this issue with duplicated keys.

- The issue with "adversarial_qa" was fixed 23 days ago: #2433. Current version of `datasets` (1.8.0) includes the patch.
- I am investigating the issue with `drop`. I'll ping you to keep you informed.

Hi @VictorSanh, the issue is already fixed and merged into master branch and will be included in our next release version 1.9.0.

thank you!
Issue: Use `Audio` features for `AutomaticSpeechRecognition` task template
State: closed
Labels: name: enhancement

In #2533 we added a task template for speech recognition that relies on the file paths to the audio files. As pointed out by @SBrandeis this is brittle as it doesn't port easily across different OS'. 

The solution is to use dedicated `Audio` features when casting the dataset. These features are not yet available in `datasets`, but should be included in the `AutomaticSpeechRecognition` template once they are.

I'm just retaking and working on #2324. 😉 

Resolved via https://github.com/huggingface/datasets/pull/4006.
Issue: Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task
State: closed
Labels: name: bug

[This colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) implements a token classification input pipeline extending the logic from [this hugging example](https://huggingface.co/transformers/custom_datasets.html#tok-ner).

The pipeline works fine with most instance in different languages, but unfortunately, [the Japanese Kana ligature (a form of abbreviation? I don't know Japanese well)](https://en.wikipedia.org/wiki/Kana_ligature) break the alignment of `return_offsets_mapping`:
![image](https://user-images.githubusercontent.com/50871412/122904371-db192700-d382-11eb-8917-1775db76db69.png)

Without the try catch block, it riase `ValueError: NumPy boolean array indexing assignment cannot assign 88 input values to the 87 output values where the mask is true`, example shown here [(another colab notebook)](https://colab.research.google.com/drive/1MmOqf3ppzzdKKyMWkn0bJy6DqzOO0SSm?usp=sharing)

It is clear that the normalizer is the process that break the alignment, as it is observed that `tokenizer._tokenizer.normalizer.normalize_str('ヿ')` return 'コト'.

One workaround is to include `tokenizer._tokenizer.normalizer.normalize_str` before the tokenizer preprocessing pipeline, which is also provided in the [first colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) with the name `udposTestDatasetWorkaround`.

I guess similar logics should be included inside the tokenizer and the offsets_mapping generation process such that user don't need to include them in their code. But I don't understand the code of tokenizer well that I think I am not able to do this.

p.s.
**I am using my own dataset building script in the provided example, but the script should be equivalent to the changes made by this [update](https://github.com/huggingface/datasets/pull/2466)**
`get_dataset `is just a simple wrapping for `load_dataset`
and the `tokenizer` is just `XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-large")`

Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**?

> Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**?

Oh, I am sorry
I would reopen the post on huggingface/transformers
Issue: Documentation Mistakes in Dataset: emotion
State: closed
Labels: name: bug

As per documentation,
Dataset: emotion
Homepage: https://github.com/dair-ai/emotion_dataset

Dataset: https://github.com/huggingface/datasets/blob/master/datasets/emotion/emotion.py

Permalink: https://huggingface.co/datasets/viewer/?dataset=emotion

Emotion is a dataset of English Twitter messages with eight basic emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. For more detailed information please refer to the paper.

But when we view the data, there are only 6 emotions, anger, fear, joy, sadness, surprise, and trust.

Hi,

this issue has been already reported in the dataset repo (https://github.com/dair-ai/emotion_dataset/issues/2), so this is a bug on their side.

The documentation has another bug in the dataset card [here](https://huggingface.co/datasets/emotion). 

In the dataset summary **six** emotions are mentioned: *"six basic emotions: anger, fear, joy, love, sadness, and surprise"*, however, in the datafields section we have only **five**:
```
label: a classification label, with possible values including sadness (0), joy (1), love (2), anger (3), fear (4).
```

@GDGauravDutta the dataset author replied in their issue: https://github.com/dair-ai/emotion_dataset/issues/2
> The dataset released is a preprocessed six emotion variant released for educational and research purposes.

@albertovilla the dataset card was fixed with 6 emotions.
Issue: Datasets with tricky task templates
State: closed
Labels: name: Dataset discussion

I'm collecting a list of datasets here that don't follow the "standard" taxonomy and require further investigation to implement task templates for.

## Text classification

* [hatexplain](https://huggingface.co/datasets/hatexplain): ostensibly a form of text classification, but not in the standard `(text, target)` format and each sample appears to be tokenized.
* [muchocine](https://huggingface.co/datasets/muchocine): contains two candidate text columns (long-form and summary) which in principle requires two `TextClassification` templates which is not currently supported 

The `task_templates` API is deprecated in favor of the `train-eval-index` YAML field, so I'm closing this issue.
Issue: seqeval metric does not work with a recent version of sklearn: classification_report() got an unexpected keyword argument 'output_dict'
State: closed
Labels: name: bug

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
seqeval = load_metric("seqeval")
seqeval.compute(predictions=[['A']], references=[['A']])
```

## Expected results
The function computes a dict with metrics

## Actual results
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-39-69a57f5cf06f> in <module>
      1 from datasets import load_dataset, load_metric
      2 seqeval = load_metric("seqeval")
----> 3 seqeval.compute(predictions=[['A']], references=[['A']])

~/p3/lib/python3.7/site-packages/datasets/metric.py in compute(self, *args, **kwargs)
    396             references = self.data["references"]
    397             with temp_seed(self.seed):
--> 398                 output = self._compute(predictions=predictions, references=references, **kwargs)
    399 
    400             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/seqeval/81eda1ff004361d4fa48754a446ec69bb7aa9cf4d14c7215f407d1475941c5ff/seqeval.py in _compute(self, predictions, references, suffix)
     95 
     96     def _compute(self, predictions, references, suffix=False):
---> 97         report = classification_report(y_true=references, y_pred=predictions, suffix=suffix, output_dict=True)
     98         report.pop("macro avg")
     99         report.pop("weighted avg")

TypeError: classification_report() got an unexpected keyword argument 'output_dict'
```

## Environment info
sklearn=0.24
datasets=1.1.3



Sorry, I was using an old version of sequeval
Issue: Add C4
State: closed
Labels: name: dataset request

## Adding a Dataset
- **Name:** *C4*
- **Description:** *https://github.com/allenai/allennlp/discussions/5056*
- **Paper:** *https://arxiv.org/abs/1910.10683*
- **Data:** *https://huggingface.co/datasets/allenai/c4*
- **Motivation:** *Used a lot for pretraining*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Should fix https://github.com/huggingface/datasets/issues/1710

Update on this: I'm computing the checksums of the data files. It will be available soon

Added in #2575 :)
Issue: Load Image Classification Dataset from Local 
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
Yes - we would like to load an image classification dataset with datasets without having to write a custom data loader.

**Describe the solution you'd like**

Given a folder structure with images of each class in each folder, the ability to load these folders into a HuggingFace dataset like "cifar10".

**Describe alternatives you've considered**

Implement ViT training outside of the HuggingFace Trainer and without datasets (we did this but prefer to stay on the main path)

Write custom data loader logic

**Additional context**

We're training ViT on custom dataset


Hi ! Is this folder structure a standard, a bit like imagenet ?
In this case maybe we can consider having a dataset loader for cifar-like, imagenet-like, squad-like, conll-like etc. datasets ?
```python
from datasets import load_dataset

my_custom_cifar = load_dataset("cifar_like", data_dir="path/to/data/dir")
```

Let me know what you think

Yep that would be sweet - closing for now as we found a workaround. 

@lhoestq I think we'll want a generic `image-folder` dataset (same as 'imagenet-like'). This is like `torchvision.datasets.ImageFolder`, and is something vision folks are used to seeing.

Opening this back up, since I'm planning on tackling this. Already posted a quick version of it on my account on the hub.

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
```

Bumping this one following our recent discussion @mariosasko @nateraw :)
Issue: SubjQA wrong boolean values in entries
State: open
Labels: name: bug

## Describe the bug
SubjQA seems to have a boolean that's consistently wrong.

It defines:
- question_subj_level: The subjectiviy level of the question (on a 1 to 5 scale with 1 being the most subjective).
- is_ques_subjective: A boolean subjectivity label derived from question_subj_level (i.e., scores below 4 are considered as subjective)

However, `is_ques_subjective` seems to have wrong values in the entire dataset.

For instance, in the example in the dataset card, we have:
- "question_subj_level": 2
- "is_ques_subjective": false

However, according to the description, the question should be subjective since the `question_subj_level` is below 4


Hi @arnaudstiegler, thanks for reporting. I'm investigating it.

@arnaudstiegler I have just checked that these mismatches are already present in the original dataset: https://github.com/megagonlabs/SubjQA

We are going to contact the dataset owners to report this.

I have:
- opened an issue in their repo: https://github.com/megagonlabs/SubjQA/issues/3
- written an email to all the paper authors

Please [see my response](https://github.com/megagonlabs/SubjQA/issues/3#issuecomment-905160010). There will be a fix in a couple of days.
Issue:  Python Programming Puzzles
State: open
Labels: name: dataset request

## Adding a Dataset
- **Name:** Python Programming Puzzles
- **Description:** Programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis
- **Paper:** https://arxiv.org/pdf/2106.05784.pdf
- **Data:** https://github.com/microsoft/PythonProgrammingPuzzles ([Scrolling through the data](https://github.com/microsoft/PythonProgrammingPuzzles/blob/main/problems/README.md))
- **Motivation:** Spans a large range of difficulty, problems, and domains. A useful resource for evaluation as we don't have a clear understanding of the abilities and skills of extremely large LMs.

Note: it's a growing dataset (contributions are welcome), so we'll need careful versioning for this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


👀 @TalSchuster

Thanks @VictorSanh!
There's also a [notebook](https://aka.ms/python_puzzles) and [demo](https://aka.ms/python_puzzles_study) available now to try out some of the puzzles
Issue: Dataset fingerprint changes after moving the cache directory, which prevent cache reload when using `map`
State: closed

`Dataset.map` uses the dataset fingerprint (a hash) for caching.
However the fingerprint seems to change when someone moves the cache directory of the dataset.

This is because it uses the default fingerprint generation:
1. the dataset path is used to get the fingerprint
2. the modification times of the arrow file is also used to get the fingerprint

To fix that we could set the fingerprint of the dataset to be a hash of (<dataset_name>, <config_name>, <version>, <script_hash>), i.e. a hash of the the cache path relative to the cache directory.
Issue: Improve docs on Enhancing performance
State: open
Labels: name: documentation

In the ["Enhancing performance"](https://huggingface.co/docs/datasets/loading_datasets.html#enhancing-performance) section of docs, add specific use cases:
- How to make datasets the fastest
- How to make datasets take the less RAM
- How to make datasets take the less hard drive mem

cc: @thomwolf 


Hi @albertvillanova, I hope you are doing well.

I am interested in this issue, is this still unresolved and open ?

The link you have provided in the above message directs to a webpage that does not exist.

Thanks and Regards

#self-assign
Issue: Allow latest pyarrow version once segfault bug is fixed
State: closed
Labels: name: enhancement

As pointed out by @symeneses (see https://github.com/huggingface/datasets/pull/2268#issuecomment-860048613), pyarrow has fixed the segfault bug present in version 4.0.0 (see https://issues.apache.org/jira/browse/ARROW-12568):
- it was fixed on 3 May 2021
- version 4.0.1 was released on 19 May 2021 with the bug fix
Issue: Implement layered building
State: open
Labels: name: enhancement

As discussed with @stas00 and @lhoestq (see also here https://github.com/huggingface/datasets/issues/2481#issuecomment-859712190):

> My suggestion for this would be to have this enabled by default.
> 
> Plus I don't know if there should be a dedicated issue to that is another functionality. But I propose layered building rather than all at once. That is:
>
> 1. uncompress a handful of files via a generator enough to generate one arrow file
> 2. process arrow file 1
> 3. delete all the files that went in and aren't needed anymore.
>
> rinse and repeat.
> 
> 1. This way much less disc space will be required - e.g. on JZ we won't be running into inode limitation, also it'd help with the collaborative hub training project
> 2. The user doesn't need to go and manually clean up all the huge files that were left after pre-processing
> 3. It would already include deleting temp files this issue is talking about
> 
> I wonder if the new streaming API would be of help, except here the streaming would be into arrow files as the destination, rather than dataloaders.
Issue: Delete extracted files to save disk space
State: closed
Labels: name: enhancement

As discussed with @stas00 and @lhoestq, allowing the deletion of extracted files would save a great amount of disk space to typical user.

My suggestion for this would be to have this enabled by default.

Plus I don't know if there should be a dedicated issue to that is another functionality. But I propose layered building rather than all at once. That is:

1. uncompress a handful of files via a generator enough to generate one arrow file
2. process arrow file 1
3. delete all the files that went in and aren't needed anymore.

rinse and repeat.

1. This way much less disc space will be required - e.g. on JZ we won't be running into inode limitation, also it'd help with the collaborative hub training project
2. The user doesn't need to go and manually clean up all the huge files that were left after pre-processing
3. It would already include deleting temp files this issue is talking about

I wonder if the new streaming API would be of help, except here the streaming would be into arrow files as the destination, rather than dataloaders.
Issue: Set download/extracted paths configurable
State: open
Labels: name: enhancement

As discussed with @stas00 and @lhoestq, setting these paths configurable may allow to overcome disk space limitation on different partitions/drives.

TODO:
- [x] Set configurable extracted datasets path: #2487
- [x] Set configurable downloaded datasets path: #2488
- [ ] Set configurable "incomplete" datasets path?

For example to be able to send uncompressed and temp  build files to another volume/partition, so that the user gets the minimal disk usage on their primary setup - and ends up with just the downloaded compressed data + arrow files, but outsourcing the huge files and building to another partition. e.g. on JZ there is a special partition for fast data, but it's also volatile, so only temp files should go there.

Think of it as `TMPDIR` so we need the equivalent for `datasets`.
Issue: Issue in timit_asr database
State: closed
Labels: name: bug

## Describe the bug
I am trying to load the timit_asr dataset however only the first record is shown (duplicated over all the rows).
I am using the next code line
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))

The above code result with the same sentence duplicated ten times.
It also happens when I use the dataset viewer at Streamlit .

## Steps to reproduce the bug
from datasets import load_dataset
dataset = load_dataset(“timit_asr”, split=“test”).shuffle().select(range(10))
data = dataset.to_pandas()

# Sample code to reproduce the bug
```

## Expected results
table with different row information

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.4.1 (also occur in the latest version)
- Platform: Linux-4.15.0-143-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 1.15.3 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


This bug was fixed in #1995. Upgrading datasets to version 1.6 fixes the issue!

Indeed was a fixed bug.
Works on version 1.8
Thanks
Issue: cache_dir parameter for load_from_disk ?
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
When using Google Colab big datasets can be an issue, as they won't fit on the VM's disk. Therefore mounting google drive could be a possible solution. Unfortunatly when loading my own dataset by using the _load_from_disk_ function, the data gets cached to the VM's disk:

`
from datasets import load_from_disk

myPreprocessedData = load_from_disk("/content/gdrive/MyDrive/ASR_data/myPreprocessedData")

`
I know that chaching on google drive could slow down learning. But at least it would run.

**Describe the solution you'd like**
Add cache_Dir parameter to the load_from_disk function.

**Describe alternatives you've considered**
It looks like you could write a custom loading script for the load_dataset function. But this seems to be much too complex for my use case. Is there perhaps a template here that uses the load_from_disk function?


Hi ! `load_from_disk` doesn't move the data. If you specify a local path to your mounted drive, then the dataset is going to be loaded directly from the arrow file in this directory. The cache files that result from `map` operations are also stored in the same directory by default.

However note than writing data to your google drive actually fills the VM's disk (see https://github.com/huggingface/datasets/issues/643)

Given that, I don't think that changing the cache directory changes anything.

Let me know what you think

Thanks for your answer! I am a little surprised since I just want to read the dataset.

After debugging a bit, I noticed that the VM’s disk fills up when the tables (generator) are converted to a list:

https://github.com/huggingface/datasets/blob/5ba149773d23369617563d752aca922081277ec2/src/datasets/table.py#L850

If I try to iterate through the table’s generator e.g.: 

`length = sum(1 for x in tables)`

the VM’s disk fills up as well.

I’m running out of Ideas 😄 

Indeed reading the data shouldn't increase the VM's disk. Not sure what google colab does under the hood for that to happen

Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636