konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Dataset Viewer issue for strombergnlp/ipm_nel : preview is empty, no error message

### Link

https://huggingface.co/datasets/strombergnlp/ipm_nel/viewer/ipm_nel/train

### Description

The viewer is blank. I tried my best to emulate a dataset with a working viewer, but this one just doesn't seem to want to come up. What did I miss?

### Owner

Yes

Hi @leondz, thanks for reporting.

Indeed, the dataset viewer relies on the dataset being streamable (passing `streaming=True` to `load_dataset`). Whereas most of the datastes are streamable out of the box (thanks to our implementation of streaming), there are still some exceptions.

In particular, in your case, that is due to the data file being TAR. This format is not streamable out of the box (it does not allow random access to the archived files), but we use a trick to allow streaming: using `dl_manager.iter_archive`.

Let me know if you need some help: I could push a commit to your repo with the fix.

Ah, right! The preview is working now, but this explanation is good to know, thank you. I'll prefer formats with random file access supported in datasets.utils.extract in future, and try out this fix for the tarfiles :)
Don't do unnecessary list type casting to avoid replacing None values by empty lists

In certain cases, `None` values are replaced by empty lists when casting feature types.

It happens every time you cast an array of nested lists like [None, [0, 1, 2, 3]] to a different type (to change the integer precision for example). In this case you'd get [[], [0, 1, 2, 3]] for example. This issue comes from PyArrow, see the discussion in https://github.com/huggingface/datasets/issues/3676

This issue also happens when no type casting is needed, because casting is supposed to be a no-op in this case. But as https://github.com/huggingface/datasets/issues/3676 shown, it's not the case and `None` are replaced by empty lists even if we cast to the exact same type.

In this PR I just workaround this bug in the case where no type casting is needed. In particular, I only call `pa.ListArray.from_arrays` only when necessary.

I also added a warning when some `None` are effectively replaced by empty lists. I wanted to raise an error in this case, but maybe we should wait a major update to do so

This PR fixes this particular case, that is occurring in `run_qa.py` in `transformers`:
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
# before:
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
#
# now:
#              b
# 0  [None, [0]]
# 1  [None, [0]]
# 2  [None, [0]]
# 3  [None, [0]]
```

cc @sgugger 

_The documentation is not available anymore as the PR was closed or merged._

Quick question about the message in the warning. You say "will be fixed in a future major version" but don't you mean "will raise an error in a future major version"?

Right ! Good catch, thanks, I updated the message to say "will raise an error in a future major version"
Enable label alignment for token classification datasets

This PR extends the `Dataset.align_labels_with_mapping()` method to support alignment of label mappings between datasets and models for token classification (e.g. NER).

Example of usage:

```python
from datasets import load_dataset

ner_ds = load_dataset("conll2003", split="train")
# returns [3, 0, 7, 0, 0, 0, 7, 0, 0]
ner_ds[0]["ner_tags"]
# hypothetical model mapping with O <--> B-LOC
label2id = {
    "B-LOC": "0",
    "B-MISC": "7",
    "B-ORG": "3",
    "B-PER": "1",
    "I-LOC": "6",
    "I-MISC": "8",
    "I-ORG": "4",
    "I-PER": "2",
    "O": "5"
  }
ner_aligned_ds = ner_ds.align_labels_with_mapping(label2id, "ner_tags")
# returns [3, 5, 7, 5, 5, 5, 7, 5, 5]
ner_aligned_ds[0]["ner_tags"]
```

Context: we need this in AutoTrain to automatically align datasets / models during evaluation. cc @abhishekkrthakur 

_The documentation is not available anymore as the PR was closed or merged._

Hmm, not sure why the Windows tests are failing with:

```
Did not find path entry C:\tools\miniconda3\bin
C:\tools\miniconda3\envs\py37\python.exe: No module named pytest
```

Edit: running the CI again fixed the problem 🙃 

> One last nit and we can merge then

Thanks, done!
CommonSenseQA has missing and inconsistent field names

## Describe the bug
In short, CommonSenseQA implementation is inconsistent with the original dataset.

More precisely, we need to:

1. Add the dataset matching "id" field. The current dataset, instead, regenerates monotonically increasing id.  
2. The [“question”][“stem”] field is flattened into "question". We should match the original dataset and unflatten it
3. Add the missing "question_concept" field in the question tree node
4. Anything else? Go over the data structure of the newly repaired CommonSenseQA and make sure it matches the original

## Expected results
Every data item of the CommonSenseQA should structurally and data-wise match the original CommonSenseQA dataset.

## Actual results
TBD

## Environment info
- `datasets` version: 2.1.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.2

Thanks for reporting, @vblagoje.

I'm opening a PR to address this.
Fix typo in logging docs

This PR fixes #4271.

_The documentation is not available anymore as the PR was closed or merged._

> This PR fixes #4271.

Things have not changed when searching "tqdm" in the Dataset document. The second result still performs as "Enable".

Hi @jiangwy99, the fix will appear on the `main` version of the docs:

![Screen Shot 2022-05-04 at 8 38 29 AM](https://user-images.githubusercontent.com/59462357/166718225-6848ab91-87d1-4572-9912-40a909af6cb9.png)


Fixed now, thanks.
Add HF Speech Bench to Librispeech Dataset Card

Adds the HF Speech Bench to Librispeech Dataset Card in place of the Papers With Code Leaderboard. Should improve usage and visibility of this leaderboard! Wondering whether this can also be done for [Common Voice 7](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) and [8](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0) through someone with permissions? 

cc @patrickvonplaten: more leaderboard promotion!

_The documentation is not available anymore as the PR was closed or merged._
Rename imagenet2012 -> imagenet-1k

On the Hugging Face Hub, users refer to imagenet2012 (from #4178 ) as imagenet-1k in their model tags.

To correctly link models to imagenet, we should rename this dataset `imagenet-1k`.

Later we can add `imagenet-21k` as a new dataset if we want.

Once this one is merged we can delete the `imagenet2012` dataset repository on the Hub.

EDIT: to complete the rationale on why we should name it `imagenet-1k`:
If users specifically added the tag `imagenet-1k` , then it could be for two reasons (not sure which one is predominant), either they
- wanted to make it explicit that it’s not 21k -> the distinction is important for the community
- or they have been following this convention from other models -> the convention implicitly exists already

> Later we can add imagenet-21k as a new dataset if we want.

isn't it what models refer to as `imagenet` already?

> isn't it what models refer to as imagenet already?

I wasn't sure, but it looks like it indeed. Therefore having a dataset `imagenet` for ImageNet 21k makes sense actually.

EDIT: actually not all `imagenet` tag refer to ImageNet 21k - we will need to correct some of them

_The documentation is not available anymore as the PR was closed or merged._

should we remove the repo mirror on the hub side or will you do it?
Add mr_polarity movie review sentiment classification

Add the MR (Movie Review) dataset. The original dataset contains sentences from Rotten Tomatoes labeled as either "positive" or  "negative". 

Homepage: [https://www.cs.cornell.edu/people/pabo/movie-review-data/](https://www.cs.cornell.edu/people/pabo/movie-review-data/)
paperswithcode: [https://paperswithcode.com/dataset/mr](https://paperswithcode.com/dataset/mr)

- [ ] I was not able to generate dummy data, the original dataset files have ".pos" and ".neg" as file extensions so the auto-generator does not work. Is it fine like this or should dummy data be added?


whoops just found https://huggingface.co/datasets/rotten_tomatoes
Fix/start token mask issue and update documentation

This pr fixes a couple bugs:

1) the perplexity was calculated with a 0 in the attention mask for the start token, which was causing high perplexity scores that were not correct
2) the documentation was not updated

_The documentation is not available anymore as the PR was closed or merged._

> Good catch ! Thanks :)
> 
> Next time can you describe your fix in the Pull Request description please ?

Thanks. Also whoops, sorry about not being very descriptive. I updated the pull request description, and will keep this in mind for future PRs.
Replace data URL in SAMSum dataset and support streaming

This PR replaces data URL in SAMSum dataset:
- original host (arxiv.org) does not allow HTTP Range requests
- we have hosted the data on the Hub (license: CC BY-NC-ND 4.0)

Moreover, it implements support for streaming.

Fix #4146.
Related to: #4236.

CC: @severo 

_The documentation is not available anymore as the PR was closed or merged._
Bump PyArrow Version to 6

Fixes #4152 

This PR updates the PyArrow version to 6 in setup.py, CI job files .circleci/config.yaml and .github/workflows/benchmarks.yaml files.
This will fix ArrayND error which exists in pyarrow 5.

_The documentation is not available anymore as the PR was closed or merged._

Updated meta.yaml as well. Thanks.

I'm OK with bumping PyArrow to version 6 to match the version in Colab, but maybe a better solution would be to stop using extension types in our codebase to avoid similar issues.

> but maybe a better solution would be to stop using extension types in our codebase to avoid similar issues.

I agree, not much attention has been payed to extension arrays in the latest developments of Arrow anyway.

Let's not use them more that what we do right now, and try to remove them at one point
The data preview of XGLUE

It seems that something wrong with the data previvew of XGLUE

![image](https://user-images.githubusercontent.com/49108847/165700611-915b4343-766f-4b81-bdaa-b31950250f06.png)


Thanks for reporting @czq1999.

Note that the dataset viewer uses the dataset in streaming mode and that not all datasets support streaming yet.

That is the case for XGLUE dataset (as the error message points out): this must be refactored to support streaming. 

Fixed, thanks @albertvillanova !

https://huggingface.co/datasets/xglue

<img width="824" alt="Capture d’écran 2022-04-29 à 10 23 14" src="https://user-images.githubusercontent.com/1676121/165909391-9f98d98a-665a-4e57-822d-8baa2dc9b7c9.png">
Support to load dataset with TSV files by passing only dataset name

This PR implements support to load a dataset (w/o script) containing TSV files by passing only the dataset name (no need to pass `sep='\t'`):
```python
ds = load_dataset("dataset/name")
```

The refactoring allows for future builder kwargs customizations based on file extension.

Related to #4238.

_The documentation is not available anymore as the PR was closed or merged._
Fix yield for crd3

Modified the `_generate_examples` function to consider all the turns for a chunk id as a single example
Modified the features accordingly

```
"turns": [
                        {
                            "names": datasets.features.Sequence(datasets.Value("string")),
                            "utterances": datasets.features.Sequence(datasets.Value("string")),
                            "number": datasets.Value("int32"),
                        }
                    ],
                }
```

I wasn't able to run `datasets-cli dummy_data datasets` command. Is there a workaround for this? 

I don't think you need to generate new dummy data, since they're in the same format as the original data.

The CI is failing because of this error:
```python
>                       turn["names"] = turn["NAMES"]
E                       TypeError: tuple indices must be integers or slices, not str
```

Do you know what could cause this ? If I understand correctly, `turn` is supposed to be a list of dictionaries right ?

> ```  
>   
> Do you know what could cause this ? If I understand correctly, turn is supposed to be a list of dictionaries right ?
> ```

This is strange. Let me look into this. As per https://github.com/RevanthRameshkumar/CRD3/blob/master/data/aligned%20data/c%3D2/C1E001_2_0.json TURNS is a list of dictionaries. So when we iterate over `row["TURNS]"` each `turn` is essentially a dictionary. Not sure why it's being considered a tuple here.
Replace data URL in big_patent dataset and support streaming

This PR replaces the Google Drive URL with our Hub one, once the data owners have approved to host their data on the Hub.

Moreover, this PR makes the dataset streamable.

Fix #4217.

_The documentation is not available anymore as the PR was closed or merged._

I first uploaded the data files to the Hub: I think it is a good option because we have git lfs to track versions and changes. Moreover people will be able to make PRs to propose updates on the data files.
- I would have preferred to upload it it to the "data" org namespace, but it is already taken (although not used): might be possible to take it?

As an alternative (and to be consistent with previous datasets), I also uploaded the data files to our AWS bucket.

We should decide which to use (now and for future datasets) and set it here before merging. We should remove the data files for the non-chosen option.

CC: @lhoestq @mariosasko @polinaeterna 

Would it make sense to make the dataset a community one (so, create an organization for it) and store the script and the data in a single repository? Just as it is for most of the datasets. That way we can also access the data using a relative path inside the repo (that's not the point though). The point is that to me it seems a bit more straightforward to store everything in one place. 

I guess the strong argument against this logic is that in this case the canonical version won't work... But maybe there is some redirecting mechanism I don't know about? :)

Anyway, I'm in favor of hosting data on the Hub instead of AWS :) 

I also think storing everything in one place/single repository is the best option.

@polinaeterna Canonical datasets also support data files (see the [`red_caps` repo](https://huggingface.co/datasets/red_caps/tree/main) for instance) 

Thanks @polinaeterna and @mariosasko for your comments.

Yes, definitely it is much better to have everything in the same repo. 

I'm transferring their data files to the Hub under "big_patent" and deleting them from the other repo and AWS.
How to load VERY LARGE dataset?

### System Info

```shell
I am using transformer trainer while meeting the issue.
The trainer requests torch.utils.data.Dataset as input, which loads the whole dataset into the memory at once. Therefore, when the dataset is too large to load, there's nothing I can do except using IterDataset, which loads samples of data seperately, and results in low efficiency. 
I wonder if there are any tricks like Sharding in huggingface trainer.
Looking forward to your reply.
```


### Who can help?

Trainer: @sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

```shell
I wonder if there are any tricks like fairseq Sharding very large datasets https://fairseq.readthedocs.io/en/latest/getting_started.html.
Thanks a lot!
```


The `Trainer` support `IterableDataset`, not just datasets.
Autoeval config

Added autoeval config to imdb as pilot

_The documentation is not available anymore as the PR was closed or merged._

Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424

The tests are failing due to the changed metadata:

```
got an unexpected keyword argument 'train-eval-index'
```

I think you can fix this by updating the `DatasetMetadata` class and implementing an appropriate `validate_train_eval_index()` function

@lhoestq we are working with an arbitrary set of tags for `autoeval config`. See https://github.com/huggingface/autonlp-backend/issues/414
I need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this?

On the question of validating the `train-eval-index` metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).

For me, the required fields include:

* `config`
* `task`
* `task_id`
* `splits` (train / validation / eval)
* `col_mapping`
* `metrics` (checking that each one has `type`, `name`) 

Here I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.

WDYT @lhoestq ?

Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it

I ended up improving the metadata validation in this PR x)

In particular:
- I added support YAML keys with dashes instead of underscores for `train-eval-index`
- I added `train-eval-index` validation  with `validate_train_eval_index`. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want

Let me know if it sounds good to you ! I think we can improve `validate_train_eval_index` in another PR

Come on windows... I didn't do anything advanced...

Anyway, will try to fix this when I get back home x)

> Come on windows... I didn't do anything advanced...
> 
> Anyway, will try to fix this when I get back home x)

Hehe, thanks!

Thanks, @lhoestq this is great! 

Did I just fix it for windows and now it fails on linux ? xD

> Did I just fix it for windows and now it fails on linux ? xD

Looks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅 

The worst is that the tests pass locally both on my windows and my linux x)

Ok fixed it, the issue came from python 3.6 that doesn't return the right `__origin__` for Dict and List types

> Alright thanks for adding the first Autoeval config ! :D

Woohoo! Thank you so much 🤗 

This is cool!
Why the `conll2003` dataset on huggingface only contains the `en` subset? Where is the  German data?

![image](https://user-images.githubusercontent.com/37113676/165416606-96b5db18-b16c-4b6b-928c-de8620fd943e.png)

But on huggingface datasets:
![image](https://user-images.githubusercontent.com/37113676/165416649-8fd77980-ca0d-43f0-935e-f398ba8323a4.png)

Where is the  German data?

Thanks for reporting @beyondguo.

Indeed, we generate this dataset from this raw data file URL: https://data.deepai.org/conll2003.zip
And that URL only contains the English version.

The German data requires payment

The [original task page](https://www.clips.uantwerpen.be/conll2003/ner/) states "The German data is a collection of articles from the Frankfurter Rundschau. The named entities have been annotated by people of the University of Antwerp. Only the annotations are available here. In order to build these data sets you need access to the ECI Multilingual Text Corpus. It can be ordered from the Linguistic Data Consortium (2003 non-member price: US$ 35.00)."

Inflation since 2003 has also affected LDC's prices, and today the dataset [LDC94T5](https://catalog.ldc.upenn.edu/LDC94T5) is available under license for $75 a copy. The [license](https://catalog.ldc.upenn.edu/license/eci-slash-mci-user-agreement.pdf) includes a non-distribution condition, which is probably why the data has not turned up openly.

The ACL hold copyright of this data; I'll mail them and anyone I can find at ECI to see if they'll open this up now. After all, it worked with Microsoft 3DMM, why not here too, after 28 years? :)


Closing this issue as we are not allowed to share publicly the German subset.
Add pearsonr mc, update functionality to match the original docs

- adds pearsonr metric card
- adds ability to return p-value
    - p-value was mentioned in the original docs as a return value, but there was no option to return it. I updated the _compute function slightly to have an option to return the p-value.

_The documentation is not available anymore as the PR was closed or merged._

thank you @lhoestq!! :hugs:
Dictionary Feature

Hi, I'm trying to create the loading script for a dataset in which one feature is a list of dictionaries, which afaik doesn't fit very well the values and structures supported by Value and Sequence. Is there any suggested workaround, am I missing something?

Thank you in advance.

Hi @jordiae,

Instead of the `Sequence` feature, you can use just a regular list: put the dict between `[` and `]`:
```python
"list_of_dict_feature": [
    {
        "key1_in_dict": datasets.Value("string"),
        "key2_in_dict": datasets.Value("int32"),
        ...
    }
],
```

Feel free to re-open this issue if that does not work for your use case.

> Hi @jordiae,
> 
> Instead of the `Sequence` feature, you can use just a regular list: put the dict between `[` and `]`:
> 
> ```python
> "list_of_dict_feature": [
>     {
>         "key1_in_dict": datasets.Value("string"),
>         "key2_in_dict": datasets.Value("int32"),
>         ...
>     }
> ],
> ```
> 
> Feel free to re-open this issue if that does not work for your use case.

Thank you
Altered faiss installation comment


_The documentation is not available anymore as the PR was closed or merged._

Hi ! Can you explain why this change is needed ?

Facebook recommends installing FAISS using conda (https://github.com/facebookresearch/faiss/blob/main/INSTALL.md). pip does not seem to have the latest version of FAISS. The latest version of faiss is 1.7.2 (https://anaconda.org/conda-forge/faiss), but the latest one available through pip is 1.5.3 (https://pypi.org/project/faiss/).
Big_Patent dataset broken

## Dataset viewer issue for '*big_patent*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/big_patent/viewer/all/train)*

*Unable to view because it says FileNotFound, also cannot download it through the python API*

Am I the one who added this dataset ? No


Thanks for reporting. The issue seems not to be directly related to the dataset viewer or the `datasets` library, but instead to it being hosted on Google Drive.

See related issues: https://github.com/huggingface/datasets/issues?q=is%3Aissue+is%3Aopen+drive.google.com

To quote [@lhoestq](https://github.com/huggingface/datasets/issues/4075#issuecomment-1087362551):

> PS: if possible, please try to not use Google Drive links in your dataset script, since Google Drive has download quotas and is not always reliable.



We should find out if the dataset license allows redistribution and contact the data owners to propose them to host their data on our Hub.

The data owners have agreed on hosting their data on the Hub.
Avoid recursion error in map if example is returned as dict value

I noticed this bug while answering [this question](https://discuss.huggingface.co/t/correct-way-to-create-a-dataset-from-a-csv-file/15686/11?u=mariosasko). 

This code replicates the bug:
```python
from datasets import Dataset
dset = Dataset.from_dict({"en": ["aa", "bb"], "fr": ["cc", "dd"]})
dset.map(lambda ex: {"translation": ex})
```
and this is the fix for it (before this PR):
```python
from datasets import Dataset
dset = Dataset.from_dict({"en": ["aa", "bb"], "fr": ["cc", "dd"]})
dset.map(lambda ex: {"translation": dict(ex)})
```

Internally, this can be fixed by merging two dicts via dict unpacking (instead of `dict.update) `in `Dataset.map`, which avoids creating recursive dictionaries.

P.S. `{**a, **b}` is slightly more performant than `a.update(b)` in my bencmarks.

_The documentation is not available anymore as the PR was closed or merged._
Add Recall Metric Card

What this PR mainly does:
- add metric card for recall metric
- update docs in recall python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)

_The documentation is not available anymore as the PR was closed or merged._

This looks good to me!
Add Precision Metric Card

What this PR mainly does:
- add metric card for precision metric
- update docs in precision python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)

_The documentation is not available anymore as the PR was closed or merged._
Update GH template for dataset viewer issues

Update template to use new issue forms instead.

With this PR we can check if this new feature is useful for us.

Once validated, we can update the other templates.

CC: @severo 

_The documentation is not available anymore as the PR was closed or merged._

You can see rendering at: https://github.com/huggingface/datasets/blob/6b48fedbdafe12a42c7b6edcecc32820af1a4822/.github/ISSUE_TEMPLATE/dataset-viewer.yml
Add remove_columns=True

This should fix all the issue we have with in place operations in mapping functions. This is crucial as where we do some weird things like:
```
def apply(batch):
    batch_size = len(batch["id"])
    batch["text"] = ["potato" for _ range(batch_size)]
    return {}

# Columns are: {"id": int}
dset.map(apply, batched=True, remove_columns="text") # crashes because `text` is not in the original columns
dset.map(apply, batched=True) # mapped datasets has `text` column
```

In this PR we suggest to have `remove_columns=True` so that we ignore the input completely, and just use the output to generate mapped dataset. This means that inplace operations won't have any effects anymore.

_The documentation is not available anymore as the PR was closed or merged._

Any reason why we can't just do `[inputs.copy()]` in this line for in-place operations to not have effects anymore:
https://github.com/huggingface/datasets/blob/bf432011ff9155a5bc16c03956bc63e514baf80d/src/datasets/arrow_dataset.py#L2232.

(in the `batched` case, we can also copy the inputs' values (list objects) to ignore in-place modifications to the inputs' columns)

I think `remove_columns=True` has no meaning, so I'm not a fan of this change.

@mariosasko copy does have a cost associated with it ... and plus you'll have to consider `deepcopy` Imagine columnds that are list of list of list of list .... Though I have to agree that `remove_columns=True` doesn't make sense (but, IMO, neither does it in its current use-case as it should refer to `input_columns`) 

Okay closing this PR for the following reasons:
 - `remove_columns=True` was expected to keep the `.update`-like operator for `.map`. I initially thought it would be a good way to ignore function side effects and only keep output of that function (cf. PR description).
 - expected `remove_columns=True` is a bad API according to @mariosasko and introduces unecessary changes for little gain (strictly equivalent to `remove_columns=dset.column_names`)
Deprecate `shard_size` in `push_to_hub` in favor of `max_shard_size`

This PR adds a `max_shard_size` param to `push_to_hub` and deprecates `shard_size` in favor of this new param to have a more descriptive name (a shard has at most the `shard_size` bytes in `push_to_hub`) for the param and to align the API with [Transformers](https://github.com/huggingface/transformers/blob/ff06b177917384137af2d9585697d2d76c40cdfc/src/transformers/modeling_utils.py#L1350).


_The documentation is not available anymore as the PR was closed or merged._
Document librispeech configs

Added an example of how to load one config or the other

I think the main purpose of #4179 was how to be able to load both configs into one, so should we maybe add this part of the code: https://github.com/huggingface/datasets/issues/4179#issuecomment-1102383717 

to the doc? 

Actually @lhoestq would this work given that they have different split names: https://huggingface.co/datasets/librispeech_asr#data-splits ? 

This doc extension does not explain why I can't simply load the whole dataset. Or what workaround I need to get the whole dataset, which is what people usually want for Librispeech.

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq, I can add a `"all"` config to Librispeech have the datasets already cached somewhere 

I'm closing this PR then, feel free to continue the discussion in https://github.com/huggingface/datasets/issues/4179
Zenodo.org download is not responding

## Describe the bug
Source download_url from zenodo.org does not respond. 
`_DOWNLOAD_URL = "https://zenodo.org/record/2787612/files/SICK.zip?download=1"`
Other datasets also use zenodo.org to store data and they cannot be downloaded as well.

It would be better to actually use more reliable way to store original data like s3 bucket.

## Steps to reproduce the bug
```python
load_dataset("sick")
```

## Expected results
Dataset should be downloaded.

## Actual results
ConnectionError: Couldn't reach https://zenodo.org/record/2787612/files/SICK.zip?download=1 (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='zenodo.org', port=443): Read timed out. (read timeout=100)")))

## Environment info
- `datasets` version: 2.1.0
- Platform: Darwin-21.4.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 7.0.0
- Pandas version: 1.3.5


[Off topic but related: Is the uptime of S3 provably better than Zenodo's?]

Hi @dkajtoch, please note that at HuggingFace we are not hosting this dataset: we are just using a script to download their data file and create a dataset from it.

It was the dataset owners decision to host their data at Zenodo. You can see this on their website: https://marcobaroni.org/composes/sick.html

And yes, you are right: Zenodo is currently having some incidents and people are reporting problems from it.

On the other hand, we could contact the data owners and propose them to host their data at our Hugging Face Hub.

@julien-c I guess so.


Thanks @albertvillanova. I know that the problem lies in the source data. I just wanted to point out that these kind of problems are unavoidable without having one place where data sources are cached. Websites may go down or data sources may move. Having a copy in Hugging Face Hub would be a great solution. 

Definitely, @dkajtoch! But we have to ask permission to the data owners. And many dataset licenses directly forbid data redistribution: in those cases we are not allowed to host their data on our Hub.

Ahhh good point! License is the problem :(
Support streaming FLEURS dataset

## Dataset viewer issue for '*name of the dataset*'

https://huggingface.co/datasets/google/fleurs

```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://storage.googleapis.com/xtreme_translations/FLEURS/af_za.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

Am I the one who added this dataset ? Yes

Can I fix this somehow in the script? @lhoestq @severo 


Yes, you just have to use `dl_manager.iter_archive` instead of `dl_manager.download_and_extract`.

That's because `download_and_extract` doesn't support TAR archives in streaming mode.

Tried to make it streamable, but I don't think it's really possible. @lhoestq @polinaeterna maybe you guys can check: 
https://huggingface.co/datasets/google/fleurs/commit/dcf80160cd77977490a8d32b370c027107f2407b 

real quick. 

I think the problem is that we cannot ensure that the metadata file is found before the audio. Or is this possible somehow @lhoestq ? 

@patrickvonplaten I think the metadata file should be found first because the audio files are contained in a folder next to the metadata files (just as in common voice), so the metadata files should be "on top of the list"   as they are closer to the root in the directories hierarchy 

@patrickvonplaten but apparently it doesn't... I don't really know why.

Yeah! Any ideas what could be the reason here? cc @lhoestq ?

The order of the files is determined when the TAR archive is created, depending on the commands the creator ran.
If the metadata file is not at the beginning of the file, that makes streaming completely inefficient. In this case the TAR archive needs to be recreated in an appropriate order.

Actually we could maybe just host the metadata file ourselves and then stream the audio data only. Don't think that this would be a problem for the FLEURS authors (I can ask them :-)) 

I made a PR to their repo to support streaming (by uploading the metadata file to the Hub). See:
- https://huggingface.co/datasets/google/fleurs/discussions/4

I'm closing this issue as the PR above has been merged.
[feat] Add ImageNet dataset

To use the dataset download the tar file
[imagenet_object_localization_patched2019.tar.gz](https://www.kaggle.com/competitions/imagenet-object-localization-challenge/data?select=imagenet_object_localization_patched2019.tar.gz) from Kaggle and then point the datasets library to it by using:

```py
from datasets import load_dataset

dataset = load_dataset("imagenet",
data_dir="/path/to/imagenet_object_localization_patched2019.tar.gz")
```

Currently train and validation splits are supported.

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the comments. I believe I have addressed all of them and also decreased the size of the dummy data file, so it should be ready for a re-review. I also made a change to allow adding synset mapping and valprep script in config in case we add ImageNet 21k some time later. 

@lhoestq I have updated the PR to address all of the review comments.
Adding missing subsets to the `SemEval-2018 Task 1` dataset

This dataset for the [1st task of SemEval-2018](https://competitions.codalab.org/competitions/17751) competition was missing all subtasks except for subtask 5. I added another two subtasks (subtask 1 and 2), which are each comprised of 12 additional data subsets: for each language in En, Es, Ar, there are 4 datasets, broken down by emotions (anger, fear, joy, sadness).

## Remaining questions

I wasn't able to find any documentation about how one should make PRs to modify datasets. Because of that, I just did my best to integrate the new data into the code, and tested locally that this worked. I'm sorry if I'm not respecting your contributing guidelines – if they are documented somewhere, I'd appreciate if you could send a pointer!

Not sure how `dataset_infos.json` and `dummy` should be updated. My understanding is that they were automatically generated at the time of the original dataset creation?

Datasets are not tracked in this repository anymore. You should move this PR to the [discussions page of this dataset](https://huggingface.co/datasets/sem_eval_2018_task_1/discussions)
Very slow between two operations

Hello, in the processing stage, I use two operations. The first one : map + filter, is very fast and it uses the full cores, while the socond step is very slow and did not use full cores. 

Also, there is a significant lag between them.  Am I missing something ?



 ```
raw_datasets = raw_datasets.map(split_func, 
                batched=False,
                num_proc=args.preprocessing_num_workers,
                load_from_cache_file=not args.overwrite_cache, 
                desc = "running split para ==>")\
                .filter(lambda example: example['text1']!='' and example['text2']!='', 
                    num_proc=args.preprocessing_num_workers, desc="filtering ==>")


    processed_datasets = raw_datasets.map(
        preprocess_function,
        batched=True, 
        num_proc=args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not args.overwrite_cache,
        desc="Running tokenizer on dataset===>",
    )
```
Stream private zipped images

As mentioned in https://github.com/huggingface/datasets/issues/4139 it's currently not possible to stream private/gated zipped images from the Hub.

This is because `Image.decode_example` does not handle authentication. Indeed decoding requires to access and download the file from the private repository.

In this PR I added authentication to `Image.decode_example` via a `token_per_repo_id` optional argument. I first wanted to just pass `use_auth_token` but a single `Image` instance can be responsible of decoding images from a combination of several datasets together (from `interleave_datasets` for example). Therefore I just used a dictionary `repo_id` -> `token` instead.

I'm getting the `repo_id` from the dataset builder (I replaced the `namepace` attribute with `repo_id`)

I did the same for `Audio.decode_example`

cc @SBrandeis @severo 

_The documentation is not available anymore as the PR was closed or merged._

oops looks like some tests are failing sorry, will fix them tomorrow

EDIT: not today but asap hopefully

cc @mariosasko this is ready for review, let me know what you think !
Optional Content Warning for Datasets

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

We now have hate speech datasets on the hub, like this one: https://huggingface.co/datasets/HannahRoseKirk/HatemojiBuild

I'm wondering if there is an option to select a content warning message that appears before the dataset preview? Otherwise, people immediately see hate speech when clicking on this dataset.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Implementation of a content warning message that separates users from the dataset preview until they click out of the warning.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Possibly just a way to remove the dataset preview completely? I think I like the content warning option better, though.

**Additional context**
Add any other context about the feature request here.


Hi! You can use the `extra_gated_prompt` YAML field in a dataset card for displaying custom messages/warnings that the user must accept before gaining access to the actual dataset. This option also keeps the viewer hidden until the user agrees to terms. 

Hi @mariosasko, thanks for explaining how to add this feature. 

If the current dataset yaml is:
```
---
annotations_creators:
- expert
language_creators:
- expert-generated
languages:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: HatemojiBuild
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- hate-speech-detection
---
```

Can you provide a minimal working example of how to added the gated prompt?

Thanks!

```
---
annotations_creators:
- expert
language_creators:
- expert-generated
languages:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: HatemojiBuild
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- hate-speech-detection
extra_gated_prompt: "This repository contains harmful content."
---
```
\+ enable `User Access requests` under the Settings pane.

There's a brief guide here https://discuss.huggingface.co/t/how-to-customize-the-user-access-requests-message/13953 , and you can see the field in action here, https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/blob/main/README.md (you need to agree the terms in the Dataset Card pane to be able to access the files pane, so this comes up 403 at first).

And a working example here! https://huggingface.co/datasets/DDSC/dkhate :) Great to be able to mitigate harms in text.

-- is there a way to gate content anonymously, i.e. without registering which users access it?

+1 to @leondz's question. One scenario is if you don't want the dataset to be indexed by search engines or viewed in browser b/c of upstream conditions on data, but don't want to collect emails. Some ability to turn off the dataset viewer or add a gating mechanism without emails would be fantastic.
Add Visual Genome


_The documentation is not available anymore as the PR was closed or merged._

Hum there seems to be some issues with tasks in test:
 - some tasks don't fit anything in `tasks.json`. Do I remove them in `task_categories`?
 - some tasks should exist, typically `visual-question-answering` (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my `master` is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?
 
 cc @mariosasko @lhoestq 

> some tasks don't fit anything in tasks.json. Do I remove them in task_categories?

You can keep them, but add `other-` as a prefix to those tasks to make the CI ignore it

> some tasks should exist, typically visual-question-answering (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my master is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?

Feel free to merge upstream/master into your branch ;)

EDIT: actually I just noticed you've already done this, thanks !

After offline discussions: will keep that image essentially it's necessary as I have a mapping that creates a mapping between url and local path (images are downloaded via a zip file) and dummy data needs to store that dummy image. The issue is when I read an annotation, I get a url, compute the local path, and basically I assume the local path exists since I've extracted all the images ... This isn't true if dummy data doesn't have all the images, so instead I've added a script that "fixes" the dummy data after using the CLI, it essentially adds the dummy image in the zip corresponding to the url.
RGBA images not showing

## Dataset viewer issue for ceyda/smithsonian_butterflies_transparent

[**Link:** *link to the dataset viewer page*](https://huggingface.co/datasets/ceyda/smithsonian_butterflies_transparent)

![image](https://user-images.githubusercontent.com/15624271/163117683-e91edb28-41bf-43d9-b371-5c62e14f40c9.png)

Am I the one who added this dataset ? Yes

👉  More of a general issue of 'RGBA' png images not being supported 
(the dataset itself is just for the huggan sprint and not that important, consider it just an example)

Thanks for reporting. It's a known issue, and we hope to fix it soon.

Fixed, thanks!
Adding STSb-TR dataset

Semantic Textual Similarity benchmark Turkish (STSb-TR) dataset introduced in our paper [Semantic Similarity Based Evaluation for Abstractive News Summarization](https://aclanthology.org/2021.gem-1.3.pdf) added.

Thanks for your contribution, @figenfikri.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Generate tasks.json taxonomy from `huggingface_hub`


_The documentation is not available anymore as the PR was closed or merged._

Ok recomputed the json file, this should be ready to review now! @lhoestq 

Note: the generated JSON from `hf/hub-docs` can be found in the output of a GitHub Action run on that repo, for instance in https://github.com/huggingface/hub-docs/runs/6006686983?check_suite_focus=true

(click on "Run export-tasks script")

Should we not add the tasks with hideInDatasets?

yes, probably true – i'll change that in a PR in `hub-docs`

Yes that's good :) feel free to merge

thanks to the both of you!
ArrayND error in pyarrow 5

As found in https://github.com/huggingface/datasets/pull/3903, The ArrayND features fail on pyarrow 5:
```python
import pyarrow as pa
from datasets import Array2D
from datasets.table import cast_array_to_feature

arr = pa.array([[[0]]])
feature_type = Array2D(shape=(1, 1), dtype="int64")
cast_array_to_feature(arr, feature_type)
```
raises
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-04610f9fa78c> in <module>
----> 1 cast_array_to_feature(pa.array([[[0]]]), Array2D(shape=(1, 1), dtype="int32"))

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in cast_array_to_feature(array, feature, allow_number_to_str)
   1806         return array_cast(array, get_nested_type(feature), allow_number_to_str=allow_number_to_str)
   1807     elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1808         return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
   1809     raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
   1810 

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_number_to_str)
   1705         array = array.storage
   1706     if isinstance(pa_type, pa.ExtensionType):
-> 1707         return pa_type.wrap_array(array)
   1708     elif pa.types.is_struct(array.type):
   1709         if pa.types.is_struct(pa_type) and (

AttributeError: 'Array2DExtensionType' object has no attribute 'wrap_array'
```

The thing is that `cast_array_to_feature` is called when writing an Arrow file, so creating an Arrow dataset using any ArrayND type currently fails.

`wrap_array` has been added in pyarrow 6, so we can either bump the required pyarrow version or fix this for pyarrow 5

Where do we bump the required pyarrow version? Any inputs on how I fix this issue? 

We need to bump it in `setup.py` as well as update some CI job to use pyarrow 6 instead of 5 in `.circleci/config.yaml` and `.github/workflows/benchmarks.yaml`