NyssaW/github-issues huggingface dataset:
Add remove_columns=True

This should fix all the issue we have with in place operations in mapping functions. This is crucial as where we do some weird things like:
```
def apply(batch):
    batch_size = len(batch["id"])
    batch["text"] = ["potato" for _ range(batch_size)]
    return {}

# Columns are: {"id": int}
dset.map(apply, batched=True, remove_columns="text") # crashes because `text` is not in the original columns
dset.map(apply, batched=True) # mapped datasets has `text` column
```

In this PR we suggest to have `remove_columns=True` so that we ignore the input completely, and just use the output to generate mapped dataset. This means that inplace operations won't have any effects anymore.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Any reason why we can't just do `[inputs.copy()]` in this line for in-place operations to not have effects anymore:
https://github.com/huggingface/datasets/blob/bf432011ff9155a5bc16c03956bc63e514baf80d/src/datasets/arrow_dataset.py#L2232.

(in the `batched` case, we can also copy the inputs' values (list objects) to ignore in-place modifications to the inputs' columns)

I think `remove_columns=True` has no meaning, so I'm not a fan of this change.

@mariosasko copy does have a cost associated with it ... and plus you'll have to consider `deepcopy` Imagine columnds that are list of list of list of list .... Though I have to agree that `remove_columns=True` doesn't make sense (but, IMO, neither does it in its current use-case as it should refer to `input_columns`) 

Okay closing this PR for the following reasons:
 - `remove_columns=True` was expected to keep the `.update`-like operator for `.map`. I initially thought it would be a good way to ignore function side effects and only keep output of that function (cf. PR description).
 - expected `remove_columns=True` is a bad API according to @mariosasko and introduces unecessary changes for little gain (strictly equivalent to `remove_columns=dset.column_names`)
Embed image and audio files in `save_to_disk`

Following https://github.com/huggingface/datasets/pull/4184, currently a dataset saved using `save_to_disk` doesn't actually contain the bytes of the image or audio files. Instead it stores the path to your local files. 

Adding `embed_external_files` and set it to True by default to save_to_disk would be kind of a breaking change since some users will get bigger Arrow files when updating the lib, but the advantages are nice:

- the resulting dataset is self contained, in case you want to delete your cache for example or share it with someone else
- users also upload these Arrow files to cloud storage via the fs parameter, and in this case they would expect to upload a self-contained dataset
- consistency with push_to_hub

This can be implemented at the same time as sharding for `save_to_disk` for efficiency, and reuse the helpers from `push_to_hub` to embed the external files.

cc @mariosasko 

Comments:
Deprecate `shard_size` in `push_to_hub` in favor of `max_shard_size`

This PR adds a `max_shard_size` param to `push_to_hub` and deprecates `shard_size` in favor of this new param to have a more descriptive name (a shard has at most the `shard_size` bytes in `push_to_hub`) for the param and to align the API with [Transformers](https://github.com/huggingface/transformers/blob/ff06b177917384137af2d9585697d2d76c40cdfc/src/transformers/modeling_utils.py#L1350).


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Document librispeech configs

Added an example of how to load one config or the other

Comments:
I think the main purpose of #4179 was how to be able to load both configs into one, so should we maybe add this part of the code: https://github.com/huggingface/datasets/issues/4179#issuecomment-1102383717 

to the doc? 

Actually @lhoestq would this work given that they have different split names: https://huggingface.co/datasets/librispeech_asr#data-splits ? 

This doc extension does not explain why I can't simply load the whole dataset. Or what workaround I need to get the whole dataset, which is what people usually want for Librispeech.

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq, I can add a `"all"` config to Librispeech have the datasets already cached somewhere 

I'm closing this PR then, feel free to continue the discussion in https://github.com/huggingface/datasets/issues/4179
Zenodo.org download is not responding

## Describe the bug
Source download_url from zenodo.org does not respond. 
`_DOWNLOAD_URL = "https://zenodo.org/record/2787612/files/SICK.zip?download=1"`
Other datasets also use zenodo.org to store data and they cannot be downloaded as well.

It would be better to actually use more reliable way to store original data like s3 bucket.

## Steps to reproduce the bug
```python
load_dataset("sick")
```

## Expected results
Dataset should be downloaded.

## Actual results
ConnectionError: Couldn't reach https://zenodo.org/record/2787612/files/SICK.zip?download=1 (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='zenodo.org', port=443): Read timed out. (read timeout=100)")))

## Environment info
- `datasets` version: 2.1.0
- Platform: Darwin-21.4.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 7.0.0
- Pandas version: 1.3.5


Comments:
[Off topic but related: Is the uptime of S3 provably better than Zenodo's?]

Hi @dkajtoch, please note that at HuggingFace we are not hosting this dataset: we are just using a script to download their data file and create a dataset from it.

It was the dataset owners decision to host their data at Zenodo. You can see this on their website: https://marcobaroni.org/composes/sick.html

And yes, you are right: Zenodo is currently having some incidents and people are reporting problems from it.

On the other hand, we could contact the data owners and propose them to host their data at our Hugging Face Hub.

@julien-c I guess so.


Thanks @albertvillanova. I know that the problem lies in the source data. I just wanted to point out that these kind of problems are unavoidable without having one place where data sources are cached. Websites may go down or data sources may move. Having a copy in Hugging Face Hub would be a great solution. 

Definitely, @dkajtoch! But we have to ask permission to the data owners. And many dataset licenses directly forbid data redistribution: in those cases we are not allowed to host their data on our Hub.

Ahhh good point! License is the problem :(
Support streaming FLEURS dataset

## Dataset viewer issue for '*name of the dataset*'

https://huggingface.co/datasets/google/fleurs

```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://storage.googleapis.com/xtreme_translations/FLEURS/af_za.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

Am I the one who added this dataset ? Yes

Can I fix this somehow in the script? @lhoestq @severo 


Comments:
Yes, you just have to use `dl_manager.iter_archive` instead of `dl_manager.download_and_extract`.

That's because `download_and_extract` doesn't support TAR archives in streaming mode.

Tried to make it streamable, but I don't think it's really possible. @lhoestq @polinaeterna maybe you guys can check: 
https://huggingface.co/datasets/google/fleurs/commit/dcf80160cd77977490a8d32b370c027107f2407b 

real quick. 

I think the problem is that we cannot ensure that the metadata file is found before the audio. Or is this possible somehow @lhoestq ? 

@patrickvonplaten I think the metadata file should be found first because the audio files are contained in a folder next to the metadata files (just as in common voice), so the metadata files should be "on top of the list"   as they are closer to the root in the directories hierarchy 

@patrickvonplaten but apparently it doesn't... I don't really know why.

Yeah! Any ideas what could be the reason here? cc @lhoestq ?

The order of the files is determined when the TAR archive is created, depending on the commands the creator ran.
If the metadata file is not at the beginning of the file, that makes streaming completely inefficient. In this case the TAR archive needs to be recreated in an appropriate order.

Actually we could maybe just host the metadata file ourselves and then stream the audio data only. Don't think that this would be a problem for the FLEURS authors (I can ask them :-)) 

I made a PR to their repo to support streaming (by uploading the metadata file to the Hub). See:
- https://huggingface.co/datasets/google/fleurs/discussions/4

I'm closing this issue as the PR above has been merged.
[feat] Add ImageNet dataset

To use the dataset download the tar file
[imagenet_object_localization_patched2019.tar.gz](https://www.kaggle.com/competitions/imagenet-object-localization-challenge/data?select=imagenet_object_localization_patched2019.tar.gz) from Kaggle and then point the datasets library to it by using:

```py
from datasets import load_dataset

dataset = load_dataset("imagenet",
data_dir="/path/to/imagenet_object_localization_patched2019.tar.gz")
```

Currently train and validation splits are supported.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the comments. I believe I have addressed all of them and also decreased the size of the dummy data file, so it should be ready for a re-review. I also made a change to allow adding synset mapping and valprep script in config in case we add ImageNet 21k some time later. 

@lhoestq I have updated the PR to address all of the review comments.
Adding missing subsets to the `SemEval-2018 Task 1` dataset

This dataset for the [1st task of SemEval-2018](https://competitions.codalab.org/competitions/17751) competition was missing all subtasks except for subtask 5. I added another two subtasks (subtask 1 and 2), which are each comprised of 12 additional data subsets: for each language in En, Es, Ar, there are 4 datasets, broken down by emotions (anger, fear, joy, sadness).

## Remaining questions

I wasn't able to find any documentation about how one should make PRs to modify datasets. Because of that, I just did my best to integrate the new data into the code, and tested locally that this worked. I'm sorry if I'm not respecting your contributing guidelines – if they are documented somewhere, I'd appreciate if you could send a pointer!

Not sure how `dataset_infos.json` and `dummy` should be updated. My understanding is that they were automatically generated at the time of the original dataset creation?

Comments:
Datasets are not tracked in this repository anymore. You should move this PR to the [discussions page of this dataset](https://huggingface.co/datasets/sem_eval_2018_task_1/discussions)
Very slow between two operations

Hello, in the processing stage, I use two operations. The first one : map + filter, is very fast and it uses the full cores, while the socond step is very slow and did not use full cores. 

Also, there is a significant lag between them.  Am I missing something ?



 ```
raw_datasets = raw_datasets.map(split_func, 
                batched=False,
                num_proc=args.preprocessing_num_workers,
                load_from_cache_file=not args.overwrite_cache, 
                desc = "running split para ==>")\
                .filter(lambda example: example['text1']!='' and example['text2']!='', 
                    num_proc=args.preprocessing_num_workers, desc="filtering ==>")


    processed_datasets = raw_datasets.map(
        preprocess_function,
        batched=True, 
        num_proc=args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not args.overwrite_cache,
        desc="Running tokenizer on dataset===>",
    )
```

Comments:
Stream private zipped images

As mentioned in https://github.com/huggingface/datasets/issues/4139 it's currently not possible to stream private/gated zipped images from the Hub.

This is because `Image.decode_example` does not handle authentication. Indeed decoding requires to access and download the file from the private repository.

In this PR I added authentication to `Image.decode_example` via a `token_per_repo_id` optional argument. I first wanted to just pass `use_auth_token` but a single `Image` instance can be responsible of decoding images from a combination of several datasets together (from `interleave_datasets` for example). Therefore I just used a dictionary `repo_id` -> `token` instead.

I'm getting the `repo_id` from the dataset builder (I replaced the `namepace` attribute with `repo_id`)

I did the same for `Audio.decode_example`

cc @SBrandeis @severo 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

oops looks like some tests are failing sorry, will fix them tomorrow

EDIT: not today but asap hopefully

cc @mariosasko this is ready for review, let me know what you think !
Optional Content Warning for Datasets

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

We now have hate speech datasets on the hub, like this one: https://huggingface.co/datasets/HannahRoseKirk/HatemojiBuild

I'm wondering if there is an option to select a content warning message that appears before the dataset preview? Otherwise, people immediately see hate speech when clicking on this dataset.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Implementation of a content warning message that separates users from the dataset preview until they click out of the warning.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Possibly just a way to remove the dataset preview completely? I think I like the content warning option better, though.

**Additional context**
Add any other context about the feature request here.


Comments:
Hi! You can use the `extra_gated_prompt` YAML field in a dataset card for displaying custom messages/warnings that the user must accept before gaining access to the actual dataset. This option also keeps the viewer hidden until the user agrees to terms. 

Hi @mariosasko, thanks for explaining how to add this feature. 

If the current dataset yaml is:
```
---
annotations_creators:
- expert
language_creators:
- expert-generated
languages:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: HatemojiBuild
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- hate-speech-detection
---
```

Can you provide a minimal working example of how to added the gated prompt?

Thanks!

```
---
annotations_creators:
- expert
language_creators:
- expert-generated
languages:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
pretty_name: HatemojiBuild
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- hate-speech-detection
extra_gated_prompt: "This repository contains harmful content."
---
```
\+ enable `User Access requests` under the Settings pane.

There's a brief guide here https://discuss.huggingface.co/t/how-to-customize-the-user-access-requests-message/13953 , and you can see the field in action here, https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/blob/main/README.md (you need to agree the terms in the Dataset Card pane to be able to access the files pane, so this comes up 403 at first).

And a working example here! https://huggingface.co/datasets/DDSC/dkhate :) Great to be able to mitigate harms in text.

-- is there a way to gate content anonymously, i.e. without registering which users access it?

+1 to @leondz's question. One scenario is if you don't want the dataset to be indexed by search engines or viewed in browser b/c of upstream conditions on data, but don't want to collect emails. Some ability to turn off the dataset viewer or add a gating mechanism without emails would be fantastic.
Add Visual Genome


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hum there seems to be some issues with tasks in test:
 - some tasks don't fit anything in `tasks.json`. Do I remove them in `task_categories`?
 - some tasks should exist, typically `visual-question-answering` (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my `master` is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?
 
 cc @mariosasko @lhoestq 

> some tasks don't fit anything in tasks.json. Do I remove them in task_categories?

You can keep them, but add `other-` as a prefix to those tasks to make the CI ignore it

> some tasks should exist, typically visual-question-answering (https://github.com/huggingface/datasets/blame/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/utils/resources/tasks.json#L195) yet the exception is failing on me. I'm guessing it's because my master is not up-to-date. However this means that the testing only tests my branch instead of the one merged with master?

Feel free to merge upstream/master into your branch ;)

EDIT: actually I just noticed you've already done this, thanks !

After offline discussions: will keep that image essentially it's necessary as I have a mapping that creates a mapping between url and local path (images are downloaded via a zip file) and dummy data needs to store that dummy image. The issue is when I read an annotation, I get a url, compute the local path, and basically I assume the local path exists since I've extracted all the images ... This isn't true if dummy data doesn't have all the images, so instead I've added a script that "fixes" the dummy data after using the CLI, it essentially adds the dummy image in the zip corresponding to the url.
RGBA images not showing

## Dataset viewer issue for ceyda/smithsonian_butterflies_transparent

[**Link:** *link to the dataset viewer page*](https://huggingface.co/datasets/ceyda/smithsonian_butterflies_transparent)

![image](https://user-images.githubusercontent.com/15624271/163117683-e91edb28-41bf-43d9-b371-5c62e14f40c9.png)

Am I the one who added this dataset ? Yes

👉  More of a general issue of 'RGBA' png images not being supported 
(the dataset itself is just for the huggan sprint and not that important, consider it just an example)

Comments:
Thanks for reporting. It's a known issue, and we hope to fix it soon.

Fixed, thanks!
Adding STSb-TR dataset

Semantic Textual Similarity benchmark Turkish (STSb-TR) dataset introduced in our paper [Semantic Similarity Based Evaluation for Abstractive News Summarization](https://aclanthology.org/2021.gem-1.3.pdf) added.

Comments:
Thanks for your contribution, @figenfikri.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Generate tasks.json taxonomy from `huggingface_hub`


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ok recomputed the json file, this should be ready to review now! @lhoestq 

Note: the generated JSON from `hf/hub-docs` can be found in the output of a GitHub Action run on that repo, for instance in https://github.com/huggingface/hub-docs/runs/6006686983?check_suite_focus=true

(click on "Run export-tasks script")

Should we not add the tasks with hideInDatasets?

yes, probably true – i'll change that in a PR in `hub-docs`

Yes that's good :) feel free to merge

thanks to the both of you!
ArrayND error in pyarrow 5

As found in https://github.com/huggingface/datasets/pull/3903, The ArrayND features fail on pyarrow 5:
```python
import pyarrow as pa
from datasets import Array2D
from datasets.table import cast_array_to_feature

arr = pa.array([[[0]]])
feature_type = Array2D(shape=(1, 1), dtype="int64")
cast_array_to_feature(arr, feature_type)
```
raises
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-04610f9fa78c> in <module>
----> 1 cast_array_to_feature(pa.array([[[0]]]), Array2D(shape=(1, 1), dtype="int32"))

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in cast_array_to_feature(array, feature, allow_number_to_str)
   1806         return array_cast(array, get_nested_type(feature), allow_number_to_str=allow_number_to_str)
   1807     elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1808         return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
   1809     raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
   1810 

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_number_to_str)
   1705         array = array.storage
   1706     if isinstance(pa_type, pa.ExtensionType):
-> 1707         return pa_type.wrap_array(array)
   1708     elif pa.types.is_struct(array.type):
   1709         if pa.types.is_struct(pa_type) and (

AttributeError: 'Array2DExtensionType' object has no attribute 'wrap_array'
```

The thing is that `cast_array_to_feature` is called when writing an Arrow file, so creating an Arrow dataset using any ArrayND type currently fails.

`wrap_array` has been added in pyarrow 6, so we can either bump the required pyarrow version or fix this for pyarrow 5

Comments:
Where do we bump the required pyarrow version? Any inputs on how I fix this issue? 

We need to bump it in `setup.py` as well as update some CI job to use pyarrow 6 instead of 5 in `.circleci/config.yaml` and `.github/workflows/benchmarks.yaml`
Inconsistent splits generation for datasets without loading script (packaged dataset puts everything into a single split)

## Describe the bug
Splits for dataset loaders without scripts are prepared inconsistently. I think it might be confusing for users.

## Steps to reproduce the bug
* If you load a packaged datasets from Hub, it infers splits from directory structure / filenames (check out the data [here](https://huggingface.co/datasets/nateraw/test-imagefolder-dataset)):
```python
ds = load_dataset("nateraw/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* If you do the same from locally stored data specifying only directory path you'll get the same:
```python
ds = load_dataset("/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 6
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 4
    })
})
```
* However, if you explicitely specify package name (like `imagefolder`, `csv`, `json`), all the data is put into a single split:
```python
ds = load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")
print(ds)
### Output:
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 10
    })
})
```

## Expected results
For `load_dataset("imagefolder", data_dir="/path/to/local/data/test-imagefolder-dataset")` I expect the same output as of the two first options.

Comments:
fix confusing bleu metric example

**Is your feature request related to a problem? Please describe.**
I would like to see the example in "Metric Card for BLEU" changed.
The 0th element in the predictions list is not closed in square brackets, and the 1st list is missing a comma.
The BLEU score are calculated correctly, but it is difficult to understand, so it would be helpful if you could correct this.
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
>>> references = [
...     [["hello", "there", "general", "kenobi"]],
...     [["foo", "bar", "foobar"]]
... ]
>>> bleu = datasets.load_metric("bleu")
>>> results = bleu.compute(predictions=predictions, references=references)
>>> print(results)
{'bleu': 0.6370964381207871, ...
```

**Describe the solution you'd like**
```
>> predictions = [
...     ["hello", "there", "general", "kenobi",       # <- no closing square bracket.
...     ["foo", "bar" "foobar"]                              # <- no comma between "bar" and "foobar"
... ]
# and
>>> print(results)
{'bleu':1.0, ...
```


Comments:
Adjust path to datasets tutorial in How-To

The link in the How-To overview page to the Datasets tutorials is currently broken. This is just a small adjustment to make it match the format used in https://github.com/huggingface/datasets/blob/master/docs/source/tutorial.md.

(Edit to add: The link in the PR deployment (https://moon-ci-docs.huggingface.co/docs/datasets/pr_4147/en/how_to) is also broken since it's actually hardcoded to `master` and not dynamic to the branch name, but other links seem to behave similarly.)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
SAMSum dataset viewer not working

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Comments:
https://huggingface.co/datasets/samsum

```
Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file
```

Currently, only the datasets that can be streamed support the dataset viewer. Maybe @lhoestq @albertvillanova or @mariosasko could give more details about why the dataset cannot be streamed.

It looks like the host (https://arxiv.org) doesn't allow HTTP Range requests, which is what we use to stream data.

This can be fix if we host the data ourselves, which is ok since the dataset is under CC BY-NC-ND 4.0
Redirect TIMIT download from LDC

LDC data is protected under US copyright laws and under various legal agreements between the Linguistic Data Consortium/the University of Pennsylvania and data providers which prohibit redistribution of that data by anyone other than LDC. Similarly, LDC's membership agreements, non-member user agreement and various corpus-specific license agreements specifically state that users cannot publish, retransmit, disclose, copy, reproduce or redistribute LDC databases to others outside their organizations.

LDC explicitly asked us to remove the download script for the TIMIT dataset. In this PR I remove all means to download the dataset, and redirect users to download the data from https://catalog.ldc.upenn.edu/LDC93S1 

Comments:
CI is failing because some tags are outdated, but they're fixed in #4067 

_The documentation is not available anymore as the PR was closed or merged._

We may do a release pretty soon (today ?), let me know if it's fine to include it in the new release

Fine to include this change!
Add ObjectFolder 2.0 dataset

## Adding a Dataset
- **Name:** ObjectFolder 2.0
- **Description:** ObjectFolder 2.0 is a dataset of 1,000 objects in the form of implicit representations. It contains 1,000 Object Files each containing the complete multisensory profile for an object instance.
- **Paper:** [*link to the dataset paper if available*](https://arxiv.org/abs/2204.02389)
- **Data:** https://github.com/rhgao/ObjectFolder

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Datasets are not tracked in this repository anymore.
Add single dataset citations for TweetEval

This PR adds single data citations as per request of the original creators of the TweetEval dataset.

This is a recent email from the creator:

> Could I ask you a favor? Would you be able to add at the end of the README the citations of the single datasets as well? You can just copy our readme maybe? https://github.com/cardiffnlp/tweeteval#citing-tweeteval
(just to be sure that the creator of the single datasets also get credits when tweeteval is used)

Please let me know if this looks okay or if any changes are needed.

Thanks,
Gunjan


Comments:
_The documentation is not available anymore as the PR was closed or merged._

The `test_dataset_cards` method is failing with the error:

```
if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           The following typing errors are found: {'annotations_creators': "(Expected `typing.List` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)\nOR\n(Expected `typing.Dict` with length > 0. Found value of type: `<class 'list'>`, with length: 0.\n)"}
```

Adding `found` as annotation creators.
dataset metadata for reproducibility

When pulling a dataset from the hub, it would be useful to have some metadata about the specific dataset and version that is used. The metadata could then be passed to the `Trainer` which could then be saved to a model card. This is useful for people who run many experiments on different versions (commits/branches) of the same dataset. 

The dataset could have a list of “source datasets” metadata and ignore what happens to them before arriving in the Trainer (i.e. ignore mapping, filtering, etc.).

Here is a basic representation (made by @lhoestq )
```python
>>> from datasets import load_dataset
>>> 
>>> my_dataset = load_dataset(...)["train"]
>>> my_dataset = my_dataset.map(...)
>>> 
>>> my_dataset.sources
[HFHubDataset(repo_id=..., revision=..., arguments={...})]
```

Comments:
+1 on this idea. This could be powerful for helping better track datasets used for model training and help with automatic model card creation. 

One possible way of doing this would be to store some/most/all the arguments passed to `load_dataset` if a hub id is passed. i.e. store the Hub ID, configuration, etc. 

cc @tomaarsen
More robust `cast_to_python_objects` in `TypedSequence`

Adds a fallback to run an expensive version of `cast_to_python_objects` which exhaustively checks entire lists to avoid the `ArrowInvalid: Could not convert` error in `TypedSequence`. Currently, this error can happen in situations where only some images are decoded in `map`, in which case `cast_to_python_objects` fails to recognize that it needs to cast `PIL.Image` objects if they are not at the beginning of the sequence and stops after the first image dictionary (e.g., if `data` is `[{"bytes": None, "path": "some path"}, PIL.Image(), ...]`)

Fix #4124

Comments:
_The documentation is not available anymore as the PR was closed or merged._
dataset viewer issue for common_voice

## Dataset viewer issue for 'common_voice'

**Link:** https://huggingface.co/datasets/common_voice

Server Error
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'

Am I the one who added this dataset ? No


Comments:
Yes, it's a known issue, and we expect to fix it soon.

Fixed.

<img width="1393" alt="Capture d’écran 2022-04-25 à 15 42 05" src="https://user-images.githubusercontent.com/1676121/165101176-d729d85b-efff-45a8-bad1-b69223edba5f.png">
Building C4 takes forever

## Describe the bug
C4-en is a 300 GB dataset. However, when I try to download it through the hub it takes over _six hours_ to generate the train/test split from the downloaded files. This is an absurd amount of time and an unnecessary waste of resources.

## Steps to reproduce the bug
```python
c4 = datasets.load("c4", "en")
```

## Expected results
I would like to be able to download pre-split data.

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.13.0-35-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Comments:
Hi @StellaAthena, thanks for reporting.

Please note, that our `datasets` library performs several operations in order to load a dataset, among them:
- it downloads all the required files: for C4 "en", 378.69 GB of JSON GZIPped files
- it parses their content to generate the dataset
- it caches the dataset in an Arrow file: for C4 "en", this file size is 1.87 TB
- it memory-maps the Arrow file

If it suits your use case, you might load this dataset in streaming mode:
- no Arrow file is generated
- you can iterate over elements immediately (no need to wait to download all the entire files)

```python
In [45]: from datasets import load_dataset
    ...: ds = load_dataset("c4", "en", split="train", streaming=True)
    ...: for item in ds:
    ...:     print(item)
    ...:     break
    ...: 
{'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.', 'timestamp': '2019-04-25T12:57:54Z', 'url': 'https://klyq.com/beginners-bbq-class-taking-place-in-missoula/'}
```
I hope this is useful for your use case.
medical_dialog zh has very slow _generate_examples

## Describe the bug
After downloading the files from Google Drive, `load_dataset("medical_dialog", "zh", data_dir="./")` takes an unreasonable amount of time. Generating the train/test split for 33% of the dataset takes over 4.5 hours.

## Steps to reproduce the bug
The easiest way I've found to download files from Google Drive is to use `gdown` and use Google Colab because the download speeds will be very high due to the fact that they are both in Google Cloud.

```python
file_ids = [
        "1AnKxGEuzjeQsDHHqL3NqI_aplq2hVL_E",
        "1tt7weAT1SZknzRFyLXOT2fizceUUVRXX",
        "1A64VBbsQ_z8wZ2LDox586JIyyO6mIwWc",
        "1AKntx-ECnrxjB07B6BlVZcFRS4YPTB-J",
        "1xUk8AAua_x27bHUr-vNoAuhEAjTxOvsu",
        "1ezKTfe7BgqVN5o-8Vdtr9iAF0IueCSjP",
        "1tA7bSOxR1RRNqZst8cShzhuNHnayUf7c",
        "1pA3bCFA5nZDhsQutqsJcH3d712giFb0S",
        "1pTLFMdN1A3ro-KYghk4w4sMz6aGaMOdU",
        "1dUSnG0nUPq9TEQyHd6ZWvaxO0OpxVjXD",
        "1UfCH05nuWiIPbDZxQzHHGAHyMh8dmPQH",
]
for i in file_ids:
    url = f"https://drive.google.com/uc?id={i}"
    !gdown $url


from datasets import load_dataset

ds = load_dataset("medical_dialog", "zh", data_dir="./")
```

## Expected results
Faster load time

## Actual results
`Generating train split: 33%: 625519/1921127 [4:31:03<31:39:20, 11.37 examples/s]`

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

@vrindaprabhu , could you take a look at this since you implemented it? I think the `_generate_examples` function might need to be rewritten

Comments:
Hi @nbroad1881, thanks for reporting.

Let me have a look to try to improve its performance. 

Thanks @nbroad1881  for reporting! I don't recall it taking so long. I will also have a look at this. 
@albertvillanova please let me know if I am doing something unnecessary or time consuming.

Hi @nbroad1881 and @vrindaprabhu,

As a workaround for the performance of the parsing of the raw data files (this could be addressed in a subsequent PR), I have found that there are also processed data files, that do not require parsing. I have added these as new configurations `processed.en` and `processed.zh`:
```python
ds = load_dataset("medical_dialog", "processed.zh")
```
Representing dictionaries (json) objects as features

In the process of adding a new dataset to the hub, I stumbled upon the inability to represent dictionaries that contain different key names, unknown in advance (and may differ between samples), original asked in the [forum](https://discuss.huggingface.co/t/representing-nested-dictionary-with-different-keys/16442).

For instance:

```
sample1 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
}}
sample2 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
}}
sample3 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
  "d": {"id": 3, "text": "text4"},
}}
```

the `nps` field cannot be represented as a Feature while maintaining its original structure.
@lhoestq suggested to add JSON as a new feature type, which will solve this problem.


It seems like an alternative solution would be to change the original data format, which isn't an optimal solution in my case. Moreover, JSON is a common structure, that will likely to be useful in future datasets as well.

Comments:
This is indeed a problem for VLMs in 2025 too! Where we expect the inputs to be of the format:
```
content = [{
"type": "text",
"text": <text>
}, {
"type": "image",
"image": <image>
}]
```
CC: @lhoestq 

The community has also been using this format instead, with the images on the side: https://huggingface.co/docs/trl/en/dataset_formats#vision-datasets

```python
dataset = Dataset.from_dict({
    "prompt": [
        [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What color is the sky in the image?"}]}],
        [{"role": "user", "content": [{"type": "text", "text": "What is the capital of France?"}]}],
    ],
    "completion": [
        [{"role": "assistant", "content": [{"type": "text", "text": "It is blue."}]}],
        [{"role": "assistant", "content": [{"type": "text", "text": "Paris."}]}],
    ],
    "images": [
        [PIL.Image.open("path/to/sky_image1.png")],
        [],
    ],
})
```

like this one: https://huggingface.co/datasets/trl-lib/rlaif-v

Hi ! with #8027 you will be able to do 
```python
from datasets import Dataset, Features, Json

features = Features({"nps": Json()})
ds = Dataset.from_list([sample1, sample2, sample3], features=features)
# OR
# ds = Dataset.from_list([sample1, sample2, sample3], on_mixed_types="use_json")

print(ds.features)
# {'nps': Json()}

for nps in ds["nps"]:
    print(nps)
# {'a': {'id': 0, 'text': 'text1'}, 'b': {'id': 1, 'text': 'text2'}}
# {'a': {'id': 0, 'text': 'text1'}, 'b': {'id': 1, 'text': 'text2'}, 'c': {'id': 2, 'text': 'text3'}}
# {'a': {'id': 0, 'text': 'text1'}, 'b': {'id': 1, 'text': 'text2'}, 'c': {'id': 2, 'text': 'text3'}, 'd': {'id': 3, 'text': 'text4'}}
```
Hotfix failing CI tests on Windows

This PR makes a hotfix for our CI Windows tests: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414

Fix #4118

I guess this issue is related to this PR:
- huggingface/huggingface_hub#815

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Pretty print dataset info files

Adds indentation to the `dataset_infos.json` file when saving for nicer diffs.

(suggested by @julien-c)

This PR also updates the info files of the GH datasets. Note that this change adds more than **10 MB** to the repo size (the total file size before the change: 29.672298 MB, after: 41.666475 MB), so I'm not sure this change is a good idea.

`src/datasets/info.py` is the only relevant file for reviewers.


Comments:
maybe just do it from now on no? (i.e. not for existing `dataset_infos.json` files)

_The documentation is not available anymore as the PR was closed or merged._

> maybe just do it from now on no? (i.e. not for existing dataset_infos.json files)

Yes, or do this only for datasets created with `push_to_hub` to (always) keep the GH datasets small? 


yep sounds good too on my side! 

I reverted the change to avoid the size increase and added the `pretty_print` flag, which pretty-prints the JSON, and that flag is only True for datasets created with `push_to_hub`.
ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

**Is your feature request related to a problem? Please describe.**
I sometimes like to peek at the dataset images from jupyterlab. thus '.ipynb_checkpoints' folder appears where my dataset is and (just realized) leads to accidental duplicate image additions. I think this is an easy enough thing to miss especially if the dataset is very large.

**Describe the solution you'd like**
maybe have an option `ignore` or something .gitignore style
`dataset = load_dataset("imagefolder", data_dir="./data/original", ignore="regex?")`

**Describe alternatives you've considered**
Could filter out manually



Comments:
Maybe it would be nice to ignore private dirs like this one (ones starting with `.`) by default. 

CC @mariosasko 

Maybe we can add a `ignore_hidden_files` flag to the builder configs of our packaged loaders (to be consistent across all of them), wdyt @lhoestq @albertvillanova? 

I think they should always ignore them actually ! Not sure if adding a flag would be helpful

@lhoestq But what if the user explicitly requests those files via regex?

`glob.glob` ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's `glob` doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

> @lhoestq But what if the user explicitly requests those files via regex?

Usually hidden files are meant to be ignored. If they are data files, they must be placed outside a hidden directory in the first place right ? I think it's more sensible to explain this than adding a flag.

> glob.glob ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's glob doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

After globbing using `fsspec`, we already ignore files that start with a `.` in `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository`, I guess we can just account for parent directories as well ?

We could open an issue on `fsspec` but I think they won't change this since it's an important breaking change for them.
Multiprocessing with FileLock fails in python 3.9

On python 3.9, this code hangs:
```python
from multiprocessing import Pool
from filelock import FileLock


def run(i):
    print(f"got the lock in multi process [{i}]")


with FileLock("tmp.lock"):
    with Pool(2) as pool:
        pool.map(run, range(2))

```

This is because the subprocesses try to acquire the lock from the main process for some reason. This is not the case in older versions of python.

This can cause many issues in python 3.9. In particular, we use multiprocessing to fetch data files when you load a dataset (as long as there are >16 data files). Therefore `imagefolder` hangs, and I expect any dataset that needs to download >16 files to hang as well.

Let's see if we can fix this and have a CI that runs on 3.9.

cc @mariosasko @julien-c 

Comments:
Closing this one because it must be used this way actually:
```python
def main():
    with FileLock("tmp.lock"):
        with Pool(2) as pool:
            pool.map(run, range(2))

if __name__ == "__main__":
    main()
```
Support huggingface_hub 0.5

Following https://github.com/huggingface/datasets/issues/4105

`huggingface_hub` deprecated some parameters in `HfApi` in 0.5. This PR updates all the calls to HfApi to remove all the deprecations, 
and I set the `hugginface_hub` requirement to `>=0.5.0`


cc @adrinjalali @LysandreJik 

Comments:
Looks like GH actions is not able to resolve `huggingface_hub` 0.5.0, I'm investivating

_The documentation is not available anymore as the PR was closed or merged._

I'm glad to see changes in `huggingface_hub` are simplifying code here.

seems to supersede #4102, feel free to close mine :)

maybe just cherry-pick the docstring fix

I think I've found the issue:
- https://github.com/huggingface/huggingface_hub/pull/790

Good catch, `huggingface_hub` doesn't support python 3.6 anymore indeed, therefore we should keep support for 0.4.0. I'm reverting the requirement version bump for now.

We can update the requirement once we drop support for python 3.6 in `datasets`

@lhoestq, I've opened this PR on `huggingface_hub`: 
- https://github.com/huggingface/huggingface_hub/pull/823

Is there any strong reason why `huggingface_hub` no longer supports Python 3.6? 

I think `datasets` can drop support for 3.6 soon. But for now maybe let's keep support for 0.4.0, python 3.6 users are not affected by https://github.com/huggingface/datasets/issues/4105 anyway.

`huggingface_hub` doesn't not have to support 3.6 again just for the CI IMO

@lhoestq I commented on the PR, that IMO it is not a good practice to drop support for Python 3.6 without a previous deprecation cycle.

Re-added support for older versions. I ended up checking `huggingface_hub` version to use the old, deprecated API for <0.5.0

I find it good practice to have all dependency version related code in a single file so that when you decide to remove support for an old version of a dependency it's easy to find and remove them, hence suggesting `utils/_fixes.py` in https://github.com/huggingface/datasets/issues/4105#issuecomment-1090041204

good idea, thanks !

I used your suggestion @adrinjalali , I just replace the try/except with a check on the version of `huggingface_hub`
How can I download only the train and test split for full numbers using load_dataset()? 

How can I download only the train and test split for full numbers using load_dataset()? 

I do not need the extra split and it will take 40 mins just to download in Colab. I have very short time in hand. Please help.

Comments:
Hi! Can you please specify the full name of the dataset? IIRC `full_numbers` is one of the configs of the `svhn` dataset, and its generation is slow due to data being stored in binary Matlab files. Even if you specify a specific split, `datasets` downloads all of them, but we plan to fix that soon and only download the requested split.

If you are in a hurry, download the `svhn` script [here](`https://huggingface.co/datasets/svhn/blob/main/svhn.py`), remove [this code](https://huggingface.co/datasets/svhn/blob/main/svhn.py#L155-L162), and run:
```python
from datasets import load_dataset
dset = load_dataset("path/to/your/local/script.py", "full_numbers")
```

And to make loading easier in Colab, you can create a dataset repo on the Hub and upload the script there. Or push the script to Google Drive and mount the drive in Colab.