KRadim/github-issues-minimal huggingface dataset:
Retry HfApi call inside push_to_hub when 504 error
State: closed

Ass suggested by @lhoestq in #3872, this PR:
- Implements a retry function
- Retries HfApi call inside `push_to_hub` when 504 error. To be agreed:
  - max_retries = 2 (at 0.5 and 1 seconds)

Fix #3872.
The metric Meteor doesn't work for nltk ==3.6.4
State: closed
Labels: name: bug

## Describe the bug
Using the metric Meteor with nltk == 3.6.4 gives a TypeError:
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

## Steps to reproduce the bug
```python
import datasets
metric = datasets.load_metric("meteor")
predictions = ["hello world"]
references = ["hello world"]
metric.compute(predictions=predictions, references=references)
```

## Expected results
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

I think this TypeError exists because input sentences are tokenized into lists of tokens and the str.lower() is applied to this list of tokens.

## Actual results
No error but a meteor score

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 7.0.0
Image process doc
State: closed
Labels: name: documentation

This PR is a first draft of how to process image data. It adds:

- Load an image dataset with `image` and `path` (adds tip about `decode=False` param to access the path and bytes, thanks to @mariosasko).
- Load an image using the `ImageFolder` builder. I know there is an [example](https://huggingface.co/docs/datasets/master/en/loading#image-folders) of this already, but I also wanted to add it here so users don't miss it. This doc seems important for centralizing all of the image-related things so far. Datasets has grown so quickly 🚀  now that I think maybe splitting up the How-to guides by modality may be better since working with vision/audio data is slightly different from what users have seen up until now. This way we can continue to scale the docs to better accommodate vision/audio things.
- Add a data augmentation with `set_transform`. There is only 1 example here so far, but we can certainly add more. 

Todo:
- [x] Couldn't figure out why my augmentation function works with `set_transform` but not `map` 🥲. Working with @mariosasko on this!
How to use Image folder
State: closed
Labels: name: question

Ran this code
```
 load_dataset("imagefolder", data_dir="./my-dataset")
```

`https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py` missing
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_33/1648737256.py in <module>
----> 1 load_dataset("imagefolder", data_dir="./my-dataset")

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1684         revision=revision,
   1685         use_auth_token=use_auth_token,
-> 1686         **config_kwargs,
   1687     )
   1688 

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, script_version, **config_kwargs)
   1511         download_config.use_auth_token = use_auth_token
   1512     dataset_module = dataset_module_factory(
-> 1513         path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
   1514     )
   1515 

/opt/conda/lib/python3.7/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_files, **download_kwargs)
   1200                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1201                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1202                     ) from None
   1203                 raise e1 from None
   1204     else:

FileNotFoundError: Couldn't find a dataset script at /kaggle/working/imagefolder/imagefolder.py or any data file in the same directory. Couldn't find 'imagefolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py
```
Change the framework switches to the new syntax
State: closed

This PR updates the syntax of the framework-specific code samples. With this new syntax, you'll be able to:

- have paragraphs of text be framework-specific instead of just code samples
- have support for Flax code samples if you want.

This should be merged after https://github.com/huggingface/doc-builder/pull/63 and https://github.com/huggingface/doc-builder/pull/130
Align metadata to DCAT/DCAT-AP
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
Align to DCAT metadata to describe datasets

**Describe the solution you'd like**
Reuse terms and structure from DCAT in the metadata file, ideally generate a json-ld file dcat compliant

**Describe alternatives you've considered**


**Additional context**
DCAT is a W3C standard extended in Europe with DCAT-AP, an example is data.europa.eu publishing datasets metadata in DCAT-AP
Module namespace cleanup for v2.0
State: closed

This is an attempt to make the user-facing `datasets`' submodule namespace cleaner:

In particular, this PR does the following:
* removes the unused `zip_nested` and `flatten_nest_dict` and their accompanying tests
* removes `pyarrow` from the top-level namespace
* properly uses `__all__` and the `from  <module> import *` syntax to avoid importing the `<module>`'s submodules
* cleans up the `utils` namespace
* moves the `temp_seed` context manage from `datasets/utils/file_utils.py` to `datasets/utils/py_utils.py`
add MSE and MAE metrics - V2
State: closed

Created a new pull request to resolve unrelated changes in PR caused due to rebasing.
Ref Older PR :  [#3845](https://github.com/huggingface/datasets/pull/3845) 

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)
HTTP error 504 Server Error: Gateway Time-out
State: closed

I am trying to push a large dataset(450000+) records with the help of `push_to_hub()`
While pushing, it gives some error like this.

```
Traceback (most recent call last):
  File "data_split_speech.py", line 159, in <module>
    data_new_2.push_to_hub("user-name/dataset-name",private=True)
  File "/opt/conda/lib/python3.8/site-packages/datasets/dataset_dict.py", line 951, in push_to_hub
    repo_id, split, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3556, in _push_parquet_shards_to_hub
    api.upload_file(
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1017, in upload_file
    raise err
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1008, in upload_file
    r.raise_for_status()
  File "/opt/conda/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/user-name/dataset-name/upload/main/data/train2-00041-of-00064.parquet

```
Can anyone help me to resolve this issue.
Making the Hub the place for datasets in Portuguese 
State: open
Labels: name: dataset request

Let's make Hugging Face Datasets the central hub for datasets in Portuguese :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the Portuguese speaking community.

What are some datasets in Portuguese worth integrating into the Hugging Face hub?

Special thanks to @augusnunes for his collaboration on identifying the first ones:

-  [NILC - USP](http://www.nilc.icmc.usp.br/nilc/index.php/tools-and-resources).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

cc @osanseviero
big_patent cased version
State: closed
Labels: name: dataset request

Hi! I am interested in working with the big_patent dataset.

In Tensorflow, there are a number of versions of the dataset:

- 1.0.0 : lower cased tokenized words
- 2.0.0 : Update to use cased raw strings
- 2.1.2 (default): Fix update to cased raw strings.

The version in the huggingface `datasets` library is the 1.0.0. I would be very interested in using the 2.1.2 cased version (used more, recently, for example in the Pegasus paper), but it does not seem to be supported (I tried using the `revision` parameter in `load_datasets`). Is there a way to already load it, or would it be possible to add that version?
Order of dataset changes due to glob.glob.
State: open
Labels: name: generic discussion

## Describe the bug

After discussion with @lhoestq, just want to mention here that `glob.glob(...)` should always be used in combination with `sorted(...)` to make sure the list of files returned by `glob.glob(...)` doesn't change depending on the OS system. 

There are currently multiple datasets that use `glob.glob()` without making use of `sorted(...)` even the streaming download manager (if I'm not mistaken): 
https://github.com/huggingface/datasets/blob/c14bfeb4af89da14f870de5ddaa584b08aa08eeb/src/datasets/utils/streaming_download_manager.py#L483
Fix push_to_hub with null images
State: closed

This code currently raises an error because of the null image:
```python
import datasets

dataset_dict = { 'name': ['image001.jpg', 'image002.jpg'], 'image': ['cat.jpg', None] }
features = datasets.Features({
    'name': datasets.Value('string'),
    'image': datasets.Image(),
})
dataset = datasets.Dataset.from_dict(dataset_dict, features)
dataset.push_to_hub("username/dataset") # this line produces an error: 'NoneType' object is not subscriptable
```
I fixed this in this PR

TODO:
- [x] add a test
Bad error message when loading private dataset
State: closed
Labels: name: bug

## Describe the bug

A pretty common behavior of an interaction between the Hub and datasets is the following.
An organization adds a dataset in private mode and wants to load it afterward.


```python
from transformers import load_dataset

ds = load_dataset("NewT5/dummy_data", "dummy")
```

This command then fails with:

```bash
FileNotFoundError: Couldn't find a dataset script at /home/patrick/NewT5/dummy_data/dummy_data.py or any data file in the same directory. Couldn't find 'NewT5/dummy_data' on the Hugging Face Hub either: FileNotFoundError: Dataset 'NewT5/dummy_data' doesn't exist on the Hub
```

**even though** the user has access to the website `NewT5/dummy_data` since she/he is part of the org. 

We need to improve the error message here similar to how @sgugger, @LysandreJik and @julien-c have done it for transformers IMO.

## Steps to reproduce the bug

E.g. execute the following code to see the different error messages between `transformes` and `datasets`.

1. Transformers
```python
from transformers import BertModel

BertModel.from_pretrained("NewT5/dummy_model")
```

The error message is clearer here - it gives:

```
OSError: patrickvonplaten/gpt2-xl is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```

Let's maybe do the same for datasets? The PR was introduced to `transformers` here:
https://github.com/huggingface/transformers/pull/15261

## Expected results

Better error message

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.15.15-76051515-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.1
load only England English dataset from common voice english dataset
State: closed
Labels: name: question

training_data = load_dataset("common_voice", "en",split='train[:250]+validation[:250]')
testing_data = load_dataset("common_voice", "en", split="test[:200]")

I'm trying to load only 8% of the English common voice data with accent == "England English." Can somebody assist me with this?

**Typical Voice Accent Proportions:**

- 24% United States English 
- 8% England English 
- 5% India and South Asia (India, Pakistan, Sri Lanka) 
- 3% Australian English 
- 3% Canadian English 
- 2% Scottish English 
- 1% Irish English 
- 1% Southern African (South Africa, Zimbabwe, Namibia) 
- 1% New Zealand English

Can we replicate this for Age as well?

**Age proportions of the common voice:-**

- 24% 19 - 29 
- 14% 30 - 39 
- 10% 40 - 49 
- 6% < 19 
- 4% 50 - 59 
- 4% 60 - 69 
- 1% 70 – 79
add ontonotes_conll dataset
State: closed

# Introduction of the dataset
OntoNotes v5.0 is the final version of OntoNotes corpus, and is a large-scale, multi-genre,
multilingual corpus manually annotated with syntactic, semantic and discourse information.

This dataset is the version of OntoNotes v5.0 extended and used in the CoNLL-2012 shared task
, includes v4 train/dev and v9 test data for English/Chinese/Arabic and corrected version v12 train/dev/test data (English only).

This dataset is widely used in name entity recognition, coreference resolution, and semantic role labeling.

In dataset loading script, I modify and use the code of [AllenNLP/Ontonotes](https://docs.allennlp.org/models/main/models/common/ontonotes/#ontonotes) to read the special conll files while don't get extra package dependency. 

# Some workarounds I did

1. task ids
I add tasks that I can't find anywhere `semantic-role-labeling`, `lemmatization`, and `word-sense-disambiguation` to the task category `structure-prediction`, because they are related to "syntax". I feel there is another good name for the task category since some tasks mentioned aren't related to structure, but I have no good idea.

2. `dl_manage.extract`
Since we'll get another zip after unzip the downloaded zip data, I have to use `dl_manager.extract` directly inside `_generate_examples`. But when testing dummy data, `dl_manager.extract` do nothing. So I make a conditional such that it  manually  extract data when testing dummy data.

# Help
Don't know how to fix the building doc error.
Redundant add dataset information and dead link.
State: closed

> Alternatively, you can follow the steps to [add a dataset](https://huggingface.co/docs/datasets/add_dataset.html) and [share a dataset](https://huggingface.co/docs/datasets/share_dataset.html) in the documentation.

The "add a dataset link" gives 404 Error, and the share_dataset link has changed. I feel this information is redundant/deprecated now since we have a more detailed guide for "How to add a dataset?".
Add "Adversarial GLUE" dataset to datasets library
State: closed

Adds the Adversarial GLUE dataset: https://adversarialglue.github.io/

```python
>>> import datasets
>>> >>> datasets.load_dataset('adv_glue')
Using the latest cached version of the module from /home/jxm3/.cache/huggingface/modules/datasets_modules/datasets/adv_glue/26709a83facad2830d72d4419dd179c0be092f4ad3303ad0ebe815d0cdba5cb4 (last modified on Mon Mar  7 19:19:48 2022) since it couldn't be found locally at adv_glue., or remotely on the Hugging Face Hub.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1657, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1510, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 1021, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 258, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 337, in _create_builder_config
    raise ValueError(
ValueError: Config name is missing.
Please pick one among the available configs: ['adv_sst2', 'adv_qqp', 'adv_mnli', 'adv_mnli_mismatched', 'adv_qnli', 'adv_rte']
Example of usage:
	`load_dataset('adv_glue', 'adv_sst2')`
>>> datasets.load_dataset('adv_glue', 'adv_sst2')['validation'][0]
Reusing dataset adv_glue (/home/jxm3/.cache/huggingface/datasets/adv_glue/adv_sst2/1.0.0/3719a903f606f2c96654d87b421bc01114c37084057cdccae65cd7bc24b10933)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 604.11it/s]
{'sentence': "it 's an uneven treat that bores fun at the democratic exercise while also examining its significance for those who take part .", 'label': 1, 'idx': 0}
```
NonMatchingChecksumError when checksum is None
State: closed
Labels: name: bug

I ran into the following error when adding a new dataset:

```bash
expected_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': None, 'num_bytes': 40662}}
recorded_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': 'efb4cbd3aa4a87bfaffc310ae951981cc0a36c6c71c6425dd74e5b55f2f325c9', 'num_bytes': 40662}}
verification_name = 'dataset source files'

    def verify_checksums(expected_checksums: Optional[dict], recorded_checksums: dict, verification_name=None):
        if expected_checksums is None:
            logger.info("Unable to verify checksums.")
            return
        if len(set(expected_checksums) - set(recorded_checksums)) > 0:
            raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
        if len(set(recorded_checksums) - set(expected_checksums)) > 0:
            raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))
        bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
        for_verification_name = " for " + verification_name if verification_name is not None else ""
        if len(bad_urls) > 0:
            error_msg = "Checksums didn't match" + for_verification_name + ":\n"
>           raise NonMatchingChecksumError(error_msg + str(bad_urls))
E           datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
E           ['https://adversarialglue.github.io/dataset/dev.zip']

src/datasets/utils/info_utils.py:40: NonMatchingChecksumError
```

## Expected results
The dataset downloads correctly, and there is no error.

## Actual results
Datasets library is looking for a checksum of None, and it gets a non-None checksum, and throws an error. This is clearly a bug.
Datasets' cache not re-used
State: open
Labels: name: bug

## Describe the bug
For most tokenizers I have tested (e.g. the RoBERTa tokenizer), the data preprocessing cache are not fully reused in the first few runs, although their `.arrow` cache files are in the cache directory.

## Steps to reproduce the bug
Here is a reproducer. The GPT2 tokenizer works perfectly with caching, but not the RoBERTa tokenizer in this example.
```python
from datasets import load_dataset
from transformers import AutoTokenizer

raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1")
# tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
text_column_name = "text"
column_names = raw_datasets["train"].column_names

def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = raw_datasets.map(
    tokenize_function,
    batched=True,
    remove_columns=column_names,
    load_from_cache_file=True,
    desc="Running tokenizer on every text in dataset",
)
```

## Expected results
No tokenization would be required after the 1st run. Everything should be loaded from the cache.

## Actual results
Tokenization for some subsets are repeated at the 2nd and 3rd run. Starting from the 4th run, everything are loaded from cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 18.04.6 LTS
- Python version: 3.6.9
- PyArrow version: 6.0.1
add RMSE and MAE metrics.
State: closed

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Please suggest any changes if required. Thank you.
Add rmse and mae metrics.
State: closed

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Any suggestions and changes required will be helpful.
Align IterableDataset.shuffle with Dataset.shuffle
State: closed

From #3444 , Dataset.shuffle can have the same API than IterableDataset.shuffle (i.e. in streaming mode).

Currently you can pass an optional seed to both if you want, BUT currently IterableDataset.shuffle always requires a buffer_size, used for approximate shuffling. I propose using a reasonable default value (maybe 1000) instead.

In this PR, I set the default `buffer_size` value to 1,000, and I reorder the `IterableDataset.shuffle` arguments to match `Dataset.shuffle`, i.e. making `seed` the first argument.
Pyright reportPrivateImportUsage when `from datasets import load_dataset` 
State: closed
Labels: name: bug

## Describe the bug
Pyright complains about module not exported. 

## Steps to reproduce the bug
Use an editor/IDE with Pyright Language server with default configuration:
```python
from datasets import load_dataset
```

## Expected results
No complain from Pyright

## Actual results
Pyright complain below:
```
`load_dataset` is not exported from module "datasets"
 Import from "datasets.load" instead [reportPrivateImportUsage] 
```
Importing from `datasets.load` does indeed solves the problem but I believe importing directly from top level `datasets` is the intended usage per the documentation.

## Environment info
- `datasets` version: 1.18.3
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0
Add a data type for labeled images (image segmentation)
State: open
Labels: name: enhancement

It might be a mix of Image and ClassLabel, and the color palette might be generated automatically.

---

### Example

every pixel in the images of the annotation column (in https://huggingface.co/datasets/scene_parse_150) has a value that gives its class, and the dataset itself is associated with a color palette (eg https://github.com/open-mmlab/mmsegmentation/blob/98a353b674c6052d319e7de4e5bcd65d670fcf84/mmseg/datasets/ade.py#L47) that maps every class with a color.

So we might want to render the image as a colored image instead of a black and white one.

<img width="785" alt="156741519-fbae6844-2606-4c28-837e-279d83d00865" src="https://user-images.githubusercontent.com/1676121/157005263-7058c584-2b70-465a-ad94-8a982f726cf4.png">

See https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/features/labeled_image.py for reference in Tensorflow
Making Hugging Face the place to go for Graph NNs datasets
State: open
Labels: name: dataset request, name: graph

Let's make Hugging Face Datasets the central hub for GNN datasets :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the GNN field.

What are some datasets worth integrating into the Hugging Face hub?

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Special thanks to @napoles-uach for his collaboration on identifying the first ones:

- [ ] [SNAP-Stanford OGB Datasets](https://github.com/snap-stanford/ogb).
- [ ] [SNAP-Stanford Pretrained GNNs Chemistry and Biology Datasets](https://github.com/snap-stanford/pretrain-gnns).
- [ ] [TUDatasets](https://chrsmrrs.github.io/datasets/) (A collection of benchmark datasets for graph classification and regression)

cc @osanseviero
when using to_tf_dataset with shuffle is true, not all completed batches are made
State: closed
Labels: name: bug

## Describe the bug
when converting a dataset to tf_dataset by using to_tf_dataset with shuffle true, the remainder is not converted to one batch

## Steps to reproduce the bug
this is the sample code below
https://colab.research.google.com/drive/1_oRXWsR38ElO1EYF9ayFoCU7Ou1AAej4?usp=sharing


## Expected results
regardless of shuffle is true or not, 67 rows dataset should be 5 batches when batch size is 16.

## Actual results
4 batches

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1
Got error when load cnn_dailymail dataset
State: closed
Labels: name: duplicate

When using datasets.load_dataset method to load cnn_dailymail dataset,  got error as below:

- windows os: FileNotFoundError: [WinError 3] 系统找不到指定的路径。: 'D:\\SourceCode\\DataScience\\HuggingFace\\Data\\downloads\\1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b\\cnn\\stories'
- google colab: NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'

The code is to load dataset:
windows os:
```
from datasets import load_dataset

dataset = load_dataset("cnn_dailymail", "3.0.0", cache_dir="D:\\SourceCode\\DataScience\\HuggingFace\\Data")

```
google colab:
```
import datasets

train_data = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")
```
[📄 Docs] Create a `datasets` performance guide.
State: open
Labels: name: enhancement

## Brief Overview
Downloading, saving, and preprocessing large datasets from the `datasets` library can often result in [performance bottlenecks](https://github.com/huggingface/datasets/issues/3735). These performance snags can be challenging to identify and to debug, especially for users who are less experienced with building deep learning experiments.

## Feature Request
Could we create a performance guide for using `datasets`, similar to:
* [Better performance with the `tf.data` API](https://github.com/huggingface/datasets/issues/3735)
* [Analyze `tf.data` performance with the TF Profiler](https://www.tensorflow.org/guide/data_performance_analysis)

This performance guide should detail practical options for improving performance with `datasets`, and enumerate any common best practices. It should also show how to use tools like the PyTorch Profiler or the TF Profiler to identify any performance bottlenecks (example below).

![image](https://user-images.githubusercontent.com/3712347/156859152-a3cb9565-3ec6-4d39-8e77-56d0a75a4954.png)

## Related Issues
* [wiki_dpr pre-processing performance #1670](https://github.com/huggingface/datasets/issues/1670)
* [Adjusting chunk size for streaming datasets #3499](https://github.com/huggingface/datasets/issues/3499)
* [how large datasets are handled under the hood #1004](https://github.com/huggingface/datasets/issues/1004)
* [using map on loaded Tokenizer 10x - 100x slower than default Tokenizer? #1830](https://github.com/huggingface/datasets/issues/1830)
* [Best way to batch a large dataset? #315](https://github.com/huggingface/datasets/issues/315)
* [Saving processed dataset running infinitely #1911](https://github.com/huggingface/datasets/issues/1911)
The Pile's _FEATURE spec seems to be incorrect
State: closed
Labels: name: bug

## Describe the bug
If you look at https://huggingface.co/datasets/the_pile/blob/main/the_pile.py:

For "all"
*  the pile_set_name is never set for data
* there's actually an id field inside of "meta" 

For subcorpora pubmed_central and hacker_news:
* the meta is specified to be a string, but it's actually a dict with an id field inside.

## Steps to reproduce the bug


## Expected results
Feature spec should match the data I'd think?

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
Add IterableDataset.filter
State: closed

_Needs https://github.com/huggingface/datasets/pull/3801 to be merged first_

I added `IterableDataset.filter` with an API that is a subset of `Dataset.filter`:
```python
def filter(self, function, batched=False, batch_size=1000, with_indices=false, input_columns=None):
```

TODO:
- [x] tests
- [x] docs

related to https://github.com/huggingface/datasets/issues/3444 and https://github.com/huggingface/datasets/issues/3753
500 internal server error when trying to open a dataset composed of Zarr stores
State: closed
Labels: name: bug

## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine. 

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format? 

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0
Add Biwi Kinect Head Pose Database
State: closed
Labels: name: dataset request, name: vision

## Adding a Dataset
- **Name:** Biwi Kinect Head Pose Database
- **Description:** Over 15K images of 20 people recorded with a Kinect while turning their heads around freely. For each frame, depth and rgb images are provided, together with ground in the form of the 3D location of the head and its rotation angles.
- **Data:** [*link to the Github repository or current dataset location*](https://icu.ee.ethz.ch/research/datsets.html)
- **Motivation:** Useful pose estimation dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Update Wikipedia dataset
State: closed

This PR combines all updates to Wikipedia dataset.

Once approved, this will be used to generate the pre-processed Wikipedia datasets.

Finally, this PR will be able to be merged into master:
- NOT using squash
- BUT a regular MERGE (or REBASE+MERGE), so that all commits are preserved

TODO:
- [x] #3435
- [x] #3789
- [x] #3825
- [x] Run to get the pre-processed data for big languages (backward compatibility)
- [x] #3958

CC: @geohci
`pubmed_qa` checksum mismatch
State: closed
Labels: name: bug, name: duplicate

## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0
Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input. 

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."], 
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```
  
**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch). 

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).
Simplify Common Voice code
State: closed

In #3736 we introduced one method to generate examples when streaming, that is different from the one when not streaming.

In this PR I propose a new implementation which is simpler: it only has one function, based on `iter_archive`. And you still have access to local audio files when loading the dataset in non-streaming mode.

cc @patrickvonplaten @polinaeterna @anton-l @albertvillanova since this will become the template for many audio datasets to come.

This change can also trivially be applied to the other audio datasets that already exist. Using this line, you can get access to local files in non-streaming mode:
```python
local_extracted_archive = dl_manager.extract(archive_path) if not dl_manager.is_streaming else None
```
Fix iter_archive getting reset
State: closed

The `DownloadManager.iter_archive` method currently returns an iterator - which is  **empty** once you iter over it once. This means you can't pass the same archive iterator to several splits.

To fix that, I changed the ouput of `DownloadManager.iter_archive` to be an iterable that you can iterate over several times, instead of a one-time-use iterator.

The `StreamingDownloadManager.iter_archive` already returns an appropriate iterable, and the code added in this PR is inspired from the one in `streaming_download_manager.py`
Handle Nones in PyArrow struct
State: closed

This PR fixes an issue introduced by #3575 where `None` values stored in PyArrow arrays/structs would get ignored by `cast_storage` or by the `pa.array(cast_to_python_objects(..))` pattern. To fix the former, it also bumps the minimal PyArrow version to v5.0.0 to use the `mask` param in `pa.SturctArray`.
Add MetaShift dataset
State: closed
Labels: name: dataset request, name: vision

## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
benchmark streaming speed with tar vs zip archives
State: closed

# do not merge
## Hypothesis 
packing data into a single zip archive could allow us not to care about splitting data into several tar archives for efficient streaming which is annoying (since data creators usually host the data in a single tar)

## Data

I host it [here](https://huggingface.co/datasets/polinaeterna/benchmark_dataset/)

## I checked three configurations:
1. All data in one zip archive, streaming only those files that exist in split metadata file (we can access them directrly with no need to iterate over full archive), see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR196)
2.  All data in three splits, the standart way to make streaming efficient, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR174)
3. All data in single tar, iterate over the full archive and take only files existing in split metadata file, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR150) 

## Results
1. one zip
![image](https://user-images.githubusercontent.com/16348744/156567611-e3652087-7147-4cf0-9047-9cbc00ec71f5.png)
2. three tars
![image](https://user-images.githubusercontent.com/16348744/156567688-2a462107-f83e-4722-8ea3-71a13b56c998.png)
3. one tar
![image](https://user-images.githubusercontent.com/16348744/156567772-1bceb5f7-e7d9-4fa3-b31b-17fec5f9a5a7.png)

didn't check on the full data as it's time consuming but anyway it's pretty obvious that one-zip-way is not a good idea. here it's even worse than full iteration over tar containing all three splits (but that would depend on the case).
Checksums didn't match for datasets on Google Drive
State: closed
Labels: name: bug, name: duplicate

## Describe the bug

Datasets hosted on Google Drive do not seem to work right now.
Loading them fails with a checksum error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
for dataset in ["head_qa", "yelp_review_full"]:
  try:
    load_dataset(dataset)
  except Exception as exception:
    print("Error", dataset, exception)
```

Here is a [colab](https://colab.research.google.com/drive/1wOtHBmL8I65NmUYakzPV5zhVCtHhi7uQ#scrollTo=cDzdCLlk-Bo4).

## Expected results

The datasets should be loaded.

## Actual results

```
Downloading and preparing dataset head_qa/es (download: 75.69 MiB, generated: 2.86 MiB, post-processed: Unknown size, total: 78.55 MiB) to /root/.cache/huggingface/datasets/head_qa/es/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Error head_qa Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
Downloading and preparing dataset yelp_review_full/yelp_review_full (download: 187.06 MiB, generated: 496.94 MiB, post-processed: Unknown size, total: 684.00 MiB) to /root/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/13c31a618ba62568ec8572a222a283dfc29a6517776a3ac5945fb508877dde43...
Error yelp_review_full Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZlU4dXhHTFhZQU0']
```

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1
Pre-Processing Cache Fails when using a Factory pattern
State: closed
Labels: name: bug

## Describe the bug
If you utilize a pre-processing function which is created using a factory pattern, the function hash changes on each run (even if the function is identical) and therefore the data will be reproduced each time.

## Steps to reproduce the bug
```python
    def preprocess_function_factory(augmentation=None):
        def preprocess_function(examples):
            # Tokenize the texts
            if augmentation:
                conversions1 = [
                    augmentation(example)
                    for example in examples[sentence1_key]
                ]
                if sentence2_key is None:
                    args = (conversions1,)
                else:
                    conversions2 = [
                        augmentation(example)
                        for example in examples[sentence2_key]
                    ]
                    args = (conversions1, conversions2)
            else:
                args = (
                    (examples[sentence1_key],)
                    if sentence2_key is None
                    else (examples[sentence1_key], examples[sentence2_key])
                )
            result = tokenizer(
                *args, padding=padding, max_length=max_seq_length, truncation=True
            )

            # Map labels to IDs (not necessary for GLUE tasks)
            if label_to_id is not None and "label" in examples:
                result["label"] = [
                    (label_to_id[l] if l != -1 else -1) for l in examples["label"]
                ]
            return result

        return preprocess_function

capitalize = lambda x: x.capitalize()
preprocess_function = preprocess_function_factory(augmentation=capitalize)
print(hash(preprocess_function)) # This will change on each run
raw_datasets = raw_datasets.map(
        preprocess_function,
        batched=True,
        load_from_cache_file=True,
        desc="Running transformation and tokenizer on dataset",
)
```

## Expected results
Running the code twice will cause the cache to be re-used.

## Actual results
Running the code twice causes the whole dataset to be re-processed
NonMatchingChecksumError in xcopa dataset
State: closed
Labels: name: bug

## Describe the bug
Loading the xcopa dataset doesn't work, it fails due to a mismatch in the checksum.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("xcopa", "it")
```

## Expected results
The dataset should be loaded correctly.

## Actual results
Fails with:
```python
in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/cambridgeltl/xcopa/archive/master.zip']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3, and 1.18.4.dev0
- Platform:
- Python version: 3.8
- PyArrow version:
Text builder with custom separator line boundaries
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted. 

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None: 
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.