josteece/github-issues huggingface dataset:
[📄 Docs] Create a `datasets` performance guide.
Opened by dynamicwebpaige (open)

## Brief Overview
Downloading, saving, and preprocessing large datasets from the `datasets` library can often result in [performance bottlenecks](https://github.com/huggingface/datasets/issues/3735). These performance snags can be challenging to identify and to debug, especially for users who are less experienced with building deep learning experiments.

## Feature Request
Could we create a performance guide for using `datasets`, similar to:
* [Better performance with the `tf.data` API](https://github.com/huggingface/datasets/issues/3735)
* [Analyze `tf.data` performance with the TF Profiler](https://www.tensorflow.org/guide/data_performance_analysis)

This performance guide should detail practical options for improving performance with `datasets`, and enumerate any common best practices. It should also show how to use tools like the PyTorch Profiler or the TF Profiler to identify any performance bottlenecks (example below).

![image](https://user-images.githubusercontent.com/3712347/156859152-a3cb9565-3ec6-4d39-8e77-56d0a75a4954.png)

## Related Issues
* [wiki_dpr pre-processing performance #1670](https://github.com/huggingface/datasets/issues/1670)
* [Adjusting chunk size for streaming datasets #3499](https://github.com/huggingface/datasets/issues/3499)
* [how large datasets are handled under the hood #1004](https://github.com/huggingface/datasets/issues/1004)
* [using map on loaded Tokenizer 10x - 100x slower than default Tokenizer? #1830](https://github.com/huggingface/datasets/issues/1830)
* [Best way to batch a large dataset? #315](https://github.com/huggingface/datasets/issues/315)
* [Saving processed dataset running infinitely #1911](https://github.com/huggingface/datasets/issues/1911)

Comments:
Hi ! Yes this is definitely something we'll explore, since optimizing processing pipelines can be challenging and because performance is key here: we want anyone to be able to play with large-scale datasets more easily.

I think we'll start by documenting the performance of the dataset transforms we provide, and then we can have some tools to help debugging/optimizing them
The Pile's _FEATURE spec seems to be incorrect
Opened by dlwh (closed)

## Describe the bug
If you look at https://huggingface.co/datasets/the_pile/blob/main/the_pile.py:

For "all"
*  the pile_set_name is never set for data
* there's actually an id field inside of "meta" 

For subcorpora pubmed_central and hacker_news:
* the meta is specified to be a string, but it's actually a dict with an id field inside.

## Steps to reproduce the bug


## Expected results
Feature spec should match the data I'd think?

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Comments:
Hi @dlwh, thanks for reporting.

Please note, that the source data files for "all" config are different from the other configurations.

The "all" config contains the official Pile data files, from https://mystic.the-eye.eu/public/AI/pile/
All data examples contain a "meta" dict with a single "pile_set_name" key:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("the_pile", "all", split="train", streaming=True)
        item = next(iter(ds))
Downloading builder script: 9.09kB [00:00, 4.42MB/s]

In [3]: item["meta"]
Out[3]: {'pile_set_name': 'Pile-CC'}
```

On the other hand, all the other subset configs data files come from the Pile preliminary components directory: https://mystic.the-eye.eu/public/AI/pile_preliminary_components/
For theses components, the "meta" field may have different keys depending on the subset: "id", "language", "pmid",... Because of that, if we had kept the `dict` data format for the "meta" field, we would have an error when trying to concatenate different subsets, whose "meta" keys are not identical. In order to avoid that, the "meta" field is cast to `str` in all these cases, so that there is no incompatibility in their "meta" data type when concatenating.

You can check, for example, that for "pubmed_central" the "meta" field is cast to `str`:
```python
In [4]: from datasets import load_dataset
        ds = load_dataset("the_pile", "pubmed_central", split="train", streaming=True)
        item = next(iter(ds))

In [5]: item["meta"]
Out[5]: "{'id': 'PMC6071596'}"
```

Feel free to reopen this issue if you have further questions.
Add IterableDataset.filter
Opened by lhoestq (closed)

_Needs https://github.com/huggingface/datasets/pull/3801 to be merged first_

I added `IterableDataset.filter` with an API that is a subset of `Dataset.filter`:
```python
def filter(self, function, batched=False, batch_size=1000, with_indices=false, input_columns=None):
```

TODO:
- [x] tests
- [x] docs

related to https://github.com/huggingface/datasets/issues/3444 and https://github.com/huggingface/datasets/issues/3753

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3826). All of your documentation changes will be reflected on that endpoint.

Indeed ! If `batch_size` is `None` or `<=0` then the full dataset should be passed. It's been mentioned in the docs for a while but never actually implemented. We can fix that later
Add Biwi Kinect Head Pose Database
Opened by osanseviero (closed)

## Adding a Dataset
- **Name:** Biwi Kinect Head Pose Database
- **Description:** Over 15K images of 20 people recorded with a Kinect while turning their heads around freely. For each frame, depth and rgb images are provided, together with ground in the form of the 3D location of the head and its rotation angles.
- **Data:** [*link to the Github repository or current dataset location*](https://icu.ee.ethz.ch/research/datsets.html)
- **Motivation:** Useful pose estimation dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Official dataset location : https://icu.ee.ethz.ch/research/datsets.html
In the "Biwi Kinect Head Pose Database" section, I do not find any information regarding "Downloading the dataset." . Do we mail the authors regarding this ?

I found the dataset on Kaggle : [Link](https://www.kaggle.com/kmader/biwi-kinect-head-pose-database) , but since 🤗 does not host any of the datasets, this would require the user to provide their Kaggle username and API key to download. 

Any inputs on how we could proceed ? Thank you.
[ Need your inputs here, @lhoestq or @mariosasko ]

Hi @dnaveenr! Thanks for tackling this issue. This link should work: https://data.vision.ee.ethz.ch/cvl/gfanelli/kinect_head_pose_db.tgz

#self-assign

Added in https://github.com/huggingface/datasets/pull/3903, thanks @dnaveenr !

@mariosasko Hi,
this link [https://data.vision.ee.ethz.ch/cvl/gfanelli/kinect_head_pose_db.tgz]
seems not working now. Do you have alternatives?

It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose

> It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose

Hi, you mean I can download it by running python script in your link?

This dataset repository only contains a script that points to the URL of the file that doesn't exist anymore, so it's not usable anymore either :/

> This dataset repository only contains a script that points to the URL of the file that doesn't exist anymore, so it's not usable anymore either :/

Then what did you mean 
"It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose"?
I don't understand the meaning of 'host'

If we find a copy of the data we can upload the file in the dataset repository on HF, since the original data is not available anymore
Update Wikipedia dataset
Opened by albertvillanova (closed)

This PR combines all updates to Wikipedia dataset.

Once approved, this will be used to generate the pre-processed Wikipedia datasets.

Finally, this PR will be able to be merged into master:
- NOT using squash
- BUT a regular MERGE (or REBASE+MERGE), so that all commits are preserved

TODO:
- [x] #3435
- [x] #3789
- [x] #3825
- [x] Run to get the pre-processed data for big languages (backward compatibility)
- [x] #3958

CC: @geohci

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm starting to generate the pre-processed data for some of the languages (for backward compatibility).

Once this merged, we will create the pre-processed data on the Hub under the Wikimedia namespace.

All steps have been properly done.

I'm merging all these commits into master.
`pubmed_qa` checksum mismatch
Opened by jon-tow (closed)

## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0


Comments:
Hi @jon-tow, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

We are planning to make a patch release today.

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Simplify Common Voice code
Opened by lhoestq (closed)

In #3736 we introduced one method to generate examples when streaming, that is different from the one when not streaming.

In this PR I propose a new implementation which is simpler: it only has one function, based on `iter_archive`. And you still have access to local audio files when loading the dataset in non-streaming mode.

cc @patrickvonplaten @polinaeterna @anton-l @albertvillanova since this will become the template for many audio datasets to come.

This change can also trivially be applied to the other audio datasets that already exist. Using this line, you can get access to local files in non-streaming mode:
```python
local_extracted_archive = dl_manager.extract(archive_path) if not dl_manager.is_streaming else None
```

Comments:
I think the script looks pretty clean and readable now! cool!
Fix iter_archive getting reset
Opened by lhoestq (closed)

The `DownloadManager.iter_archive` method currently returns an iterator - which is  **empty** once you iter over it once. This means you can't pass the same archive iterator to several splits.

To fix that, I changed the ouput of `DownloadManager.iter_archive` to be an iterable that you can iterate over several times, instead of a one-time-use iterator.

The `StreamingDownloadManager.iter_archive` already returns an appropriate iterable, and the code added in this PR is inspired from the one in `streaming_download_manager.py`

Comments:
Handle Nones in PyArrow struct
Opened by mariosasko (closed)

This PR fixes an issue introduced by #3575 where `None` values stored in PyArrow arrays/structs would get ignored by `cast_storage` or by the `pa.array(cast_to_python_objects(..))` pattern. To fix the former, it also bumps the minimal PyArrow version to v5.0.0 to use the `mask` param in `pa.SturctArray`. 



Comments:
Looks like I added my comments while you were editing - sorry about that
Add MetaShift dataset
Opened by osanseviero (closed)

## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
I would like to take this up and give it a shot. Any image specific - dataset guidelines to keep in mind ? Thank you.

#self-assign

I've started working on adding this dataset. I require some inputs on the following : 

Ref for the initial draft [here](https://github.com/dnaveenr/datasets/blob/add_metashift_dataset/datasets/metashift/metashift.py)
1. The dataset does not have a typical - train/test/val split. What do we do for the _split_generators() function ? How do we go about this ?
2. This dataset builds on the Visual Genome dataset, using a metadata file. The dataset is generated using generate_full_MetaShift.py script. By default, the authors choose to generate the dataset only for a SELECTED_CLASSES. The following script is used : 
Code : https://github.com/Weixin-Liang/MetaShift/blob/main/dataset/generate_full_MetaShift.py 
Info : https://metashift.readthedocs.io/en/latest/sub_pages/download_MetaShift.html#generate-the-full-metashift-dataset
Can I just copy over the required functions into the metashift.py to generate the dataset ?
3. How do we complete the _generate_examples for this dataset ?

The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Inputs, suggestions would be helpful. Thank you.

I think @mariosasko and @lhoestq should be able to help here 😄 

Hi ! Thanks for adding this dataset :) Let me answer your questions:

1. in this case you can put everything in the "train" split
2. Yes you can copy the script (provided you also include the MIT license of the code in the file header for example). Though we ideally try to not create new directories nor files when generating dataset, so if possible this script should be adapted to not create the file structure they mentioned, but instead yield the images one by one in `_generate_examples`. Let me know if you think this is feasible
3. see point 2 haha

> The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Yup ! We can also define a `selected_classes` parameter such that users can do
```python
load_dataset("metashift", selected_classes=["cat", "dog", ...])
```

Great. This is helpful. Thanks @lhoestq .
Regarding Point 2, I'll try using yield instead of creating the directories and see if its feasible. selected_classes config sounds good.

Closed via #3900
benchmark streaming speed with tar vs zip archives
Opened by polinaeterna (closed)

# do not merge
## Hypothesis 
packing data into a single zip archive could allow us not to care about splitting data into several tar archives for efficient streaming which is annoying (since data creators usually host the data in a single tar)

## Data

I host it [here](https://huggingface.co/datasets/polinaeterna/benchmark_dataset/)

## I checked three configurations:
1. All data in one zip archive, streaming only those files that exist in split metadata file (we can access them directrly with no need to iterate over full archive), see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR196)
2.  All data in three splits, the standart way to make streaming efficient, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR174)
3. All data in single tar, iterate over the full archive and take only files existing in split metadata file, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR150) 

## Results
1. one zip
![image](https://user-images.githubusercontent.com/16348744/156567611-e3652087-7147-4cf0-9047-9cbc00ec71f5.png)
2. three tars
![image](https://user-images.githubusercontent.com/16348744/156567688-2a462107-f83e-4722-8ea3-71a13b56c998.png)
3. one tar
![image](https://user-images.githubusercontent.com/16348744/156567772-1bceb5f7-e7d9-4fa3-b31b-17fec5f9a5a7.png)

didn't check on the full data as it's time consuming but anyway it's pretty obvious that one-zip-way is not a good idea. here it's even worse than full iteration over tar containing all three splits (but that would depend on the case).


Comments:
I'm closing the PR since we're not going to merge it
Checksums didn't match for datasets on Google Drive
Opened by muelletm (closed)

## Describe the bug

Datasets hosted on Google Drive do not seem to work right now.
Loading them fails with a checksum error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
for dataset in ["head_qa", "yelp_review_full"]:
  try:
    load_dataset(dataset)
  except Exception as exception:
    print("Error", dataset, exception)
```

Here is a [colab](https://colab.research.google.com/drive/1wOtHBmL8I65NmUYakzPV5zhVCtHhi7uQ#scrollTo=cDzdCLlk-Bo4).

## Expected results

The datasets should be loaded.

## Actual results

```
Downloading and preparing dataset head_qa/es (download: 75.69 MiB, generated: 2.86 MiB, post-processed: Unknown size, total: 78.55 MiB) to /root/.cache/huggingface/datasets/head_qa/es/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Error head_qa Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
Downloading and preparing dataset yelp_review_full/yelp_review_full (download: 187.06 MiB, generated: 496.94 MiB, post-processed: Unknown size, total: 684.00 MiB) to /root/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/13c31a618ba62568ec8572a222a283dfc29a6517776a3ac5945fb508877dde43...
Error yelp_review_full Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZlU4dXhHTFhZQU0']
```

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Comments:
Hi @muelletm, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

Until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Pre-Processing Cache Fails when using a Factory pattern
Opened by Helw150 (closed)

## Describe the bug
If you utilize a pre-processing function which is created using a factory pattern, the function hash changes on each run (even if the function is identical) and therefore the data will be reproduced each time.

## Steps to reproduce the bug
```python
    def preprocess_function_factory(augmentation=None):
        def preprocess_function(examples):
            # Tokenize the texts
            if augmentation:
                conversions1 = [
                    augmentation(example)
                    for example in examples[sentence1_key]
                ]
                if sentence2_key is None:
                    args = (conversions1,)
                else:
                    conversions2 = [
                        augmentation(example)
                        for example in examples[sentence2_key]
                    ]
                    args = (conversions1, conversions2)
            else:
                args = (
                    (examples[sentence1_key],)
                    if sentence2_key is None
                    else (examples[sentence1_key], examples[sentence2_key])
                )
            result = tokenizer(
                *args, padding=padding, max_length=max_seq_length, truncation=True
            )

            # Map labels to IDs (not necessary for GLUE tasks)
            if label_to_id is not None and "label" in examples:
                result["label"] = [
                    (label_to_id[l] if l != -1 else -1) for l in examples["label"]
                ]
            return result

        return preprocess_function

capitalize = lambda x: x.capitalize()
preprocess_function = preprocess_function_factory(augmentation=capitalize)
print(hash(preprocess_function)) # This will change on each run
raw_datasets = raw_datasets.map(
        preprocess_function,
        batched=True,
        load_from_cache_file=True,
        desc="Running transformation and tokenizer on dataset",
)
```

## Expected results
Running the code twice will cause the cache to be re-used.

## Actual results
Running the code twice causes the whole dataset to be re-processed


Comments:
Ok - this is still an issue but I believe the root cause is different than I originally thought. I'm now able to get caching to work consistently with the above example as long as I fix the python hash seed `export PYTHONHASHSEED=1234`

Hi! 

Yes, our hasher should work with decorators. For instance, this dummy example:
```python
def f(arg):
    def f1(ex):
        return {"a": ex["col1"] + arg}
    return f1
```
gives the same hash across different Python sessions (`datasets.fingerprint.Hasher.hash(f("string1")` returns `"408c9059f89dbd6c"` on my machine).

Could you please make the example self-contained? This way, we can reproduce the bug. Additionally, you can try to find the problematic object yourself by testing their hash with `datasets.fingerprint.Hasher.hash(obj)`

This could be related to https://github.com/huggingface/datasets/issues/3638.

#3638 was indeed my issue. Thanks!