SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: NonMatchingChecksumError when checksum is None

Comment:
At a higher level, also note that we are preparing the release of `datasets` version 2.0, and some docs are being updated...

In order to add a dataset, I think the most updated instructions are in our official documentation pages: https://huggingface.co/docs/datasets/share
Issue: NonMatchingChecksumError when checksum is None

Comment:
Hi @jxmorris12, we have discovered the bug why `None` checksums wrongly appeared when generating the `dataset_infos.json` file:
- #3892

The fix will be accessible once this PR merged. And we are planning to do our 2.0 release today.

We are also working on updating all our docs for our release today.
Issue: Datasets' cache not re-used

Comment:

I think this is because the tokenizer is stateful and because the order in which the splits are processed is not deterministic. Because of that, the hash of the tokenizer may change for certain splits, which causes issues with caching.

To fix this we can try making the order of the splits deterministic for map.
Issue: Datasets' cache not re-used

Comment:
Actually this is not because of the order of the splits, but most likely because the tokenizer used to process the second split is in a state that has been modified by the first split.

Therefore after reloading the first split from the cache, then the second split can't be reloaded since the tokenizer hasn't seen the first split (and therefore is considered a different tokenizer).

This is a bit trickier to fix, we can explore fixing this next week maybe
Issue: Datasets' cache not re-used

Comment:
Sorry didn't have the bandwidth to take care of this yet - will re-assign when I'm diving into it again !
Issue: Datasets' cache not re-used

Comment:
I had this issue with `run_speech_recognition_ctc.py` for wa2vec2.0 fine-tuning. I made a small change and the hash for the function (which includes tokenisation) is now the same before and after pre-porocessing. With the hash being the same, the caching works as intended.

Before:
```
    def prepare_dataset(batch):
        # load audio
        sample = batch[audio_column_name]

        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
        batch["input_values"] = inputs.input_values[0]
        batch["input_length"] = len(batch["input_values"])

        # encode targets
        additional_kwargs = {}
        if phoneme_language is not None:
            additional_kwargs["phonemizer_lang"] = phoneme_language

        batch["labels"] = tokenizer(batch["target_text"], **additional_kwargs).input_ids

        return batch

    with training_args.main_process_first(desc="dataset map preprocessing"):
        vectorized_datasets = raw_datasets.map(
            prepare_dataset,
            remove_columns=next(iter(raw_datasets.values())).column_names,
            num_proc=num_workers,
            desc="preprocess datasets",
        )
```
After:
```
    def prepare_dataset(batch, feature_extractor, tokenizer):
        # load audio
        sample = batch[audio_column_name]

        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
        batch["input_values"] = inputs.input_values[0]
        batch["input_length"] = len(batch["input_values"])

        # encode targets
        additional_kwargs = {}
        if phoneme_language is not None:
            additional_kwargs["phonemizer_lang"] = phoneme_language

        batch["labels"] = tokenizer(batch["target_text"], **additional_kwargs).input_ids

        return batch

    pd = lambda batch: prepare_dataset(batch, feature_extractor, tokenizer)

    with training_args.main_process_first(desc="dataset map preprocessing"):
        vectorized_datasets = raw_datasets.map(
            pd,
            remove_columns=next(iter(raw_datasets.values())).column_names,
            num_proc=num_workers,
            desc="preprocess datasets",
        )
```
Issue: Datasets' cache not re-used

Comment:
Not sure why the second one would work and not the first one - they're basically the same with respect to hashing. In both cases the function is hashed recursively, and therefore the feature_extractor and the tokenizer are hashed the same way.

With which tokenizer or feature extractor are you experiencing this behavior ?

Do you also experience this ?
> Tokenization for some subsets are repeated at the 2nd and 3rd run. Starting from the 4th run, everything are loaded from cache.
Issue: Datasets' cache not re-used

Comment:
Thanks ! Hopefully this can be useful to others, and also to better understand and improve hashing/caching
Issue: Datasets' cache not re-used

Comment:
Thanks for investigating ! Does that mean that `save_pretrained`() produces non-deterministic tokenizers on disk ? Or is it `from_pretrained()` which is not deterministic given the same files on disk ?

I think one way to fix this would be to make save/from_pretrained deterministic, or make the pickling of `transformers.tokenization_utils.Trie` objects deterministic (this could be implemented in `transformers`, but maybe let's discuss in an issue in `transformers` before opening a PR)
Issue: Datasets' cache not re-used

Comment:
Late to the party but everything should be deterministic (afaik at least).

But `Trie` is a simple class object, so afaik it's hash function is linked to its `id(self)` so basically where it's stored in memory, so super highly non deterministic. Could that be the issue ?
Issue: Datasets' cache not re-used

Comment:
> But Trie is a simple class object, so afaik it's hash function is linked to its id(self) so basically where it's stored in memory, so super highly non deterministic. Could that be the issue ?

We're computing the hash of the pickle dump of the class so it should be fine, as long as the pickle dump is deterministic
Issue: Datasets' cache not re-used

Comment:
I've ported wav2vec2.0 fine-tuning into Optimum-Graphcore which is where I found the issue. The majority of the script was copied from the Transformers version to keep it similar, [here is the tokenizer loading section from the source](https://github.com/huggingface/transformers/blob/f0982682bd6fd0b438dda79ec45f3a8fac83a985/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L531).

In the last comment I have two loaded tokenizers, one from run 'N' of the script and one from 'N+1'. I think what's happening is that when you add special tokens (e.g. PAD and UNK) another AddedToken object is appended when tokenizer is saved regardless of whether special tokens are there already. 

If there is a AddedTokens cleanup at load/save this could solve the issue, but then is Trie going to cause hash to be different? I'm not sure.
Issue: Datasets' cache not re-used

Comment:
Which Python version are you using ?

The trie is basically a big dict of dics, so deterministic nature depends on python version:
https://stackoverflow.com/questions/2053021/is-the-order-of-a-python-dictionary-guaranteed-over-iterations

Maybe the investigation is actually not finding the right culprit though (the memory id is changed, but `datasets` is not using that to compare, so maybe we need to be looking within  `datasets` so see where the comparison fails)
Issue: Datasets' cache not re-used

Comment:
Similar issue found on `BartTokenizer`. You can bypass the bug by loading a fresh new tokenizer everytime.

```
    dataset = dataset.map(lambda x: tokenize_func(x, BartTokenizer.from_pretrained(xxx)),
                          num_proc=num_proc, desc='Tokenize')
```
Issue: Pyright reportPrivateImportUsage when `from datasets import load_dataset` 

Comment:
Hi! 

This issue stems from `datasets` having `py.typed` defined (see https://github.com/microsoft/pyright/discussions/3764#discussioncomment-3282142) - to avoid it, we would either have to remove `py.typed` (added to be compliant with PEP-561) or export the names with `__all__`/`from .submodule import name as name`.

Transformers is fine as it no longer has `py.typed` (removed in https://github.com/huggingface/transformers/pull/18485)

WDYT @lhoestq @albertvillanova @polinaeterna 

@sgugger's point makes sense - we should either be "properly typed" (have py.typed + mypy tests) or drop `py.typed` as Transformers did (I like this option better).

(cc @Wauplin since `huggingface_hub` has the same issue.)
Issue: Pyright reportPrivateImportUsage when `from datasets import load_dataset` 

Comment:
> (cc @Wauplin since huggingface_hub has the same issue.)

Hmm maybe we have the same issue but I haven't been able to reproduce something similar to `"load_dataset" is not exported from module "datasets"`  message (using VSCode+Pylance -that is powered by Pyright). `huggingface_hub` contains a `py.typed` file but the package itself is actually typed. We are running `mypy` in our CI tests since ~3 months and so far it seems to be ok. But happy to change if it causes some issues with linters.

Also the top-level [`__init__.py`](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/__init__.py) is quite different in `hfh` than `datasets` (at first glance). We have a section at the bottom to import all high level methods/classes in a `if TYPE_CHECKING` block.
Issue: Pyright reportPrivateImportUsage when `from datasets import load_dataset` 

Comment:
@Wauplin I only get the error if I use Pyright's CLI tool or the Pyright extension (not sure why, but Pylance also doesn't report this issue on my machine)

> Also the top-level [`__init__.py`](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/__init__.py) is quite different in `hfh` than `datasets` (at first glance). We have a section at the bottom to import all high level methods/classes in a `if TYPE_CHECKING` block.

I tried to fix the issue with `TYPE_CHECKING`, but it still fails if `py.typed` is present.
Issue: Pyright reportPrivateImportUsage when `from datasets import load_dataset` 

Comment:
@mariosasko thank for the tip. I have been able to reproduce the issue as well. I would be up for including a (huge) static `__all__` variable in the `__init__.py` (since the file is already generated automatically in `hfh`) but honestly I don't think it's worth the hassle. 

I'll delete the `py.typed` file in `huggingface_hub` to be consistent between HF libraries. I opened a PR here: https://github.com/huggingface/huggingface_hub/pull/1329
Issue: Pyright reportPrivateImportUsage when `from datasets import load_dataset` 

Comment:
I am getting this error in google colab today:

![image](https://user-images.githubusercontent.com/3464445/219883967-c7193a23-0388-4ba3-b00c-a53883fb6512.png)

The code runs just fine too.
Issue: Making Hugging Face the place to go for Graph NNs datasets

Comment:
It will be indeed really great to add support to GNN datasets. Big :+1: for this initiative.
Issue: Making Hugging Face the place to go for Graph NNs datasets

Comment:
@napoles-uach identifies the [TUDatasets](https://chrsmrrs.github.io/datasets/) (A collection of benchmark datasets for graph classification and regression). 

Added to the Tasks in the initial issue.
Issue: Making Hugging Face the place to go for Graph NNs datasets

Comment:
Great initiative! Let's keep this issue for these 3 datasets, but moving forward maybe let's create a new issue per dataset :rocket: great work @napoles-uach and @omarespejel!
Issue: when using to_tf_dataset with shuffle is true, not all completed batches are made

Comment:
Hi @greenned, this is expected behaviour for `to_tf_dataset`. By default, we drop the smaller 'remainder' batch during training (i.e. when `shuffle=True`). If you really want to keep that batch, you can set `drop_remainder=False` when calling `to_tf_dataset()`.
Issue: Got error when load cnn_dailymail dataset

Comment:
Hi @jon-tow, thanks for reporting. And hi @dynamicwebpaige, thanks for your investigation. 

This issue was already reported 
- #3784

and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

We are planning to make a patch release today (indeed, we were planning to do it last Friday).

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

CC: @lhoestq
Issue: [📄 Docs] Create a `datasets` performance guide.

Comment:
Hi ! Yes this is definitely something we'll explore, since optimizing processing pipelines can be challenging and because performance is key here: we want anyone to be able to play with large-scale datasets more easily.

I think we'll start by documenting the performance of the dataset transforms we provide, and then we can have some tools to help debugging/optimizing them
Issue: The Pile's _FEATURE spec seems to be incorrect

Comment:
Hi @dlwh, thanks for reporting.

Please note, that the source data files for "all" config are different from the other configurations.

The "all" config contains the official Pile data files, from https://mystic.the-eye.eu/public/AI/pile/
All data examples contain a "meta" dict with a single "pile_set_name" key:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("the_pile", "all", split="train", streaming=True)
        item = next(iter(ds))
Downloading builder script: 9.09kB [00:00, 4.42MB/s]

In [3]: item["meta"]
Out[3]: {'pile_set_name': 'Pile-CC'}
```

On the other hand, all the other subset configs data files come from the Pile preliminary components directory: https://mystic.the-eye.eu/public/AI/pile_preliminary_components/
For theses components, the "meta" field may have different keys depending on the subset: "id", "language", "pmid",... Because of that, if we had kept the `dict` data format for the "meta" field, we would have an error when trying to concatenate different subsets, whose "meta" keys are not identical. In order to avoid that, the "meta" field is cast to `str` in all these cases, so that there is no incompatibility in their "meta" data type when concatenating.

You can check, for example, that for "pubmed_central" the "meta" field is cast to `str`:
```python
In [4]: from datasets import load_dataset
        ds = load_dataset("the_pile", "pubmed_central", split="train", streaming=True)
        item = next(iter(ds))

In [5]: item["meta"]
Out[5]: "{'id': 'PMC6071596'}"
```

Feel free to reopen this issue if you have further questions.
Issue: 500 internal server error when trying to open a dataset composed of Zarr stores

Comment:
Hi @jacobbieker, thanks for reporting!

I have transferred this issue to our Hub team and they are investigating it. I keep you informed.
Issue: 500 internal server error when trying to open a dataset composed of Zarr stores

Comment:
Hi @jacobbieker, we are investigating this issue on our side and we'll see if we can fix it, but please note that your repo is considered problematic for git. Here are the results of running https://github.com/github/git-sizer on it:

```
Processing blobs: 147448                        
Processing trees: 27                        
Processing commits: 4                        
Matching commits to trees: 4                        
Processing annotated tags: 0                        
Processing references: 3                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |   167 k   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of files        [2] |   189 k   | ***                            |

[1]  aa057d2667c34c70c6146efc631f5c9917ff326e (refs/heads/main:2016.zarr/unknown)
[2]  6897b7bf6440fdd16b2c39d08085a669e7eaa59d (refs/heads/main^{tree})
```

You can check https://github.com/github/git-sizer for more information on how to avoid such pathological structures.
Issue: 500 internal server error when trying to open a dataset composed of Zarr stores

Comment:
Hi, thanks for getting back to me so quick! And yeah, I figured that was probably the problem. I was going to try to delete the repo, but couldn't through the website, so if that's the easiest way to solve it, I can regenerate the dataset in a different format with less tiny files, and you guys can delete the repo as it is. Zarr just saves everything as lots of small files to make chunks easy to load, which is why I was preferring that format, but maybne that just doesn't work well for HF datasets.
Issue: 500 internal server error when trying to open a dataset composed of Zarr stores

Comment:
Hi @jacobbieker,

For future use cases, our Hub team is still pondering whether to limit the maximum number of files per repo to avoid technical issues...

On the meantime, they have made a fix and your dataset is working: https://huggingface.co/datasets/openclimatefix/mrms
Issue: Add Biwi Kinect Head Pose Database

Comment:
Official dataset location : https://icu.ee.ethz.ch/research/datsets.html
In the "Biwi Kinect Head Pose Database" section, I do not find any information regarding "Downloading the dataset." . Do we mail the authors regarding this ?

I found the dataset on Kaggle : [Link](https://www.kaggle.com/kmader/biwi-kinect-head-pose-database) , but since 🤗 does not host any of the datasets, this would require the user to provide their Kaggle username and API key to download. 

Any inputs on how we could proceed ? Thank you.
[ Need your inputs here, @lhoestq or @mariosasko ]
Issue: `pubmed_qa` checksum mismatch

Comment:
Hi @jon-tow, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

We are planning to make a patch release today.

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Issue: Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI

Comment:
Hi, thanks for reporting! We can add a `sources: datasets.Value("string")` feature to the `Features` dict in the `SARI` script to fix this. Would you be interested in submitting a PR?
Issue: Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI

Comment:
Hi Mario,

Thanks for your message. I did try to add `sources` into the `Features` dict using a script for the metric:
```
            features=datasets.Features(
                {
                    "sources": datasets.Value("string", id="sequence"),
                    "predictions": datasets.Value("string", id="sequence"),
                    "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
                }
            ),
```

But that only avoids a failure in `encode_batch` in the `add_batch` method:
```
        batch = {"predictions": predictions, "references": references}
        batch = self.info.features.encode_batch(batch)
```

The real problem is that `add_batch()`, `add()` and `compute()` does not receive a `sources` param:
```
def add_batch(self, *, predictions=None, references=None):
def add(self, *, prediction=None, reference=None):
def compute(self, *, predictions=None, references=None, **kwargs)
```

And then, it fails:
`TypeError: add_batch() got an unexpected keyword argument sources`

I need this for adding any metric based on SARI or alike, not only for sari.py :)

Let me know if I understood correctly the proposed solution.
Issue: Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI

Comment:
The `Metric` class has been modified recently to support this use-case, but the `add_batch` + `compute` pattern still doesn't work correctly. I'll open a PR.
Issue: Add MetaShift dataset

Comment:
I would like to take this up and give it a shot. Any image specific - dataset guidelines to keep in mind ? Thank you.
Issue: Add MetaShift dataset

Comment:
I've started working on adding this dataset. I require some inputs on the following : 

Ref for the initial draft [here](https://github.com/dnaveenr/datasets/blob/add_metashift_dataset/datasets/metashift/metashift.py)
1. The dataset does not have a typical - train/test/val split. What do we do for the _split_generators() function ? How do we go about this ?
2. This dataset builds on the Visual Genome dataset, using a metadata file. The dataset is generated using generate_full_MetaShift.py script. By default, the authors choose to generate the dataset only for a SELECTED_CLASSES. The following script is used : 
Code : https://github.com/Weixin-Liang/MetaShift/blob/main/dataset/generate_full_MetaShift.py 
Info : https://metashift.readthedocs.io/en/latest/sub_pages/download_MetaShift.html#generate-the-full-metashift-dataset
Can I just copy over the required functions into the metashift.py to generate the dataset ?
3. How do we complete the _generate_examples for this dataset ?

The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Inputs, suggestions would be helpful. Thank you.
Issue: Add MetaShift dataset

Comment:
Hi ! Thanks for adding this dataset :) Let me answer your questions:

1. in this case you can put everything in the "train" split
2. Yes you can copy the script (provided you also include the MIT license of the code in the file header for example). Though we ideally try to not create new directories nor files when generating dataset, so if possible this script should be adapted to not create the file structure they mentioned, but instead yield the images one by one in `_generate_examples`. Let me know if you think this is feasible
3. see point 2 haha

> The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Yup ! We can also define a `selected_classes` parameter such that users can do
```python
load_dataset("metashift", selected_classes=["cat", "dog", ...])
```
Issue: Add MetaShift dataset

Comment:
Great. This is helpful. Thanks @lhoestq .
Regarding Point 2, I'll try using yield instead of creating the directories and see if its feasible. selected_classes config sounds good.
Issue: Checksums didn't match for datasets on Google Drive

Comment:
Hi @muelletm, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

Until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```