SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: sqaud_v2 dataset contains misalignment between the answer text and the context value at the answer index

Comment:
If you are going to be altering the data cleaning from the source Squad dataset, here is one thing to consider.
There are occasional double spaces separating words which it might be nice to get rid of. 

Either way, thank you.
Issue: Error iteration over IterableDataset using Torch DataLoader

Comment:
Hi ! This is because you first need to format the dataset for pytorch:

```python
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
>>> torch_iterable_dataset = dataset.with_format("torch")
>>> assert isinstance(torch_iterable_dataset, torch.utils.data.IterableDataset)
>>> dataloader = torch.utils.data.DataLoader(torch_iterable_dataset, batch_size=4)
>>> next(iter(dataloader))
{'id': tensor([0, 1, 2, 3]), 'text': ['Mtendere Village was inspired...]}
```

This is because the pytorch dataloader expects a subclass of `torch.utils.data.IterableDataset`. Since you can't pass an arbitrary iterable to a pytorch dataloader, you first need to build an object that inherits from `torch.utils.data.IterableDataset` using `with_format("torch")` for example.
Issue: Error iteration over IterableDataset using Torch DataLoader

Comment:
Thank you for that and the example! 

What you said makes total sense; I just somehow missed that and assumed HF IterableDataset was a subclass of Torch IterableDataset.
Issue: Finding right block-size with JSON loading difficult for user

Comment:
This was actually a second error arising from a too small block-size in the json reader.

Finding the right block size is difficult for the layman user
Issue: Support Zstandard compressed files

Comment:
I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

```
!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)

WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported
```
Issue: Support Zstandard compressed files

Comment:
I just tried on google colab and this works:
```python
!pip install zstandard
!pip install datasets
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(lds))
```

Can you check that you have a correct installation of `zstandard` ?
Issue: Support Zstandard compressed files

Comment:
> I just tried on google colab and this works:
> 
> ```python
> !pip install zstandard
> !pip install datasets
> from datasets import load_dataset
> 
> lds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(lds))
> ```
> 
> Can you check that you have a correct installation of `zstandard` ?

I was downloading datasets first then was doing zstandard installation and that was causing the issue. This was highlighted by the Hugging Face staff and that helped. Now the issue is resolved. Thank you.
Issue: Weights of model checkpoint not initialized for RobertaModel for Bertscore

Comment:
Hi @suzyahyah, thanks for reporting.

The message you get is indeed not an error message, but a warning coming from Hugging Face `transformers`. The complete warning message is:
```
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

In this case, this behavior IS expected and you can safely ignore the warning message.

The reason is that you are just using RoBERTa to get the contextual embeddings of the input sentences/tokens, thus leaving away its head layer, whose weights are ignored.

Feel free to reopen this issue if you need further explanations.
Issue: Weights of model checkpoint not initialized for RobertaModel for Bertscore

Comment:
Hi @suzyahyah, I have created a Pull Request to filter out that warning message in this specific case, since the behavior is as expected and the warning message can only cause confusion for users (as in your case).
Issue: concatenate_datasets for iterable datasets

Comment:
It is probably worth noting here that the [documentation](https://huggingface.co/docs/datasets/process#concatenate) is misleading (indicating that it does work for IterableDatasets):

> You can also mix several datasets together by taking alternating examples from each one to create a new dataset. This is known as interleaving, and you can use it with [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets). **Both [interleave_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.interleave_datasets) and [concatenate_datasets()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.concatenate_datasets) will work with regular [Dataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset) and [IterableDataset](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.IterableDataset) objects**. Refer to the [Stream](https://huggingface.co/docs/datasets/stream#interleave) section for an example of how it’s used.
Issue: Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`

Comment:
Hi ! I just tried to reproduce what you said:
- create a local builder class
- use `load_dataset`
- update the builder class code
- use `load_dataset` again (with or without `ignore_verifications=True`)
And it creates a new cache, as expected.

What modifications did you do to your builder's code ?
Issue: Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`

Comment:
Hi @lhoestq. Thanks for your reply. I just did minor modifications for which it should not regenerate cache (for e.g. Adding a print statement). Overall, regardless of cache miss, there should be an explicit option to allow reuse of existing cache if author knows cache shouldn't be affected.
Issue: Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`

Comment:
The cache is based on the hash of the dataset builder's code, so changing the code makes it recompute the cache.

You could still rename the cache directory of your previous computation to the new expected cache directory if you want to avoid having to recompute it and if you're sure that it would generate the exact same result.

The verifications are data integrity verifications: it checks the checksums of the downloaded files, as well as the size of the generated splits.
Issue: Existing cache for local dataset builder file updates is ignored with `ignore_verifications=True`

Comment:
Hi @apsdehal,

If you decide to follow @lhoestq's suggestion to rename the cache directory of your previous computation to the new expected cache directory, you can do the following to get the name of the new expected cache directory once #2500 is merged:
```python
from datasets import load_dataset_builder
dataset_builder = load_dataset_builder("path/to/your/dataset")
print(dataset_builder.cache_dir)
```

This way, you don't have to recompute the hash of the dataset script yourself each time you modify the script.
Issue: Memory usage consistently increases when processing a dataset with `.map`

Comment:
Hi ! Can you share the function you pass to `map` ?
I know you mentioned it would be hard to share some code but this would really help to understand what happened
Issue: Memory usage consistently increases when processing a dataset with `.map`

Comment:
This is the same behavior as in #4883, so I'm closing this issue as a duplicate.
Issue: Better DuplicateKeysError error to help the user debug the issue

Comment:
excuse me, my `datasets` version is `2.2.2`, but I also just see the error info like 
```
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```
Issue: Better DuplicateKeysError error to help the user debug the issue

Comment:
Hi ! for which dataset do you have this error ?

Also note that this issue is just about improving the error message, which is not very friendly x)
Issue: Better DuplicateKeysError error to help the user debug the issue

Comment:
@lhoestq  I would like to take a hit at improving the error message. Will open a draft PR and will reach out to you for review
Issue: Better DuplicateKeysError error to help the user debug the issue

Comment:
> DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.

@lhoestq  when you mention 42th and 1337th in the above case , are these values the examples' "id" or are they the examples' index ?
Issue: Better DuplicateKeysError error to help the user debug the issue

Comment:
Hi ! Thanks @VijayKalmath :)

In the general case, examples don't have an "id" field, so I think it should correspond to the index
Issue: Better DuplicateKeysError error to help the user debug the issue

Comment:
@lhoestq  , I have opened a draft PR for this Issue. 

I wanted to check with you if there is a way to get `<path/to/the/dataset/script>` currently or do I need to add extra code to find that. 

If I need to find the script , I can assume that the generator function will always be in `datasets/{dataset_name}/{dataset_name}.py`.
Issue: Better DuplicateKeysError error to help the user debug the issue

Comment:
Thanks !

> I wanted to check with you if there is a way to get <path/to/the/dataset/script> currently or do I need to add extra code to find that.

You don't have access to this info inside the ArrowWriter unfortunately. This info is available in builder.py in the DatasetBuilder code that uses the ArrowWriter though, maybe a try-catch there can do the job
Issue: Multilabel metrics not supported

Comment:
Hi @GuillemGSubies, thanks for reporting.

I have made a PR to fix this issue and allow metrics to be computed also for multilabel classification problems.
Issue: Multilabel metrics not supported

Comment:
Sorry for reopening but I just noticed that the `_compute` method for the F1 metric is still not good enough for multilabel problems:

https://github.com/huggingface/datasets/blob/92a3ee549705aa0a107c9fa5caf463b3b3da2616/metrics/f1/f1.py#L115

Somehow we should be able to change the parameter `average` at least
Issue: load_dataset("web_nlg") NonMatchingChecksumError

Comment:
Hi ! Thanks for reporting. This is due to the WebNLG repository that got updated today.
I just pushed a fix at #2558 - this shouldn't happen anymore in the future.
Issue: Keys should be unique error on code_search_net

Comment:
Two questions:
- with `datasets-cli env` we don't have any information on the dataset script version used. Should we give access to this somehow? Either as a note in the Error message or as an argument with the name of the dataset to `datasets-cli env`?
- I don't really understand why the id is duplicated in the code of `code_search_net`, how can I debug this actually?
Issue: Keys should be unique error on code_search_net

Comment:
Thanks for reporting. There was indeed an issue with the keys. The key was the addition of the file id and row id, which resulted in collisions. I just opened a PR to fix this at https://github.com/huggingface/datasets/pull/2555

To help users debug this kind of errors we could try to show a message like this
```python
DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>
```

This way users who what to look for if they want to debug this issue. I opened an issue to track this: https://github.com/huggingface/datasets/issues/2556
Issue: Keys should be unique error on code_search_net

Comment:
and are we sure there are not a lot of datasets which are now broken with this change?
Issue: Keys should be unique error on code_search_net

Comment:
Thanks to the dummy data, we know for sure that most of them work as expected.
`code_search_net` wasn't caught because the dummy data only have one dummy data file while the dataset script can actually load several of them using `os.listdir`. Let me take a look at all the other datasets that use `os.listdir` to see if the keys are alright
Issue: Keys should be unique error on code_search_net

Comment:
I found one issue on `fever` (PR here: https://github.com/huggingface/datasets/pull/2557)
All the other ones seem fine :)
Issue: Keys should be unique error on code_search_net

Comment:
Hi! Got same error when loading other dataset:
```python3
load_dataset('wikicorpus', 'raw_en')
```

tb:
```pytb
---------------------------------------------------------------------------
DuplicatedKeysError                       Traceback (most recent call last)
/opt/conda/lib/python3.8/site-packages/datasets/builder.py in _prepare_split(self, split_generator)
   1109                     example = self.info.features.encode_example(record)
-> 1110                     writer.write(example, key)
   1111             finally:

/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py in write(self, example, key, writer_batch_size)
    341             if self._check_duplicates:
--> 342                 self.check_duplicate_keys()
    343                 # Re-intializing to empty list for next batch

/opt/conda/lib/python3.8/site-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
    352             if hash in tmp_record:
--> 353                 raise DuplicatedKeysError(key)
    354             else:

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 519
Keys should be unique and deterministic in nature
```

Version: datasets==1.11.0
Issue: Keys should be unique error on code_search_net

Comment:
The wikicorpus issue has been fixed by https://github.com/huggingface/datasets/pull/2844

We'll do a new release of `datasets` soon :)
Issue: Handling unlabeled datasets

Comment:
Hi @nelson-liu,

You can pass the parameter `features` to `load_dataset`: https://huggingface.co/docs/datasets/_modules/datasets/load.html#load_dataset

If you look at the code of the MNLI script you referred in your question (https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py#L62-L77), you can see how the Features were originally specified. 

Feel free to use it as a template, customize it and pass it to `load_dataset` using the parameter `features`.
Issue: Field order issue in loading json

Comment:
Hi @luyug, thanks for reporting.

The good news is that we fixed this issue only 9 days ago: #2507.

The patch is already in the master branch of our repository and it will be included in our next `datasets` release version 1.9.0.

Feel free to reopen the issue if the problem persists.
Issue: Dataset load_from_disk is too slow

Comment:
Hi ! It looks like an issue with the virtual disk you are using.

We load datasets using memory mapping. In general it makes it possible to load very big files instantaneously since it doesn't have to read the file (it just assigns virtual memory to the file on disk).
However there happens to be issues with virtual disks (for example on spot instances), for which memory mapping does a pass over the entire file, and this takes a while. We are discussing about this issue here: #2252 

Memory mapping is something handled by the OS so we can't do much about it, though we're still trying to figure out what's causing this behavior exactly to see what we can do.
Issue: Dataset load_from_disk is too slow

Comment:
Okay, that's exactly my case, with spot instances... Therefore this isn't something we can change in any way to be able to load the dataset faster? I mean, what do you do internally at huggingface for being able to use spot instances with datasets efficiently?
Issue: Dataset load_from_disk is too slow

Comment:
There are no solutions yet unfortunately.
We're still trying to figure out a way to make the loading instantaneous on such disks, I'll keep you posted
Issue: switching some low-level log.info's to log.debug?

Comment:
Hi @stas00, thanks for pointing out this issue with logging.

I agree that `datasets` can sometimes be too verbose... I can create a PR and we could discuss there the choice of the log levels for different parts of the code.
Issue: `datasets.keyhash.DuplicatedKeysError` for `drop` and `adversarial_qa/adversarialQA`

Comment:
Hi @VictorSanh, thank you for reporting this issue with duplicated keys.

- The issue with "adversarial_qa" was fixed 23 days ago: #2433. Current version of `datasets` (1.8.0) includes the patch.
- I am investigating the issue with `drop`. I'll ping you to keep you informed.
Issue: `datasets.keyhash.DuplicatedKeysError` for `drop` and `adversarial_qa/adversarialQA`

Comment:
Hi @VictorSanh, the issue is already fixed and merged into master branch and will be included in our next release version 1.9.0.
Issue: Loading partial dataset when debugging

Comment:
Hi ! `load_dataset` downloads the full dataset once and caches it, so that subsequent calls to `load_dataset` just reloads the dataset from your disk.
Then when you specify a `split` in `load_dataset`, it will just load the requested split from the disk. If your specified split is a sliced split (e.g. `"train[:10]"`), then it will load the 10 first rows of the train split that you have on disk.

Therefore, as long as you don't delete your cache, all your calls to `load_dataset` will be very fast. Except the first call that downloads the dataset of course ^^
Issue: Loading partial dataset when debugging

Comment:
Hi @reachtarunhere.

Besides the above insights provided by @lhoestq and @thomwolf, there is also a Dataset feature in progress (I plan to finish it this week): #2249, which will allow you, when calling `load_dataset`, to pass the option to download/preprocess/cache only some specific split(s), which will definitely speed up your workflow.

If this feature is interesting for you, I can ping you once it will be merged into the master branch.
Issue: Loading partial dataset when debugging

Comment:
Thanks all for responding.

Hey @albertvillanova 

Thanks. Yes, I would be interested.

@lhoestq I think even if a small split is specified it loads up the full dataset from the disk (please correct me if this is not the case). Because it does seem to be slow to me even on subsequent calls. There is no repeated downloading so it seems that the cache is working.

I am not aware of the streaming feature @thomwolf mentioned. So I might need to read up on it.
Issue: Loading partial dataset when debugging

Comment:
If I want to create a dataset, containing only the 10 elements of a given dataset (slice it), how do I do that?
Issue: Loading partial dataset when debugging

Comment:


> ```python
> small_ds = ds.select(range(10))
> ```

Thanks, but this doesn't help me to save time during initial loading, right?
Issue: Loading partial dataset when debugging

Comment:
Indeed by default load_dataset would download and prepare everything as Arrow files. And passing `split=train[:10]` memory maps only the beginning of the full dataset that has been prepared on disk.

If you don't want to download everything, you can use streaming : 
```python 
ids = load_dataset(..., streaming=True)
first_samples = list(ids["train"].take(10))
```

To get a Dataset you can use 
```python 
ds = Dataset.from_generator(ids.take(10).__iter__)
```

edit: fixed small bug
Issue: Loading partial dataset when debugging

Comment:
Thanks @lhoestq, but I don't think it is 100% accurate, as it doesn't keep the dataset structure exactly the same.
To load the full dataset, I do:
```
data = load_dataset("json", data_files="a.json")
train_data = data["train"].shuffle()
```

But when I am changing it as per your instructions: 
```
ids = load_dataset("json", data_files="a.json", streaming=True)
data = Dataset.from_generator(ids["train"].take(1).__iter__)
train_data = data["train"].shuffle()
```
It throws KeyError.
I need a simple way, like you suggested, to have a subset of a Dataset, which exactly the same attributes.
Issue: Loading partial dataset when debugging

Comment:
Whoops I fixed my code sorry
```diff
- ds = Dataset.from_generator(ids["train"].take(10).__iter__)
+ ds = Dataset.from_generator(ids.take(10).__iter__)
```

in your case that means running
```python
train_data = data.shuffle()
```

without `["train"]`
Issue: Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task

Comment:
Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**?
Issue: Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task

Comment:
> Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**?

Oh, I am sorry
I would reopen the post on huggingface/transformers
Issue: Add COCO datasets

Comment:
I'm currently adding it, the entire dataset is quite big around 30 GB so I add splits separately. You can take a look here https://huggingface.co/datasets/merve/coco
Issue: Add COCO datasets

Comment:
I talked to @lhoestq and it's best if I download this dataset through TensorFlow datasets instead, so I'll be implementing that one really soon.
@NielsRogge
Issue: Add COCO datasets

Comment:
I started adding COCO, will be done tomorrow EOD
my work so far https://github.com/merveenoyan/datasets (my fork)
Issue: Add COCO datasets

Comment:
Hi Merve @merveenoyan , thank you so much for your great contribution! May I ask about the current progress of your implementation? Cuz I see the pull request is still in progess here. Or can I just run the COCO scripts in your fork repo?
Issue: Add COCO datasets

Comment:
Hello @yixuanren I had another prioritized project about to be merged, but I'll start continuing today will finish up soon.
Issue: Add COCO datasets

Comment:
> Hello @yixuanren I had another prioritized project about to be merged, but I'll start continuing today will finish up soon.

It's really nice of you!! I see you've commited another version just now