SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Add COCO datasets

Comment:
Hi @NielsRogge and @merveenoyan, did you find a way to load a dataset with COCO annotations to HF's hub?
I have a panoptic segmentation dataset in COCO format and would like to share it with the community.
Thanks in advance :)
Issue: Add COCO datasets

Comment:
The COCO format is not supported out of the box in the HF's hub - you'd need to reformat it to an [ImageFolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder) with metadata format, or write a [loading script](https://huggingface.co/docs/datasets/image_dataset#loading-script)
Issue: Add COCO datasets

Comment:
> The COCO format is not supported out of the box in the HF's hub - you'd need to reformat it to an [ImageFolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder) with metadata format, or write a [loading script](https://huggingface.co/docs/datasets/image_dataset#loading-script)

Hi @lhoestq , thank you for your quick reply.
I've correctly created a metadata.jsonl file for a dataset with instance segmentation annotations [here](https://huggingface.co/datasets/lombardata/data_2017)
but do not understand how I can integrate panoptic annotations with the metadata format of ImageFolder datasets. The "problem" with panoptic annotations is that we have a folder with images, a json file with annotations and another folder with png annotations.

I checked between all the datasets already published on HuggingFace and, the only one who has uploaded a correct panoptic dataset is @NielsRogge [here](https://huggingface.co/datasets/nielsr/coco-panoptic-val2017) and [here](https://huggingface.co/datasets/nielsr/ade20k-panoptic-demo). Indeed he accomplished to have three fields : 
1.image (image)
2.label (image)
3.segments_info (list)
but I not find the corresponding code that allows to upload a panoptic dataset from this 3 sources.
Can you please share an example code?
Thanks !
Issue: Add COCO datasets

Comment:
Both were uploaded using `ds.push_to_hub()` :)

You can get a Dataset from a python dictionary using `ds = Dataset.from_dict(...)` and casts the paths to images to the `Image()` type using `ds = ds.cast_column("image", Image())`.

```python
from datasets import Dataset, Image

ds = Dataset.from_dict(...)
ds = ds.cast_column("image", Image())
ds = ds.cast_column("label", Image())
ds.push_to_hub(...)
```
Issue: Add COCO datasets

Comment:
> Both were uploaded using `ds.push_to_hub()` :)
> 
> You can get a Dataset from a python dictionary using `ds = Dataset.from_dict(...)` and casts the paths to images to the `Image()` type using `ds = ds.cast_column("image", Image())`.
> 
> ```python
> from datasets import Dataset, Image
> 
> ds = Dataset.from_dict(...)
> ds = ds.cast_column("image", Image())
> ds = ds.cast_column("label", Image())
> ds.push_to_hub(...)
> ```

Thank you very much @lhoestq , I succesfully created a hf dataset [here](https://huggingface.co/datasets/lombardata/panoptic_2023_06_21) with the two fields :
1.image (image)
2.label (image)
following your suggestions. Now still remain the problem of uploading **segments_info** information to the dataset.
There is a function that easily imports the _panoptic_coco_annotation.json_ file to a segment_info field?
I think we must define a **list_of_segment**, i.e. a list of lists  of this type : 
```python
[ { "area": 214858, "bbox": [ 0, 0, 511, 760 ], "category_id": 0, "id": 7895160, "iscrowd": 0 }, { "area": 73067, "bbox": [ 98, 719, 413, 253 ], "category_id": 3, "id": 3289680, "iscrowd": 0 }, { "area": 832, "bbox": [ 53, 0, 101, 16 ], "category_id": 5, "id": 5273720, "iscrowd": 0 }, { "area": 70668, "bbox": [ 318, 60, 191, 392 ], "category_id": 8, "id": 15132390, "iscrowd": 0 }, { "area": 32696, "bbox": [ 0, 100, 78, 872 ], "category_id": 18, "id": 472063, "iscrowd": 0 }, { "area": 76045, "bbox": [ 42, 48, 264, 924 ], "category_id": 37, "id": 16713830, "iscrowd": 0 }, { "area": 27103, "bbox": [ 288, 482, 216, 306 ], "category_id": 47, "id": 16753408, "iscrowd": 0 } ]
```
and then apply again the **cast_column** function [here](https://github.com/huggingface/datasets/blob/2.13.0/src/datasets/arrow_dataset.py#L2060) but with a list as a second argument, like : 
```python
from datasets import Dataset, Image
ds = ds.cast_column("image", Image())
ds = ds.cast_column("label", Image())
ds = ds.cast_column("segments_info", list)
```
but I do not see how to transfer the information of the _panoptic_coco_annotation.json_ to a list of lists of this type : 
```python
[ { "area": 214858, "bbox": [ 0, 0, 511, 760 ], "category_id": 0, "id": 7895160, "iscrowd": 0 }, { "area": 73067, "bbox": [ 98, 719, 413, 253 ], "category_id": 3, "id": 3289680, "iscrowd": 0 }, { "area": 832, "bbox": [ 53, 0, 101, 16 ], "category_id": 5, "id": 5273720, "iscrowd": 0 }, { "area": 70668, "bbox": [ 318, 60, 191, 392 ], "category_id": 8, "id": 15132390, "iscrowd": 0 }, { "area": 32696, "bbox": [ 0, 100, 78, 872 ], "category_id": 18, "id": 472063, "iscrowd": 0 }, { "area": 76045, "bbox": [ 42, 48, 264, 924 ], "category_id": 37, "id": 16713830, "iscrowd": 0 }, { "area": 27103, "bbox": [ 288, 482, 216, 306 ], "category_id": 47, "id": 16753408, "iscrowd": 0 } ]
```
like @NielsRogge has done  [here](https://huggingface.co/datasets/nielsr/coco-panoptic-val2017) and [here](https://huggingface.co/datasets/nielsr/ade20k-panoptic-demo).
Thank you again for your help and have a good day !
Issue: Add COCO datasets

Comment:
You can pass this data in .from_dict() - no need to cast anything for this column

```python
ds = Dataset.from_dict({
    "image": [...],
    "label": [...],
    "segments_info": [...],
)}
```

where `segments_info` is the list of the segment_infos of all the examples in the dataset, and therefore is a list of lists of dicts.
Issue: Add COCO datasets

Comment:
> You can pass this data in .from_dict() - no need to cast anything for this column
> 
> ```python
> ds = Dataset.from_dict({
>     "image": [...],
>     "label": [...],
>     "segments_info": [...],
> )}
> ```
> 
> where `segments_info` is the list of the segment_infos of all the examples in the dataset, and therefore is a list of lists of dicts.

Thank you for the quick reply @lhoestq , but then how to generate the `segments_info` list of lists of dicts starting from a _panoptic_coco_annotation.json_ file ?
Issue: Add COCO datasets

Comment:
You read the JSON file and transform the data yourself. I don't think there's an automatic converter somewhere
Issue: Add COCO datasets

Comment:
> You read the JSON file and transform the data yourself. I don't think there's an automatic converter somewhere

Perfect, I've done it and succesfully uploaded a new dataset [here](https://huggingface.co/datasets/lombardata/panoptic_2023_06_22), but I've (I hope) a last problem.
The dataset has currently 302 images and, when I upload it to the hub, only the first page of images is correctly uploaded.
When I try to see the second/third/fourth page of items of my dataset, I can see that the fields **segments_info** and **image_name** are correctly uploaded, while the images are not (the "null" string is shown everywhere).

I've checked the path of images that are not uploaded and they exists, is there a problem with the size of the dataset ?
How can I upload the whole dataset to the hub ?
Thank you again @lhoestq and have a good day !
Issue: Add COCO datasets

Comment:
Awesome ! Your dataset looks all good 🤗 

The `null` in the viewer is a bug on our side, let me investigate
Issue: Documentation Mistakes in Dataset: emotion

Comment:
Hi,

this issue has been already reported in the dataset repo (https://github.com/dair-ai/emotion_dataset/issues/2), so this is a bug on their side.
Issue: Documentation Mistakes in Dataset: emotion

Comment:
The documentation has another bug in the dataset card [here](https://huggingface.co/datasets/emotion). 

In the dataset summary **six** emotions are mentioned: *"six basic emotions: anger, fear, joy, love, sadness, and surprise"*, however, in the datafields section we have only **five**:
```
label: a classification label, with possible values including sadness (0), joy (1), love (2), anger (3), fear (4).
```
Issue: Documentation Mistakes in Dataset: emotion

Comment:
@GDGauravDutta the dataset author replied in their issue: https://github.com/dair-ai/emotion_dataset/issues/2
> The dataset released is a preprocessed six emotion variant released for educational and research purposes.

@albertovilla the dataset card was fixed with 6 emotions.
Issue: Datasets with tricky task templates

Comment:
The `task_templates` API is deprecated in favor of the `train-eval-index` YAML field, so I'm closing this issue.
Issue: datasets.map pickle issue resulting in invalid mapping function

Comment:
Hi ! `map` calls `__getstate__` using `dill` to hash your map function. This is used by the caching mechanism to recover previously computed results. That's why you don't see any `__setstate__` call.

Why do you change an attribute of your tokenizer when `__getstate__` is called ?
Issue: datasets.map pickle issue resulting in invalid mapping function

Comment:
@lhoestq because if I try to pickle my custom tokenizer (it contains a pure python pretokenization step in an otherwise rust backed tokenizer) I get

> Exception: Error while attempting to pickle Tokenizer: Custom PreTokenizer cannot be serialized

So I remove the Custom PreTokenizer in `__getstate__` and then restore it in `__setstate__` (since it doesn't contain any state). This is what my `__getstate__`  / `__setstate__` looks like:

    def __getstate__(self):
        """
        Removes pre_tokenizer since it cannot be pickled
        """
        logger.debug("Copy state dict")
        out = self.__dict__.copy()
        logger.debug("Detaching pre_tokenizer")
        out['_tokenizer'].pre_tokenizer = tokenizers.pre_tokenizers.Sequence([]) 
        return out

    def __setstate__(self, d):
        """
        Reinstates pre_tokenizer
        """
        logger.debug("Reattaching pre_tokenizer")
        self.__dict__ = d
        self.backend_tokenizer.pre_tokenizer = self._pre_tokenizer()

If this is the case can you think of another way of avoiding my issue?
Issue: datasets.map pickle issue resulting in invalid mapping function

Comment:
Actually, maybe I need to deep copy `self.__dict__`? That way `self` isn't modified. That was my intention and I thought it was working - I'll double-check after the weekend.
Issue: datasets.map pickle issue resulting in invalid mapping function

Comment:
Doing a deep copy results in the warning:

> 06/20/2021 16:02:15 - WARNING - datasets.fingerprint -   Parameter 'function'=<function tokenize_function at 0x7f1e95f05d40> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.


```
def __getstate__(self):
    """
    Removes pre_tokenizer since it cannot be pickled
    """
    logger.debug("Copy state dict")
    out = copy.deepcopy(self.__dict__)
    logger.debug("Detaching pre_tokenizer")
    out['_tokenizer'].pre_tokenizer = tokenizers.pre_tokenizers.Sequence([]) 
    return out
```
Issue: datasets.map pickle issue resulting in invalid mapping function

Comment:
Looks like there is still an object that is not pickable in your `tokenize_function` function.

You can test if an object can be pickled and hashed by using 
```python
from datasets.fingerprint import Hasher

Hasher.hash(my_object)
```

Under the hood it pickles the object to compute its hash, so it calls `__getstate__` when applicable.
Issue: datasets.map pickle issue resulting in invalid mapping function

Comment:
I figured it out, the problem is deep copy itself uses pickle (unless you implement `__deepcopy__`). So when I changed `__getstate__` it started throwing an error.

I'm sure there's a better way of doing this, but in order to return the `__dict__` without the non-pikelable pre-tokeniser and without modifying self I removed the pre-tokenizers, did a deep copy and then re-generated it.

It does work -  although I noticed Hasher doesn't call `__hash__` if the object being hashed implements it which I feel it should? If it did I could return a hash of the tokenizers.json file instead.

```
    def __getstate__(self):
        """
        Removes pre_tokenizer since it cannot be pickled
        """
        logger.debug("Copy state dict")
        self.backend_tokenizer.pre_tokenizer = tokenizers.pre_tokenizers.Sequence([])
        out = copy.deepcopy(self.__dict__)  #self.__dict__.copy()
        self.backend_tokenizer.pre_tokenizer = self._pre_tokenizer()

        return out
```
Issue: datasets.map pickle issue resulting in invalid mapping function

Comment:
I'm glad you figured something out :)

Regarding hashing: we're not using hashing for the same purpose as the python `__hash__` purpose (which is in general for dictionary lookups). For example it is allowed for python hashing to not return the same hash across sessions, while our hashing must return the same hashes across sessions for the caching to work properly.
Issue: Can datasets remove duplicated rows?

Comment:
Hi ! For now this is probably the best option.
We might add a feature like this in the feature as well.

Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases
Issue: Can datasets remove duplicated rows?

Comment:
Yes, I'd like to work on this feature once I'm done with #2500, but first I have to do some research, and see if the implementation wouldn't be too complex.

In the meantime, maybe [this lib](https://github.com/TomScheffers/pyarrow_ops) can help. However, note that this lib operates directly on pyarrow tables and relies only on `hash` to find duplicates (e.g. `-1` and `-2` have the same hash in Python 3, so this lib will treat them as duplicates), which doesn't make much sense.
Issue: Can datasets remove duplicated rows?

Comment:
> Hi ! For now this is probably the best option.
> We might add a feature like this in the feature as well.
> 
> Do you know any deduplication method that works on arbitrary big datasets without filling up RAM ?
> Otherwise we can have do the deduplication in memory like pandas but I feel like this is going to be limiting for some cases

Great if this is can be done. Thanks!!

Not sure if you are asking me. In any case I don't know of any unfortunately :( in practice if data is really large we normally do it with spark (only for info. I understand this is not useful in developing this library..)
Issue: Can datasets remove duplicated rows?

Comment:
Hello,

I'm also interested in this feature.
Has there been progress on this issue?

Could we use a similar trick as above, but with a better hashing algorithm like SHA?

We could also use a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter), should we care a lot about collision in this case?
Issue: Can datasets remove duplicated rows?

Comment:
For reference, we can get a solution fairly easily if we assume that we can hold in memory all unique values. 

```python
from datasets import Dataset
from itertools import cycle
from functools import partial

memory = set()
def is_unique(elem:Any , column: str, memory: set) -> bool:
    if elem[column] in memory:
        return False
    else:
        memory.add(elem[column])
        return True

# Example dataset
ds = Dataset.from_dict({"col1" : [sent for i, sent in zip(range(10), cycle(["apple", "orange", "pear"]))],
                                      "col2": [i % 5 for i in range(10)]})

# Drop duplicates in `ds` on "col1"
ds2 = ds.filter(partial(is_unique, column="col1", memory=memory))
```

Of course, we can improve the API so that we can introduce `Dataset.drop_duplicates`.
For the parallel version, we can use a shared memory set.
Issue: Can datasets remove duplicated rows?

Comment:
An approach that works assuming you can hold the all the unique document hashes in memory:
```python
from datasets import load_dataset

def get_hash(example):
    """Get hash of content field."""
    return {"hash": hash(example["content"])} # can use any hashing function here
    
def check_uniques(example, uniques):
    """Check if current hash is still in set of unique hashes and remove if true."""
    if example["hash"] in uniques:
        uniques.remove(example["hash"])
        return True
    else:
        return False

ds = load_dataset("some_dataset")
ds = ds.map(get_hash)
uniques = set(ds.unique("hash"))
ds_filter = ds.filter(check_uniques, fn_kwargs={"uniques": uniques})
```
If the `uniques` could be stored in arrow then no additional memory would used at all but I don't know if this is possible.
Issue: Can datasets remove duplicated rows?

Comment:
@lvwerra hey, could you tell me how reliable is this deduplication method. i am currently using the same deduplication strategy to deduplicate a large text corpus to pretrain LLMs ~ 11B to 20B. just needed to ensure if this strategy would be fine on large datasets for LLMs pretraining.
Issue: Add C4

Comment:
Update on this: I'm computing the checksums of the data files. It will be available soon
Issue: Load Image Classification Dataset from Local 

Comment:
Hi ! Is this folder structure a standard, a bit like imagenet ?
In this case maybe we can consider having a dataset loader for cifar-like, imagenet-like, squad-like, conll-like etc. datasets ?
```python
from datasets import load_dataset

my_custom_cifar = load_dataset("cifar_like", data_dir="path/to/data/dir")
```

Let me know what you think
Issue: Load Image Classification Dataset from Local 

Comment:
@lhoestq I think we'll want a generic `image-folder` dataset (same as 'imagenet-like'). This is like `torchvision.datasets.ImageFolder`, and is something vision folks are used to seeing.
Issue: Load Image Classification Dataset from Local 

Comment:
Opening this back up, since I'm planning on tackling this. Already posted a quick version of it on my account on the hub.

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
```
Issue: SubjQA wrong boolean values in entries

Comment:
@arnaudstiegler I have just checked that these mismatches are already present in the original dataset: https://github.com/megagonlabs/SubjQA

We are going to contact the dataset owners to report this.
Issue: SubjQA wrong boolean values in entries

Comment:
I have:
- opened an issue in their repo: https://github.com/megagonlabs/SubjQA/issues/3
- written an email to all the paper authors
Issue:  Python Programming Puzzles

Comment:
Thanks @VictorSanh!
There's also a [notebook](https://aka.ms/python_puzzles) and [demo](https://aka.ms/python_puzzles_study) available now to try out some of the puzzles
Issue: Improve torch formatting performance

Comment:
That’s interesting thanks, let’s see what we can do. Can you detail your last sentence? I’m not sure I understand it well.
Issue: Improve torch formatting performance

Comment:
Hi ! I just re-ran a quick benchmark and using `to_numpy()` seems to be faster now:

```python
import pyarrow as pa  # I used pyarrow 3.0.0
import numpy as np

n, max_length = 1_000, 512
low, high, size = 0, 2 << 16, (n, max_length)

table = pa.Table.from_pydict({
    "input_ids": np.random.default_rng(42).integers(low=low, high=high, size=size).tolist()
})


%%timeit
_ = table.to_pandas()["input_ids"].to_numpy()
# 1.44 ms ± 80.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_pandas().to_numpy()
# 461 µs ± 14.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_numpy()
# 317 µs ± 5.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Currently the conversion from arrow to numpy is done in the NumpyArrowExtractor here:

https://github.com/huggingface/datasets/blob/d6d0ede9486ffad7944642ca9a326e058b676788/src/datasets/formatting/formatting.py#L143-L166

Let's update the NumpyArrowExtractor to call `to_numpy` directly and see how our github benchmarks evolve ?__
Issue: Improve torch formatting performance

Comment:
Sounds like a plan @lhoestq  If you create a PR I'll pick it up and try it out right away!
Issue: Improve torch formatting performance

Comment:
I’m not exactly sure how to read the graph but it seems that to_categorical take a lot of time here. Could you share more informations on the features/stats of your datasets so we could maybe design a synthetic datasets that looks more similar for debugging testing?
Issue: Improve torch formatting performance

Comment:
> I’m not exactly sure how to read the graph but it seems that to_categorical take a lot of time here. Could you share more informations on the features/stats of your datasets so we could maybe design a synthetic datasets that looks more similar for debugging testing?

@thomwolf starting from the top, each rectangle represents the cumulative amount of it takes to execute the method call. Therefore, format_batch in torch_formatter.py takes ~20 sec, and the largest portion of that call is taken by to_pandas call and the smaller portion (grey rectangle) by the other method invocation(s) in format_batch (series_to_numpy etc). 

Features of the dataset are BERT pre-training model input columns i.e:
```
f = Features({            
 "input_ids": Sequence(feature=Value(dtype="int32")),           
 "attention_mask": Sequence(feature=Value(dtype="int8")),            
 "token_type_ids": Sequence(feature=Value(dtype="int8")),           
 "labels": Sequence(feature=Value(dtype="int32")),           
 "next_sentence_label": Value(dtype="int8")
})
```

I'll work with @lhoestq till we get to the bottom of this one.
Issue: Improve torch formatting performance

Comment:
@lhoestq the proposed branch is faster, but overall training speedup is a few percentage points. I couldn't figure out how to include the GitHub branch into setup.py, so I couldn't start NVidia optimized Docker-based pre-training run.  But on bare metal, there is a slight improvement. I'll do some more performance traces.
Issue: Improve torch formatting performance

Comment:
Hi @vblagoje, to install Datasets from @lhoestq PR reference #2505, you can use:
```shell
pip install git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head#egg=datasets
```
Issue: Improve torch formatting performance

Comment:
Hey @albertvillanova yes thank you, I am aware, I can easily pull it from a terminal command line but then I can't automate docker image builds as dependencies are picked up from setup.py and for some reason setup.py doesn't accept this string format.
Issue: Improve torch formatting performance

Comment:
@vblagoje in that case, you can add this to your `setup.py`:
```python
    install_requires=[
        "datasets @ git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head",
```