Multi-image loading in Imagefolder dataset
Opened by vvvm23 (open)

### Feature request

Extend the `imagefolder` dataloading script to support loading multiple images per dataset entry.

This only really makes sense if a metadata file is present.

Currently you can use the following format (example `metadata.jsonl`:
```
{'file_name': 'path_to_image.png', 'metadata': ...}
...
```
which will return a batch with key `image` and any other metadata.

I would propose extending `file_name` to also accept a list of files, which would return a batch with key `images` and any other metadata.

### Motivation

This is useful for example in segmentation tasks in computer vision models, or in text-to-image models that also accept conditioning signals such as another image, feature map, or similar. Currently if I want to do this, I would need to write a custom dataset, rather than just use `imagefolder`.

### Your contribution

Would be open to doing a PR, but also happy for someone else to take it as I am not familiar with the datasets library.

Comments:
Supporting this could be useful (I remember a use-case for this on the Hub). Do you agree @polinaeterna? 

Implementing this should be possible if we iterate over metadata files and build image/audio file paths instead of iterating over image/audio files and looking for the corresponding entries in metadata files.

I've build a similar feature from scratch and would be interested to combine it with the datasets package.

My solution works something like this:
Interpret the first element of each column as a file path. If the path exists and is a file, (try to) load the files for the entire column. Thereby, one isn't restricted to a particular column name, with comes in handy when dealing with multiple file columns.

I've looked into the code to try to implement this, but didn't find the right places. I'm also open to contribute, but will need some guidance.

Required here: https://discuss.huggingface.co/t/dataset-repo-requires-arbitrary-python-code-execution/59346/14

+1

Is the only way to do this right now to write a custom dataset loader script?

Also: be able to have input and output images for each row. Asked here: https://discuss.huggingface.co/t/how-to-structure-image-files-for-datasets-load-dataset-imagefolder-when-you-have-input-and-output-images-like-in-instruct-pix2pix/82467

👀 I encountered the same problem. Is the only way to solve it by writing a custom dataset loader script?

Yes I had to use script at the end

Sent from Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Yunzhuo Hao ***@***.***>
Sent: Sunday, December 1, 2024 11:52:21 AM
To: huggingface/datasets ***@***.***>
Cc: Sushant Gautam ***@***.***>; Manual ***@***.***>
Subject: Re: [huggingface/datasets] Multi-image loading in Imagefolder dataset (Issue #5760)


👀 I encountered the same problem. Is the only way to solve it by writing a custom dataset loader script?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5760#issuecomment-2509684773>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AD7SQP5KY67NTZ7ELAGSIJ32DLS6LAVCNFSM6AAAAABSZRPAQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGY4DINZXGM>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>


@SushantGautam Thanks for your reply!! In addition, can the dataset shared by writing such a script be displayed in the Viewer? And if it is convenient, can you please show me your script? Thanks again!!

Yes will be visible!
The script docs is at
https://huggingface.co/docs/datasets/en/dataset_script



Sincerely,
*Sushant Gautam*

📧: ***@***.*** | ***@***.***
🌐: sushant.info.np | simula.no/people/sushant
<https://www.simula.no/people/sushant>
[image: Simula Metropolitan Center for Digital Engineering]
<https://www.simulamet.no/>
Stensberggata 27, 0170 Oslo
Find me on:* LinkedIn <https://www.linkedin.com/in/eSushant> | Facebook
<https://www.facebook.com/eSushant> | Twitter
<https://twitter.com/esushant> *


On Sun, 1 Dec 2024 at 12:02, Yunzhuo Hao ***@***.***> wrote:

> @SushantGautam <https://github.com/SushantGautam> Thanks for your reply!!
> In addition, can the dataset shared by writing such a script be displayed
> in the Viewer? And if it is convenient, can you please show me your script?
> Thanks again!!
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5760#issuecomment-2509692544>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AD7SQP2XGIKBCSSSNW2S6K32DLUFDAVCNFSM6AAAAABSZRPAQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGY4TENJUGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


I see. Thank you very very much!!!

Sushant Gautam ***@***.***>于2024年12月1日 周日18:54写道：

> Yes I had to use script at the end
>
> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Yunzhuo Hao ***@***.***>
> Sent: Sunday, December 1, 2024 11:52:21 AM
> To: huggingface/datasets ***@***.***>
> Cc: Sushant Gautam ***@***.***>; Manual ***@***.***>
> Subject: Re: [huggingface/datasets] Multi-image loading in Imagefolder
> dataset (Issue #5760)
>
>
> 👀 I encountered the same problem. Is the only way to solve it by writing
> a custom dataset loader script?
>
> —
> Reply to this email directly, view it on GitHub<
> https://github.com/huggingface/datasets/issues/5760#issuecomment-2509684773>,
> or unsubscribe<
> https://github.com/notifications/unsubscribe-auth/AD7SQP5KY67NTZ7ELAGSIJ32DLS6LAVCNFSM6AAAAABSZRPAQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGY4DINZXGM>.
>
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5760#issuecomment-2509685427>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AVYJNSJ6UQZ6G6VGSB5RGXD2DLTGFAVCNFSM6AAAAABSZRPAQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGY4DKNBSG4>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>