Add note about loading image / audio files to docs
State: closed | Author: lewtun (MEMBER)

This PR adds a small note about how to load image / audio datasets that have multiple splits in their dataset structure.

Related forum thread: https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447

cc @NielsRogge 

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4925). All of your documentation changes will be reflected on that endpoint.

Thanks for the feedback @polinaeterna ! I've reworded the docs a bit to integrate your comments and this should be ready for another review :)

> I've just realized that there is another PR about audio documentation open: #4872
> and there the more detailed description on how to use `audiofolder` is moved to another section ("Create an audio dataset")

Ah yes, let's add a comment to #4872 - that will be simpler than the alternatives :)

@polinaeterna @lhoestq What do you think about adding support for the metadata format from Kaggle (one metadata file for each split with the name equal to the split name) to ImageFolder/AudioFolder? I also think we can relax some requirements a bit by:
* allowing `filename` as the name of the main metadata column (currently, only `file_path` is allowed)
* not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using `_check_if_features_can_be_aligned` + `_align_features`. The rationale is that train/val metadata often has extra columns compared to test metadata.

These changes would allow us to load the Kaggle dataset linked in the forum thread without any "interventions".

PS: this metadata format for ImageFolder was also proposed by @abhishekkrthakur initially.


Can you give more details about the Kaggle format ? I'm down to discuss it in a separate issue if you don't mind.

> allowing filename as the name of the main metadata column (currently, only file_path is allowed)

`filename` refers to the name of the file, so there's no logic about relative path or directories. If I recall correctly this is what we're doing right now so why not

> not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using _check_if_features_can_be_aligned + _align_features. The rationale is that train/val metadata often has extra columns compared to test metadata.

+1 and we can set to None the missing features

I'm not sure if this is worth opening a new issue :).

What I mean by the Kaggle format is the structure like this one (the name of a metadata file is equal to the directory it "references"):
```
- train
    - img1.jpeg
    - img2.jpeg
    - ...
- test
    - img1.jpeg
    - img2.jpeg
    - ...   
- train.csv
- test.csv
```




Sounds nice !

@mariosasko +1 to allowing different features set and metadata filenames corresponding to split names

Considering filename column - right now it's even called `file_name` now, which is not nice because in fact it's a relative file path indeed, so I think it should be `file_path` (and I don't know why I haven't thought about it before the release...)

@lewtun don't you mind if I close this pull request as I've integrated your changes in https://github.com/huggingface/datasets/pull/4872 ? (it doesn't have a link to a kaggle example though)