load_dataset can't work correct with my image data 
Opened by WiNE-iNEFF (closed)

I have about 20000 images in my folder which divided into 4 folders with class names.
When i use load_dataset("my_folder_name", split="train") this function create dataset in which there are only 4 images, the remaining 19000 images were not added there. What is the problem and did not understand. Tried converting images and the like but absolutely nothing worked

Comments:
Can you post a reproducible code snippet of what you tried to do?



> Can you post a reproducible code snippet of what you tried to do?
> 
> 

```python
from datasets import load_dataset

dataset = load_dataset("my_folder_name", split="train")
```

hi @WiNE-iNEFF ! can you please also tell a bit more about how your data is structured (directory structure and filenames patterns)?

> hi @WiNE-iNEFF ! can you please also tell a bit more about how your data is structured (directory structure and filenames patterns)?

All file have format .png converted in RGBA. 
In main folder "MyData" contain 4 folder with images. In function load_dataset i use folder "MyData"

@WiNE-iNEFF I'm sorry there is still not enough information to answer your question :( For now I can only assume that your [filenames contain split names](https://huggingface.co/docs/datasets/repository_structure#splits-and-file-names) which are somehow incorrectly parsed. 
What would be the output if you omit `split` while loading? Like just
```python
ds = load_dataset("MyData")
print(ds)
```



> @WiNE-iNEFF I'm sorry there is still not enough information to answer your question :( For now I can only assume that your [filenames contain split names](https://huggingface.co/docs/datasets/repository_structure#splits-and-file-names) which are somehow incorrectly parsed. 
> What would be the output if you omit `split` while loading? Like just
> ```python
> ds = load_dataset("MyData")
> print(ds)
> ```
> 
> 

```python
Dataset({
    features: ['image', 'label'],
    num_rows: 4
})
```

@WiNE-iNEFF  My only guess is that 4 images in your data have `"train"` string in their names (something like `"train_image_0.png"`) and others do not and the loader ignores all the files that do not contain split name in filename. If it's true, please try to remove "train" from filenames. Or maybe they are inside a directory named "train", then the directory should be renamed (unless you want to put only these 4 specific images to the train but apparently you do not).

If there is a bug I cannot investigate it unfortunately because I cannot reproduce your case without some data samples. 

> @WiNE-iNEFF  My only guess is that 4 images in your data have `"train"` string in their names (something like `"train_image_0.png"`) and others do not and the loader ignores all the files that do not contain split name in filename. If it's true, please try to remove "train" from filenames. Or maybe they are inside a directory named "train", then the directory should be renamed (unless you want to put only these 4 specific images to the train but apparently you do not).
> 
> If there is a bug I cannot investigate it unfortunately because I cannot reproduce your case without some data samples. 

I checked my files and some of them do have the words train, valid and test in their names, but the number of such images is more than 500, not 4.

@WiNE-iNEFF Probably they are named inconsistently so that the correct pattern for which files should correspond to which split cannot be inferred. You can make it clearer to the loader by removing split names from filenames and putting files in separate folder for each split (you can take a look at the [documentation for imagefolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder)):
```
 Fuaimeanna2/
├─ test
│   ├─ label_0
│   │   ├── filename_0.jpg
│   │   └── filename_1.jpg
│   │   └── ...
│   ├─ label_1
│   │   └── ...
│   ├─ label_2
│   │   └── ...
│   └─ label_3
│          └── ...
├─ train
│   ├─ label_0
│   │   └── ...
│   ├─ label_1
│   │   └── ...
│   ├─ label_2
│   │   └── ...
│   └─ label_3
│          └── ...
└── validation
      ├─ label_0
      │   └── ...
      ├─ label_1
      │   └── ...
      ├─ label_2
      │   └── ...
      └─ label_3
            └── ...
```

> @WiNE-iNEFF Probably they are named inconsistently so that the correct pattern for which files should correspond to which split cannot be inferred. You can make it clearer to the loader by removing split names from filenames and putting files in separate folder for each split (you can take a look at the [documentation for imagefolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder)):
> ```
>  Fuaimeanna2/
> ├─ test
> │   ├─ label_0
> │   │   ├── filename_0.jpg
> │   │   └── filename_1.jpg
> │   │   └── ...
> │   ├─ label_1
> │   │   └── ...
> │   ├─ label_2
> │   │   └── ...
> │   └─ label_3
> │          └── ...
> ├─ train
> │   ├─ label_0
> │   │   └── ...
> │   ├─ label_1
> │   │   └── ...
> │   ├─ label_2
> │   │   └── ...
> │   └─ label_3
> │          └── ...
> └── validation
>       ├─ label_0
>       │   └── ...
>       ├─ label_1
>       │   └── ...
>       ├─ label_2
>       │   └── ...
>       └─ label_3
>             └── ...
> ```

I have read this documentation more than once. It just wasn't a problem before.

Hi,

You need to use:
```
from datasets import load_dataset

dataset = load_dataset("imagefolder", split="train", data_dir="path_to_your_folder")
```
instead of 
```
from datasets import load_dataset

dataset = load_dataset("my_folder_name", split="train")
```
To create an image dataset from your local folders.

> Hi,
> 
> You need to use:
> 
> ```
> from datasets import load_dataset
> 
> dataset = load_dataset("imagefolder", split="train", data_dir="path_to_your_folder")
> ```
> 
> instead of
> 
> ```
> from datasets import load_dataset
> 
> dataset = load_dataset("my_folder_name", split="train")
> ```
> 
> To create an image dataset from your local folders.

Thank you, but even using the method that you wrote above absolutely nothing changes, especially without using data_dir on my other data everything works fine

@WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 


> even using the method that you wrote above absolutely nothing changes

fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.

Oh I didn't know that! OK but in any case, not sure why the image builder isn't working for @WiNE-iNEFF. But it's hard for us to help if we can't reproduce. I'd just check the structure of the folders, see if the splits are correctly set up, etc.

> @WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 
> 
> 
> > even using the method that you wrote above absolutely nothing changes
> 
> fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.

I'll try to try your method over the next few days, then I'll write it turned out 

> @WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 
> 
> 
> > even using the method that you wrote above absolutely nothing changes
> 
> fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.

I tried creating a `train` folder and put my image folders in it. As a result, all 18,000 images were loaded. 

@WiNE-iNEFF great! So to explain what happened according to my assumptions:

When you use a standard packaged loader (like `imagefolder`, `csv`, `jsonl`, and so on) and load your data like `load_dataset("my_folder_name")` or `load_dataset("imagefolder", data_dir="my_folder_name"`, the library searches for patterns to divide files into splits. This is described a bit in [this doc](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#splits-and-file-names). And the order to search for patterns is the following:
1. first it checks for [pattern like `data/<split_name>-xxxxx-of-xxxxx`](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#custom-split-names) (which allows to pass custom split names)
2. then for directories named as splits (if you have directories named `train`, `test` etc.)
3. then for [splits in filenames](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#splits-and-file-names) (like if you have files named `train-image.jpg`, `test_0.jpg`, ...)
4. then if no pattern was found, it treats all files as belonging to a single `train` split

The code is [here](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L215).
So I assume that in your case, since you didn't have directories for splits (pattern 2), some files that included split keywords (pattern 3) were included and others were ignored as not matching the pattern. And when you added `train` directory, the pattern for directories (pattern 2) was triggered first and everything worked as expected. Everything worked in your previous cases probably because you didn't have split names keywords in filenames, so all the files ended up being a part of a single train split (pattern 4).

Another way to mitigate this apart from structuring your data according to the patterns is to explicitly state with files belong to which splits by passing them with `data_files` parameter:
```python
load_dataset("my_folder_name", data_files={"train": "**"})   # to tell that all files should be included  
```

Now I see that this order should be explained in documentation and also referenced in sections for packaged modules like `imagefolder`, thank you for pointing this out.  

 


@NielsRogge @polinaeterna I have a similar problem when reading my dataset. I want to use DETR for object detection, but my data is in YOLO format. With a dataset of 10k images, yolo format involves having 10k labels. As far as I read regarding [COCO format](https://auto.gluon.ai/stable/tutorials/multimodal/object_detection/data_preparation/convert_data_to_coco_format.html), there must be one JSON per split. However, as I post in the [Hugging Face forum](https://discuss.huggingface.co/t/prepare-dataset-from-yolo-format-to-coco-for-detr/34894), when it is read, the number of rows is 1, which does not make sense. 
The instruction to read the train-val-test splits are: 
```python
from datasets import load_dataset
data_files = {
	"train": './train_labels.json',
	"validation": './val_labels.json',
	"test": './test_labels.json'
}
dataset = load_dataset("json", data_files=data_files)
```
An example of the short version of the json file I read, to reproduce my error, is the following: 

``` json
{
    "info": {},
    "licenses": [],
    "images": [
        {
            "id": 1,
            "file_name": "aceca_100.mp4frame21.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null
            },
        {
            "id": 2,
            "file_name": "aceca_100.mp4frame24.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null
        },
        {
            "id": 3,
            "file_name": "aceca_100.mp4frame25.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null}],
            "annotations": [
                {
                    "id": 1,
                    "image_id": 1,
                    "category_id": 0,
                    "bbox": [0.0, 278.21896388398557, 86.94096523844935, 156.0293445072134],
                    "area": 13565.341816979679,
                    "iscrowd": 0
                },
                {
                    "id": 2,
                    "image_id": 2,
                    "category_id": 0,
                    "bbox": [149.28851295721816, 297.6359759754418, 34.76802347007475, 98.03908698442889],
                    "area": 3408.625277259324,
                    "iscrowd": 0
                },
                {
                    "id": 3,
                    "image_id": 3,
                    "category_id": 0,
                    "bbox": [153.3817197549372, 300.168969412891, 31.787555842913775, 89.69583163436312],
                    "area": 2851.2112569539095,
                    "iscrowd": 0
                }
            ],
            "categories": [
                {
                    "id": 0, "name": "person"
                }
            ]
        }
```
If full files required, my email is aruigui98@gmail.com

Hi @Alberto1404, to load an object detection dataset it's recommended to make use of the metadata feature as explained [here](https://huggingface.co/docs/datasets/image_dataset#object-detection).  

Thank you @NielsRogge! It works!!!

You can now refer to https://huggingface.co/docs/datasets/repository_structure to learn about the `datasets`' data files inference, so I'm closing this issue.