Pytorch torchvision.datasets.ImageFolder FileNotFoundError: Found no valid file for the classes

username: Niraja
cooked: <p>Tried to load training data with PyTorch torch.datasets.ImageFolder in Colab. I am using Wiki crop data for age and gender prediction.<br>
I am try to fix this error but I couldn’t.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88a65999e988c857232fcf9748e63c3c7b0b9a41.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/88a65999e988c857232fcf9748e63c3c7b0b9a41" title="ImageFolderError"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a65999e988c857232fcf9748e63c3c7b0b9a41_2_690x265.jpeg" alt="ImageFolderError" data-base62-sha1="juRl1YeB5d3pyOfsRtxlOj48YrD" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a65999e988c857232fcf9748e63c3c7b0b9a41_2_690x265.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a65999e988c857232fcf9748e63c3c7b0b9a41_2_1035x397.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88a65999e988c857232fcf9748e63c3c7b0b9a41_2_1380x530.jpeg 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ImageFolderError</span><span class="informations">1837×706 207 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have uploaded my code here,<br>
‘’’</p>
<pre><code class="lang-auto">WikiCrop_df = pd.read_csv('/content/drive/MyDrive/wiki_crop/wikiCrop.csv')

#Define the path to the directory containing the images
src_folder = '/content/drive/MyDrive/wiki_crop/'

# Define the path to the train directory
train_dir = '/content/drive/MyDrive/WikiCrop_TrainDirData'

# Define the path to the test directory
test_dir = '/content/drive/MyDrive/WikiCrop_TestDirData'

import os

# Create the train and test directories
if not os.path.exists(train_dir):
    os.makedirs(train_dir)
if not os.path.exists(test_dir):
    os.makedirs(test_dir)

#Now, we can create subdirectories in the train and test directories for each label using age and gender information in your data frame:

for age in filtered_df.age.unique():
    os.mkdir(os.path.join(train_dir, str(age)))
    os.mkdir(os.path.join(test_dir, str(age)))

for gender in filtered_df.gender.unique():
    os.mkdir(os.path.join(train_dir, str(gender)))
    os.mkdir(os.path.join(test_dir, str(gender)))

# Split the data frame into train and test sets, with a specified ratio (e.g., 80/20):

train_ratio = 0.8

train_df = WikiCrop_df.sample(frac=train_ratio, random_state=42)
test_df = WikiCrop_df.drop(train_df.index)

# Create the train and test directories
if not os.path.exists(train_dir):
    os.makedirs(train_dir)
if not os.path.exists(test_dir):
    os.makedirs(test_dir)

for index, row in train_df.iterrows():
    src_path = os.path.join(src_folder, row['path'])
    dst_path = os.path.join(train_dir, str(row['age']), str(row['gender']), str(row['path']))
    if src_path != dst_path:
        try:
            copyfile(src_path, dst_path)
        except FileExistsError as e:
            print(f"Error copying {src_path} to {dst_path}: {e}")
    

for index, row in test_df.iterrows():
    src_path = os.path.join(src_folder, row['path'])
    dst_path = os.path.join(test_dir, str(row['age']), str(row['gender']), str(row['path']))
    if src_path != dst_path:
        try:
            copyfile(src_path, dst_path)
        except FileExistsError:
            pass

#train and test data directory
train_data_dir = '/content/drive/MyDrive/WikiCrop_TrainDirData'
test_data_dir = '/content/drive/MyDrive/WikiCrop_TestDirData'

import torch
import torchvision
from torchvision import transforms
from torchvision.datasets import ImageFolder

#load the train and test data

train_dataset = ImageFolder(train_data_dir, transform = transforms.Compose([
transforms.Resize((112,112)), transforms.ToTensor()
]))

test_dataset = ImageFolder(test_data_dir, transform = transforms.Compose([
transforms.Resize((112,112)), transforms.ToTensor()
]))
</code></pre>
<p>Please tell me where my code is wrong. I tried to fix this error since long time but I couldn’t. I have uploaded how my WikiCrop_TrainDirData looks like.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ee4cbad27efe24a754b45e70a3a267265779b5e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6ee4cbad27efe24a754b45e70a3a267265779b5e" title="ImageFolderCode4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ee4cbad27efe24a754b45e70a3a267265779b5e_2_227x500.jpeg" alt="ImageFolderCode4" data-base62-sha1="fP0IgWs0VWGgTUZY79U9FVcJox8" width="227" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6ee4cbad27efe24a754b45e70a3a267265779b5e_2_227x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ee4cbad27efe24a754b45e70a3a267265779b5e.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6ee4cbad27efe24a754b45e70a3a267265779b5e.jpeg 2x" data-dominant-color="F6F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ImageFolderCode4</span><span class="informations">287×631 17.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thank you in advance.</p>

username: ptrblck
cooked: <p><code>ImageFolder</code> expects images in each of the class folders supporting the file extensions mentioned in the error message.<br>
Could you check what each folder contains and could you make sure these are valid images?</p>

username: Niraja
cooked: <p>These folders does not contain anything. For Wiki Crop data I have convert .mat file to csv.<br>
‘’’<br>
import numpy as np<br>
from <a href="http://scipy.io" rel="noopener nofollow ugc">scipy.io</a> import loadmat<br>
import pandas as pd<br>
import datetime as date<br>
from dateutil.relativedelta import relativedelta</p>
<p>cols = [‘age’, ‘gender’, ‘path’]</p>
<p>wiki_mat = ‘/content/drive/MyDrive/wiki_crop/wiki.mat’</p>
<p>wiki_data = loadmat(wiki_mat)</p>
<p>del wiki_mat</p>
<p>wiki = wiki_data[‘wiki’]</p>
<p>wiki_photo_taken = wiki[0][0][1][0]<br>
wiki_full_path = wiki[0][0][2][0]<br>
wiki_gender = wiki[0][0][3][0]<br>
<span class="hashtag">#wiki_face_score1</span> = wiki[0][0][6][0]<br>
<span class="hashtag">#wiki_face_score2</span> = wiki[0][0][7][0]</p>
<p>wiki_path = []</p>
<p>for path in wiki_full_path:<br>
wiki_path.append(‘/content/drive/MyDrive/wiki_crop/’ + path[0])</p>
<p>wiki_dob = []</p>
<p>for file in wiki_path:<br>
wiki_dob.append(file.split(‘_’)[2])</p>
<p>wiki_age = []</p>
<p>for i in range(len(wiki_dob)):<br>
try:<br>
d1 = date.datetime.strptime(wiki_dob[i], ‘%Y-%m-%d’)<br>
d2 = date.datetime.strptime(str(wiki_photo_taken[i]), ‘%Y’)<br>
rdelta = relativedelta(d2, d1)<br>
diff = rdelta.years<br>
except Exception as ex:<br>
print(ex)<br>
diff = -1<br>
wiki_age.append(diff)</p>
<p><span class="hashtag">#final_imdb</span> = np.vstack((imdb_age, imdb_genders, imdb_path, imdb_face_score1, imdb_face_score2)).T<br>
final_wiki = np.vstack((wiki_age, wiki_gender, wiki_path)).T</p>
<p>final_wiki_df = pd.DataFrame(final_wiki)</p>
<p>final_wiki_df.columns = cols</p>
<p>meta = pd.concat([final_wiki_df])</p>
<p><span class="hashtag">#meta</span> = meta[meta[‘face_score1’] != ‘-inf’]<br>
<span class="hashtag">#meta</span> = meta[meta[‘face_score2’] == ‘nan’]</p>
<p><span class="hashtag">#meta</span> = meta.drop([‘face_score1’, ‘face_score2’], axis=1)</p>
<p>meta = meta.sample(frac=1)</p>
<p>meta.to_csv(‘meta.csv’, index=False)<br>
‘’’<br>
and convert thus .csv to dataframe using pandas and used for train test classification and ImageFolder</p>

username: Niraja
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68cae45b170befeaf0b09101d9254906ef9c2405.png" data-download-href="https://discuss.pytorch.org/uploads/default/68cae45b170befeaf0b09101d9254906ef9c2405" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68cae45b170befeaf0b09101d9254906ef9c2405.png" alt="image" data-base62-sha1="eX2lLulXdKi9PrRq7H3saSvTALr" width="690" height="350" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">751×382 9.04 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This I have used for train, test split and ImageFolder</p>

username: ptrblck
cooked: <p>Your code is not properly formatted so hard to read, but:</p>
<aside class="quote no-group" data-username="Niraja" data-post="3" data-topic="178047">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niraja/48/61924_2.png" class="avatar"> Niraja:</div>
<blockquote>
<p>These folders does not contain anything.</p>
</blockquote>
</aside>
<p>sounds weird as I don’t understand why you would use the <code>ImageFolder</code> dataset in this case.<br>
Check the <a href="https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html#imagefolder">docs</a> to real more about its use case, but as already mentioned it expects each subfolder to represent a class and to contain all corresponding images.<br>
If you are dealing with a custom dataset stored in csv files you might want to create a custom <code>Dataset</code> as described in <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>.</p>

username: RAN123
cooked: <p>wow.amazing.<br>
I am a chinese student. I am studying it now.</p>

username: Niraja
cooked: <p>I used ImageFolder to create Train and Test data directories and apply transform to resize the images and convert it to ToTensor.</p>

username: Niraja
cooked: <p>My wiki crop folder contain these type of images. Is my image format is wrong so that it could create the error while I am running ImageFolder instruction.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af67d716b7961dda716faa20811b8fe74f949284.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/af67d716b7961dda716faa20811b8fe74f949284" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af67d716b7961dda716faa20811b8fe74f949284_2_690x347.jpeg" alt="image" data-base62-sha1="p1I26lNZ1r86Wqe687fNqdd34dS" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af67d716b7961dda716faa20811b8fe74f949284_2_690x347.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af67d716b7961dda716faa20811b8fe74f949284_2_1035x520.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af67d716b7961dda716faa20811b8fe74f949284.jpeg 2x" data-dominant-color="BCB4B1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1123×566 233 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Niraja
cooked: <p>This is my code part where I have convert .mat file to .csv file.</p>
<pre><code class="lang-auto">'''
import numpy as np
from scipy.io import loadmat
import pandas as pd
import datetime as date
from dateutil.relativedelta import relativedelta

cols = ['age', 'gender', 'path']

wiki_mat = '/content/drive/MyDrive/wiki_crop/wiki.mat'
wiki_data = loadmat(wiki_mat)
wiki = wiki_data['wiki']

wiki_photo_taken = wiki[0][0][1][0]
wiki_full_path = wiki[0][0][2][0]
wiki_gender = wiki[0][0][3][0]
#wiki_face_score1 = wiki[0][0][6][0]
#wiki_face_score2 = wiki[0][0][7][0]

wiki_path = []

for path in wiki_full_path:
    wiki_path.append('/content/drive/MyDrive/wiki_crop/' + path[0])

wiki_dob = []


for file in wiki_path:
    wiki_dob.append(file.split('_')[2])

wiki_age = []

for i in range(len(wiki_dob)):
    try:
        d1 = date.datetime.strptime(wiki_dob[i], '%Y-%m-%d')
        d2 = date.datetime.strptime(str(wiki_photo_taken[i]), '%Y')
        rdelta = relativedelta(d2, d1)
        diff = rdelta.years
    except Exception as ex:
        print(ex)
        diff = -1
    wiki_age.append(diff)

#final_imdb = np.vstack((imdb_age, imdb_genders, imdb_path, imdb_face_score1, imdb_face_score2)).T
final_wiki = np.vstack((wiki_age, wiki_gender, wiki_path)).T

final_wiki_df = pd.DataFrame(final_wiki)

final_wiki_df.columns = cols

meta = pd.concat([final_wiki_df])
#meta = meta[meta['face_score1'] != '-inf']
#meta = meta[meta['face_score2'] == 'nan']

#meta = meta.drop(['face_score1', 'face_score2'], axis=1)

meta = meta.sample(frac=1)

meta.to_csv('meta.csv', index=False)
'''
</code></pre>
<p>and after this I filtered out nan values from this and used this for train, test split and make directories using ImageFolder.  Please tell where i did mistake.</p>

username: Niraja
cooked: <p>okay. Now I got where I did the mistake. Thank you.</p>