Imagenet folders problem

rahul: <p>The imagenet dataset that I downloaded from ILSVRC has two sub-folders</p>
<ol>
<li>Annotations</li>
<li>Data</li>
<li>ImageSets<br>
Out of the above 3, the second folder, i.e. “Data” folder seems to have the .jpeg files.<br>
This folder has a single subfolder called CLS-LOC, which further has 3 subfolders</li>
</ol>
<p>2 a) test<br>
2 b) train<br>
2 c)val</p>
<p>While training and testing, I use the following for loading datasets</p>
<pre><code>traindir = '/home/ec2-user/anaconda3/ILSVRC/Data/CLS-LOC/train'
trainset = datasets.ImageFolder(traindir,t_train)    
train_set = torch.utils.data.DataLoader(trainset,batch_size=bsz,shuffle=True,num_workers=4,pin_memory=True) 
</code></pre>
<p>while for validation, I use the following</p>
<pre><code>testdir = '/home/ec2-user/anaconda3/ILSVRC/Data/CLS-LOC/test'
testset = datasets.ImageFolder(valdir,t_test)
test_set = \
torch.utils.data.DataLoader(testset,batch_size=bsz_test,shuffle=False,num_workers=2,pin_memory=True)
</code></pre>
<p>However, I get the following error</p>
<p>“RuntimeError: Found 0 files in subfolders of: /home/ec2-user/anaconda3/ILSVRC/Data/CLS-LOC/test<br>
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif”</p>
<p>I am confused what to do?</p>

ptrblck: <p>Could you check, if the download was successful?<br>
Are you seeing files with a size &gt; 0kB?</p>

rahul: <p>The val folder size is 6.4 G, and train folder size is approximately 280 G (which includes tar files, and also their extracted version).</p>

ptrblck: <p>What does <code>ls /home/ec2-user/anaconda3/ILSVRC/Data/CLS-LOC/test</code> in your terminal return?</p>

rahul: <p>it gives me the following error</p>
<p>““RuntimeError: Found 0 files in subfolders of: /home/ec2-user/anaconda3/ILSVRC/Data/CLS-LOC/test<br>
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif””</p>

ptrblck: <p>This seems to be the error thrown by <code>ImageFolder</code>.<br>
Could you type the command into your terminal (bash) and see, if there are folders inside the path?</p>

rahul: <p>I didnt understand…I am attaching the snapshot of my val folder here…please find it attached here…<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/dbdb016ab4051dc2955c3a82b062b5dddb78b7a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/dbdb016ab4051dc2955c3a82b062b5dddb78b7a7" title="Screenshot%20from%202018-06-26%2013-16-07"><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/dbdb016ab4051dc2955c3a82b062b5dddb78b7a7.png" alt="Screenshot%20from%202018-06-26%2013-16-07" data-base62-sha1="vmVMyHGFIJmJddkQQFsfFjzSOx1" width="684" height="500" data-dominant-color="1F2B38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202018-06-26%2013-16-07</span><span class="informations">1318×963 205 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>What’s inside these folders?<br>
Could you look into some with <code>ls</code>?</p>

rahul: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="20256">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>ls /home/ec2-user/anaconda3/ILSVRC/Data/CLS-LOC/test</code></p>
</blockquote>
</aside>
<p>Hi, I am using the 2012 dataset now…there doesnt seem to be the same issue…i did an ls, and am attaching screenshot of results…however, the accuracy on train and val tests are differing a lot (50% versus 1%).<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a25fe8c8b46b8e17cac0fd61bc6be80afecf3419.png" data-download-href="https://discuss.pytorch.org/uploads/default/a25fe8c8b46b8e17cac0fd61bc6be80afecf3419" title="Screenshot%20from%202018-06-26%2015-14-27"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a25fe8c8b46b8e17cac0fd61bc6be80afecf3419_2_690x155.png" alt="Screenshot%20from%202018-06-26%2015-14-27" data-base62-sha1="naqPLC2FQ1g4kBzh79ubJBRF9bz" width="690" height="155" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a25fe8c8b46b8e17cac0fd61bc6be80afecf3419_2_690x155.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/a25fe8c8b46b8e17cac0fd61bc6be80afecf3419_2_1035x232.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/a25fe8c8b46b8e17cac0fd61bc6be80afecf3419.png 2x" data-dominant-color="2B2D2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202018-06-26%2015-14-27</span><span class="informations">1330×300 70 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Maybe there are still some empty folders?<br>
Could you try to find them with <code>find . -type d -empty</code>?</p>

rahul: <p>I typed the above command in the val folder, and hit enter…it printed nothing (I am assuming it means all folders are non empty)…also, a print out of the total count of the validation set images while validating, is around 50k…</p>

Priya: <p>Hi<br>
I have faced same problem when I’m trying to load data which is present in a folder. The thing is create a folder again in the target folder with that data… Now in your case /test/test_dummy…<br>
Test_dummy contains same as your test data. So that torch ImageFolder can find test_dummy as a subfolder inside the test folder. Now it doesn’t give any error. Even though larger datasets require more time to copy there is no other option I have found. I hope you find it useful.<br>
Thank you. Have a try and let me know.</p>