Dataloader map-style dataset, how does it work?

username: lucastononrodrigues
cooked: <p>I’m having trouble trying to fine-tune a model.<br>
My main problem is that I cannot verify if the data is being correctly interpreted by the code.<br>
The model will not advance and have a significant result, so I suspect the labels are messed up.</p>
<p>To set the data for training and validation I used the following code found in the Pytorch documentation.(<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Finetuning Torchvision Models — PyTorch Tutorials 2.1.1+cu121 documentation</a>)</p>
<blockquote>
<h1>Create training and validation datasets</h1>
</blockquote>
<p>image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>) for x in [‘train’, ‘val’]}</p>
<blockquote>
<h1>Create training and validation dataloaders</h1>
</blockquote>
<p>dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>, batch_size=batch_size, shuffle=True, num_workers=0) for x in [‘train’, ‘val’]}</p>
<p>My data is organized in folders in “./database/train” and “./database/val”.<br>
I’m treating different shoe models in different folders that I’d like to classify.</p>
<p>example of folders: “./database/train/ADIDAS - Yeezy 500 Bone White” and “./database/train/NIKE - OFF WHITE - Air Max 90 Desert Ore”</p>
<p>Where I’ve inserted multiple images of a given category (which is the name of the model).</p>
<p>So my main question is: when I use the torch.utils.data.Dataloader(), is it considering the name of the folder as the label and treating it automatically what is inside the folder? That’s what I thought it would do, given the map-style dataset functionality (<a href="https://pytorch.org/docs/stable/data.html#map-style-datasets" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 2.1 documentation</a>)</p>
<p>Thank you!!</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> does not interpret or create the targets, but creates batches of samples using the <code>Dataset</code>.<br>
<code>ImageFolder</code>, which you have used as the dataset, will sort the folders and assign a class label to each folder.<br>
You could check some samples by trying to visualize a few random indices:</p>
<pre><code class="lang-python">x, y = image_datasets['train'][0]
plt.imshow(x.permute(1, 2, 0).numpy())
print(y)
print(image_datasets['train'].class_to_index)
</code></pre>
<p>Note that the visualization might look distorted, if you’ve normalized the data, so you could either remove the normalization for the sake of debugging or “unnormalize” the data.</p>

username: lucastononrodrigues
cooked: <p>Thank you!! That was really helpful, so if I just separate the folders of different classes with different names this should take care of it for me?<br>
The reason I asked it is because I though the Data loader in Pytorch was messing up the labels.<br>
I’m fine tuning vgg11, resnet and other models in order to classify 77 shoe classes where I have about 100 to 150 samples of each class (about 12k images in total), about 70% for training and 30% for testing, the results wont get any better than 1/77=1.3% and I find it really awkward, even though I know I don’t have a good data-set.<br>
Do you think that’s normal?<br>
Thank you again and again, I really appreciate your help !!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="lucastononrodrigues" data-post="3" data-topic="80040">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucastononrodrigues/48/23856_2.png" class="avatar"> lucastononrodrigues:</div>
<blockquote>
<p>so if I just separate the folders of different classes with different names this should take care of it for me?</p>
</blockquote>
</aside>
<p>Yes, this should be the case.</p>
<aside class="quote no-group" data-username="lucastononrodrigues" data-post="3" data-topic="80040">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucastononrodrigues/48/23856_2.png" class="avatar"> lucastononrodrigues:</div>
<blockquote>
<p>Do you think that’s normal?</p>
</blockquote>
</aside>
<p>I would recommend to scale down the problem a bit and maybe just use two classes with 5 samples each. Once you’ve created this dummy dataset, try to overfit your model so that you reach ~100% accuracy by changing the learning rate etc.<br>
If that’s not possible, there might be a bug in your code somewhere or the model architecture is “bad”, but I doubt the second point as VGG and ResNet models are heavily used.</p>