Creating a training directory

username: shafei
cooked: <p>Hi,</p>
<p>I have the following directory</p>
<p>data/<br>
├── images/         # dir for jpg files<br>
├── aeroplane.txt   # aeroplane object class labels<br>
├── bicycle.txt     # bicycle object class labels<br>
├── bird.txt        # bird object class labels<br>
├── boat.txt        # boat object class labels<br>
├── bottle.txt      # bottle object class labels<br>
├── bus.txt         # bus object class labels<br>
├── car.txt         # car object class labels<br>
├── cat.txt         # cat object class labels<br>
├── chair.txt       # chair object class labels<br>
├── cow.txt         # cow object class labels<br>
├── diningtable.txt # dining table object class labels<br>
├── dog.txt         # dog object class labels<br>
├── horse.txt       # horse object class labels<br>
├── motorbike.txt   # motorbike object class labels<br>
├── person.txt      # person object class labels<br>
├── pottedplant.txt # potted plant object class labels<br>
├── sheep.txt       # sheep object class labels<br>
├── sofa.txt        # sofa object class labels<br>
├── train.txt       # train object class labels<br>
├── tvmonitor.txt   # TV monitor object class labels</p>
<p>I have 17000 pictures that I want to classify the objects it has according to the directory above whether a sofa train or sheep, ect… is present.</p>
<p>I have set up the code below but I am having difficulty setting up a training folder that would randomly take 25% of the data set present for example.</p>
<p>import torch<br>
import torchvision.transforms as transforms<br>
import torchvision.models as models<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
import numpy as np<br>
from PIL import Image<br>
import numpy as np<br>
import matplotlib.pyplot as plt</p>
<p>pip install torchvision</p>
<p>from torchvision import models<br>
dir(models)</p>
<p>resnet = models.resnet101(pretrained=True)</p>
<p>from torchvision import transforms</p>
<h1>Image transformations</h1>
<p>image_transforms = {<br>
# Train uses data augmentation<br>
‘train’:<br>
transforms.Compose([<br>
transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0)),<br>
transforms.RandomRotation(degrees=15),<br>
transforms.ColorJitter(),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.CenterCrop(size=224),  # Image net standards<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])  # Imagenet standards<br>
]),<br>
# Validation does not use augmentation<br>
‘valid’:<br>
transforms.Compose([<br>
transforms.Resize(size=256),<br>
transforms.CenterCrop(size=224),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
}</p>
<p>import torchvision.datasets as datasets</p>
<p>train_set = datasets.ImageFolder(“root/label/train”, transform = image_transforms[‘train’])<br>
val_set = datasets.ImageFolder(“root/label/valid”, transform = image_transforms[‘valid’])</p>
<p>FileNotFoundError                         Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 train_set = datasets.ImageFolder(“root/label/train”, transform = image_transforms[‘train’])<br>
2 val_set = datasets.ImageFolder(“root/label/valid”, transform = image_transforms[‘valid’])</p>
<p>2 frames<br>
/usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py in _find_classes(self, dir)<br>
120         if sys.version_info &gt;= (3, 5):<br>
121             # Faster and available in Python 3.5 and above<br>
–&gt; 122             classes = [d.name for d in os.scandir(dir) if d.is_dir()]<br>
123         else:<br>
124             classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]</p>
<p>FileNotFoundError: [Errno 2] No such file or directory: ‘root/label/train’</p>

username: ptrblck
cooked: <p>Did you store the paths for each image in the corresponding <code>.txt</code> file or what do you mean by “class label” in the comments?</p>
<p>If you’ve stored the paths, I would recommend to write a custom <code>Dataset</code> (as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>).</p>
<p>You could read all file paths and create the target tensor in a separate tensor in the <code>__init__</code> method.<br>
Once you have a list of all paths and the <code>target</code>, you could lazily load and process the data in <code>__getitem__</code>.</p>
<p>Also, if you want to use a subet of the data, you could manually e.g. use <code>random_split</code> on the <code>Dataset</code>, use a <code>Subset</code> with your manually created indices or use a <code>SubsetRandomSampler</code> again with your indices.</p>
<p>Let me know, if that would work for you of if you need more information.</p>

username: shafei
cooked: <p>Thanks,</p>
<p>Each image is labelled with one of three numbers for each object class:</p>
<ul>
<li>-1 (no objects of this class feature in the image)</li>
<li>1 (at least one object of this class features in the image)</li>
<li>0 (at least one object of this class features in the image but they are all difficult to recognise)</li>
</ul>
<p>Each object class file (e.g.  <code>aeroplane.txt</code> ) contains the name of the image without the extension (e.g.  <code>2008_007739</code> ) followed by a space and then the class label (e.g.  <code>-1</code> ).</p>
<p>structure of the data in one of the labels:</p>
<p>2008_000002  -1<br>
0      2008_000003  -1<br>
1      2008_000007  -1<br>
2      2008_000008  -1<br>
3      2008_000009  -1<br>
4      2008_000015  -1<br>
…            …  …<br>
11534  2011_003269  -1<br>
11535  2011_003271  -1<br>
11536  2011_003274  -1<br>
11537  2011_003275  -1<br>
11538  2011_003276  -1</p>
<p>I am thinking of setting up a single labels folder that would include the name of the object the image file name and then in front of it a 0 or a 1 (0 means the object exists with a low probability and one means it exists with a high probability)  that single labels file should work with the option you guided me to.</p>
<p>Only problem I am thinking about can I use the 0 and the 1 (high and low probability) to set up a better preforming network.</p>

username: ptrblck
cooked: <p>The general workflow sounds good.<br>
However, I’m not sure about this part:</p>
<aside class="quote no-group" data-username="shafei" data-post="3" data-topic="73446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e274bd/48.png" class="avatar"> shafei:</div>
<blockquote>
<p>(0 means the object exists with a low probability and one means it exists with a high probability)</p>
</blockquote>
</aside>
<p>Would this mean that you won’t pass any images without the class?<br>
Your model would thus only learn to predict the object class for each sample.</p>
<aside class="quote no-group" data-username="shafei" data-post="3" data-topic="73446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e274bd/48.png" class="avatar"> shafei:</div>
<blockquote>
<p>Only problem I am thinking about can I use the 0 and the 1 (high and low probability) to set up a better preforming network.</p>
</blockquote>
</aside>
<p>You could try to apply a loss weighting based on the low and high probability, i.e. you could reduce the loss for a low probability class to decrease the penalty for your model.</p>

username: shafei
cooked: <p>thanks, so for the problem I have in hand I don’t have a labelled data set they all have a -1 labels coming in the txt files after the name of each image. So I don’t have a ground truth I want to pass next to each image name a 1 or a 0 if the class exits in the image.</p>
<p>I was having two difficulties the first is the structure of my directory matching the imagenet directory to use the dataloader features root/class/imagename, so does the directory and data frame I have work or do I need to change it?</p>
<p>The second difficulty is the fact that I don’t have any labels for the class so it is an unsupervised problem as when I was exploring the data set using pandas I found that all the class files are just filled with the image names and have a -1. Any idea what is the best way to set up an unsupervised problem?</p>
<p>data/<br>
├── images/ # dir for jpg files<br>
├── aeroplane.txt # aeroplane object class labels<br>
├── bicycle.txt # bicycle object class labels<br>
├── bird.txt # bird object class labels<br>
├── boat.txt # boat object class labels<br>
├── bottle.txt # bottle object class labels<br>
├── bus.txt # bus object class labels<br>
├── car.txt # car object class labels<br>
├── cat.txt # cat object class labels</p>

username: ptrblck
cooked: <p>I would assume it would be helpful.<br>
If you just pass class0 and class1, your model will basically always output the same class, no?<br>
The difference between these classes is just the “difficulty” to recognize the object.</p>
<p>Let me know, if I’m misunderstanding the use case.</p>

username: shafei
cooked: <p>thanks, so for the problem I have in hand I don’t have a labelled data set they all have a -1 labels coming in the txt files after the name of each image. So I don’t have a ground truth I want to pass next to each image name a 1 or a 0 if the class exits in the image.</p>
<p>I was having two difficulties the first is the structure of my directory matching the imagenet directory to use the dataloader features root/class/imagename, so does the directory and data frame I have work or do I need to change it?</p>
<p>The second difficulty is the fact that I don’t have any labels for the class so it is an unsupervised problem as when I was exploring the data set using pandas I found that all the class files are just filled with the image names and have a -1. Any idea what is the best way to set up an unsupervised problem? I would classify the image and change its name if it has a certain class.</p>
<p>data/<br>
├── images/ # dir for jpg files<br>
├── aeroplane.txt # aeroplane object class labels<br>
├── bicycle.txt # bicycle object class labels<br>
├── bird.txt # bird object class labels<br>
├── boat.txt # boat object class labels<br>
├── bottle.txt # bottle object class labels<br>
├── bus.txt # bus object class labels<br>
├── car.txt # car object class labels<br>
├── cat.txt # cat object class labels</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shafei" data-post="7" data-topic="73446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e274bd/48.png" class="avatar"> shafei:</div>
<blockquote>
<p>I would classify the image and change its name if it has a certain class.</p>
</blockquote>
</aside>
<p>Do you want to label each sample manually or are you planning on using an unsupervised learning approach?<br>
I’m unfortunately not deeply familiar with the current state-of-the-art unsupervised learning methods and am not sure, what would work the best. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<p>I’m still unsure to understand the data completely.<br>
It seems you have text files with names of different classes.<br>
Does e.g. <code>bird.txt</code> contain all image paths containing the bird class?<br>
If so, then you already have the labels, no?</p>

username: shafei
cooked: <p>Thanks, no bird.txt only has the all the image names, so I am researching how to preform unsupervised clustering right now.</p>
<p>Once I am done with the unsupervised clustering I hope mu current code will work.</p>
<p>would a data frame of root/class/imagename be the right way I should cluster my data, so that I don’t run in any further problems after I preform the unsupervised clustering.</p>

username: ptrblck
cooked: <p>What kind of image names does <code>bird.txt</code> contain?<br>
Only the image names with images containing birds?<br>
If so, is <code>bird</code> a class you would like to use in training?</p>

username: shafei
cooked: <p>no unfortunately bird.txt and all the .txt files have the same list of all images not just the images with a certain class.</p>
<p>So unsupervised clustering is the way to go. how do you think the structure of the data frame should be so that once I have it ready I can use the dataloader.</p>
<p>Thanks,</p>
<p>Aly</p>

username: ptrblck
cooked: <p>Ah OK, thanks for the clarification.<br>
Since you don’t have the labels yet, you could use a single list with the paths to all files and use an unsupervised learning method directly.</p>
<p>I would recommend to create a custom <code>Dataset</code> as explained <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a> and load the images lazily in the <code>__getitem__</code>.<br>
Since you don’t have the labels, you can just return the image tensor.</p>