Load big amount of unlabeled images and predict classes

adama: <p>I have a finetuned model and want to apply it to unlabeled images. The images are located in one directory with several subfolders. Each suchfolder can contain several subfolders as well. So I have to get them recursivly.</p>
<p>I have around 23.000 images to make a binary classification on. I thought it would be more efficient to load the data with a dataloader into my network rather than loading each image after another.</p>
<p>If the image gets classified as True (1) I want to copy the original image into another folder.</p>
<p>To get the data I am using <code>os.walk()</code>.</p>
<pre><code class="lang-auto">rootpath = &lt;mypath&gt;
paths = []
for subdir, dirs, files in os.walk(rootpath):
    for file in files:
        #print os.path.join(subdir, file)
        filepath = subdir + os.sep + file
        paths.append(filepath)
</code></pre>
<p>How can I bring this into a pytorch <code>Dataset</code> where I can feed it into the <code>DataLoader</code>?</p>

arya47: <p>This a one of several approaches you can take</p>
<p>Here in the <strong>init</strong> method you can create a structure which stores paths to all images, any structure that can be indexed.<br>
eg: list of image paths [’/img/1.png’ , ‘/img/2.png’ …]<br>
Then in the <strong>getitem</strong> method you can load the corresponding image to a index x</p>
<p>(Add parameters to the init function if you want to)</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset

class DataClass(Dataset):
    def __init__(self):
        self.list_of_paths = ... # Here create a list of all image paths / paths..

    def  __len__(self):
        return len(self.list_of_paths)

    def __getitem__(self, x):
        image_path = self.list_of_paths[x] # Gives the path to an image
        image = load_image ... # Here load your image using your path
        
        return image

</code></pre>
<p>then create a ds</p>
<pre><code class="lang-auto">train_ds = DataClass()
train_dl = DataLoader(train_ds ...)
</code></pre>

adama: <p>Thank you for your hint.<br>
I made following script:</p>
<pre><code class="lang-auto">rootpath = '&lt;mypath&gt;'
paths = []
for subdir, dirs, files in os.walk(rootpath):
    for file in files:
        #print os.path.join(subdir, file)
        filepath = subdir + os.sep + file
        paths.append(filepath)


class ImageDataset(Dataset):
    def __int__(self):
        self.imagelist = paths

    def __len__(self):
        return len(self.imagelist)

    def __getitem__(self, index):
        self.imagepath = self.imagelist[index]
        self.image = Image.open(self.imagepath)

        transform = transforms.Compose([
                    transforms.ToPILImage(),
                    transforms.Resize(256),
                    transforms.CenterCrop(224),
                    transforms.ToTensor(),
                    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
                    ])

        i = transform(self.image)
        return i


imagedataset = ImageDataset()

imagedl = torch.utils.data.DataLoader(imagedataset, batch_size=300, drop_last=True)

for index, img in enumerate(imagedl):
    print(index, img)
</code></pre>
<p>And i get following error if I want to iterate over the dataloader:</p>
<p><code>AttributeError: 'ImageDataset' object has no attribute 'imagelist'</code></p>
<p>Why? I declare <code>self.imagelist</code> as a list of all my imagepaths</p>

adama: <pre><code class="lang-auto">transform = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
            ])

class CustomDataSet(Dataset):
    def __init__(self, imagelist, transform):
        self.imagepaths = imagelist
        self.transform = transform

    def __len__(self):
        return len(self.imagepaths)

    def __getitem__(self, index):
        self.imagepath = self.imagepaths[index]
        self.image = Image.open(self.imagepath)

        self.i = self.transform(self.image)
        return self.i

imagedataset = CustomDataSet(paths, transform)
imagedl = torch.utils.data.DataLoader(imagedataset, batch_size=300, drop_last=True)
</code></pre>
<p>Yeah, can´t work if I spelled call my <code>__init__</code> function <code>int()</code>. Changed that and now it´s working fine.</p>