How to load images from different folders in the same batch?

Harry-675: <p>Hi,I need to load images from different folders,for example:batch_size=8,so I need to load 8 *3 images from 8 different folders,and load 3 images from each folder,all these images combined one batch.How to realize this?<br>
I will be grateful for your help!</p>

crcrpar: <p>Hi</p>
<p>It would be enough to define your original <code>torch.utils.data.DataSet</code> subclass to handle 8 folders simultaneously. I write down one naive not complete example below.</p>
<p>Let the folder structure be</p>
<pre><code class="lang-auto">- train
    - folder_1
    - ...
    - folder_8
</code></pre>
<p>In the above setting, you can list all the image files in each folder by <code>os.listdir('train/folder_1)</code>.<br>
Also you can override <code>torch.utils.data.DataSet</code> class as below and pass your dataset instance to <code>DataLoader</code> setting <code>batch_size=3</code></p>
<pre><code class="lang-python">import os
from torch.utils.data import DataSet

class ImageDataSet(DataSet):

    def __init__(self, root='train', image_loader=None, transform=None):
        self.root = root
        self.image_files = [os.listdir(os.path.join(self.root, 'folder_{}'.format(i)) for i in range(1, 9)]
        self.loader = image_loader
        self.transform = transform
    def __len__(self):
        # Here, we need to return the number of samples in this dataset.
        return sum([len(folder) for folder in self.image_files])

    def __getitem__(self, index):
        images = [self.loader(os.path.join(root, 'folder_{}'.format(i), self.image_files[i][index])) for i in range(1, 9)]
        if self.transform is not None:
            images = [self.transform(img) for img in images]
        return images
</code></pre>

Harry-675: <p>Thanks for your help! I am a quite newer to pytorch,it really help me a lot!</p>

crcrpar: <p><a class="mention" href="/u/harry-675">@Harry-675</a><br>
I’m really sorry it might not work. (I’m not sure)<img src="https://discuss.pytorch.org/images/emoji/apple/bowing_man.png?v=5" title=":bowing_man:" class="emoji" alt=":bowing_man:"></p>
<p>If the above doesn’t work, try the below please.<br>
More naive solution is that preparing DataSet and DataLoader for each folder. Then you loop over all the dataloaders like this <a href="https://discuss.pytorch.org/t/train-simultaneously-on-two-datasets/649/9?u=crcrpar">Train simultaneously on two datasets</a> if you don’t care the order of sampling in each folder.</p>
<p>So,</p>
<pre><code class="lang-auto">class ImageData(torch.utils.data.DataSet):
    def __init__(self, root='train/folder_1', loader=image_load_func, transform=None):
        self.root = root
        self.files = os.listdir(self.root)
        self.loader = loader
        self.transform = transform
    def __len__(self):
        return len(self.files)
    def __getitem__(self, index):
        return self.transform(self.loader(os.path.join(self.root, self.files[index])))

loader_1 = DataLoader(ImageData('train/folder_1'), batch_size=3)
...
loader_8 = DataLoader(ImageData('train/folder_8'), batch_size=3)

for batch in zip(loader_1, ..., loader_8):
    batch = torch.cat(batch, dim=0)
</code></pre>

RoseFun: <p>Hi, how to definite the <code>image_load_func</code> ? ( I just got started <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:">)</p>

vaasudev96: <p>If anyone is looking for a different method, torchvision has a utility to accomplish this task cleanly.<br>
<a href="https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder</a></p>
<p>There’s a small caveat with this though, images in a particular batch are not guaranteed to come from different classes.</p>
<p>It assumes the images are arranged in the following manner:</p>
<pre><code class="lang-auto">root_dir/
    class_1/
        001.png
        002.png
        ...
    class_2/
        001.png
        002.png
        ...
    ...
</code></pre>
<pre><code class="lang-auto">from torchvision import datasets, transforms
from torch.utils import data

dataset = datasets.ImageFolder(root = root_dir, 
                transform = transforms.ToTensor())

loader = data.DataLoader(dataset, batch_size = 8, shuffle = True)
</code></pre>

Nimai_Patel: <p>Thanks a lot, this worked for me!</p>

demonvferrari: <p><a class="mention" href="/u/vaasudev96">@vaasudev96</a>  How to use root for the custom dataset(ImageDataSet) defined by <a class="mention" href="/u/crcrpar">@crcrpar</a> ?</p>
<p>loader_1 = DataLoader(ImageData.ImageFolder(root=‘train/folder_1’ , batch_size=3)</p>