A dataloader for multiple similar inputs

styx97: <p>Hi All,<br>
I have a network that takes <strong>three</strong> images in the input layer. Now the three images must be frames of the same video (this can be known only from the filename of the image). I understood how to change the dataloader’s <code>__getitem__</code> to send multiple inputs from <a href="https://discuss.pytorch.org/t/upload-a-customize-data-set-for-multi-regression-task/43413/2">here</a>. But how do I make sure three images from the same video are chosen and that none of them are repeated in the same epoch ?</p>

ptrblck: <p>It depends a bit on the current way of storing these files.<br>
Did you store each image from a video in a corresponding folder?<br>
If so, how would you like to sample the files, if their amount is not divisible without a remainder by 3? Would you like to drop the last files or fill it up with repetition?</p>
<p>Should the frames be contiguous in a sample or would you like to shuffle the frames from a single video?<br>
Also, would you like to shuffle the video folders or should the videos be loaded in a consecutive way?</p>

styx97: <p>Thanks for replying, <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Currently all images are together in test, train and val folders. If amount not divisible by 3, dropping the last frame is no problem . The frames from a single video need not be contiguous. It’s fine as long as they are from the same video.</p>
<p>The video folders (I can change my dataset to that format) can be shuffled freely. There’s no order among the videos, and the order of the frames in a single video can also be ignored.</p>

ptrblck: <p>Here is a small dummy example using multiple video folders.<br>
Note that I’ve used tensors directly, so you should add your frame loading logic into the <code>Dataset</code>.</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, videos, transform=None, nb_frames=3):
        self.nb_frames = nb_frames
        self.transform = transform
        
        # Crop data to multiple of nb_frames
        self.data = [v[:-(len(v)%self.nb_frames)] if len(v)%self.nb_frames!=0 else v 
                     for v in videos]
        
        # calculate lengths
        self.lens = [len(d)//self.nb_frames for d in self.data]
        # calculate offsets
        self.offsets = np.concatenate(([0], np.cumsum(self.lens[:-1])))
        
    
    def __getitem__(self, index):
        # subtract offset
        print('index: {}'.format(index))
        # get corresponding video file
        found = False
        for i, offset in enumerate(self.offsets):
            if index &lt; offset:
                print('subtracting {} from index'.format(self.offsets[i-1]))
                index -= self.offsets[i-1]
                index *= self.nb_frames
                found = True
                break
        # handle last video separately
        if not found:
            index -= self.offsets[-1]
            index *= self.nb_frames
            i += 1

        # select correspondind data
        print('selecing video {}'.format(i-1))
        data = self.data[i-1]
        # get frames
        print('reading frames {}'.format([idx for idx in range(index, index+self.nb_frames)]))
        x = []
        for idx in range(index, index+self.nb_frames):
            tmp = data[idx]
            if self.transform:
                tmp = self.transform(tmp)
            x.append(tmp)
        x = torch.cat(x)
        
        return x
        
    def __len__(self):
        return np.sum(self.lens)


videos = [torch.ones(torch.randint(3, 12, (1,)), 1)*i for i in range(5)]

dataset = MyDataset(videos)
for data in dataset:
    print(data)

loader = DataLoader(
    dataset,
    batch_size=2,
    shuffle=True)
for data in loader:
    print(data)
</code></pre>
<p>The code currently uses <code>nb_frames</code> consecutive frames for each video folder and removes the trailing frames.<br>
Shuffling using a <code>DataLoader</code> will work.<br>
I’ve also tried to add some debug print statement for better understanding, but let me know, if your need more information about this code.</p>

styx97: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I was wondering how the concept of ordinary dataloaders fit with this particular Dataset. That is, how the ordinary structure of -</p>
<pre><code class="lang-auto">train/  
   classA/
   classB/
val/
   classA/
   classB/ 
test/ 
   classA/ 
   classB/ 
</code></pre>
<p>fit with this type of loading. Inside a particular train, test or val folder, do we have to further pass the paths of the subfolders corresponding to each of the classes ? (this was automatically detected in the default approach)</p>

ptrblck: <p>I would create separate <code>Datasets</code> for the train, val, and test folders, so that you could stick yo your current <code>Dataset</code> implementation and have a clean cut between the data splits to avoid data leakage.</p>

styx97: <p>Thanks, that makes much more sense. Should I worry about the labels or will it  be handled automatically ?</p>

ptrblck: <p>In my code snippet I’m not handling the labels currently, so you would need to add them for your use case.</p>

styx97: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,I noticed in the source <a href="https://pytorch.org/docs/stable/_modules/torchvision/datasets/folder.html#ImageFolder" rel="nofollow noopener">here</a> that the <code>__getitem__</code> method of the image folder class returns a <code>sample, target</code> pair where sample is the image and target is the target label. In case of my dataset, how do I handle the case of three images having the same label ?</p>
<p>The <code>forward</code> method of my final model looks like this -</p>
<pre><code class="lang-auto">def forward(self,image1,image2,image3):
        x1 = self.model1(image1)
        x2 = self.model2(image2)
        x3 = self.model3(image3)
        x4 = torch.cat((x1, x2, x3), dim=1)
        x5 = self.classifier1(F.relu(x4))
        x6 = self.classifier2(F.relu(x5))
        x7 = self.classifier3(F.relu(x6))
        x8 = self.logSoftmax(x7)
        #print(x6.shape)
        return x8
</code></pre>
<p>In the train function, the images are fetched like -</p>
<pre><code class="lang-auto">for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)
                .....
</code></pre>
<p>If I concatenate the three images in one tensor and pass along a single label, does the model understand it’s actually three images and treat it as such ? I feel a bit lost here.</p>

ptrblck: <p>I would rather write a custom <code>Dataset</code> and return these three images in your <code>__getitem__</code> method:</p>
<pre data-code-wrap="python"><code class="lang-python">def __getitem__(self, index):
    # load the images and your labels according to your code logic
    ...
    reutrn image1, image2, image3, target
    # or concatenate the images and return them as a single one
    return images, target
</code></pre>
<aside class="quote no-group" data-username="styx97" data-post="10" data-topic="51284">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/styx97/48/13651_2.png" class="avatar"> styx97:</div>
<blockquote>
<p>does the model understand it’s actually three images and treat it as such ?</p>
</blockquote>
</aside>
<p>I’m not sure to understand this question properly.<br>
Your current code snippet will work, but I’m not sure what your use case is.</p>

styx97: <p>My use case is that I’m cropping certain portions of frames from a video and storing them in folders (hence the images have to be from the same folder) . Then three images (which are frames of the same video) are passed through three models before the models join at the fc layers.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="51284">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Your current code snippet will work, but I’m not sure what your use case is.</p>
</blockquote>
</aside>
<p>I was not sure if it would work, thank you.</p>

ptrblck: <aside class="quote no-group" data-username="styx97" data-post="12" data-topic="51284">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/styx97/48/13651_2.png" class="avatar"> styx97:</div>
<blockquote>
<p>Then three images (which are frames of the same video) are passed through three models before the models join at the fc layers.</p>
</blockquote>
</aside>
<p>That should work with your code snippet, however you could alternatively also use a single base model and pass these 3 frames through the same model and concatenate the output afterwards.</p>

styx97: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I tried both <code>return x, target</code>  and return <code>x[0],x[1],x[2],target</code> and the error I got was -</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-83-e03efae61dea&gt; in &lt;module&gt;
----&gt; 1 model, history_model = train_model(model, criterion, optimizer, exp_lr_scheduler, num_epochs=10)

&lt;ipython-input-47-f9a9be299e22&gt; in train_model(model, criterion, optimizer, scheduler, num_epochs)
     34                 # track history if only in train
     35                 with torch.set_grad_enabled(phase == 'train'):
---&gt; 36                     outputs = model(inputs)
     37                     #print(outputs.shape)
     38                     _, preds = torch.max(outputs, 1)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

TypeError: forward() missing 2 required positional arguments: 'image2' and 'image3'
</code></pre>
<p>in both the cases.</p>

ptrblck: <p>Since your <code>forward</code> definition is defined as:</p>
<pre><code class="lang-python">def forward(self,image1,image2,image3)
</code></pre>
<p>you should pass <code>inputs</code> separately or unwrap it:</p>
<pre><code class="lang-python">outputs = model(*inputs)
# or
outputs = model(inputs[0], inputs[1], inputs[3]) # index in the dimemsion you've concatenated the inputs
</code></pre>

styx97: <p>Thanks a lot.</p>
<p>Let me try that and get back to you.</p>

styx97: <p>For some reason, the size of <code>inputs</code> in <code>model(*inputs)</code> is getting dependent on the batch size.</p>
<p>For a batch size of 16, doing <code>model(*inputs)</code> gives me <code>Expected 3 inputs, got 16</code>. I must have  made a mistake.</p>

ptrblck: <p>My code snippet was probably wrong.<br>
Could you check the shape of <code>inputs</code> and split it in the right dimension?<br>
E.g. if your inputs have the shape <code>[batch_size, 3, ...]</code> split it in dim1.</p>

styx97: <p>Yes the inputs have the shape - <code>torch.Size([16, 3, 224, 224])</code></p>
<p>What do I put as the <code>split size or sections</code> in <code>torch.split( inputs , split_size_or_sections, dim=1 )</code> ?</p>

ptrblck: <p>You could use the following (depending on the expected shape inside <code>forward</code>):</p>
<pre><code class="lang-python">outputs = model(*x.split(1, dim=1))  # each input will have shape [16, 1, 224, 224]
outputs = model(x[:, 0], x[:, 1], x[:, 2])  # each input will have shape [16, 224, 224]
</code></pre>

styx97: <p>Using the first option, I get the error</p>
<hr>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-118-e03efae61dea&gt; in &lt;module&gt;
----&gt; 1 model, history_model = train_model(model, criterion, optimizer, exp_lr_scheduler, num_epochs=10)

&lt;ipython-input-117-f1d1a61a074d&gt; in train_model(model, criterion, optimizer, scheduler, num_epochs)
     36                 with torch.set_grad_enabled(phase == 'train'):
     37                     #outputs = model(inputs[:, 0], inputs[:, 1], inputs[:, 2])
---&gt; 38                     outputs = model(*inputs.split(1, dim=1))
     39                     #outputs = model(inputs)
     40                     #print(outputs.shape)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

&lt;ipython-input-11-f590b21f7c95&gt; in forward(self, image1, image2, image3)
     11 
     12     def forward(self,image1,image2,image3):
---&gt; 13         x1 = self.model1(image1)
     14         x2 = self.model2(image2)
     15         x3 = self.model3(image3)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torchvision/models/vgg.py in forward(self, x)
     40 
     41     def forward(self, x):
---&gt; 42         x = self.features(x)
     43         x = self.avgpool(x)
     44         x = x.view(x.size(0), -1)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     90     def forward(self, input):
     91         for module in self._modules.values():
---&gt; 92             input = module(input)
     93         return input
     94 

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    336                             _pair(0), self.dilation, self.groups)
    337         return F.conv2d(input, self.weight, self.bias, self.stride,
--&gt; 338                         self.padding, self.dilation, self.groups)
    339 
    340 

RuntimeError: Given groups=1, weight of size 64 3 3 3, expected input[16, 1, 224, 224] to have 3 channels, but got 1 channels instead
</code></pre>
<p>This is strange, the torch tensor originally had a size of [16,3,224,224]. Is the problem in the architecture of the model ?</p>
<p>Using the second option I get a similar error where it says the expected tensor is 4d and [16, 224, 224] is 3d.</p>
<p>​</p>