Loading image frames with corresponding heartrate data in csv file (Python Pytorch Model Implementation)

username: Parminder_kaur
cooked: <p>Hi there,</p>
<p>I have images in folder and having corresponding hr_data labels in csv file. I want to use dataloader to load images with corresponding target labels. Do I need to implement custom dataset class for it? I am new to pytorch. Please help me in this regard. I would really appreciate it. Can anyone share the method to do so?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>I would recommend to write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.</p>
<p>The general workflow is:</p>
<ul>
<li>load the data paths or the data directly (if it fits into memory and/or is small) in the <code>__init__</code> method</li>
<li>return the length of the dataset in <code>__len__</code>
</li>
<li>load, process, and return each data-target pair in <code>__getitem__</code> using the passed <code>index</code>.</li>
</ul>
<p>In your case you would most likely load the <code>csv</code> data in the <code>__init__</code> method and index it in the <code>__getitem__</code> using the <code>index</code>.<br>
Also, the lazy image loading and processing would take place in the <code>__getitem__</code> method.</p>

username: Parminder_kaur
cooked: <p>I tried the way you told me but I am not getting the correct results.</p>
<p>Note: I want my image tensor to be of shape [3, T, 128,128] where T is no of images and 3 is channels and 128, 128 are width and height</p>
<p><strong>My code:</strong></p>
<p>transform = transforms.Compose(</p>
<pre><code>[transforms.ToTensor(),

 transforms.Resize(128)

])
</code></pre>
<p>from torch.utils.data import Dataset, DataLoader</p>
<p>from torch import from_numpy, tensor</p>
<p>import numpy as np</p>
<p>class UBFCDataset(Dataset):</p>
<pre><code>def __init__(self):

    xy = np.loadtxt('/content/drive/My Drive/Subject3hr.csv')

    self.x_data = torch.from_numpy(xy[:])

    self.len = xy.shape[0]

def __len__(self):

    return self.len

def __getitem__(self, index):

    train_data_tensor = torchvision.datasets.ImageFolder("/content/drive/My Drive/Meta-rPPG-master/da", transform=transforms)

    return train_data_tensor, self.x_data
</code></pre>
<p>dataset = UBFCDataset()</p>
<p>train_loader = DataLoader(dataset=dataset,batch_size=32,shuffle=True)</p>
<p>for data in train_loader:</p>
<pre><code>    inputs, labels = data

    inputs, labels = tensor(inputs), tensor(labels)
</code></pre>
<p><strong>Error:</strong></p>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘torchvision.datasets.folder.ImageFolder’&gt;</p>

username: ptrblck
cooked: <p>The <code>ImageFolder</code> creation should be in the <code>__init__</code> and in <code>__getitem__</code> you would index the <code>ImageFolder</code>.<br>
Alternatively, you could also load the images manually using a <code>list</code> of image paths in <code>__getitem__</code>.</p>

username: Parminder_kaur
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I am getting the proper tensors for my images tensor [batch,channel,width,height]  and also getting my label tensor but When I am trying to feed that into my model I am getting the following error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88bc7787ae16b43021c9632fe0b799bfdd66674f.png" data-download-href="https://discuss.pytorch.org/uploads/default/88bc7787ae16b43021c9632fe0b799bfdd66674f" title="Screenshot (101)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88bc7787ae16b43021c9632fe0b799bfdd66674f_2_690x326.png" alt="Screenshot (101)" data-base62-sha1="jvCIUUoombbNVX0AExxucAbIauj" width="690" height="326" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88bc7787ae16b43021c9632fe0b799bfdd66674f_2_690x326.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88bc7787ae16b43021c9632fe0b799bfdd66674f_2_1035x489.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88bc7787ae16b43021c9632fe0b799bfdd66674f_2_1380x652.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (101)</span><span class="informations">1920×908 67.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The <code>ImageFolder</code> will return a data and target sample, which will be stored in <code>frame</code>, while <code>labels</code> contain your “real” target tensor.<br>
I guess you want to remove the target tensor returned by the <code>ImageFolder</code> dataset, so use:</p>
<pre><code class="lang-python">frame = frame[0]
</code></pre>
<p>to only keep the returned data tensor and pass it to the model.</p>
<p>PS: it’s better to post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Parminder_kaur
cooked: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>. Your solution worked. Now my model is not accepting my inputs tensor shape. It is throwing following error.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-31-84d3b7e54446&gt; in &lt;module&gt;()
     44         # here: 178 samples, batch_size = 4, n_iters=178/4=44.5 -&gt; 45 iterations
     45         model = PhysNet_padding_Encoder_Decoder_MAX(frames=128)
---&gt; 46         rPPG, x_visual, x_visual3232, x_visual1616 = model(inputs)

1 frames
&lt;ipython-input-23-fe7087e31db6&gt; in forward(self, x)
     85     def forward(self, x):               # x [3, T, 128,128]
     86         x_visual = x
---&gt; 87         [batch,channel,length,width,height] = x.shape
     88 
     89         x = self.ConvBlock1(x)               # x [3, T, 128,128]

ValueError: not enough values to unpack (expected 5, got 4)
</code></pre>

username: ptrblck
cooked: <p><code>x</code> has 4 dimensions, while you are trying to unpack the <code>size</code> into 5 values.<br>
I guess the <code>length</code> dimension is not defined.</p>

username: Parminder_kaur
cooked: <p>Yeah, but I need to get the length dimension which means total length of the inputs, How can I get the tensor shape right?</p>
<p>Here is the model in which I am trying to feed my inputs:</p>
<pre><code class="lang-auto">
Only for research purpose, and commercial use is not allowed.
MIT License
Copyright (c) 2019 
'''



import math
import torch.nn as nn
from torch.nn.modules.utils import _triple
import torch
import pdb



class PhysNet_padding_Encoder_Decoder_MAX(nn.Module):
    def __init__(self, frames=128):  
        super(PhysNet_padding_Encoder_Decoder_MAX, self).__init__()
        
        self.ConvBlock1 = nn.Sequential(
            nn.Conv3d(3, 16, [1,5,5],stride=1, padding=[0,2,2]),
            nn.BatchNorm3d(16),
            nn.ReLU(inplace=True),
        )

        self.ConvBlock2 = nn.Sequential(
            nn.Conv3d(16, 32, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(32),
            nn.ReLU(inplace=True),
        )
        self.ConvBlock3 = nn.Sequential(
            nn.Conv3d(32, 64, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
        )
        
        self.ConvBlock4 = nn.Sequential(
            nn.Conv3d(64, 64, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
        )
        self.ConvBlock5 = nn.Sequential(
            nn.Conv3d(64, 64, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
        )
        self.ConvBlock6 = nn.Sequential(
            nn.Conv3d(64, 64, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
        )
        self.ConvBlock7 = nn.Sequential(
            nn.Conv3d(64, 64, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
        )
        self.ConvBlock8 = nn.Sequential(
            nn.Conv3d(64, 64, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
        )
        self.ConvBlock9 = nn.Sequential(
            nn.Conv3d(64, 64, [3, 3, 3], stride=1, padding=1),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True),
        )
        
        self.upsample = nn.Sequential(
            nn.ConvTranspose3d(in_channels=64, out_channels=64, kernel_size=[4,1,1], stride=[2,1,1], padding=[1,0,0]),   #[1, 128, 32]
            nn.BatchNorm3d(64),
            nn.ELU(),
        )
        self.upsample2 = nn.Sequential(
            nn.ConvTranspose3d(in_channels=64, out_channels=64, kernel_size=[4,1,1], stride=[2,1,1], padding=[1,0,0]),   #[1, 128, 32]
            nn.BatchNorm3d(64),
            nn.ELU(),
        )
 
        self.ConvBlock10 = nn.Conv3d(64, 1, [1,1,1],stride=1, padding=0)
        
        self.MaxpoolSpa = nn.MaxPool3d((1, 2, 2), stride=(1, 2, 2))
        self.MaxpoolSpaTem = nn.MaxPool3d((2, 2, 2), stride=2)
        
        
        #self.poolspa = nn.AdaptiveMaxPool3d((frames,1,1))    # pool only spatial space 
        self.poolspa = nn.AdaptiveAvgPool3d((frames,1,1))

        
    def forward(self, x):	    	# x [3, T, 128,128]
        x_visual = x
        [batch,channel,length,width,height] = x.shape
          
        x = self.ConvBlock1(x)		     # x [3, T, 128,128]
        x = self.MaxpoolSpa(x)       # x [16, T, 64,64]
        
        x = self.ConvBlock2(x)		    # x [32, T, 64,64]
        x_visual6464 = self.ConvBlock3(x)	    	# x [32, T, 64,64]
        x = self.MaxpoolSpaTem(x_visual6464)      # x [32, T/2, 32,32]    Temporal halve
        
        x = self.ConvBlock4(x)		    # x [64, T/2, 32,32]
        x_visual3232 = self.ConvBlock5(x)	    	# x [64, T/2, 32,32]
        x = self.MaxpoolSpaTem(x_visual3232)      # x [64, T/4, 16,16]
        

        x = self.ConvBlock6(x)		    # x [64, T/4, 16,16]
        x_visual1616 = self.ConvBlock7(x)	    	# x [64, T/4, 16,16]
        x = self.MaxpoolSpa(x_visual1616)      # x [64, T/4, 8,8]

        x = self.ConvBlock8(x)		    # x [64, T/4, 8, 8]
        x = self.ConvBlock9(x)		    # x [64, T/4, 8, 8]
        x = self.upsample(x)		    # x [64, T/2, 8, 8]
        x = self.upsample2(x)		    # x [64, T, 8, 8]
        
        
        x = self.poolspa(x)     # x [64, T, 1,1]    --&gt;  groundtruth left and right - 7 
        x = self.ConvBlock10(x)    # x [1, T, 1,1]
        
        rPPG = x.view(-1,length)            
        

        return rPPG, x_visual, x_visual3232, x_visual1616
</code></pre>

username: ptrblck
cooked: <p>You could write a custom <code>Dataset</code> and concatenate multiple images to the desired shape.<br>
<code>ImageFolder</code> will create samples using single image tensors and their corresponding target.</p>
<p>The more interesting part would be, how should the <code>length</code> be defined? Would you like to concatenate random images? If so, you could also increase the batch size and reshape the data such that the batch size will be split into a smaller batch size and the length dimension.<br>
On the other hand, if the length dimension would have any specific “logic” such as sequential frames, I would recommend to stick to the custom <code>Dataset</code> approach and implement the loading logic in its <code>__getitem__</code>.</p>

username: Parminder_kaur
cooked: <p>I currently have following tensor with dimensions[batch_size, channel, width, height] but I want to add one dimension to the above tensors at third index as [batch_size, channel, length, width, height] where length is the dimension that I want to add into my tensor. Length value will be defined as the total length of inputs Can you please tell me how should I add need dimension and set its value. Please help me with this regard.</p>

username: ptrblck
cooked: <p>You can add the dimension via <code>tensor = tensor.unsqueeze(2)</code>, but this would only add this dimension with a size of 1.<br>
I don’t know how this dimension should be filled, but if you would like to use multiple image tensors and stack them in this dimension, you could use:</p>
<pre><code class="lang-python">data = [torch.randn(1, 3, 24, 24) for _ in range(10)]
x = torch.stack(data, dim=2)
print(x.shape)
&gt; torch.Size([1, 3, 10, 24, 24])
</code></pre>

username: Parminder_kaur
cooked: <p>Thank you I have got the following code working for my model. Can you help me in getting the accuracy for the model after every epoch. I am getting tensors as a result. How to compare and get the results. I am using 1792 input images corresponding to labels.</p>
<pre><code class="lang-auto">for i in range(EPOCHS):
  for batch_num in range(total_batch_num):
      inputs=np.reshape(x_train[batch_num,:,:,:,:], [1,CHANNELS,BATCH_SIZE,IMAGE_HEIGHT,IMAGE_WIDTH])
      Target = np.reshape(y_train[batch_num,:],[1,BATCH_SIZE])
      inputs = torch.from_numpy(inputs).float()
      Target = torch.from_numpy(Target).float()
      rPPG, x_visual, x_visual3232, x_visual1616 = model(inputs)
      rPPG = (rPPG-torch.mean(rPPG)) /torch.std(rPPG)	 	# normalize   
      Target = (Target-torch.mean(Target)) /torch.std(Target)    
      loss = criterion(rPPG,Target)     
      loss = loss.detach().numpy()
      print('Training: loss=' + str(loss))
      torch.save(model, 'save.model')
      print('A model has been saved')
</code></pre>
<p>where predicted value is rPPG and true value is Target</p>

username: ptrblck
cooked: <p><code>rPPG</code> doesn’t seem to contain the predicted class indices and based on the operation on it and the target (mean subtraction) I guess you are working with a regression task?<br>
If so, then you could calculate the loss (e.g. via <code>nn.MSELoss</code> or any other criterion suitable for a regression task) as I’m unsure how the accuracy would be defined in this case.</p>

username: Parminder_kaur
cooked: <p>This is a regression task. I have already calculated the loss but was wondering if I can find out the accuracy of it. Can I define confusion matrix for it? If yes how can I do that?</p>

username: ptrblck
cooked: <p>I’m still unsure how you would calculate the accuracy and and how the confusion matrix should look like.<br>
Since you are working with (infinite) floating point values, comparing the prediction and target directly won’t work without binning and also the confusion matrix would have an infinite shape.<br>
Do you want to transform this regression task somehow into a classification?</p>