3D CNN overfittting issue

Mukesh1729: <p>Hi,<br>
I am trying to retrain a 3D CNN model from a research article and I run into overfitting issues even upon implementing data augmentation on the fly to avoid overfitting.</p>
<p>I can see that my model learns and then starts to oscillate along the same loss numbers.</p>
<p>Any suggestions on how to improve or how I should proceed in preventing the model from overfitting will be of great help. For further reference, I am posting my code below and the loss. Thanks</p>
<p>What I tried to avoid overfitting:</p>
<ul>
<li>image augmentation,</li>
<li>regularization.</li>
<li>Dropout layers</li>
<li>Increasing the batch size</li>
<li>Shuffling the data-loader</li>
</ul>
<p>Training dataset size: approx 358 images+augmentation(every epoch)</p>
<pre><code class="lang-auto">class T_LEAP(torch.nn.Module):
    """T_LEAP ARCHITECTURE"""

    def __init__(self):
        super(T_LEAP, self).__init__()
        self.encoder = torch.nn.Sequential(
                                          torch.nn.Conv3d(3, 64, kernel_size=(3,3,3), stride=(1,1,1), padding=(1,1,1)),
                                          torch.nn.BatchNorm3d(num_features=64),
                                          torch.nn.ReLU(inplace=True),
                                          torch.nn.MaxPool3d(kernel_size=(1,2,2),stride=(1,2,2)),
                                          
                                          
                                           
                                          torch.nn.Conv3d(64, 128, kernel_size=(3,3,3), stride=(1,1,1), padding=(1,1,1)),                                         
                                          torch.nn.BatchNorm3d(num_features=128),
                                          torch.nn.ReLU(inplace=True),
                                          torch.nn.MaxPool3d(kernel_size=(2,2,2),stride=(2,2,2)),
                                          
                                           
                                          torch.nn.Conv3d(128, 256, kernel_size=(3,3,3), stride=(1,1,1), padding=(1,1,1)),
                                          torch.nn.BatchNorm3d(num_features=256),
                                          torch.nn.ReLU(inplace=True),
                                          torch.nn.MaxPool3d(kernel_size=(1,2,2),stride=(1,2,2)),
                                         
                                           
                                          torch.nn.Conv3d(256, 512, kernel_size=(1,3,3), stride=(1,1,1), padding=(1,1,1)),
                                          torch.nn.BatchNorm3d(num_features=512),
                                          torch.nn.ReLU(inplace=True),
                                          
                                          
                                          )
        
        self.decoder = torch.nn.Sequential(
                                          torch.nn.ConvTranspose3d(512, 256, kernel_size=(1,3,3), stride = (1,2,2), padding=(1,1,1), output_padding=(0,1,1)),
                                          torch.nn.BatchNorm3d(num_features=256), 
                                          torch.nn.ReLU(inplace=True),
                                     
                                          torch.nn.Conv3d(256, 256, kernel_size=(1,3,3), stride=(1,1,1), padding=(1,1,1)),
                                          torch.nn.BatchNorm3d(num_features=256),
                                          torch.nn.ReLU(inplace=True),
        
                                          torch.nn.ConvTranspose3d(256,128, kernel_size=(1,3,3), stride = (1,2,2), padding=(1,1,1),output_padding=(0,1,1)), 
                                          torch.nn.BatchNorm3d(num_features=128),
                                          torch.nn.ReLU(inplace=True),
                                          
                                          torch.nn.Conv3d(128, 128, kernel_size=(1,3,3), stride=(1,1,1), padding=(1,1,1)),
                                          torch.nn.BatchNorm3d(num_features=128),
                                          torch.nn.ReLU(inplace=True),
                                          
                                          torch.nn.ConvTranspose3d(128,17, kernel_size=(1,3,3), stride = (1,2,2), padding=(1,1,1),output_padding=(0,1,1)),
                                          torch.nn.BatchNorm3d(num_features=17),
                                          torch.nn.ReLU(inplace=True),
                                          )           
               
    def forward(self, image):
        """PUTTING THE MODEL TOGETHER"""
        encoder = self.encoder(image)
        decoder = self.decoder(encoder)
        out     = torch.nn.Softmax(dim=1)(decoder)
        return out


model = T_LEAP()
if torch.cuda.is_available():
  input = torch.rand(1,3, 2, 200, 200).cuda()
  model = model.cuda()
  print(model(input).shape)
else:
  summary(model,
          input_size=(3, 2, 200, 200),
          batch_size=1
          )
</code></pre>
<p>Training loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e0c27c37f23108da63e2888e056c701e69fcac1.png" alt="image" data-base62-sha1="b8rhGVTduTgVEUCO8tHrX2LUYHT" width="249" height="292"></p>
<p>other stuff i checked:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57b8fdd0593fb885f9a5884878000023249078ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/57b8fdd0593fb885f9a5884878000023249078ec" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57b8fdd0593fb885f9a5884878000023249078ec.png" alt="image" data-base62-sha1="cw1SHzZZQBSMfDUr9Otwg4iWOhS" width="330" height="500" data-dominant-color="EBEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">434×657 11.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Training loop:</p>
<pre><code class="lang-auto">DEVICE     = "cuda" if torch.cuda.is_available() else "cpu"
EPOCHS     = 35
lr         = 1e-3
model      = T_LEAP()
optimizer  = torch.optim.Adam(model.parameters(), lr=lr)
model = model.to(DEVICE)
criterion  = torch.nn.MSELoss()

train_loss = []
for epoch in range(EPOCHS): 
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # Get[inputs, labels]
        inputs, labels = data["images"], data["heatmaps"]

        inputs = inputs.permute(0,2,1,3,4).to(DEVICE,dtype=torch.float)
        labels = labels.unsqueeze(0).permute(1,4,0,2,3).to(DEVICE,dtype=torch.float)
        
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

    print("loss:",running_loss/len(train_loader))
    train_loss.append(running_loss)
        
print('Finished Training')
</code></pre>

my3bikaht: <p>How do you estimate overfitting? I don’t see any validation loss here.</p>

Mukesh1729: <p>This was one of the runs that i made on the data and the model performs like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bd00a15ce6a27e3d7eb4eafc68901d610bdd3b.png" data-download-href="https://discuss.pytorch.org/uploads/default/09bd00a15ce6a27e3d7eb4eafc68901d610bdd3b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09bd00a15ce6a27e3d7eb4eafc68901d610bdd3b_2_690x401.png" alt="image" data-base62-sha1="1o9escd431N3ro4IIY9bdGopJOP" width="690" height="401" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/09bd00a15ce6a27e3d7eb4eafc68901d610bdd3b_2_690x401.png, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bd00a15ce6a27e3d7eb4eafc68901d610bdd3b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bd00a15ce6a27e3d7eb4eafc68901d610bdd3b.png 2x" data-dominant-color="FBFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">731×425 19.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

my3bikaht: <p>It doesn’t look like overfitting since model is unable to generalize at all, according to val loss.<br>
Your weights are rather small, usually this happens when model decides that best choice here is to provide constant output (usually zero). I’d look into scaling/normalizing data, simplifying model and descreasing Adam lr further.</p>

Mukesh1729: <p>I have scaled the data using ToTensor() and have varied the learning rate of the optimizer from 1e-3 to 1e5 and the train loss decreases at higher learning rates 1e-5 but doesn’t learn much  as compared to a learning rate of 1e-3.</p>
<p>I also noticed that when I print my loss.backward() while training and I get none as the output. Would this mean that my model is not learning at all?</p>

JuanFMontesinos: <p>To me it seems strange you are using softmax + mse loss.<br>
You are computing some sort of probabilities but then you use a cartesian distance? Strange</p>
<p>Are you using the same dataset or a different one? You have convolutions with 512 filters which leads to a huge amount of a parameters for a 3D nn. Be sure you have the proper amount of parameters given the size of your dataset as 3D convolutions tend to overfit.</p>
<p>Despite the scale of the plot is not proper, it seems even the training is not learning much if it can converge in 10 epochs.<br>
Lastly it would be nice if you use any other metric to measure “overfitting”. Some sort of accuracy would be nice as you can measure if it better than by chance.</p>
<p>Lastly, just random suggestion, are you using a validation set extracted from the same type of data? If you use 2 different datasets (rather than splitting one into train and val) you may find this.</p>

Mukesh1729: <p>Hey Juan,<br>
Thanks for getting back and sharing your suggestions.</p>
<ul>
<li>
<p><strong>To me it seems strange you are using softmax + mse loss.</strong><br>
<strong>You are computing some sort of probabilities but then you use a cartesian distance?</strong> - The reason for using softmax plus mse loss is a direct implementation of the paper (<a href="https://arxiv.org/abs/2104.08029" class="inline-onebox" rel="noopener nofollow ugc">[2104.08029] T-LEAP: occlusion-robust pose estimation of walking cows using temporal information</a>). It is for a pose estimation problem so that I can have confidence maps as the output.</p>
</li>
<li>
<p><strong>Lastly, just random suggestion, are you using a validation set extracted from the same type of data? If you use 2 different datasets (rather than splitting one into train and Val) you may find this. -</strong>  I am using a personal dataset and splitting the same data into an 80:20 train and validation dataset. I am using different videos as one dataset</p>
</li>
<li>
<p><strong>Despite the scale of the plot is not proper, it seems even the training is not learning much if it can converge in 10 epochs.</strong> - I have trained for 80 epochs and the loss remains on the same scale. However, I realized that when I print loss.backward() i get None value as output which also means that the network is not training.</p>
</li>
</ul>

JuanFMontesinos: <p>Well I’m not sure about what’s the return of loss.backward()<br>
It can be None if the method “backward” returns nothing . The requirement is loss is not NaN.<br>
Soo if you are using a custom dataset, how big is it compared to the original one? You may want to reduce the amount of convolutions / filters you are using.</p>
<p>If your loss or any of the inner computations were NaN or Inf, wou values of the weights wo. Would also be NaN. Basically, when a NaN falls in the process it spreads. I think it’s just regular overfitting.</p>
<p>Ofc I’m neither sure about the range of values your loss should be reaching but 10e-3 seems already small. I would bet it reached those values within the first epoch.</p>

Mukesh1729: <ul>
<li>
<strong>Soo if you are using a custom dataset, how big is it compared to the original one</strong>- The paper says they have used samples of four consecutive frames resulting in 1059 samples(assuming 1059*4 frames). In my case, I am using 387 images + data augmentation per epoch. With data augmentations, I once generate 1059 frames and it still got stuck. Like you said the network might need more data or need to be reduced in size</li>
<li>
<strong>Ofc I’m neither sure about the range of values your loss should be reaching but 10e-3 seems already small. I would bet it reached those values within the first epoch.</strong> - For 1e-3 it reaches the values in the first epoch and then the loss oscillates around the next values for a long time and diminishes very slowly</li>
<li>One of my last resort ideas was to eventually increase the training data to see if it helps in any way. However, using augmentation does the trick</li>
</ul>

JuanFMontesinos: <p>Augmentation helps but it’s not magic.<br>
I would count the parameters and try with a smaller network. But still, try to measure the performance with any other mertric than the loss. All the networks (moreover the 3D CNN ones) may overfit.</p>

J_Johnson: <p>I’m more curious about your gradient sizes and model length. Maybe your model needs some split connections? Print a sum of your gradients at each layer.</p>

Mukesh1729: <p>Could you suggest any other metric? Thanks</p>

Mukesh1729: <p><strong>Model summary</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/7305b77a6870393a8a6be7ef229e19df7a80bdef.png" data-download-href="https://discuss.pytorch.org/uploads/default/7305b77a6870393a8a6be7ef229e19df7a80bdef" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/7305b77a6870393a8a6be7ef229e19df7a80bdef.png" alt="image" data-base62-sha1="gpx9JTf7J8TLBFo4U27wB9n9iP5" width="354" height="500" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">534×754 13.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried to do <strong>loss.register_hook(lambda grad: print(grad))</strong>:<br>
How do you compute the gradients at each layer?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/957727fccaba18f8d80812ad2ae10770d6da50de.png" alt="image" data-base62-sha1="lkeqUBtQWWwNJ6JjCCH68RNNtmK" width="226" height="322"></p>

JuanFMontesinos: <p>Well given the fact you are stimating joints you just need to have a look at other papers like openpose</p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1812.08008v2.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1812.08008v2.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1812.08008v2.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1812.08008v2.pdf" target="_blank" rel="noopener nofollow ugc">1812.08008v2.pdf</a></h3>

  <p class="filesize">7.88 MB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So you can basically compute accuracy (are the predicted joints matched with the proper ones), the joint-wise error, recall etcetera…</p>

Mukesh1729: <p>Hey Johnson, by split connections do you mean skip connections? I tried to add skip connections and test them.</p>
<p>My output looks like for the below architecture:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c19f0890b112a3e31769a76bbbc523572c56c5d8.png" alt="image" data-base62-sha1="rCQSwrHmwG0RnmjHMprHcjCky5W" width="476" height="248"></p>
<pre><code class="lang-auto">class T_LEAP_SKIP(torch.nn.Module):
    """T_LEAP ARCHITECTURE"""

    def __init__(self):
        super(T_LEAP_SKIP, self).__init__()
        
        self.upconv1 = torch.nn.Sequential(torch.nn.Conv3d(3, 64, kernel_size=(3,3,3), 
                                                stride=(1,1,1), padding=(1,1,1)),
                                                torch.nn.BatchNorm3d(num_features=64),
                                                torch.nn.ReLU(inplace=True),
                                                )
        self.pool1   = torch.nn.MaxPool3d(kernel_size=(1,2,2),stride=(1,2,2))

        self.upconv2 = torch.nn.Sequential(torch.nn.Conv3d(64, 128,kernel_size=(3,3,3), 
                                                    stride=(1,1,1), padding=(1,1,1)), 
                                                                            
                                      torch.nn.BatchNorm3d(num_features=128),
                                      torch.nn.ReLU(inplace=True),
                                      
                                    )
        self.pool2   = torch.nn.MaxPool3d(kernel_size=(2,2,2),stride=(2,2,2))

        self.upconv3 = torch.nn.Sequential(torch.nn.Conv3d(128, 256,
                                                           kernel_size=(3,3,3), 
                                                           stride=(1,1,1), 
                                                           padding=(1,1,1)), 
                                                                            
                                      torch.nn.BatchNorm3d(num_features=256),
                                      torch.nn.ReLU(inplace=True),
                                      
                                    )
        
        self.pool3   = torch.nn.MaxPool3d(kernel_size=(1,2,2),stride=(1,2,2))

        self.upconv4 = torch.nn.Sequential(torch.nn.Conv3d(256, 
                                                           512,
                                                           kernel_size=(1,3,3), 
                                                           stride=(1,1,1),
                                                           padding=(1,1,1)), 
                                                                            
                                      torch.nn.BatchNorm3d(num_features=512),
                                      torch.nn.ReLU(inplace=True),
                               
                                    )
        #-----------------------------------------------------------------------
        #1st conv-transpose layer 
        self.convt1 = torch.nn.Sequential(torch.nn.ConvTranspose3d(512, 
                                                                   256, 
                                                                   kernel_size=(1,3,3),
                                                                   stride = (1,2,2), 
                                                                   padding=(1,1,1),
                                                                   output_padding=(0,1,1)),
                                          torch.nn.BatchNorm3d(num_features=256), 
                                          torch.nn.ReLU(inplace=True)
                                          )
        self.down1  = torch.nn.Sequential(torch.nn.Conv3d(512, 256, kernel_size=(1,3,3), stride=(1,1,1), padding=(1,1,1)),
                                          torch.nn.BatchNorm3d(num_features=256),
                                          torch.nn.ReLU(inplace=True))
        
        
        #2nd conv-transpose layer 
        self.convt2 = torch.nn.Sequential(torch.nn.ConvTranspose3d(256, 
                                                                   128, 
                                                                   kernel_size=(1,3,3),
                                                                   stride = (1,2,2), 
                                                                   padding=(1,1,1),
                                                                   output_padding=(0,1,1)),
                                          torch.nn.BatchNorm3d(num_features=128), 
                                          torch.nn.ReLU(inplace=True)
                                          )
        
        self.down2  = torch.nn.Sequential(torch.nn.Conv3d(256,128, kernel_size=(1,3,3), stride=(1,1,1), padding=(1,1,1)),
                                          torch.nn.BatchNorm3d(num_features=128),
                                          torch.nn.ReLU(inplace=True)) 

        #Final conv-transpose layer                                   
        self.convt3 = torch.nn.Sequential(torch.nn.ConvTranspose3d(128, 
                                                                   17, 
                                                                   kernel_size=(1,3,3),
                                                                   stride = (1,2,2), 
                                                                   padding=(1,1,1),
                                                                   output_padding=(0,1,1)),
                                          torch.nn.BatchNorm3d(num_features=17), 
                                          torch.nn.ReLU(inplace=True)
                                          )

    def forward(self, image):
        """PUTTING THE MODEL TOGETHER"""
        
        # Encoder 
        block1 = self.upconv1(image)
        pool1  = self.pool1(block1)

        block2 = self.upconv2(pool1)
        pool2  = self.pool2(block2)

        block3 = self.upconv3(pool2)
        pool3  = self.pool3(block3)
      
        block4 = self.upconv4(pool3)


        #Decoder
        trans1 = self.convt1(block4)
        skip1   = torch.cat([trans1, block3], axis=1)
        up1     = self.down1(skip1)

        trans2 = self.convt2(up1)
        skip2  = torch.cat([trans2, block2[:,:,1:,:]], axis=1)   #trouble with matching depth
        up2    = self.down2(skip2)

        trans3 = self.convt3(up2)
        output = torch.nn.Softmax(dim=1)(trans3)
        return output

model = T_LEAP_SKIP()
input = torch.rand(1,3, 2, 200, 200)
model = model
print("Model output", model(input).shape)

# summary(model,
#         input_size=(3, 2, 200, 200),
#         batch_size=1
#         )
</code></pre>

J_Johnson: <p>You can get the gradients from a tensor by</p>
<p><code>tensor.grad</code></p>
<p>For the skip connections, I’d try adding between a Linear or Conv3d keeping the size identical with padding and a relu activation and batchnorm. Highway Networks are an option, too.</p>
<p>But it seems you have an issue elsewhere. I notice your targets are under the variable “labels”. So why are you using MSELoss? If this is a classification problem, shouldn’t you be using CrossEntropyLoss or BCELoss?</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a></p>

J_Johnson: <p>Nevermind, I see the “heatmaps” reference now. So you’re wanting the NN to replicate the heatmap from something like an MRI. Correct? Why are the number of features out equal to 17?<br>
That seems a little odd. A heatmap should have just one channel.</p>

Mukesh1729: <p>I have a heatmap for each joint, 17 in total that’s why 17 channels. I want to be able to compare 17 channels for each joint from the networks prediction to the target/label for a pose estimation task.</p>

J_Johnson: <p>Since it’s a 3d mapping of a heatmap, the loss could be “low” simply due to much of the space being zeroes. As dimensionality increases, so does empty space.</p>
<p>As such, the mean loss will be lower.</p>

Mukesh1729: <p>But generally pose estimation models using heatmaps are trained this way. I woul suspect that the network is too deep and has very little data to learn from.</p>