Any suggestions for regression task in deep learning?

username: Wong_Marco
cooked: <h3><a name="background-1" class="anchor" href="#background-1"></a>background</h3>
<p>I am studying how to use video data to predict temporal regression value. Firstly, I tried using some backbone with pretrained parameters. Here is a part of my code.</p>
<pre><code class="lang-auto">class Mynet_LSTM(nn.Module):
   def __init__(self, model, num_classes):
       super().__init__()
       self.featureModel = nn.Sequential(*list(model.children())[:-1])
       self.linear = nn.Sequential(nn.LayerNorm(2048), nn.Linear(2048, 256))
       self.rnn = nn.LSTM(256, 128, 2, bidirectional=True, dropout=0.3)
       self.classifier = nn.Sequential(nn.LayerNorm(256), nn.Linear(256, num_classes))

       for m in self.modules():
           if isinstance(m, nn.Linear):
               n = m.in_features
               y = 1.0 / np.sqrt(n)
               m.weight.data.uniform_(-y, y)
               m.bias.data.fill_(0)

   def forward(self, x):
       b, c, t, h, w = x.shape
       x = einops.rearrange(x, 'b c t h w -&gt; (b t) c h w', b=b, t=t)
       x = self.featureModel(x).squeeze()  # -&gt; (b t), 2048
       x = self.linear(x)  # -&gt; (b t), 256
       x = einops.rearrange(x, '(b t) d -&gt; b t d', b=b, t=t)
       x, _ = self.rnn(x)
       x = einops.rearrange(x, 'b t d -&gt; (b t) d', b=b, t=t)
       x = self.classifier(x)
       x = einops.rearrange(x, '(b t) d -&gt; b t d', b=b, t=t)
       return x

# this is an example how to use my Net
model = torch.hub.load('facebookresearch/WSL-Images', 'resnext101_32x8d_wsl')
mymodel = Mynet_LSTM(model, 2)
frame_for_example = torch.rand(2, 3, 16, 400, 400) # b, c, t, h, w
</code></pre>
<p>The result are not good as I expected as shown in the picture below. In the training time, it worked so efficiently, but then the performing became worse during the validating time. After few days of debugging, I found out that this is because no matter what the input data is, my network will always show the output as “the last given ground_truth”. By the way, I am using nn.MSELoss() as my criterion.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72560163131aa51d94602c4e8afad176e0b0ffc5.png" data-download-href="https://discuss.pytorch.org/uploads/default/72560163131aa51d94602c4e8afad176e0b0ffc5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72560163131aa51d94602c4e8afad176e0b0ffc5_2_690x476.png" alt="image" data-base62-sha1="gjsHeLqR44ddA0Hpps1N5oYtHmJ" width="690" height="476" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72560163131aa51d94602c4e8afad176e0b0ffc5_2_690x476.png, https://discuss.pytorch.org/uploads/default/original/3X/7/2/72560163131aa51d94602c4e8afad176e0b0ffc5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/2/72560163131aa51d94602c4e8afad176e0b0ffc5.png 2x" data-dominant-color="EEEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">706×488 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>the blue line is my target value and the orange one is my prediction. Using MSE as my criterion.</p>
</blockquote>
<p>In the case of using CCC as my criterion, the metrics for my tasks should commonly be CCC as well. My results are shown below; even keep training for more than 5-6 epoch, the results are still the same.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f5c51ac0cebb2e956f608bf2b60087da88f31f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f5c51ac0cebb2e956f608bf2b60087da88f31f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f5c51ac0cebb2e956f608bf2b60087da88f31f9.png" alt="image" data-base62-sha1="2bSXbKuwJSOdWbddiuv08bIDsut" width="574" height="500" data-dominant-color="F5D4BD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">740×644 22.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>the blue line is my target value and the orange one is my prediction. Using CCC I found in the github as my criterion.</p>
</blockquote>
<h3><a name="question-2" class="anchor" href="#question-2"></a>question</h3>
<p>I would like to know why this strange result happened?<br>
And is there any suggestion for me to train the regression network?<br>
Thanks<br>
–Joanna</p>