FC layer(linear) output and train loss increasing

username: banikr
cooked: <p>Hello All,<br>
I am using ResNet3D architecture with some changes:</p>
<pre><code>    x = self.lastConv2(x)
    x = self.relu(x)
    x = self.lastConv1(x)
    x = self.relu(x)
    x = self.lastConv0(x)
    x = self.relu(x)
    x = x.view(x.size(0), -1)
    x = self.dropout(x) #20%
    x = self.fc(x)
    x = torch.tanh(x)
    return x #torch.tanh(x)
</code></pre>
<p>I am facing a strange issue like the output of the FC layer is increasing in every epoch which is increasing the train MSE loss.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cd796a8ebc69178813fac9921b1af4476929e6a.png" data-download-href="https://discuss.pytorch.org/uploads/default/9cd796a8ebc69178813fac9921b1af4476929e6a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cd796a8ebc69178813fac9921b1af4476929e6a_2_464x500.png" alt="image" data-base62-sha1="mnunF4EOUp5LLf6WJ2BsjnXmC82" width="464" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cd796a8ebc69178813fac9921b1af4476929e6a_2_464x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cd796a8ebc69178813fac9921b1af4476929e6a_2_696x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cd796a8ebc69178813fac9921b1af4476929e6a.png 2x" data-dominant-color="FCE1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">812×997 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And the output of tanh ultimately becomes very clipped at +1 to -1 which I understand that tanh will be clipped at +1 and -1 since the inputs are very high.</p>
<p>I am using Xavier weight initialization and all the inputs are normalized from -1 to +1 scale.</p>
<pre><code class="lang-auto">def weights_init(m):
    if isinstance(m, nn.modules.conv._ConvNd): #nn.Conv3d
        init.xavier_uniform_(m.weight.data, gain = np.sqrt(2.0))
        # m.bias.data.fill_(0) #todo: if bias = False then comment out the line
        # torch.nn.init.xavier_uniform_(m.bias.data)
    elif isinstance(m, nn.modules.batchnorm._BatchNorm):
        # print('OK')
        m.weight.data.normal_(mean=1.0, std=0.02)
        # m.bias.data.fill_(0) #todo: if bias = False then comment out the line
    elif isinstance(m, nn.Linear):
        # m.weight.data.normal_(0.0, 0.02)
        # init.xavier_uniform_(m.weight.data)
        y = 1/np.sqrt(m.in_features)
        m.weight.data.uniform_(-y, y)
        m.bias.data.fill_(0) #0.01
</code></pre>
<p>And for the network and loss:</p>
<pre><code class="lang-auto">    Net = ResNet(Bottleneck, [1, 1, 1, 1]).to(device)
    Net.apply(weights_init)
    Optimizer = optim.Adam(Net.parameters(), lr=0.0005)
    Criterion = nn.MSELoss()
    Criterion = Criterion.cuda()
</code></pre>
<p>And about normalizing:</p>
<pre><code class="lang-auto">from skimage.exposure import rescale_intensity
MRimage = rescale_intensity(MRimage, in_range=(MRimage.min(), MRimage.max()),out_range=(-1,1))
target = rescale_intensity(target, in_range=(target.min(), target.max()), out_range=(-1,1))
</code></pre>
<p>What could be happening inside??</p>

username: ptrblck
cooked: <p>What range are your targets using?<br>
If the targets are “classification” targets, i.e. zeros and ones, I would expect the model to try to output very high activations to saturate the <code>tanh</code> and this match the target values.</p>

username: banikr
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
The targets are in range of -1 to +1.<br>
<code>target = rescale_intensity(target, in_range=(target.min(), target.max()),                             out_range=(-1, 1))</code><br>
So the targets are not simply 0 or 1. Rather they are some values.</p>

username: banikr
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I think I have figured the mishap I was making.<br>
I did not use Batchnorm after the lastConv layers and without normalization, I guess that is the reason the outputs were rocketing.</p>
<pre><code class="lang-auto">        x = self.bnl2(self.lastConv2(x))
        x = self.relu(x)
        x = self.bnl1(self.lastConv1(x))
        x = self.relu(x)
        x = self.bnl0(self.lastConv0(x))
        x = self.relu(x)
        x = x.view(x.size(0), -1)
        x = self.dropout(x)
        x = self.fc1(x)
        x = self.dropout(x) %40
        x = self.fc2(x)
        return x
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f283557d6d2fd28f7138f92697d89ebbc382a7f.png" alt="image" data-base62-sha1="2a5j6jMWBEWsMpRlgmqiPMP7H4z" width="320" height="240"></p>
<p>Though I guess the network overfits right after epoch: 2 but this is kinda normal problem.</p>
<p>Let me know your thoughts… I will provide more details if you need.<br>
Thanks</p>