A strange kinf of overfitting

username: bruce
cooked: <p>here is my train loss and val loss, val loss is a bit of strange.<br>
however, It is definitely overfitting. My datasets only contains 1500 's 30×30 grayscale image.</p>
<p>I wonder that is there any wrong data is my datasets according to the val loss?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93af219e0c37c25c08700cb77f3e3b4c00cf9751.png" data-download-href="https://discuss.pytorch.org/uploads/default/93af219e0c37c25c08700cb77f3e3b4c00cf9751" title="gg"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93af219e0c37c25c08700cb77f3e3b4c00cf9751_2_296x500.png" alt="gg" data-base62-sha1="l4tp7S9ElQAgPR0tSXlDooUr5FD" width="296" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93af219e0c37c25c08700cb77f3e3b4c00cf9751_2_296x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/3/93af219e0c37c25c08700cb77f3e3b4c00cf9751.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/3/93af219e0c37c25c08700cb77f3e3b4c00cf9751.png 2x" data-dominant-color="FAF7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gg</span><span class="informations">386×651 67.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mailcorahul
cooked: <p>The network seems to be overfitting. Also if any garbage data is present, that could show up in your train loss.</p>
<ol>
<li>You can apply more augmentations to train.</li>
<li>Dropouts can help.</li>
<li>Like you said, you can check your val data distribution and also look for any errors in it.</li>
</ol>

username: bruce
cooked: <pre><code>class PReNet_LSTM(nn.Module):
    def __init__(self, recurrent_iter=6, use_GPU=True):
        super(PReNet_LSTM, self).__init__()
        self.iteration = recurrent_iter
        self.use_GPU = use_GPU

        self.conv0 = nn.Sequential(
            nn.Conv2d(2, 32, 3, 1, 1),
            nn.ReLU()
            )
        self.res_conv1 = nn.Sequential(
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU()
            )
        self.res_conv2 = nn.Sequential(
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU()
            )
        self.res_conv3 = nn.Sequential(
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU()
            )
        self.res_conv4 = nn.Sequential(
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU()
            )
        self.res_conv5 = nn.Sequential(
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(32, 32, 3, 1, 1),
            nn.ReLU()
            )
        self.conv_i = nn.Sequential(
            nn.Conv2d(32 + 32, 32, 3, 1, 1),
            nn.Sigmoid()
            )
        self.conv_f = nn.Sequential(
            nn.Conv2d(32 + 32, 32, 3, 1, 1),
            nn.Sigmoid()
            )
        self.conv_g = nn.Sequential(
            nn.Conv2d(32 + 32, 32, 3, 1, 1),
            nn.Tanh()
            )
        self.conv_o = nn.Sequential(
            nn.Conv2d(32 + 32, 32, 3, 1, 1),
            nn.Sigmoid()
            )
        self.conv = nn.Sequential(
            nn.Conv2d(32, 1, 3, 1, 1),
            )

    def forward(self, input):
        batch_size, row, col = input.size(0), input.size(2), input.size(3)

        x = input
        h = Variable(torch.zeros(batch_size, 32, row, col))
        c = Variable(torch.zeros(batch_size, 32, row, col))

        if self.use_GPU:
            h = h.cuda()
            c = c.cuda()

#        x_list = []
        for i in range(self.iteration):
#            x1 = x
            x = torch.cat((input, x), 1)
            x = self.conv0(x)

            x = torch.cat((x, h), 1)
            i = self.conv_i(x)
            f = self.conv_f(x)
            g = self.conv_g(x)
            o = self.conv_o(x)
            c = f * c + i * g
            h = o * torch.tanh(c)

            x = h
            resx = x
            x = F.relu(self.res_conv1(x) + resx)
            resx = x
            x = F.relu(self.res_conv2(x) + resx)
            resx = x
            x = F.relu(self.res_conv3(x) + resx)
            resx = x
            x = F.relu(self.res_conv4(x) + resx)
            resx = x
            x = F.relu(self.res_conv5(x) + resx)
            x = self.conv(x)

#            x_list.append(x)

        return x</code></pre>

username: bruce
cooked: <p>above is the model architecture I applied. which place should I insert dropout layer?</p>

username: Dexter
cooked: <p>You seem to have only convolutions only in your network without any <code>maxpooling</code> . I would suggest applying <code>strided-convolution</code> or <code>maxpooling</code> . In case of <code>Dropout</code> apply it after the convolution and just before a dense layer and in-between some dense layers</p>