How to overcome this overfitting?

hiqmat_nisa1: <p>Hi,<br>
I am trying to train a model for Handwritten text recognition task using encoder-decoder with attention. The loss of validtion is not decressing after reaching some value and train loss is getting decreseing and there is huge gap between them.<br>
I have tried Dropout 0.5 after CNN fc layer and at the input of decoder. A Different weight decay from 0.1 to 0.000001 and also data augumenation have been applied.<br>
I am using optim.Adam and  nn.CrossEntropyLoss. My training exapmles are 8161 and validtion samples are 900.<br>
Can anyone help me what is going wrong ?</p>
<p>Here is the loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b076e41a9cc89138a35730f233cfb7719e9d450.png" alt="Weight Decay-0.000001" data-base62-sha1="hymEeVlkWZg3bvPlDJ1bA46cHQc" width="640" height="480"></p>
<p>here is the Encoder:<br>
class Encoder(nn.Module):<br>
def <strong>init</strong>(self,height, width, enc_hid_dim, dec_hid_dim, dropout):<br>
super().<strong>init</strong>()<br>
self.height= height<br>
self.enc_hid_dim=enc_hid_dim<br>
self.width=width<br>
self.num_layers=1</p>
<pre><code>    self.layer0 = nn.Sequential(
            nn.Conv2d(1, 8, kernel_size=(3,3),stride =(1,1), padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(8),
            nn.MaxPool2d(2,2),
            )
    self.layer1 = nn.Sequential(
            nn.Conv2d(8, 32, kernel_size=(3,3),stride =(1,1), padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(2,2),
            )
    self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=(3,3),stride =(1,1), padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(2,2)
            )
    self.rnn=nn.LSTM(self.height//8*64, self.enc_hid_dim, num_layers= self.num_layers, bidirectional=True)
    
    self.fc = nn.Linear(self.enc_hid_dim * 2, dec_hid_dim)
    
    self.dropout_layer = nn.Dropout(dropout)
    
def forward(self, src, in_data_len, train):
    batch_size = src.shape[0]
    out = self.layer0(src)
    out = self.layer1(out)
    out = self.layer2(out)
    if train:
        out = self.dropout_layer(out) # torch.Size([batch, channel, h, w])

    out = out.permute(3, 0, 2, 1) # (width, batch, height, channels)
    out.contiguous()
    out = out.reshape(-1, batch_size, self.height//8*64) #(w,batch, (height, channels)) 
    width = out.shape[0]
    src_len = in_data_len.numpy()*(width/self.width)
    src_len = src_len + 0.999 # in case of 0 length value from float to int
    src_len = src_len.astype('int')
    out = pack_padded_sequence(out, src_len.tolist(), batch_first=False)
    outputs, hidden_out = self.rnn(out)
    

    hidden=hidden_out[0]
    cell=hidden_out[1]
    # output: t, b, f*2  hidden: 2, b, f
    outputs, output_len = pad_packed_sequence(outputs, batch_first=False)
        
    #outputs = [src len, batch size, hid dim * num directions]
    #hidden = [n layers * num directions, batch size, hid dim]
    #hidden is stacked [forward_1, backward_1, forward_2, backward_2, ...]
    #outputs are always from the last layer
    
    #hidden [-2, :, : ] is the last of the forwards RNN 
    #hidden [-1, :, : ] is the last of the backwards RNN
    
    
    hidden = torch.tanh(self.fc(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)))
    cell = torch.tanh(self.fc(torch.cat((cell[-2,:,:], cell[-1,:,:]), dim = 1)))

    return outputs, hidden, cell
</code></pre>

AdamTogether: <aside class="quote no-group" data-username="hiqmat_nisa1" data-post="1" data-topic="105206">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hiqmat_nisa1/48/31972_2.png" class="avatar"> hiqmat_nisa1:</div>
<blockquote>
<p>My training exapmles are 8161 and validtion samples are 900.</p>
</blockquote>
</aside>
<p>Awesome that you’ve implemented augmenting!</p>
<p><em>Main Suggestion:</em> Given that the dataset is very small, <strong>reducing the overall complexity of your model</strong> should help to avoid overfitting. Perhaps try simply dropping one of the convolutional/sequential layers from your model and see what happens <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging.png?v=12" title=":man_shrugging:" class="emoji" alt=":man_shrugging:" loading="lazy" width="20" height="20">. (Note that to do this you’ll probably have to adjust the corresponding input layer of the LSTM.)</p>
<p><em>Side Note:</em> Adjusting the Learning Rate <em>α</em> — and to a lesser extent, weight decay <em>λ</em> — in your call to optim.Adam() will help if you are having convergence issues, but it seems that your model is doing okay in that regard. (Commonly, <em>λ</em> ranges from <code>1e-5</code> to <code>1e-6</code>.) However, given the spike around epoch 15 it may still be worth looking into if the main suggestion doesn’t help. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

hiqmat_nisa1: <p><a class="mention" href="/u/adamtogether">@AdamTogether</a> Thanks for the reply.  I am using only One layer BLSTM. First, I have tried 2 BLSTM layers in encoder, it was worse then 1 layer BLSTM.<br>
So how we can be sure that the particualr problem is overfitting problem?  As from the graph the validation loss is not increasing.<br>
secondly, What does these spikes actaully represents? Why we are getting these spikes?</p>

AdamTogether: <aside class="quote no-group" data-username="hiqmat_nisa1" data-post="3" data-topic="105206">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hiqmat_nisa1/48/31972_2.png" class="avatar"> hiqmat_nisa1:</div>
<blockquote>
<p>Thanks for the reply. I am using only One layer BLSTM. First, I have tried 2 BLSTM layers in encoder, it was worse then 1 layer BLSTM.</p>
</blockquote>
</aside>
<p>No problem - good that you checked different numbers of layers in BLSTM! If you have the choice of changing architectures, using transformers instead of BLSTM would likely perform much better in this case - though implementation can be a bit more difficult. However, I understand that sometimes we don’t have that choice <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="hiqmat_nisa1" data-post="3" data-topic="105206">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hiqmat_nisa1/48/31972_2.png" class="avatar"> hiqmat_nisa1:</div>
<blockquote>
<p>So how we can be sure that the particualr problem is overfitting problem? As from the graph the validation loss is not increasing.</p>
</blockquote>
</aside>
<p>Generally, if you are getting convergence on the training loss, but validation loss does not converge (in this case we see it increasing and decreasing rapidly), then your model is likely overfitting to the training dataset. (Assuming validation loss is being calculated correctly - which it is always good to double check lol.)</p>
<p>In terms of reducing model complexity, I would suggest you start by removing this layer:</p>
<pre data-code-wrap="python"><code class="lang-python">    self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=(3,3),stride =(1,1), padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(2,2)
            )
</code></pre>
<p>and maybe even removing this layer as well:</p>
<pre data-code-wrap="python"><code class="lang-python">    self.layer1 = nn.Sequential(
            nn.Conv2d(8, 32, kernel_size=(3,3),stride =(1,1), padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(2,2),
            )
</code></pre>
<aside class="quote no-group" data-username="hiqmat_nisa1" data-post="3" data-topic="105206">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hiqmat_nisa1/48/31972_2.png" class="avatar"> hiqmat_nisa1:</div>
<blockquote>
<p>secondly, What does these spikes actaully represents? Why we are getting these spikes?</p>
</blockquote>
</aside>
<p>In this case, I suspect the spike could be an indication that your Learning Rate <em>α</em> is a bit too large. (Learning rates that are too big will cause the gradient to explode during backpropagation, causing the training loss to oscillate.) In some cases however, seeing many small spikes in the training loss could simply be a result of <a href="https://stats.stackexchange.com/questions/303857/explanation-of-spikes-in-training-loss-vs-iterations-with-adam-optimizer" rel="noopener nofollow ugc">optim.Adam using mini-batch</a>. Perhaps try using different batch_sizes if reducing model complexity doesn’t help <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

hiqmat_nisa1: <aside class="quote no-group" data-username="AdamTogether" data-post="4" data-topic="105206">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adamtogether/48/31975_2.png" class="avatar"> AdamTogether:</div>
<blockquote>
<p>backpropagation</p>
</blockquote>
</aside>
<p>Thanks for the detail explanation.  I will check that.</p>

basilique: <p>Hello,<br>
Could you solve the problem? If so, I would appreciate if you could share it with us here.</p>

ykemiche: <p><a class="mention" href="/u/hiqmat_nisa1">@hiqmat_nisa1</a>  try to increase the number of epochs, you may be able to have a better  validation loss and see an epoch-wise double descent( section 6 of <a href="https://arxiv.org/pdf/1912.02292.pdf" rel="noopener nofollow ugc">this paper</a> )</p>