Loss spikes late in the training process

Kaga: <p>My model is given by:</p>
<pre><code class="lang-auto">class MyNetQ(nn.Module):
    def __init__(self):
        super(MyNetQ, self).__init__()
        self.myp=0.5
        self.flatten = nn.Flatten()
        self.linear_LeakyReLU_stack = nn.Sequential(
            nn.Linear(8*68, 8*60),
            nn.LeakyReLU(),
            nn.Dropout(p=self.myp),
            nn.Linear(8*60, 8*50),
            nn.LeakyReLU(),
            nn.Dropout(p=self.myp),
            nn.Linear(8*50, 8*40),
            nn.LeakyReLU(),
            nn.Dropout(p=self.myp),
            nn.Linear(8*40, 8*30),
            nn.LeakyReLU(),
            nn.Dropout(p=self.myp),
            nn.Linear(8*30, 8*20),
            nn.LeakyReLU(),
            nn.Dropout(p=self.myp),
            nn.Linear(8*20, 8*15),
            nn.LeakyReLU(),
            nn.Linear(8*15, 11)
        )
    def forward(self, x):
        x = self.flatten(x)
        out = self.linear_LeakyReLU_stack(x)
        return out
</code></pre>
<p>During the training process, the soft cross entropy loss exhibits huge spikes after seemingly having made good progress:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/284821f7b209170dc3f2a80fc6d340a6956f539c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/284821f7b209170dc3f2a80fc6d340a6956f539c" title="2021-09-25 00_41_57-Figure 1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/284821f7b209170dc3f2a80fc6d340a6956f539c_2_690x381.jpeg" alt="2021-09-25 00_41_57-Figure 1" data-base62-sha1="5KlDPkdh0UduznbhKwVmNINwVRi" width="690" height="381" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/284821f7b209170dc3f2a80fc6d340a6956f539c_2_690x381.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/284821f7b209170dc3f2a80fc6d340a6956f539c_2_1035x571.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/284821f7b209170dc3f2a80fc6d340a6956f539c_2_1380x762.jpeg 2x" data-dominant-color="F6F5F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2021-09-25 00_41_57-Figure 1</span><span class="informations">1497×828 194 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could be the reason for this behavior? Is there any way I could make the training process more stable?</p>

Kaga: <p>A hint, anyone? I suspect there may be a flaw in how I set up the model, but I don’t see it.</p>

Sayed_Nadim: <p>I think your model is overfitting. What’s your validation phase behavior?</p>

Kaga: <p>The blue line in the plot is the average training loss during each epoch, and the orange line is the average validation loss for each epoch. Validation does not use the dropout layers while training does use them. Actually, since training and validation loss are so close together at all times, I don’t think overfitting is an issue. There must be something else going on?</p>

Sayed_Nadim: <p>My bad. I didn’t see the legends. Did you experiment with learning rate decay? Maybe after some epochs, your lr is too high to find optimal minima.</p>

Kaga: <p>I am using <code>optimizer = torch.optim.Adam(mymodel.parameters(), lr=0.003)</code>. Good point, I’ll try to experiment with different learning rates, thanks for the suggestion.</p>

Rohan_Rathore: <p>Use a <code>Learning Rate Schedular</code>. Should help with convergence.</p>