LSTM loss fluctuating with slight decrease and then increases

username: Tauren_franklin
cooked: <p>I am trying to train a bi-directional LSTM. My input data is in shape [batch_size, 8(seq), 6], my output is in shape- [batch_size, 2] My code:</p>
<pre><code class="lang-auto">class MODEL(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes, device):
        super(MODEL, self).__init__()
        torch.manual_seed(0)
        self.device = device
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=True).to(self.device)
        #self.fc = nn.Linear(hidden_size*2, num_classes).to(self.device)
        self.dropout = nn.Dropout(0.2)
        self.activation = nn.Sigmoid()

    def forward(self, x):
        h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(self.device)
        c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(self.device)
        out, _ = self.lstm(x, (h0, c0))
        out = out[:,-1,:]
        out = self.activation(self.fc1(out))
        return out
</code></pre>
<p>I am using Adam optimiser with 0.0015 LR and SmoothL1Loss. For more reference, I am trying to implement this paper - <a href="https://christopherlu.github.io/files/papers/%5BAAAI2018%5Dionet.pdf" rel="noopener nofollow ugc">IONet</a> - except their sequence length for data is 200.</p>
<p>I have also tried using MSELoss for the purpose, but still the loss keeps fluctuating and does not decrease.  I also referred to a previous raise issue, which sounds similar to mine(<a href="https://discuss.pytorch.org/t/lstm-loss-keeps-fluctuating/75373" class="inline-onebox">LSTM loss keeps fluctuating</a>) , but it seems hasn’t been solved yet.<br>
My loss function keeps fluctuating it initially decreases, but then it starts to increase. Any help is appreciated. Thanks.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/8785fa2c11a55f1e87c104ac26c55aa40cc0d017.png" data-download-href="https://discuss.pytorch.org/uploads/default/8785fa2c11a55f1e87c104ac26c55aa40cc0d017" title="Figure_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/8785fa2c11a55f1e87c104ac26c55aa40cc0d017_2_666x500.png" alt="Figure_1" data-base62-sha1="jkTvd7kksqAnBPSmctierV9nRbx" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/8785fa2c11a55f1e87c104ac26c55aa40cc0d017_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/8785fa2c11a55f1e87c104ac26c55aa40cc0d017_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/7/8785fa2c11a55f1e87c104ac26c55aa40cc0d017.png 2x" data-dominant-color="F6F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_1</span><span class="informations">1280×960 70.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>