Loss doesn't decrease, model analysis help!

username: ykukkim
cooked: <p>Hi all,</p>
<p>I have built an LSTM model for binary class detection, and I am training with different hyperparameters but the loss just does not decrease.</p>
<p>I am feeding several time-series data through LSTM, then to predict true positive output[0,0,0,…1,0,0].</p>
<p>I am not too sure what else I am missing, if anyone could give me a piece of advice it will helpful</p>
<p>Thanks,</p>
<p>Here is my model,<br>
`    self.lstm = nn.LSTM(self.input_size, self.hidden_size, self.num_layers, dropout = self.drop_out, batch_first=True)</p>
<pre><code>def forward(self, x):
    hidden, cell = self.init_hidden()
    out, (hn, cn) = self.lstm(x, (hidden, cell))
    return out[:,:,-1]

def init_hidden(self):
    weight = next((self.parameters())).data
    
    hidden, cell = (weight.new(self.num_layers, self.batch_size, self.hidden_size).zero_().to(self.device),
                    weight.new(self.num_layers, self.batch_size, self.hidden_size).zero_().to(self.device))
    return hidden, cell
</code></pre>
<p>and Training Loop</p>
<blockquote>
<p>def train(self,input_size, batch_size,seq_length,hidden_size,num_layers,lr, epoch):</p>
</blockquote>
<pre><code>  self.model.train()

  for batch_idx, (data, target) in enumerate(self.train_loader):
      self.optimizer.zero_grad()
      data, target = data.to(self.device), target.to(self.device)
      predictions = self.model(data.float())
      self.globaliter += 1
      loss = self.criterion(predictions.float(), target.float())
      loss.backward()

      self.optimizer.step()

      pred = predictions.detach()
      pred[pred &gt;0.5] = 1
      pred[pred&lt;=0.5] = 0

      correct_indx = pred[target == 1]
      out_size = correct_indx[correct_indx == 1]
      accuracy = out_size.shape[0] / target.shape[0] 

      if batch_idx % self.log_interval == 0:
          print('\nTrain Epoch: {}\tLoss: {:.6f}\tAccuracy: {} '.format(epoch, loss.item(), accuracy * 100))
            
          with self.train_summary_writer.as_default():
              summary.scalar('loss', loss.item(), step=self.globaliter)
              summary.scalar('accuracy', accuracy, step=self.globaliter)
</code></pre>

username: quanguet
cooked: <p>Could you give detailed implementation of your model. One possibility is the mismatch between the ground truth labels and the predictions.<br>
But there is a high chance that you missed a <code>linear layer</code> to convert hidden output of LSTM to class prediction.</p>
<p>Try the following:</p>
<pre><code class="lang-python">
class Model(nn.Module):

    def __init__(self, input_size, output_size, hidden_size):
        super().__init__()

        self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.linear = nn.Linear(hidden_size, output_size)
    
    def forward(self, x):
        out, (hn, cn) = self.lstm(x)
        # out shape: [bs, seq, hidden_size]

        logits = self.linear(out)
        return logits
</code></pre>
<p>Hidden states are initialized by default.</p>

username: ykukkim
cooked: <p><a class="mention" href="/u/quanguet">@quanguet</a></p>
<p>Thank you for your reply.</p>
<p>I am not too sure what you mean by a detailed implementation. However, I missed out on giving my loss and optimizer function.</p>
<pre><code class="lang-auto">        self.model = Network(config).float()
        self.model = self.model.to(self.device)
        print(self.model)
        # Optimizer and Loss
        self.optimizer = optim.Adam(self.model.parameters(), lr=lr)
        self.pos_weight_factor = torch.tensor(120)
        # self.criterion = nn.BCELoss(reduction='none').to(self.device)
        self.criterion = nn.BCEWithLogitsLoss(pos_weight= self.pos_weight_factor)
</code></pre>
<p>What is the purpose of <code>linear layer</code> with hidden output? With <code>BCEWithLogitsLoss</code> I thought that I would need to put <code>out</code> as an input[predictions] to the <code>loss = self.criterion(predictions.float(), target.float())</code>. This is because <code>BCEWIthLogisloss</code> has intenal sigmoid function.</p>
<p>If you need any more information, please let me know</p>

username: quanguet
cooked: <p>In fact, the <code>out</code> is the hidden state from <code>self.lstm</code>, we use this hidden state to make a prediction by forwarding it into the Linear layer. Even you only has one label, then the Linear layer should be <code>nn.Linear(hidden_size, 1)</code>. On the other hands, when you extract <code>out[:, :, -1]</code>, you only return one value of a hidden state. It may be not enough to learn anything.</p>

username: ykukkim
cooked: <p><a class="mention" href="/u/quanguet">@quanguet</a></p>
<p>Thank you for your explanation. Regards to your comment about <code>out[:,:,-1]</code>, the full shape of out is fed to linear then <code>logtis[:,:,-1]</code> is extracted to the loss function. Will this still not be enough to learn anything?</p>
<pre><code class="lang-auto"> def forward(self, x):
        out, (hn, cn) = self.lstm(x)
        # out shape: [bs, seq, hidden_size]

        logits = self.linear(out)
        return logits[:,:,-1]
</code></pre>
<p>If so, what will be the best way to tackle this? as I only have one dimension of label. Will large dataset can solve the issue?</p>
<p>Thank you!</p>

username: quanguet
cooked: <p>Since your model learned to predict the binary class. <code>logits</code> will have the shape of <code>[batch_size, seq_len, 1]</code> already. Indexing <code>logits[:, :, -1]</code> will basically return the same values but with different shape <code>[batch_size, seq_len]</code>.</p>
<p>In order to forward these <code>logits</code> into some loss like <code>nn.BCEWithLogitsLoss()</code>, you may need to reshape the input of logits into <code>[batch_size * seq_len]</code>, which can be done by</p>
<pre><code class="lang-python">return logits[-1]

</code></pre>