RNN model: training loss does not decrease

lorenzo_fabbri: <p>I’m trying to train a simple RNN model (plain RNN or LSTM). My mode is this:</p>
<pre><code>class Sequence(nn.Module):

    def __init__(self, input_dim, embedding_dim, hidden_dim, 
                 num_classes):
        super(Sequence, self).__init__()

        self.n_layers = 1
        self.hidden_size = hidden_dim

        self.embedding = nn.Embedding(input_dim, embedding_dim)

        self.encoder = nn.LSTM(embedding_dim, hidden_dim, 
                              num_layers=self.n_layers, dropout=0, 
                              batch_first=True)

        self.classify = nn.Linear(hidden_dim, num_classes)

    def forward(self, smi):
        smi = self.embedding(smi.long())

        smi, hidden = self.encoder(smi, hidden)

        out = self.classify(smi[:, -1, :])

        return out, hidden

    def init_weights(self):
        nn.init.xavier_uniform_(self.embedding.weight)

        nn.init.xavier_uniform_(self.classify.weight)
        nn.init.constant_(self.classify.bias, 0)

        for name, param in self.encoder.named_parameters():
            if 'weight' in name:
                nn.init.orthogonal_(param)

            elif 'bias' in name:
                nn.init.constant_(param, 0)
                r_gate = param[int(0.25 * len(param)):int(0.5 * len(param))]
                nn.init.constant_(r_gate, 1)

    def init_hidden(self, bsz, device):
        return (torch.zeros(self.n_layers, bsz, self.hidden_size).to(device), 
                torch.zeros(self.n_layers, bsz, self.hidden_size).to(device))
</code></pre>
<p>I don’t think I have to explain that much. The input is an array of integers (I encode each character of each sequence to an integer so that a string of N characters is encoded as an array of N integers): I pass it to an embedding layer, then an LSTM and finally a Linear layer for classification (it’s a multi-class classification task).</p>
<p>The problem is that the training loss is not decreasing, it just fluctuates a lot around the same value. I checked the weights and I cannot notice anything strange.</p>
<p>First, does anyone notice anything wrong in the model itself? It’s the first time I use these models and I’m not sure about the inputs. The input to the model has size <code>torch.Size([batch_size, 70])</code>, the output of <code>forward</code> is <code>torch.Size([batch_size, num_classes])</code> and the labels have dimensions <code>torch.Size([batch_size])</code>.</p>
<p>Another thing I still don’t get is the <code>hidden</code> tensor, which is pass to an RNN model: is it useful or not? Why some people use it and others don’t?</p>

phan_phan: <p>I’m not sure why your loss would not decrease. Depending on your machine, and your initialization, LSTM can take a very long time to learn ! I remember once seeing no improvement for hours.</p>
<p>The <code>hidden</code> argument passed to LSTM is a tuple with the initial tensors (h, c). If you do not give this argument, they will be initialized to zero, which is fine.</p>

e4exp: <p>Lowering the learning rate, or using gradient clipping by a smaller value than 1.0 may help.</p>
<aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn/352037#352037">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn/352037#352037" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/22311/sycorax" target="_blank" rel="noopener nofollow ugc">
    <img alt="Sycorax" src="https://i.stack.imgur.com/fX0Q3.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn/352037#352037" target="_blank" rel="noopener nofollow ugc">What should I do when my neural network doesn't learn?</a>
</h4>

<div class="tags">
  <strong>neural-networks, faq</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stats.stackexchange.com/users/22311/sycorax" target="_blank" rel="noopener nofollow ugc">
    Sycorax
  </a>
  on <a href="https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn/352037#352037" target="_blank" rel="noopener nofollow ugc">12:26AM - 19 Jun 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>