One layer BiLSTM is not learning

marcog64: <p>Hi, I am testing the single layer BiLSTM. My expectation is that it should learn.<br>
This is the model:</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(LSTMModel, self).__init__()
        self.hidden_dim = hidden_dim
        self.directions = 2      
        self.layer_dim = layer_dim        
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(hidden_dim, output_dim)
    
    def forward(self, x):
        batch, seq_len, input_size = x.shape
        
        h0 = torch.zeros(self.layer_dim*self.directions, batch, self.hidden_dim)
        c0 = torch.zeros(self.layer_dim*self.directions, batch, self.hidden_dim)
        
        out, (hn, cn) = self.lstm(x, (h0, c0))
        out = out.view(batch, seq_len, self.directions, self.hidden_dim)
        fwd = out[:, :, 0, :]
        bwd = out[:, :, 1, :]
        out = fwd + bwd
        
        out = self.fc(out) 
        return out
</code></pre>
<p>The result is quite disappointing, it tries to follow the last step:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/8889ebe7c9cce6b299b790c2df2e380990b78d84.png" data-download-href="https://discuss.pytorch.org/uploads/default/8889ebe7c9cce6b299b790c2df2e380990b78d84" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8889ebe7c9cce6b299b790c2df2e380990b78d84_2_690x348.png" alt="image" data-base62-sha1="jtSqKtak98B4xcWjfuIs8O3QPe4" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8889ebe7c9cce6b299b790c2df2e380990b78d84_2_690x348.png, https://discuss.pytorch.org/uploads/default/original/3X/8/8/8889ebe7c9cce6b299b790c2df2e380990b78d84.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/8889ebe7c9cce6b299b790c2df2e380990b78d84.png 2x" data-dominant-color="FAF6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">912×461 45.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What is wrong?</p>
<p>The notebook is available here:</p>
<aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1epJ3OtEhL-DljUXKwPrZc7lPaN4iAJBj/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1epJ3OtEhL-DljUXKwPrZc7lPaN4iAJBj/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1epJ3OtEhL-DljUXKwPrZc7lPaN4iAJBj/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1epJ3OtEhL-DljUXKwPrZc7lPaN4iAJBj/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">pytorch_bilstm_sunspot.ipynb</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


marcog64: <p>it seems that results are very much better when I use the following input/target sequence:</p>
<p>INPUT             TARGET<br>
x1 x2 x3 x4 → x5 x6 x7 x8<br>
x2 x3 x4 x5 → x6 x7 x8 x9<br>
x3 x4 x5 x6 → x7 x8 x9 x10</p>
<p>instead of:</p>
<p>INPUT             TARGET<br>
x1 x2 x3 x4 → x2 x3 x4 x5<br>
x2 x3 x4 x5 → x3 x4 x5 x6<br>
x3 x4 x5 x6 → x4 x5 x6 x7</p>
<p>But I am noticing better fitting when the value is between 0-1:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e485a2a33a03e7f5ad45c1cbbc79e512ddf19a02.png" data-download-href="https://discuss.pytorch.org/uploads/default/e485a2a33a03e7f5ad45c1cbbc79e512ddf19a02" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e485a2a33a03e7f5ad45c1cbbc79e512ddf19a02_2_690x219.png" alt="image" data-base62-sha1="wBBaSVmT3NQre2cJOqMB8b7nvns" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e485a2a33a03e7f5ad45c1cbbc79e512ddf19a02_2_690x219.png, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e485a2a33a03e7f5ad45c1cbbc79e512ddf19a02.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e485a2a33a03e7f5ad45c1cbbc79e512ddf19a02.png 2x" data-dominant-color="FDFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1006×320 22.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and worst result when it is in -1-0:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe0bed41413c965afe46adeec6a82ce0d26fc37b.png" data-download-href="https://discuss.pytorch.org/uploads/default/fe0bed41413c965afe46adeec6a82ce0d26fc37b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe0bed41413c965afe46adeec6a82ce0d26fc37b_2_690x348.png" alt="image" data-base62-sha1="AfoPuqpFIZG95xulCK4148IdOfV" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe0bed41413c965afe46adeec6a82ce0d26fc37b_2_690x348.png, https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe0bed41413c965afe46adeec6a82ce0d26fc37b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe0bed41413c965afe46adeec6a82ce0d26fc37b.png 2x" data-dominant-color="FBF8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">992×501 40.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am also getting worst validation result.<br>
The activation function for LSTM should be tanh and this should not be the problem. Any one can help me?</p>