Gradient almost zero in case of simple LSTM Network

username: aquorio15
cooked: <p>I am training two simple NN one is a simple RNN network other is a LSTM based network, But after computing the gradients and plotting them, i am getting almost zero gradient flow in case of LSTM<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8ce96ffaed9efe50b4f3beabb9db8bf1c8234ffa.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ce96ffaed9efe50b4f3beabb9db8bf1c8234ffa" title="lstm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8ce96ffaed9efe50b4f3beabb9db8bf1c8234ffa_2_507x500.png" alt="lstm" data-base62-sha1="k6yYXyeVN86eLkpDr4emx3zxBma" width="507" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8ce96ffaed9efe50b4f3beabb9db8bf1c8234ffa_2_507x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/c/8ce96ffaed9efe50b4f3beabb9db8bf1c8234ffa.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/c/8ce96ffaed9efe50b4f3beabb9db8bf1c8234ffa.png 2x" data-dominant-color="D9D9EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm</span><span class="informations">580×571 37.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
almost similar to rnn<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bd807ed8f867df52668fbb36f4065430462d971.png" data-download-href="https://discuss.pytorch.org/uploads/default/3bd807ed8f867df52668fbb36f4065430462d971" title="rnn"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3bd807ed8f867df52668fbb36f4065430462d971_2_514x499.png" alt="rnn" data-base62-sha1="8xp0Vyi37L5xyZwySX9KKYsbRdf" width="514" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3bd807ed8f867df52668fbb36f4065430462d971_2_514x499.png, https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bd807ed8f867df52668fbb36f4065430462d971.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bd807ed8f867df52668fbb36f4065430462d971.png 2x" data-dominant-color="D8D8EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rnn</span><span class="informations">580×564 41.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>These are the simple models that i have used</p>
<pre><code class="lang-auto">class RNNModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, num_layers=1):
        super(RNNModel, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.rnn = nn.RNN(input_size=input_dim, hidden_size=hidden_dim, num_layers=num_layers, batch_first=True)
        self.linear = nn.Linear(hidden_dim, output_dim)
    def forward(self, x):
        x, _ = self.rnn(x)
        x = self.linear(x)
        return x
</code></pre>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, num_layers=1):
        super(LSTMModel, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(input_size=input_dim, hidden_size=hidden_dim, num_layers=num_layers, batch_first=True)
        self.linear = nn.Linear(hidden_dim, output_dim)
    def forward(self, x):
        x, _ = self.lstm(x)
        x = self.linear(x)
        return x
</code></pre>
<p>Can anyone tell me why the gradient is so bad in case of lstm which i think shold be better than rnn</p>