Why LSTM stops learning if I do not set a hidden state to zero?

odats: <p>If I remove 2 lines <code>h0=torch.zeros.. c0=torch.zeros</code> and<code> batch_first=True</code> my network stops learning.</p>
<p>I thought that a zero initial hidden state is by default in nn.LSTM if you don’t pass in a hidden state .</p>
<pre><code class="lang-auto">class ModelLSTMFSM(nn.Module):
    def __init__(self, input_size=MAX_STRING_SIZE, hidden_size=256, num_layers=2, states_size=MAX_STATES_SIZE):
        super(ModelLSTMFSM, self).__init__()
        self.states_size = states_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, self.states_size * self.states_size * 2)
    
    def forward(self, x):
        x = x.reshape(-1, INPUT_SIZE, MAX_STRING_SIZE)
        
        # Set initial hidden and cell states 
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device) 
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
        
        # Forward propagate LSTM
        out, _ = self.lstm(x, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size)
        
        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :])
        return out.reshape(-1, self.states_size, self.states_size, 2)
</code></pre>

vdw: <aside class="quote no-group" data-username="odats" data-post="1" data-topic="92725">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/odats/48/21970_2.png" class="avatar"> odats:</div>
<blockquote>
<p>batch_first=True</p>
</blockquote>
</aside>
<p>If your remove <code>batch_first=True</code> it’s of course <code>batch_first=False</code> by default. In this case you would need to change <code> out = self.fc(out[:, -1, :])</code> to <code>out = self.fc(out[-1])</code></p>
<p>I don’t what you’re trying to learn and how your data looks like, but <code>x = x.reshape(-1, INPUT_SIZE, MAX_STRING_SIZE)</code> looks a but suspicious. Firstly, why do you need to infer the batch size (I assume <code>batch_first=True</code> and the first dimension is for the batch). And secondly, <code>input_size</code> is expected to be in the 3rd dimension of the input tensors for a LSTM/GRU.</p>
<p>I’m not sure about your <code>reshape()</code> in the <code>return</code> line either. If you’re not careful <code>reshape()</code> and <code>view()</code> can quickly <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">mess up your data</a>.</p>

odats: <p>Thank you for your suggestions. I did code refactoring and have 2 simple models: with <code>batch_first=True</code> trains and <code>batch_first=False</code> does not train (no errors, just a loss is not decreasing). What can be wrong?</p>
<p>Seems that there is no issue with zero hidden state.</p>
<p>I use CrossEntropyLoss. Target shape is (states_size, 2)</p>
<pre><code class="lang-auto">class ModelLSTMFSM_TRAINS(nn.Module):
    def __init__(self, input_size=MAX_STRING_SIZE, hidden_size=256, num_layers=2, states_size=MAX_STATES_SIZE):
        super(ModelLSTMFSM_TRAINS, self).__init__()
        self.states_size = states_size
        self.input_size = input_size
        
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, self.states_size * self.states_size * 2)
    
    def forward(self, x):
        x = x.view(x.size(0), -1, self.input_size)
        
        out, _ = self.lstm(x)
        
        out = self.fc(out[:, -1, :])
    
        return out.view(out.size(0), self.states_size, self.states_size, 2)
    
    
class ModelLSTMFSM_NOT_TRAINS(nn.Module):
    def __init__(self, input_size=MAX_STRING_SIZE, hidden_size=256, num_layers=2, states_size=MAX_STATES_SIZE):
        super(ModelLSTMFSM_NOT_TRAINS, self).__init__()
        self.states_size = states_size
        self.input_size = input_size
        
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers)
        self.fc = nn.Linear(hidden_size, self.states_size * self.states_size * 2)
    
    def forward(self, x):
        x = x.view(-1, x.size(0), self.input_size)
        
        out, _ = self.lstm(x)

        out = self.fc(out[-1])
        
        return out.view(out.size(0), self.states_size, self.states_size, 2)
</code></pre>

mobius: <p>Have you initialized the hidden state of your lstm layer? Also you probably need to get the hidden state at each iteration of your training and use it in the next forward() call, otherwise you might be starting from a new hidden state at each iteration (and your model will not converge)</p>
<p>I would suggest to take a look at the word language model example at the pytorch github</p>

odats: <p>In my previous comment, I have provided 2 examples of NN. The difference only in batch_first=False/True way of initialization. One is learning but the other one is not.</p>

vdw: <p>Without knowing what exactly you’re trying to learn, it’s basically impossible to tell for sure what is off. Also, how does your input <code>x</code> look like, i.e., what is the kind of data and what is its shape. At the moment I can only make the vaguely informed guess that your <code>view()</code> commands cause the problem. Any need for something like <code>x.view(batch_size, ...)</code> should not be needed and only does the right thing by chance.</p>
<p>Debugging neural nets can be tricky:</p>
<ul>
<li>
<p>Errors are only most commonly thrown when the shapes do not match the expected input. This is the easy case since, well, nothing works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
</li>
<li>
<p>If there are no errors but the network is not training, i.e., the loss is not going down, then there can be many issues, but at east there’s an indicator that something is wrong.</p>
</li>
<li>
<p>However, even if the loss does goes down doesn’t mean that the training is done correctly!!! The network only looks for pattern which it can fined even in messed up data/batches. And this can easily happen when using <code>view()</code> or <code>reshape()</code> when one is not being careful.</p>
</li>
</ul>
<p>I would almost betting 5 bucks that your network that is learning something, is still not necessarily learning correctly. But again, without knowing what you’re trying to do and what your input <code>x</code> looks like, it’s impossible to tell for sure.</p>
<p>For both networks can you change your forward methods to:</p>
<pre><code class="lang-auto">def forward(self, x):
    print(x.shape)
    x = x.view(...)
    print(x.shape)
    ...
</code></pre>
<p>and report the output of the <code>print</code> statements? Also, what do the dimensions of the input <code>x</code> mean?</p>

odats: <aside class="quote no-group" data-username="vdw" data-post="6" data-topic="92725">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>Any need for something like <code>x.view(batch_size, ...)</code> should not be needed and only does the right thing by chance.</p>
</blockquote>
</aside>
<p>You are right. There is no need for reshaping in this model (<code>ModelLSTMFSM_TRAINS</code>).</p>
<p>trains:</p>
<pre><code class="lang-auto">before torch.Size([64, 21, 8])
after torch.Size([64, 21, 8])
</code></pre>
<p>not trains:</p>
<pre><code class="lang-auto">before torch.Size([64, 21, 8])
after torch.Size([21, 64, 8])
</code></pre>
<p><code>batch_size=64</code></p>
<p>x is a matrix, with 21 rows. The size of each row is 8. Each row value can be 0 or 1 and padded with 2. Padded binary strings.</p>
<pre><code class="lang-auto">tensor([[0., 2., 2., 2., 2., 2., 2., 2.],
        [0., 0., 0., 1., 0., 0., 0., 0.],
        [0., 0., 0., 1., 1., 2., 2., 2.],
        [0., 0., 1., 1., 0., 1., 0., 0.], ....
</code></pre>
<p>The output is 3 by 2 matrix. In my case it is Finite-state machine:<br>
<code>[[0, 1], [2, 1], [2, 1]]</code></p>

odats: <aside class="quote no-group" data-username="vdw" data-post="2" data-topic="92725">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>I’m not sure about your <code>reshape()</code> in the <code>return</code> line either. If you’re not careful <code>reshape()</code> and <code>view()</code> can quickly <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">mess up your data</a></p>
</blockquote>
</aside>
<p>I have found the issue: Do not use view() or reshape() to swap dimensions of tensors!</p>
<p><code>x = x.view(-1, x.size(0), self.input_size) # very bad idea, do not use it</code><br>
<code>x = x.permute(1,0,2) # fixes everething</code></p>

vdw: <p>Glad it worked out in the end. You’d be surprised how many people use <code>view()</code> or <code>reshape()</code> to squash their tensors into the right shape without double-checking if preserves integrity.</p>

odats: <p>Your post about this issue is super helpful. Many thanks.</p>
<p>What is the right way to check if the data preserves integrity after <code>view()</code>  or  <code>reshape()</code>?</p>

vdw: <p>I don’t really work with PyTorch that often and it’s usually just basic stuff. And the first question is always: Do I really need <code>view()</code> or <code>reshape()</code>. And as an NLP working with RNNs, the answer is  basically always No. The only exception is if I need to resolve the output of a, say, GRU, where the hidden state <code>h_n</code> has a shape of <code>(num_layers * num_directions, batch, hidden_size)</code> and I need to “dissolve” <code>num_layers</code> and <code>num_directions</code> with</p>
<pre><code>h_n = h_n.view(num_layers, num_directions, batch, hidden_size) 
</code></pre>
<p>But this is well document and I trust the the PyTorch people are smart.</p>
<p>I once had to use it to flatten and unflatten an LSTM/GRU hidden state for an (Variational) Autoencoder. There the important part was to ensure that the tensor for flattening and after unflattening was indeed the same, which I double-checked with a toy tenser to simplify testing. Again, the Github code I found to help did it incorrectly, i.e., the tensor looked different after unflattening.</p>
<p>That’s the problem: once the shapes are correct, the network throws no errors. And even if then tensor is messed up, the network still might learn something.</p>