Runtime error (backprop) while training an RNN. Loss does not decrease

scarecrow21: <p>Following is the error I get while training an RNN:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>Input sizes:<br>
X_train.shape:  (90, 10)<br>
y_train.shape:  (90, 1)</p>
<p>RNN Config:</p>
<pre><code class="lang-auto">batch_size = 1
input_size = 1
sequence_length = 10
hidden_size = 1
num_layer = 10
</code></pre>
<p>RNN Class:</p>
<pre><code class="lang-auto">class ModelRnn(nn.Module):
    def __init__(self):
        super(ModelRnn, self).__init__()
        
        self.rnn = nn.RNN(input_size=input_size, hidden_size=hidden_size, batch_first=True, num_layers=num_layer)
        
    def forward(self, x, hidden):
        x = x.view(batch_size, sequence_length, input_size)
        
        out, hidden = self.rnn(x, hidden)
        
        return hidden, out
    
    def init_hidden(self):
        hidden_state = Variable(torch.zeros(num_layer, batch_size, hidden_size))
        
        return (hidden_state)
</code></pre>
<p>Train loop:</p>
<pre><code class="lang-auto">hidden = model.init_hidden()    

for epoch in range(5):
    for x_batch, y_batch in train_loader:
        model.zero_grad()
        hidden, output = model(x_batch, hidden)
        optimizer.zero_grad()
        loss = criterion(output, y_batch)
        
    print(f"{epoch+1} epoch | loss = {loss}")
    loss.backward()
    optimizer.step()
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">1 epoch | loss = 96040000.0
2 epoch | loss = 96040000.0

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-90-8a1b1c3e9bec&gt; in &lt;module&gt;()
      9 
     10     print(f"{epoch+1} epoch | loss = {loss}")
---&gt; 11     loss.backward()
     12     optimizer.step()

~/anaconda3/envs/arena-py3.6/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    100                 products. Defaults to ``False``.
    101         """
--&gt; 102         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    103 
    104     def register_hook(self, hook):

~/anaconda3/envs/arena-py3.6/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 90         allow_unreachable=True)  # allow_unreachable flag
     91 
     92 

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>I tries the solution given here: <a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time-while-using-custom-loss-function/12360/2" class="inline-onebox">"RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time" while using custom loss function</a></p>
<p>I don’t understand why it’s happening. Doing <code>.backward(retain_graph=True)</code> solves the error, but then the loss does not decrease no matter how many epochs.<br>
It’s maybe because I don’t fully understand the “detachment part” in the computational graph.</p>