Is flatten, pool or Conv leading to error?

Chanu_hcu: <p>I developed a neural network in PyTorch. While backpropagating loss using <code>.backward</code>, I got the following error</p>
<blockquote>
<p>Variable._execution_engine.run_backward(</p>
<p>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate<br>
results have already been freed. Specify retain_graph=True when<br>
calling .backward() or autograd.grad() the first time.</p>
</blockquote>
<p>I tried many ways and figured out some minimal steps, for me, that is responsible for the error. But I am not sure which line among these four is exactly creating the issue and why. The lines of code that are leading to error are given below</p>
<pre><code>    t = self.c1(t1)
    t = self.pool(t)
    t = torch.flatten(t,1)
    t = F.relu((self.linear(t)))
</code></pre>
<p>here t1 is tensor. c1 is a convolutional layer. I don’t think that last line is causing the issue as there are several <code>linear</code> layers before and after the piece of code I provided that are not causing any issue.</p>
<p>Which line may be causing the error? If there is a need for more information, I am ready to provide it.</p>
<hr>
<p><strong>Observation</strong>:</p>
<p>I came across the following warning in the <a href="https://pytorch.org/docs/master/notes/autograd.html" rel="noopener nofollow ugc">AUTOGRAD MECHANICS</a> article of PyTorch</p>
<blockquote>
<p><strong>Performing inplace operations on the input of any of the functions is<br>
forbidden as they may lead to unexpected side-effects</strong>. PyTorch will<br>
throw an error if the input to a pack hook is modified inplace but<br>
does not catch the case where the input to an unpack hook is modified<br>
inplace.</p>
</blockquote>
<p>Does it have any relation with the error I am getting? Although there are multiple usages of ReLu in my NN, I checked with the above usage of Relu with <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.relu.html" rel="noopener nofollow ugc"><code>inplace=False</code></a> and the error didn’t go.</p>

thecho7: <p>I reproduced your model like this,</p>
<pre><code class="lang-auto">  1 import torch
  2 import torch.nn as nn
  3 import torch.nn.functional as F
  4
  5 class model(nn.Module):
  6     def __init__(self):
  7         super(model, self).__init__()
  8         self.c1 = nn.Conv2d(1, 3, 1)
  9         self.pool = nn.MaxPool2d(2)
 10         self.linear = nn.Linear(48, 32)
 11
 12     def forward(self, t1):
 13         t = self.c1(t1)
 14         t = self.pool(t)
 15         t = torch.flatten(t, 1)
 16         t = F.relu(self.linear(t))
 17
 18         return t
 19
 20 m = model()
 21 a = torch.rand(1, 1, 8, 8)
 22 out = m(a)
 23 loss = out.sum()
 24 loss.backward()
</code></pre>
<p>It works well without the backprop error.<br>
I guess the error does not come from layers but the loss function.</p>

Chanu_hcu: <p><code>loss = -model.forward(....inputs.....).mean()</code></p>
<p>I am using <code>.mean</code> instead of sum. So, I don’t think it is leading to error. If I comment out those layers I mentioned, the program is running without an error. So, I think the layers are somehow responsible for the error I am getting.</p>

Chanu_hcu: <p>Recently I updated all the <code>.backward</code> in my code with <code>retain_graph=True</code>.  And then placed the backward step crating issue inside <code>with torch.autograd.set_detect_anomaly(True):</code> Then the error got narrowed down and is as follows</p>
<blockquote>
<p>[W python_anomaly_mode.cpp:85] Warning: Error detected in CudnnConvolutionBackward. No forward pass information available. Enable detect anomaly during forward pass for more information. (function _print_stack)</p>
<p>…</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 32, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>As I suspected, the issue is all due to some inplace operation. But if I search for the inplace operation with the given tensor size, I cannot able to find any.</p>
<p>Now, I have two questions:</p>
<p>1 Why the code is working well if I comment on the layers I mentioned in the first post?</p>
<p>2 How to find the exact inplace operation leading to error?</p>

Chanu_hcu: <p>To know the exact line, I tried to introduce a random tensor after the pool operation, and then the code led to the error shown below. I tried with random tensors after and before every operation. Except after pool everywhere, the same error (mentioned in post <span class="hashtag-raw">#1</span>) is occurring.</p>
<pre><code class="lang-auto">    t = self.c1(t1)
    t = self.pool(t)
    t = torch.FloatTensor(t.shape).to('cuda:0')
    t = torch.flatten(t,1)
    t = F.relu((self.linear(t)))
</code></pre>
<p><strong>Error</strong></p>
<blockquote>
<p>RuntimeError: Function ‘AddmmBackward0’ returned nan values in its 2th output.</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="Chanu_hcu" data-post="4" data-topic="156861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/47e85d/48.png" class="avatar"> Chanu_hcu:</div>
<blockquote>
<p>Recently I updated all the <code>.backward</code> in my code with <code>retain_graph=True</code>.</p>
</blockquote>
</aside>
<p>Don’t use <code>retain_graph=True</code> unless you explicitly need it and can explain why, as it usually yields other errors.</p>
<aside class="quote no-group" data-username="Chanu_hcu" data-post="4" data-topic="156861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/47e85d/48.png" class="avatar"> Chanu_hcu:</div>
<blockquote>
<p>As I suspected, the issue is all due to some inplace operation. But if I search for the inplace operation with the given tensor size, I cannot able to find any.</p>
</blockquote>
</aside>
<p>This is one expected error of using <code>retain_graph=True</code> as the <code>optimizer.step()</code> will update the parameters inplace. The <code>backward</code> call in the next iteration will then try to use the already updated parameters with the old forward activations of the first iteration to calculate the gradients, which is mathematically wrong.</p>
<aside class="quote no-group" data-username="Chanu_hcu" data-post="5" data-topic="156861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/47e85d/48.png" class="avatar"> Chanu_hcu:</div>
<blockquote>
<p>To know the exact line, I tried to introduce a random tensor after the pool operation</p>
</blockquote>
</aside>
<p>Note that you are not creating a random tensor, but an uninitialized one which can contain invalid values (such as NaN, Inf, etc.).</p>

Chanu_hcu: <p>The error is not due to the layers I used, but the input tensor t1 is leading to error. As t1 is the output of another neural network. It is necessary to call <code>.detach()</code>  on t1 before passing it to the current neural network.</p>