Variables needed for gradient computation has been modified by an inplace operation

yoad.tewel: <p>Hey!<br>
So I got pytorch 1.7.1 with cuda 11, rtx 3080 and ubuntu 20.04 (Installed from binaries).</p>
<p>I tried to train the model from this git: <a href="https://github.com/leftthomas/SRGAN" rel="noopener nofollow ugc">https://github.com/leftthomas/SRGAN</a><br>
But I got the following error:</p>
<pre><code class="lang-auto">&gt; Traceback (most recent call last):
&gt;   File "/home/work/projects/SRGAN/train.py", line 88, in &lt;module&gt;
&gt;     g_loss.backward()
&gt;   File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
&gt;     torch.autograd.backward(self, gradient, retain_graph, create_graph)
&gt;   File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
&gt;     Variable._execution_engine.run_backward(
&gt; RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 1024, 1, 1]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>So I ran the code with anomaly detections and this is the error:</p>
<pre><code class="lang-auto">  0%|          | 0/261 [00:00&lt;?, ?it/s][W python_anomaly_mode.cpp:104] Warning: Error detected in CudnnConvolutionBackward. Traceback of forward call that caused the error:
  File "/home/work/projects/SRGAN/train.py", line 80, in &lt;module&gt;
    fake_out = netD(fake_img).mean()
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/work/projects/SRGAN/model.py", line 84, in forward
    return torch.sigmoid(self.net(x).view(batch_size))
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
 (function _print_stack)
  0%|          | 0/261 [00:00&lt;?, ?it/s]
Traceback (most recent call last):
  File "/home/work/projects/SRGAN/train.py", line 90, in &lt;module&gt;
    g_loss.backward()
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/yoad/anaconda3/envs/torch1.7.1/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 1024, 1, 1]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>Not sure where to look for the inplace operation, any Ideas?</p>
<p>thanks!</p>

ptrblck: <p>I think you might be running into a similar issue as described <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">here</a>.<br>
By skimming through the code I’m unsure, why <code>retain_graph</code> is used in <code>d_loss.backward(retain_graph=True)</code> and why <code>fake_img</code> as well as <code>fake_out</code> are recomputed at the end before the <code>optimizerG.step()</code>. Could you explain this workflow a bit and check, if the issue from the link might also apply to your use case?</p>

yoad.tewel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for the response, yes I agree that it’s weird that they computed <code>fake_img</code> and <code>fake_out</code> <strong>after</strong>  the loss and <code>optimizerG.step()</code>.<br>
I tried 2 different changes that seems to work and while the training now works, I’m not sure if my changes afftect the overall intended training loop and model’s accuracy:<br>
(Changes can be seen here: <a href="https://github.com/YoadTew/SRGAN/blob/master/train.py" rel="noopener nofollow ugc">https://github.com/YoadTew/SRGAN/blob/master/train.py</a>)</p>
<ol>
<li>I changed the order of fake_img and fake_out, and it fixed the problem.</li>
</ol>
<pre><code class="lang-auto">            netG.zero_grad()

            fake_img = netG(z)
            fake_out = netD(fake_img).mean()

            g_loss = generator_criterion(fake_out, fake_img, real_img)

            g_loss.backward()
            optimizerG.step()
</code></pre>
<ol start="2">
<li>I removed <code>retain_grph=True</code> in <code>d_loss.backward()</code>, and sent <code>fake_img.detach()</code> to create new fake_out for <code>netD</code>.</li>
</ol>
<p>The thing is, I still dont know the reason for the error in the first place, and if my changes will change the intended flow of the training script.</p>

ptrblck: <p>The error might have been raised, if you tried to compute gradients from already updated parameters and thus stale intermediate activations as explained in the linked post.<br>
Did you compare your current code logic to the example I’ve posted in the other topic?</p>