"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1024, 1]] is at version 2; expected version 1 instead."

Priyank_Pathak: <p>I have cascaded SR GAN (two generators and two discriminator)<br>
(single generator and discriminator code is working fine, not uploaded here).</p>
<p>The essentials are as follows :</p>
<pre><code class="lang-auto">self.no_of_gan = [16, 64, 256]
with g_optim_wrapper.optim_context(self):
      batch_outputs = self.generator(batch_inputs)
      # list of 2 generator outputs : (B, 3, 64, 64) &amp; (B, 3, 256, 256) : GAN : 4x Super-resolution
all_parsed_losses_g = 0 
set_requires_grad(self.discriminator, False)

for i in range(2):
      req_size = self.no_of_gan[i+1]
      gt_resize = torch.nn.functional.interpolate(batch_gt_data, size=req_size)
      # batch_gt_data:  B, 3, 256, 256
      parsed_losses_g, log_vars_d = self.g_step_with_optim(
                batch_outputs=batch_outputs[i], batch_gt_data=gt_resize,
                optim_wrapper=optim_wrapper, index=i)
            
      all_parsed_losses_g += parsed_losses_g
      log_vars.update(log_vars_d)

</code></pre>
<p>The above calculates GAN error <strong>(but doesn’t update the parameter though)</strong>. Then I calculate the discriminator error and update the weights of discriminator.</p>
<pre><code class="lang-auto">set_requires_grad(self.discriminator, True)

for i in range(self.n_layers):
      req_size = self.no_of_gan[i+1]
      gt_resize = torch.nn.functional.interpolate(batch_gt_data, size=req_size)
     
       log_vars_d = self.d_step_with_optim(
                batch_outputs=batch_outputs[i].detach(),
                batch_gt_data=gt_resize,
                optim_wrapper=optim_wrapper, index=i)

        log_vars.update(log_vars_d)

set_requires_grad(self.discriminator, False)
all_parsed_losses_g.backward()
set_requires_grad(self.discriminator, True)
</code></pre>
<p>This should work as updated weights of discriminator have nothing to do with generator loss backward.</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1024, 1]] is at version 2; expected version 1 instead.
</code></pre>
<p>I have uploaded the code for error reproducibility:</p>
<aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/100lDWaKZAO7hp9NLd5tkvXMLmnF4lSWq/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/100lDWaKZAO7hp9NLd5tkvXMLmnF4lSWq/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/100lDWaKZAO7hp9NLd5tkvXMLmnF4lSWq/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/100lDWaKZAO7hp9NLd5tkvXMLmnF4lSWq/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">debug.py</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><strong>NOTE1:</strong> If I move the generator update step before the discriminator update step, the error is resolved. If updating the discriminator weight is the problem, then why not doing the cascade (1 generator and 1 discriminator) not see the same error?</p>
<p><strong>NOTE2:</strong> Updating Generator weights after discriminator works as long as I’m not involving Discriminator based loss. (Pixel loss, Perception error etc. all work fine till I do discriminator loss)</p>

ptrblck: <p>You might be running into <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this error</a> if stale forward activations are used during a <code>backward</code> call.</p>

Priyank_Pathak: <p>If I’m interpreting your answer correctly, doing <code>.backward()</code> from stale output should throw this error. That is,<br>
“Updating the discriminator is somehow updating the generator’s weights, which in turn when doing the generator’s loss  <code>.backward()</code> after the discriminator’s weight update raises the error due to stale generator.”</p>
<p>Before discriminator update, I did:</p>
<pre><code>    model_weights = {}
    for p in self.generator.named_parameters():
        model_weights[p[0]] = p[1].mean().item() 
</code></pre>
<p>And After Discriminator update</p>
<pre><code class="lang-auto">new_model_weights = {}
for p in self.generator.named_parameters():
    new_model_weights[p[0]] = p[1].mean().item() 
    assert new_model_weights[p[0]]  == model_weights[p[0]]
</code></pre>
<p>The other possibility is Generator weights use Discriminator’s output, which becomes stale after Discriminator’s weight update step. This also is unlikely as <code>set_requires_grad(self.discriminator, False)</code> forces Gradient independence from Discriminators activations (hopefully) as verified by</p>
<pre><code class="lang-auto">for child in self.discriminator.children():
    for name, param in child.named_parameters():
        if param.requires_grad:
            print("******", name, param.shape)
            # should not print anything here
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Priyank_Pathak" data-post="3" data-topic="184543">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/priyank_pathak/48/19974_2.png" class="avatar"> Priyank_Pathak:</div>
<blockquote>
<p>This also is unlikely as <code>set_requires_grad(self.discriminator, False)</code> forces Gradient independence from Discriminators activations (hopefully) as verified by</p>
</blockquote>
</aside>
<p>That’s not the case and the <code>backward</code> call could still try to compute gradients in <code>self.discriminator</code> if a valid computation graph was created before you have frozen the parameters.<br>
Here is a small example showing the error:</p>
<pre data-code-wrap="python"><code class="lang-python">modelA = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1))

modelB = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1))

optA = torch.optim.SGD(modelA.parameters(), lr=1.)
optB = torch.optim.SGD(modelB.parameters(), lr=1.)

x = torch.randn(1, 1)
out = modelA(x)
out.mean().backward(retain_graph=True)
optA.step()

for param in modelA.parameters():
    param.requires_grad = False
out = modelB(out)#.detach())
out.mean().backward()
</code></pre>
<p>If you properly <code>.detach()</code> the input to <code>modelB</code> the code works fine.</p>