RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:.... is at version 2; expected version 1 instead.ected version 1 instead

Ruffybeo: <p>Hi everyone!</p>
<p>Currently, I’m trying to replicate a DeblurGanV2 network (<a href="https://github.com/KupynOrest/DeblurGAN" rel="noopener nofollow ugc">link</a>). At the moment, I’m working on performing the training. Here is my current status of my trainings pipeline:</p>
<blockquote>
<p>torch.autograd.set_detect_anomaly(mode=True)<br>
total_generator_loss = 0<br>
total_discriminator_loss = 0<br>
psnr_score = 0.0<br>
used_loss_function = ‘wgan_gp_loss’<br>
for epoch in range(n_epochs):</p>
<p><span class="hashtag-raw">#set</span> to train mode<br>
generator.train(); discriminator.train()<br>
tqdm_bar = tqdm(train_loader, desc=f’Training Epoch {epoch} ', total=int(len(train_loader)))<br>
for batch_idx, imgs in enumerate(tqdm_bar):</p>
<pre><code>#load imgs to cpu
blurred_images = imgs["blurred"].cuda()
sharped_images = imgs["sharp"].cuda()

# generator output
deblurred_img = generator(blurred_images)

# denormalize
with torch.no_grad():
  denormalized_blurred = denormalize(blurred_images)
  denormalized_sharp = denormalize(sharped_images)
  denormalized_deblurred = denormalize(deblurred_img)

# get D's output
sharp_discriminator_out = discriminator(sharped_images)
deblurred_discriminator_out = discriminator(deblurred_img)

# set critic_updates
if used_loss_function== 'wgan_gp_loss':
  critic_updates = 5
else:
    critic_updates = 1

#train discriminator
discriminator_loss = 0
for i in range(critic_updates):
  discriminator_optimizer.zero_grad()
  # train discriminator on real and fake
  if used_loss_function== 'wgan_gp_loss':
    gp_lambda = 10
    alpha = random.random()
    interpolates = alpha * sharped_images + (1 - alpha) * deblurred_img
    interpolates_discriminator_out = discriminator(interpolates)
    kwargs = {'gp_lambda': gp_lambda,
               'interpolates': interpolates,
               'interpolates_discriminator_out': interpolates_discriminator_out,
               'sharp_discriminator_out': sharp_discriminator_out,
               'deblurred_discriminator_out': deblurred_discriminator_out
                }
    wgan_loss_d, gp_d = wgan_gp_loss('D', **kwargs)
    discriminator_loss_per_update = wgan_loss_d + gp_d

  discriminator_loss_per_update.backward(retain_graph=True)
  discriminator_optimizer.step()
  discriminator_loss += discriminator_loss_per_update.item()
</code></pre>
</blockquote>
<p>But when I run this code, I receive the following error message:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 512, 4, 4]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<blockquote>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
62 #       discriminator_loss_per_update = gan_loss_d<br>
63<br>
—&gt; 64       discriminator_loss_per_update.backward(retain_graph=True)<br>
65       discriminator_optimizer.step()<br>
66       discriminator_loss += discriminator_loss_per_update.item()</p>
<p>1 frames<br>
/usr/local/lib/python3.7/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)<br>
243                 create_graph=create_graph,<br>
244                 inputs=inputs)<br>
 → 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
246<br>
247     def register_hook(self, hook):</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)<br>
145     Variable.<em>execution_engine.run_backward(<br>
146         tensors, grad_tensors</em>, retain_graph, create_graph, inputs,<br>
 → 147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag<br>
148<br>
149</p>
</blockquote>
<p>Unfortunately, I can’t really trace the in-place operation that would cause this error. Does anybody maybe has an idea or advice for me? I would appreciate any input <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>