Inplace error when using backward() function on discriminator

username: Nervous_Hero
cooked: <p>I got this error :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e28513a479b825e9a3bb205f45f8e40dee87a848.png" data-download-href="https://discuss.pytorch.org/uploads/default/e28513a479b825e9a3bb205f45f8e40dee87a848" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e28513a479b825e9a3bb205f45f8e40dee87a848_2_690x167.png" alt="image" data-base62-sha1="wjT1tRnHociPwSN97JqFiSSXoxy" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e28513a479b825e9a3bb205f45f8e40dee87a848_2_690x167.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e28513a479b825e9a3bb205f45f8e40dee87a848_2_1035x250.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e28513a479b825e9a3bb205f45f8e40dee87a848_2_1380x334.png 2x" data-dominant-color="222323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1811×440 26.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The problem occur in training loop when using backward() for discriminator not for generator</p>
<pre><code class="lang-auto">for epoch in range(0, EPOCH):
  print("EPOCH number = %d" % epoch)
  for x in iter(dataloader):
    with torch.autograd.set_detect_anomaly(True):

      G_x = model(x)
      
      # Generator train
      disc_fake_wav = wave_disc(G_x)
      disc_real_wav = wave_disc(x)

      disc_fake_stft = stft_disc(G_x)
      disc_real_stft = stft_disc(x)

      gen_loss = L_G(disc_fake_wav, disc_real_wav, disc_fake_stft, disc_real_stft, x, G_x)
      disc_loss = L_D(disc_fake_wav, disc_real_wav, disc_fake_stft, disc_real_stft)

      generator_optim.zero_grad()
      gen_loss.backward(retain_graph=True)
      generator_optim.step()

      # Discriminator train
      disc_fake_wav = wave_disc(G_x)
      disc_real_wav = wave_disc(x)

      disc_fake_stft = stft_disc(G_x)
      disc_real_stft = stft_disc(x)

      disc_loss = L_D(disc_fake_wav, disc_real_wav, disc_fake_stft, disc_real_stft)

      discriminator_optim.zero_grad()
      disc_loss.backward(retain_graph=True) # Error occurred here
      discriminator_optim.step()

      print("Training generator loss = %d" % gen_loss)
      print("Training discriminator loss = %d" % disc_loss)  
</code></pre>
<p>Full Traceback :</p>
<p>c:\Python\Python39\lib\site-packages\torch\autograd_<em>init</em>_.py:154: UserWarning: Error detected in MkldnnConvolutionBackward0. Traceback of forward call that caused the error:<br>
File “c:\Python\Python39\lib\runpy.py”, line 197, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “c:\Python\Python39\lib\runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “c:\Python\Python39\lib\site-packages\ipykernel_launcher.py”, line 16, in <br>
app.launch_new_instance()<br>
File “c:\Python\Python39\lib\site-packages\traitlets\config\application.py”, line 846, in launch_instance<br>
app.start()<br>
File “c:\Python\Python39\lib\site-packages\ipykernel\kernelapp.py”, line 677, in start<br>
self.io_loop.start()<br>
File “c:\Python\Python39\lib\site-packages\tornado\platform\asyncio.py”, line 199, in start<br>
self.asyncio_loop.run_forever()<br>
File “c:\Python\Python39\lib\asyncio\base_events.py”, line 596, in run_forever<br>
self._run_once()<br>
File “c:\Python\Python39\lib\asyncio\base_events.py”, line 1890, in _run_once<br>
handle._run()<br>
File “c:\Python\Python39\lib\asyncio\events.py”, line 80, in _run<br>
self._context.run(self._callback, *self._args)<br>
File “c:\Python\Python39\lib\site-packages\ipykernel\kernelbase.py”, line 457, in dispatch_queue<br>
await self.process_one()<br>
File “c:\Python\Python39\lib\site-packages\ipykernel\kernelbase.py”, line 446, in process_one<br>
await dispatch(*args)<br>
File “c:\Python\Python39\lib\site-packages\ipykernel\kernelbase.py”, line 353, in dispatch_shell<br>
await result<br>
File “c:\Python\Python39\lib\site-packages\ipykernel\kernelbase.py”, line 648, in execute_request<br>
reply_content = await reply_content<br>
File “c:\Python\Python39\lib\site-packages\ipykernel\ipkernel.py”, line 353, in do_execute<br>
res = shell.run_cell(code, store_history=store_history, silent=silent)<br>
File “c:\Python\Python39\lib\site-packages\ipykernel\zmqshell.py”, line 533, in run_cell<br>
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)<br>
File “c:\Python\Python39\lib\site-packages\IPython\core\interactiveshell.py”, line 2914, in run_cell<br>
result = self._run_cell(<br>
File “c:\Python\Python39\lib\site-packages\IPython\core\interactiveshell.py”, line 2960, in _run_cell<br>
return runner(coro)<br>
File “c:\Python\Python39\lib\site-packages\IPython\core\async_helpers.py”, line 78, in <em>pseudo_sync_runner<br>
coro.send(None)<br>
File “c:\Python\Python39\lib\site-packages\IPython\core\interactiveshell.py”, line 3185, in run_cell_async<br>
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,<br>
File “c:\Python\Python39\lib\site-packages\IPython\core\interactiveshell.py”, line 3377, in run_ast_nodes<br>
if (await self.run_code(code, result,  async</em>=asy)):<br>
File “c:\Python\Python39\lib\site-packages\IPython\core\interactiveshell.py”, line 3457, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “C:\Users\ACHYUT~1\AppData\Local\Temp/ipykernel_11308/2867750421.py”, line 6, in <br>
G_x = model(x)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “e:\Programming\Final Major\model.py”, line 43, in forward<br>
return self.decode(frames)[:, :, :x.shape[-1]].clone()<br>
File “e:\Programming\Final Major\model.py”, line 32, in decode<br>
return self._decode_frame(encoded_frames)<br>
File “e:\Programming\Final Major\model.py”, line 38, in _decode_frame<br>
out = self.decoder(emb)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “e:\Programming\Final Major\decoder.py”, line 41, in forward<br>
return self.layers(x)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\container.py”, line 141, in forward<br>
input = module(input)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “e:\Programming\Final Major\conv1d.py”, line 46, in forward<br>
return self.conv1d(x)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\conv.py”, line 301, in forward<br>
return self._conv_forward(input, self.weight, self.bias)<br>
File “c:\Python\Python39\lib\site-packages\torch\nn\modules\conv.py”, line 297, in _conv_forward<br>
return F.conv1d(input, weight, bias, self.stride,<br>
(Triggered internally at  …\torch\csrc\autograd\python_anomaly_mode.cpp:104.)<br>
Variable._execution_engine.run_backward(</p>

username: srishti-git1110
cooked: <p>Could you please try removing <code>retain_graph=True</code> from the line that gives the error?</p>
<p>I’m not sure why it’s needed there, could you explain?</p>

username: Nervous_Hero
cooked: <p>I solved the problem by removing retain_graph and detaching the fake tensor while calculating<br>
discriminator loss.<br>
code snippet :-</p>
<pre><code class="lang-auto">      # Discriminator train
      disc_fake_wav = wave_disc(G_x.detach())
      disc_real_wav = wave_disc(x)

      disc_fake_stft = stft_disc(G_x.detach())
      disc_real_stft = stft_disc(x)
</code></pre>