RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 512, 4, 4]] is at version 3; expected version 2 instead

Walter_sh: <p>hello!<br>
I’m training my GAN code(pix2pixHD code<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e37ab18349bcd613e5d0e67134ec25b9584229f2.png" data-download-href="https://discuss.pytorch.org/uploads/default/e37ab18349bcd613e5d0e67134ec25b9584229f2" title="loss function"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e37ab18349bcd613e5d0e67134ec25b9584229f2_2_684x500.png" alt="loss function" data-base62-sha1="wsnfP1A1zvRnZ8jyzOqmNmq0keu" width="684" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e37ab18349bcd613e5d0e67134ec25b9584229f2_2_684x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e37ab18349bcd613e5d0e67134ec25b9584229f2_2_1026x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/3/e37ab18349bcd613e5d0e67134ec25b9584229f2.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss function</span><span class="informations">1111×811 93.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f077aa8a5506adf306886cbb8ed4c537c095399d.png" data-download-href="https://discuss.pytorch.org/uploads/default/f077aa8a5506adf306886cbb8ed4c537c095399d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f077aa8a5506adf306886cbb8ed4c537c095399d.png" alt="image" data-base62-sha1="yjgYxFFIPFMMV06eAVZMM99UNBP" width="690" height="111" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">925×150 7.14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
) and I have an error about inplace operation.<br>
how can I correct this problem?<br>
Thanks!</p>

eqy: <p>What is the output when you set<br>
<code>torch.autograd.set_detect_anomaly(True)</code>?</p>

Walter_sh: <p>I didn’t set the torch.autograd.set_detect_anomaly(True).<br>
Where should I write the code?</p>

eqy: <p>I think you can set it any time before you start model training.</p>

Walter_sh: <p>Thanks for the advice!<br>
I set the detect _anomaly code at the start of the training part and the here is the error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9843d36024be72f86d62573b5eaddb0b3a792766.png" data-download-href="https://discuss.pytorch.org/uploads/default/9843d36024be72f86d62573b5eaddb0b3a792766" title="autograd_set_error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9843d36024be72f86d62573b5eaddb0b3a792766.png" alt="autograd_set_error" data-base62-sha1="lIZTgqowJaHZeawniO6hoyvgFcG" width="690" height="417" data-dominant-color="101010"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">autograd_set_error</span><span class="informations">972×588 22.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

eqy: <p>Ok, it looks like the issue is in a conv layer of the model. Can you show the model definition?</p>

Walter_sh: <p>Here is the layer code I implemented.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/261d2c7ed558c6d5f0a6f17023c7d8ea1b2c5132.png" data-download-href="https://discuss.pytorch.org/uploads/default/261d2c7ed558c6d5f0a6f17023c7d8ea1b2c5132" title="discriminator"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/261d2c7ed558c6d5f0a6f17023c7d8ea1b2c5132_2_690x255.png" alt="discriminator" data-base62-sha1="5raEcKkavAH1Vd9EkOvmA9UjF7k" width="690" height="255" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/261d2c7ed558c6d5f0a6f17023c7d8ea1b2c5132_2_690x255.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/261d2c7ed558c6d5f0a6f17023c7d8ea1b2c5132_2_1035x382.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/6/261d2c7ed558c6d5f0a6f17023c7d8ea1b2c5132.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">discriminator</span><span class="informations">1104×408 47.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/665a75752747b20cb16e3e6ca3f770a84fc043cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/665a75752747b20cb16e3e6ca3f770a84fc043cd" title="generator"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/665a75752747b20cb16e3e6ca3f770a84fc043cd_2_690x407.png" alt="generator" data-base62-sha1="eBsvz1v95HtAqWy1sfrgPyFDIdv" width="690" height="407" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/665a75752747b20cb16e3e6ca3f770a84fc043cd_2_690x407.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/665a75752747b20cb16e3e6ca3f770a84fc043cd_2_1035x610.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/6/665a75752747b20cb16e3e6ca3f770a84fc043cd.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">generator</span><span class="informations">1100×649 74.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9eaf1769c6750f6b891ada0d6f3e61b28e5c68c.png" data-download-href="https://discuss.pytorch.org/uploads/default/c9eaf1769c6750f6b891ada0d6f3e61b28e5c68c" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9eaf1769c6750f6b891ada0d6f3e61b28e5c68c_2_674x500.png" alt="loss" data-base62-sha1="sOfkM2M7KsfwGd7Ve1F3sbrDStm" width="674" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9eaf1769c6750f6b891ada0d6f3e61b28e5c68c_2_674x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9eaf1769c6750f6b891ada0d6f3e61b28e5c68c_2_1011x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9eaf1769c6750f6b891ada0d6f3e61b28e5c68c.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1104×818 94.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c70afa1e9534414eb95be3af006f45e0efbd1e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c70afa1e9534414eb95be3af006f45e0efbd1e1" title="patchdiscriminator"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c70afa1e9534414eb95be3af006f45e0efbd1e1_2_690x478.png" alt="patchdiscriminator" data-base62-sha1="mjVUG4ai7DqvTcqlp5QgMFxZuut" width="690" height="478" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c70afa1e9534414eb95be3af006f45e0efbd1e1_2_690x478.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c70afa1e9534414eb95be3af006f45e0efbd1e1_2_1035x717.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c70afa1e9534414eb95be3af006f45e0efbd1e1.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">patchdiscriminator</span><span class="informations">1104×765 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>My best guess here would be that you compute all the forward and losses first, then you do a serie of backward/step for each loss one after the other?<br>
The problem is that the step function modifies the parameters of your net inplace, but these values are needed to run the backward (by conv for example).</p>
<p>You want to make sure that you don’t do a step operation between the forward and the backward.</p>