RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 64, 3, 3, 3]] is at version 2; expected version 1 instead

username: JohnMin
cooked: <p>I was trying to make 3D GAN with FMRI dataset with codes from <a href="https://github.com/cyclomon/3dbraingen/blob/master/VAEGAN_ADNI_train.ipynb" class="inline-onebox" rel="noopener nofollow ugc">3dbraingen/VAEGAN_ADNI_train.ipynb at master · cyclomon/3dbraingen · GitHub</a></p>
<p>And it keeps making errors that I don’t understand…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e83df58c6e4e0644d767d149204f9da9b6706cf5.png" data-download-href="https://discuss.pytorch.org/uploads/default/e83df58c6e4e0644d767d149204f9da9b6706cf5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e83df58c6e4e0644d767d149204f9da9b6706cf5.png" alt="image" data-base62-sha1="x8vwf7qkWp6yUuc6005QMNdHrG5" width="464" height="500" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">748×806 21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35ac5644d8588fd2abdd653404147fe6342139aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/35ac5644d8588fd2abdd653404147fe6342139aa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35ac5644d8588fd2abdd653404147fe6342139aa.png" alt="image" data-base62-sha1="7EOxfg4HSjFlD8cLWFFsIL3gG4W" width="418" height="500" data-dominant-color="262727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">691×825 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61f37570edfd98ec688c8964d7c34fa1191c5137.png" data-download-href="https://discuss.pytorch.org/uploads/default/61f37570edfd98ec688c8964d7c34fa1191c5137" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61f37570edfd98ec688c8964d7c34fa1191c5137.png" alt="image" data-base62-sha1="dYvVcvCBnEkrLfqp8U8DhxNQfEX" width="690" height="188" data-dominant-color="232424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1494×408 22.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>can anybody help me to solve this problem?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Have you tried enabling anomaly mode as suggested in the error message?</p>

username: JohnMin
cooked: <p>yes i had!</p>
<p>But it keeps making errors <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: albanD
cooked: <p>It is not supposed to remove the error but give you more information about where it comes from.</p>

username: JohnMin
cooked: <p>hmm i think tried .mean() → torch.mean() and inplace to True for nn.ReLU and Leaky …</p>
<p>any other suggestions?</p>

username: albanD
cooked: <p>Well, it is hard to say without the second stack trace that the anomaly mode will give.<br>
You should enable anomaly mode as suggested and share the second stack trace here.</p>

username: JohnMin
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd625e5c9fea5f8aa02c3339c98be10d6410d1a3.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd625e5c9fea5f8aa02c3339c98be10d6410d1a3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd625e5c9fea5f8aa02c3339c98be10d6410d1a3.png" alt="image" data-base62-sha1="tiUDnHjLJ2aS3Y5GV5qnnG5NMpd" width="690" height="273" data-dominant-color="292A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1530×607 40.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I applied the anomaly mode ‘True’</p>
<p>should I put “torch.autograd.set_detect_anomaly(True)” in training steps?</p>

username: albanD
cooked: <p>Hey,</p>
<p>Given the stack, it looks like the issue is that the input/output to the conv layer it points to has been modified inplace.<br>
Do you use inplace versions of activations in your code</p>

username: JohnMin
cooked: <p>When I used the model that makes no error(Alpha_GAN), it keeps making it error.</p>
<p>So I am not really sure that its a conv layers’ problem</p>
<p>The model that I am currently using looks like this.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/cyclomon/3dbraingen/blob/master/Model_alphaWGAN.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/cyclomon/3dbraingen/blob/master/Model_alphaWGAN.py" target="_blank" rel="noopener nofollow ugc">cyclomon/3dbraingen/blob/master/Model_alphaWGAN.py</a></h4>
<pre><code class="lang-py">import numpy as np
import torch
import os
from torch import nn
from torch import optim
from torch.nn import functional as F

#***********************************************
#Encoder and Discriminator has same architecture
#***********************************************
class Discriminator(nn.Module):
    def __init__(self, channel=512,out_class=1,is_dis =True):
        super(Discriminator, self).__init__()
        self.is_dis=is_dis
        self.channel = channel
        n_class = out_class 
        
        self.conv1 = nn.Conv3d(1, channel//8, kernel_size=4, stride=2, padding=1)
        self.conv2 = nn.Conv3d(channel//8, channel//4, kernel_size=4, stride=2, padding=1)
        self.bn2 = nn.BatchNorm3d(channel//4)
</code></pre>

  This file has been truncated. <a href="https://github.com/cyclomon/3dbraingen/blob/master/Model_alphaWGAN.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: JohnMin
cooked: <p>The training cod is here.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/cyclomon/3dbraingen/blob/master/Alpha_WGAN_ADNI_train.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/cyclomon/3dbraingen/blob/master/Alpha_WGAN_ADNI_train.ipynb" target="_blank" rel="noopener nofollow ugc">cyclomon/3dbraingen/blob/master/Alpha_WGAN_ADNI_train.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import numpy as np\n",
    "import torch\n",
    "import os\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "from torch.nn import functional as F\n",
    "from torch import autograd\n",
    "from torch.autograd import Variable\n",
    "import nibabel as nib\n",
    "from torch.utils.data.dataset import Dataset\n",
    "from torch.utils.data import dataloader\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/cyclomon/3dbraingen/blob/master/Alpha_WGAN_ADNI_train.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>