RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation. AddBackward0

ZdsAlpha: <p>I am getting this error because of the following code.<br>
<strong>"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8, 16, 240, 427]], which is output 0 of AddBackward0, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)."</strong></p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.activation(self.in_conv(x))
        for i, conv in enumerate(self.mid_conv):
            x += self.activation(conv(x))
        return self.out_conv(x)
</code></pre>
<p>if I change the code into this it works fine:</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.activation(self.in_conv(x))
        for i, conv in enumerate(self.mid_conv):
            x = self.activation(conv(x))
        return self.out_conv(x)
</code></pre>
<p>Here self.activation is F.relu, self.mid_conv is nn.Sequential</p>

ptrblck: <p>As the error states, you’ve modified a variable by an inplace operation by using <code>x += ...</code>.<br>
If you want to sum the <code>conv</code> output to <code>x</code>, you should write out the operation as:</p>
<pre><code class="lang-python">x = x + self.activation(conv(x))
</code></pre>

JosueCom: <p>I got a similar error and I have no idea what it mean?  Any insights? I am not using an inplace operation like +=. After following the hint, I traced it to <code>return th.stack(neighs).sum(dim = 0)</code> where neighs is a list of 1D tensors.</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [50]], which is output 0 of AsStridedBackward, is at version 
2256; expected version 2255 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>It seems that adding values one by one in a for loop works. But, this is not a clean solution.</p>

ptrblck: <p>Have you tried enabling anomaly detection as suggested in the error message and if so, was the stack trace pointing to <code>torch.stack</code>?<br>
This would be unusual, as <code>torch.stack</code> wouldn’t modify the tensors inplace assuming <code>neighs</code> is e.g. a <code>list</code> of separate tensors. In case you get stuck, could you try to come up with a minimal executable code snippet so that we could try to debug it?</p>

JosueCom: <p>I enabled anomaly detection and it indicated <code>torch.stack</code>. I was using cuda and mixed precision training at the same time before. I think it had to do something with it.</p>
<p>I tried to create a simple problem but I cannot cause the error to appear again. However, in the bigger problem, I have fixed the issue by restructuring my code so I can set <code>neighs</code> to <code>all_neighbors[[indexes of local neighbors]]</code>.</p>
<p>This is the full output:</p>
<pre><code class="lang-auto">[W ..\torch\csrc\autograd\python_anomaly_mode.cpp:60] Warning: Error detected in StackBackward. Traceback of forward call that caused the error:
  File ".\cora.py", line 127, in &lt;module&gt;
    sub_graph_size=SUBGRPAH_SIZE)
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\lign\train.py", line 126, in superv
    out = base(full_graph, inp) if is_base_gcn else base(inp)
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\torch\nn\modules\module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\lign\models\CORA.py", line 20, in forward
    x = F.relu(self.unit3(g, x))
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\torch\nn\modules\module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\lign\nn.py", line 19, in forward
    g.push(func = self.aggregation, data = "__hidden__")
    out = func(out)
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\lign\utils\functions.py", line 71, in sum_tensors
    return th.stack(neighs).sum(dim = 0)
 (function print_stack)
Traceback (most recent call last):
  File ".\cora.py", line 127, in &lt;module&gt;
    sub_graph_size=SUBGRPAH_SIZE)
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\lign\train.py", line 133, in superv
    scaler.scale(loss).backward()
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "C:\Users\josue\anaconda3\envs\LIGN\lib\site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [50]], which is output 0 of AsStridedBackward, is at version 
2256; expected version 2255 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>If you still want to see the full project, I have it on <a href="https://github.com/JosueCom/Lign/tree/c82901a937a2637d8217f80c1b3129a45315771f" rel="noopener nofollow ugc">Github</a>. It is an older commit now. To produce the error, run <code>performance/cora.py</code> while inside the <code>performance</code> directory.</p>

jmjkx111: <p>Thank you very much.<br>
You are pytorch god.</p>