Variable modified by an inplace operation error although using non-inplace operation

Sam-gege: <p>Hi, the following piece of code:</p>
<pre><code class="lang-python">import torch
x=torch.tensor([-1.2, 0.6], requires_grad=True)
y=1+x
y[0]=torch.abs(y[0])
y.sum().backward()
</code></pre>
<p>gives error that says:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead.
</code></pre>
<p>However, if you change the fourth line to <code>y[0]=torch.abs_(y[0])</code> everything works again. But according to the doc, <code>abs_</code> is inplace version of <code>abs</code>. How come it works while <code>abs</code> won’t?</p>

thecho7: <p>Let’s look that into step by step.</p>
<p>You should know what <code>torch.abs</code> operation returns.</p>
<pre><code class="lang-auto">a1 = torch.rand(2)
a1.requires_grad_(True)
b1 = a1 + 1
c1 = torch.abs(b1[0])
print(b1[0].data_ptr(), c1.data_ptr()) # They are the different

a2 = torch.rand(2)
a2.requires_grad_(True)
b2 = a2 + 1
c2 = torch.abs_(b2[0])
print(b2[0].data_ptr(), c2.data_ptr()) # They are the same
</code></pre>
<p>Copy the above code and see how those two printed values different.<br>
<code>in-place</code> returns exactly the same tensor (value, address)<br>
Consequently, <code>c2 = torch.abs_(b2[0])</code> is ignored because they are pointing the same data(address).<br>
So, <code>c2 = ~~~</code> (in-place operation) doesn’t occur at all.</p>

Sam-gege: <p>Thanks for your reply. Is there a way to avoid such “variable modified by an inplace operation” error? I encounter this error when I’m implementing YOLO v1, recall they take sqrt of width and height in calculating loss:</p>
<pre><code class="lang-python">import torch
x=torch.randn(2,4, requires_grad=True) * 1
x[:,2:]=torch.sqrt(torch.abs(x[:,2:])+1e-8) * torch.sign(x[:,2:]) # take sqrt of w and h
y=x.sum()
y.backward()
</code></pre>
<p>This results in error. what may be a good way to solve it?</p>

thecho7: <p>How about separate them?</p>
<pre><code class="lang-auto">a = x[:, :2]
b = x[:, 2:]
b = ~~~
c = torch.cat(a, b, dim=1)
c = c.sum()
c.backward()
</code></pre>
<p>I’m not sure it works</p>

KFrank: <p>Hi Sam (and Suho)!</p>
<aside class="quote no-group" data-username="Sam-gege" data-post="1" data-topic="176419" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam-gege/48/42338_2.png" class="avatar"> Sam-gege:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead.
</code></pre>
<p>However, if you change the fourth line to <code>y[0]=torch.abs_(y[0])</code> everything works again.</p>
</blockquote>
</aside>
<p>This requires a few words of explanation.</p>
<p>The short story is that <code>abs()</code> requires an unmodified copy of its input<br>
tensor in order to compute its gradient during <code>backward()</code> (as does<br>
<code>abs_()</code>).  <code>abs()</code> relies on its input still being available in unmodified<br>
form during backward, while <code>abs_()</code> <em>knows</em> that it is modifying its input<br>
inplace so it stores an unmodified <em>copy</em> of its input in the computation<br>
graph for use during <code>backward()</code>.</p>
<p>When you switch from <code>abs()</code> to <code>abs_()</code> you cause an unmodified <em>copy</em><br>
of <code>y</code> to be stored, thus fixing the inplace modification error.</p>
<p>(Reminder:  Modifying a tensor inplace is not necessarily an error.  It<br>
depends on whether that tensor is needed during the backward pass.)</p>
<p>A tweaked version of your example starts to show what is going on:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.0.0
&gt;&gt;&gt;
&gt;&gt;&gt; # using abs_ without assignment
&gt;&gt;&gt; x = torch.tensor ([-1.2, -2.6], requires_grad = True)
&gt;&gt;&gt; y = x + 1
&gt;&gt;&gt; y, y._version, y.data_ptr()   # initial, unmodified version of y
(tensor([-0.2000, -1.6000], grad_fn=&lt;AddBackward0&gt;), 0, 1833055763904)
&gt;&gt;&gt; torch.abs_ (y[0])
tensor(0.2000, grad_fn=&lt;AsStridedBackward0&gt;)
&gt;&gt;&gt; y *= 1
&gt;&gt;&gt; y, y._version, y.data_ptr()   # y has been modified inplace once
(tensor([ 0.2000, -1.6000], grad_fn=&lt;MulBackward0&gt;), 2, 1833055763904)
&gt;&gt;&gt; y.sum().backward()            # backward() works
&gt;&gt;&gt; x.grad
tensor([-1.,  1.])
&gt;&gt;&gt;
&gt;&gt;&gt; # using abs_ with assignment
&gt;&gt;&gt; x = torch.tensor ([-1.2, -2.6], requires_grad = True)
&gt;&gt;&gt; y = x + 1
&gt;&gt;&gt; y, y._version, y.data_ptr()   # initial, unmodified version of y
(tensor([-0.2000, -1.6000], grad_fn=&lt;AddBackward0&gt;), 0, 1833055761472)
&gt;&gt;&gt; y[0] = torch.abs_ (y[0])
&gt;&gt;&gt; y, y._version, y.data_ptr()   # y has been modified inplace twice
(tensor([ 0.2000, -1.6000], grad_fn=&lt;CopySlices&gt;), 2, 1833055761472)
&gt;&gt;&gt; y.sum().backward()            # backward() works
&gt;&gt;&gt; x.grad
tensor([-1.,  1.])
&gt;&gt;&gt;
&gt;&gt;&gt; # using abs with assignment
&gt;&gt;&gt; x = torch.tensor ([-1.2, -2.6], requires_grad = True)
&gt;&gt;&gt; y = x + 1
&gt;&gt;&gt; y, y._version, y.data_ptr()   # initial, unmodified version of y
(tensor([-0.2000, -1.6000], grad_fn=&lt;AddBackward0&gt;), 0, 1833055763200)
&gt;&gt;&gt; y[0] = torch.abs (y[0])
&gt;&gt;&gt; y, y._version, y.data_ptr()   # y has been modified inplace once
(tensor([ 0.2000, -1.6000], grad_fn=&lt;CopySlices&gt;), 1, 1833055763200)
&gt;&gt;&gt; y.sum().backward()            # and backward() fails
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
&gt;&gt;&gt; x.grad                        # grad is None because backward() failed
&gt;&gt;&gt; 
</code></pre>
<p>It’s easier to see what’s going on if we just look at <code>abs_()</code> without<br>
also assigning into the indexed view of  y<code>, </code>y[0] = torch.abs (y[0])`.</p>
<p>The following example shows how an explicit inplace-modification<br>
with <code>abs()</code> breaks <code>backward()</code>, but switching to <code>abs_()</code> fixes things:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; # modify input to abs_() inplace separately from calling abs_()
&gt;&gt;&gt; x = torch.tensor ([-1.2, 0.6], requires_grad = True)
&gt;&gt;&gt; z = x + 1
&gt;&gt;&gt; y = z.abs_()                   # modifies z inplace, so it stores a copy of z
&gt;&gt;&gt; y
tensor([0.2000, 1.6000], grad_fn=&lt;AbsBackward0&gt;)
&gt;&gt;&gt; z *= 1                         # modifies z again, but its copy remains unmodified for backward()
&gt;&gt;&gt; z._version                     # z modified twice
2
&gt;&gt;&gt; y.sum().backward()             # backward() works
&gt;&gt;&gt; x.grad
tensor([-1.,  1.])
&gt;&gt;&gt;
&gt;&gt;&gt; # modify input to abs() inplace separately from calling abs()
&gt;&gt;&gt; x = torch.tensor ([-1.2, 0.6], requires_grad = True)
&gt;&gt;&gt; z = x + 1
&gt;&gt;&gt; y = z.abs()                    # doesn't modify z -- no copy stored
&gt;&gt;&gt; y
tensor([0.2000, 1.6000], grad_fn=&lt;AbsBackward0&gt;)
&gt;&gt;&gt; z *= 1                         # modifies z, but no separate copy, so breaks backward() through abs()
&gt;&gt;&gt; z._version                     # (z modified once)
1
&gt;&gt;&gt; y.sum().backward()             # backward() fails
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2]], which is output 0 of MulBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

thecho7: <p>Good explanation <a class="mention" href="/u/kfrank">@KFrank</a>, thanks</p>

Sam-gege: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> thanks for your detailed explanation, that makes it really clear! upon some testing, I still have some doubts, can you please share some insights?</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="176419">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>The short story is that <code>abs()</code> requires an unmodified copy of its input<br>
tensor in order to compute its gradient during <code>backward()</code></p>
</blockquote>
</aside>
<p>from what I understand, during backward it needs to know where in the original data <code>x</code> did <code>abs</code> change, so that it inverts sign at the same position at the incoming gradient. i.e. the <code>x.grad += incoming_gradient[x&lt;0]*-1</code>. If this is the case, what about other functions that also requires input data for gradient calculation, such as <code>sqrt</code> (df/dx=0.5/sqrt(x))?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
x_init=torch.tensor([1., 4], requires_grad=True)
x=x_init+0
y=torch.sqrt(x)
x[0]=0
y.sum().backward()
print(f'x version: {x._version}')
print(f'y version: {y._version}')
print(f'x grad: {x_init.grad}')
</code></pre>
<p>The above code works and gives the correct <code>grad</code> value at <code>x</code>. However, if you change the third line to <code>y=torch.abs(x)</code>(or <code>y=torch.log(x)</code> etc.), it gives the <code>modified by inplace operation</code> error. What happens here? Thanks!</p>

KFrank: <p>Hi Sam!</p>
<aside class="quote no-group" data-username="Sam-gege" data-post="7" data-topic="176419" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam-gege/48/42338_2.png" class="avatar"> Sam-gege:</div>
<blockquote>
<p>what about other functions that also requires input data for gradient calculation, such as <code>sqrt</code> (df/dx=0.5/sqrt(x))?</p>
</blockquote>
</aside>
<p>The point here is that <code>sqrt()</code> saves its output, rather than its input, for<br>
use in the backward pass.  (<code>sqrt (x)</code> could save its input, <code>x</code>, but then<br>
in would have to recompute <code>sqrt (x)</code> from <code>x</code> in order to compute its<br>
gradient.  To be a little more efficient, it stores its output, <code>sqrt (x)</code>,<br>
instead.)</p>
<p>Try modifying <code>sqrt (x)</code>'s output, <code>y</code>, inplace, and see what happens.</p>
<p>Here is a post that goes through a few more examples of this sort of<br>
thing:</p>
<aside class="quote quote-modified" data-post="3" data-topic="169583">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/when-inplace-operation-are-allowed-and-when-not/169583/3">When inplace operation are allowed and when not</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Jie (and Srishti)! 

The part that can be confusing is that a given tensor used in the forward 
pass may or may not be used in the backward pass.  It depends on the math 
of the gradient computation (and the details of how the relevant backward() 
function is implemented).  It’s not always obvious. 
Here are some explanatory examples: 
The derivative of exp (x) is exp (x).  So it makes sense that pytorch saves 
the output of exp (x) when it is computed during the forward pass so that it 
can …
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

Sam-gege: <aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="176419">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>while <code>abs_()</code> <em>knows</em> that it is modifying its input<br>
inplace so it stores an unmodified <em>copy</em> of its input in the computation<br>
graph for use during <code>backward()</code>.</p>
</blockquote>
</aside>
<p>Hi Frank, one last question, if abs_() stores a copy of its input in the computation graph, wouldn’t this makes no different than its outplace version memory-wise? For example, for <code>y = abs(x)</code> there will be two tensors <code>x</code> and <code>y</code> in the computation graph; whereas for <code>abs_(x)</code> there is only one tensor <code>x</code>, but still you need to save an <code>x</code> value at version 0 for backward. So, what’s the point of such inplace ops?</p>
<p>From my understanding, there’s only a few functions that can really save memory by using their inplace version, such as sqrt, exp, tan, tanh, sigmoid, ReLU etc, because those derivatives can be expressed using output only. The only drawback is that you can’t do further inplace ops, because that would change the output. Am I correct?</p>
<p>Thanks again for your time!<br>
Regards,<br>
Sam</p>

soulitzer: <p>Indeed the memory savings from in-place is less pronounced when intermediary results are needed for backward anyway. This also kind of applies to the case where an in-place operation saves the outputs instead, since that prevents you from doing another in-place on that output.</p>
<p>In general though, in-place will still be useful for in two ways (1) memory savings for inference. (2) supporting the common pattern of modifying a small slice of a large tensor. (there should be memory savings here since in-place only needs to clone the small slice instead of the entire tensor in the case of an out-of-place)</p>