Gradient computaion quesiton

bingqing_liu: <pre><code class="lang-auto">a=nn.Parameter(torch.tensor([[2.]]))
b=a*2
print(b)
d=nn.Parameter(torch.tensor(3.))
print(d)
e=b[[0],:]*d
b[0]=e#inplaced
print(e)
loss1=e.sum()
e=b[[0],:]*d
b[0]=e
print(e)
loss1+=e.sum()
</code></pre>
<p><code>loss1.backward()</code> works well,why?the initial b[0](</p>
<blockquote>
<p>tensor([[4.]], grad_fn=&lt;MulBackward0&gt;</p>
</blockquote>
<p>) has been inplaced by e,how can this loss backward to a?</p>
<p>i just find the gradient to a works well,but do not know why?becuase i konw pytorch always encounter problems when using inplace operation,but this case works well,why?</p>

KFrank: <p>Hi Bingqing!</p>
<aside class="quote no-group quote-modified" data-username="bingqing_liu" data-post="1" data-topic="146142" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bingqing_liu/48/31455_2.png" class="avatar"> bingqing_liu:</div>
<blockquote>
<p>becuase i konw pytorch always encounter problems when using inplace operation,but this case works well,why?<br>
…</p>
<pre><code class="lang-auto">e=b[[0],:]*d
b[0]=e#inplaced
</code></pre>
</blockquote>
</aside>
<p>The <code>b[[0],:]</code> in the line <code>e=b[[0],:]*d</code> creates a new tensor, rather<br>
than just a view into <code>b</code>, and it is this new tensor that is stored in the<br>
computation graph and then used for  backpropagation.  So, even<br>
though <code>b[0]=e#inplaced</code> does modify <code>b</code> in place, the actual tensor<br>
stored in the computation graph is not modified in place.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.10.2
&gt;&gt;&gt;
&gt;&gt;&gt; a = torch.nn.Parameter (torch.tensor ([[2.]]))
&gt;&gt;&gt; b = a * 2
&gt;&gt;&gt; c_new =  b[[0], :]   # c_new is a new tensor
&gt;&gt;&gt; c_view = b[0]        # c_view is a view into b
&gt;&gt;&gt; print (b)
tensor([[4.]], grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; print (c_new)
tensor([[4.]], grad_fn=&lt;IndexBackward0&gt;)
&gt;&gt;&gt; print (c_view)
tensor([4.], grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; b[0] = 99.9          # inplace modification of b
&gt;&gt;&gt; print (b)
tensor([[99.9000]], grad_fn=&lt;CopySlices&gt;)
&gt;&gt;&gt; print (c_new)        # c_new is not modified -- won't break backpropagation
tensor([[4.]], grad_fn=&lt;IndexBackward0&gt;)
&gt;&gt;&gt; print (c_view)       # c_view reflects the modification -- breaks backpropagation
tensor([99.9000], grad_fn=&lt;AsStridedBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; a = torch.nn.Parameter (torch.tensor ([[2.]]))
&gt;&gt;&gt; b = a * 2
&gt;&gt;&gt; print (b)
tensor([[4.]], grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; d = torch.nn.Parameter (torch.tensor (3.))
&gt;&gt;&gt; print (d)
Parameter containing:
tensor(3., requires_grad=True)
&gt;&gt;&gt; e = b[[0], :] * d    # b[[0], :] is actually a new tensor in the computation graph
&gt;&gt;&gt; b[0] = e             # inplaced, but doesn't break backpropagation
&gt;&gt;&gt; print (e)
tensor([[12.]], grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; loss1 = e.sum()
&gt;&gt;&gt; loss1.backward()
&gt;&gt;&gt;
&gt;&gt;&gt; a = torch.nn.Parameter (torch.tensor ([[2.]]))
&gt;&gt;&gt; b = a * 2
&gt;&gt;&gt; print (b)
tensor([[4.]], grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; d = torch.nn.Parameter (torch.tensor (3.))
&gt;&gt;&gt; print (d)
Parameter containing:
tensor(3., requires_grad=True)
&gt;&gt;&gt; e = b[0] * d         # b[0] is a view into b
&gt;&gt;&gt; b[0] = e             # inplaced -- breaks backpropagation
&gt;&gt;&gt; print (e)
tensor([12.], grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; loss1 = e.sum()
&gt;&gt;&gt; loss1.backward()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 156, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Best.</p>
<p>K. Frank</p>