Confused about in-place operation in .backward()

njucckevin: <p>I encounter this problem when trying to assign value for tensor like below (RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation) . Notice that my in-place operation is before calculate <code>loss = (b*b).mean()</code>.</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)  # 1.8.1
torch.autograd.set_detect_anomaly(True)
a = torch.rand(3, requires_grad=True)
b = torch.sigmoid(a)
b[0] = 1
loss = (b*b).mean()
b.retain_grad()
loss.backward()  # RuntimeError: one of the variables needed for gradient computation has been modified by an in-place operation...
</code></pre>
<p>After that, I found that if replace <code>torch.sigmoid()</code> step by step like below, the .backward() could work successfully.</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)  # 1.8.1
torch.autograd.set_detect_anomaly(True)
a = torch.rand(3, requires_grad=True)
# b = torch.sigmoid(a)
b = 1 / (1+torch.exp(-a))
b[0] = 1
loss = (b*b).mean()
b.retain_grad()
loss.backward()  # success
print(b.grad)  # tensor([0.6667, 0.4498, 0.4630])
print(a.grad)  # tensor([0.0000, 0.0987, 0.0982])
</code></pre>
<p>In addition, I also found that using <code>.index_fill()</code> like below, the code could also work.</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)  # 1.8.1
torch.autograd.set_detect_anomaly(True)
a = torch.rand(3, requires_grad=True)
b = torch.sigmoid(a)
# b[0] = 1
b = b.index_fill(0, torch.LongTensor([0]), 1)
loss = (b*b).mean()
b.retain_grad()
loss.backward()  # success
print(b.grad)  # tensor([0.6667, 0.4498, 0.4630])
print(a.grad)  # tensor([0.0000, 0.0987, 0.0982])
</code></pre>
<p>In conclusion, I have the following questions:</p>
<ol>
<li>Why is there in-place error with using torch.sigmoid()? But the step by step has no problem.</li>
<li>Why after using index_fill the code could work?</li>
</ol>
<p>Thanks for reply!</p>

soulitzer: <ol>
<li>because <code>torch.sigmoid</code> saves its <strong>output</strong> to use in the backward computation, modifying the output of sigmoid “b” in-place will result in an error. If you split torch.sigmoid such that the last out of place operation is a division, that actually saves its <strong>input</strong> for backward, so it doesn’t care if you modified the output in-place.</li>
<li>index_fill is an out-of-place op, when you do <code>b = b.index_fill(...)</code> you are just changing what object “b” points to.</li>
</ol>

njucckevin: <ol>
<li>Why does <code>output = torch.sigmoid(input)</code> design this way? Actually it can do backward computation without using output (by using just input).</li>
<li>In my opinion, the reason for we can’t using in-place op is that the variable value changes between forward and backward. So why out-of-place op is ok for this situation? Maybe even we have <code>b = b.index_fill(...)</code>, the previous b (without index_fill) is still saved in the computation graph?</li>
</ol>

soulitzer: <ol>
<li>its convenient to use the output because sigmoid’s derivative is <code>sigmoid(x)(1 - sigmoid(x))</code>
</li>
<li>yes the reason we can’t use in-place op is because you are modifying the saved variable in-place. If you do an out-of-place op here, yes the previous b is still saved in the computation graph, except now you are creating a new tensor object, and pointing b to the new tensor object. The saved variable shares storage with the old tensor object, so it is fine.</li>
</ol>

njucckevin: <p>I understand. You help me a lot, thanks!</p>