Question about the inplace operation

wwwangzhch: <p>In my forward function, I want to set all masked value be -inf after a sigmoid operation, here is my code:</p>
<pre><code class="lang-auto">    def forward(self, x, mask):
        x = self.bn(self.linear1(x))
        x = self.linear2(x).squeeze(-1)
        score = torch.sigmoid(x)
        score[mask] = -math.inf
        return score
</code></pre>
<p>when execute the backward propogation, I got</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>but if I change the order of inplace operation and sigmoid function, this error will not occur:</p>
<pre><code class="lang-auto">    def forward(self, x, prev_w, mask):
        x = self.bn(self.linear1(x))
        x = self.linear2(x).squeeze(-1)
        x[mask] = -math.inf
        score = torch.sigmoid(x)
        return score
</code></pre>
<p>under these circumstances, I also execute an inplace operation(x[mask] = -math.inf), but it works well in backpropagation (though the results are not  what I want since the masked value are set to 0 after sigmoid operation, not -inf), so here is my question:</p>
<ol>
<li>what is the difference between these two types of codes when executing backpropagation in Pytorch? I know inplace operation is not allowed in backpropagation, but in my understanding,</li>
</ol>
<blockquote>
<p>x[mask] = -math.inf</p>
</blockquote>
<p>is also an inplace operation, right?</p>
<ol start="2">
<li>how can I correctly get what I want, in other words, set all masked value to be -inf after sigmoid operation and can also properly execute backpropagation?</li>
</ol>
<p>Thanks for your help!</p>

wwwangzhch: <p>I found this issue on Github and maybe it’s related to my problem.<br>
<a href="https://github.com/pytorch/pytorch/issues/9391" rel="nofollow noopener">backward pass different behaviors with inplace operation</a></p>
<p>To my understand, when calculate the derivative of  score, Pytorch’s autograd mechanism uses the formula <code>S'(x) = S(x)(1-S(x))</code>, which means we cannot change the S(x) because we need it. In my second case, the inplace operation is before the sigmoid operation, so the second case doesn’t raise error.</p>
<p>But here comes another question (<img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:">), why the inplace operation on x doesn’t raise error? or more clearly, why the following codes doesn’t raise error?</p>
<pre><code class="lang-auto">a = torch.randn((5), requires_grad=True)
b = 2*a + 1
c = b*b + 1
b[1] = 0.
d = torch.mean(b)
d.backward()
</code></pre>
<p>in this case, <code>b</code>  is needed to correctly compute the gradient of  <code>c</code>, but <code>b</code> is modified after <code>c</code>,  why this case doesn’t raise error?</p>

albanD: <blockquote>
<p>why the inplace operation on x doesn’t raise error?</p>
</blockquote>
<p>Because in your second example, you change x <em>before</em> the sigmoid. So that is fine.</p>
<p>If you were modifying <code>x</code> after doing the sigmoid, it wouldn;t be a problem either as the sigmoid only requires its output to compute the gradient, not its input.<br>
If you want to see the details, you can see <a href="https://github.com/pytorch/pytorch/blob/56de8853dad19a050bc9ce5f89c3f7572ca8a15c/tools/autograd/derivatives.yaml#L755-L756">here</a> that <code>result</code> is used to compute the gradients but not <code>self</code>. So you can modify self as much as you want.</p>
<blockquote>
<p>Your example code</p>
</blockquote>
<p>I think you have a typo as you never use <code>c</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>