Clone to avoid in-place operations and runtime error during gradient calculation

username: marioo
cooked: <p>Hi,</p>
<p>From this link:<br>
<a href="https://discuss.pytorch.org/t/encounter-the-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/836">https://discuss.pytorch.org/t/encounter-the-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/836</a><br>
I learned that sometimes we cannot use in-place operations in the forward because it will yield an error during backpropagation, and it is suggested to use <code>.clone()</code> before in-place operations. However, I don’t understand well when it is needed to use <code>.clone()</code>. For instance, in the following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

def mut(x, w, mask): return w*x[mask]

# Weights
w1 = torch.ones(5, requires_grad=True)
w2 = torch.ones(3, requires_grad=True)
w3 = torch.ones(3, requires_grad=True)

x = 2*torch.ones(5)
mask = torch.tensor([True, False, True, False, True], dtype=torch.bool)


x = w1*x
x = F.selu(x).clone()
x[mask] = mut(x, w2, mask)
x[mask] = F.selu(x[mask])
x[mask] = mut(x, w3, mask)
x.mean().backward()
</code></pre>
<p>I need to add <code>.clone()</code> after the first SeLU, if I added it in the next line: <code>x[mask] = mut(x.clone(), w2, mask)</code> it does not work. Why is this?</p>
<p>Also, it seems that I only need to use  <code>.clone()</code> before the first call to <code>mut() </code>, but not before the second time. Why?</p>
<p>Probably I am missing something. I would be very grateful for an explanation to where and when it is needed to clone.</p>
<p>Thanks a lot,</p>
<p>Mario</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="marioo" data-post="1" data-topic="117354">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marioo/48/13447_2.png" class="avatar"> marioo:</div>
<blockquote>
<p>I need to add <code>.clone()</code> after the first SeLU, if I added it in the next line: <code>x[mask] = mut(x.clone(), w2, mask)</code> it does not work. Why is this?</p>
</blockquote>
</aside>
<p>let’s disambiguate things first, this is working:<br>
a = F.selu(x)<br>
b = a.clone()<br>
b[mask] = mut(b, w2, mask)<br>
b[mask] = F.selu(b[mask])<br>
b[mask] = mut(b, w3, mask)</p>
<p>your breaking change:<br>
a = F.selu(x)<br>
b = a.clone()<br>
a[mask] = mut(b, w2, mask)<br>
a[mask] = F.selu(a[mask])<br>
a[mask] = mut(a, w3, mask)</p>
<p>This suggests that “a” is referenced in some backward function, that prevents "a[mask] = " ops, but this problematic reference doesn’t exist in the first code with “b”.</p>
<p>Obvious culprit is F.selu itself, that keeps a reference to its output. It is hard to tell when this is the case - autograd code generation manages stored tensors, AFAIK this is not documented.</p>
<p>Relevant entry in tools/autograd/derivatives.yaml:<br>
self: elu_backward(grad, alpha, scale, input_scale, <strong>result</strong>)</p>
<aside class="quote no-group" data-username="marioo" data-post="1" data-topic="117354">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marioo/48/13447_2.png" class="avatar"> marioo:</div>
<blockquote>
<p>Also, it seems that I only need to use <code>.clone()</code> before the first call to <code>mut() </code>, but not before the second time. Why?</p>
</blockquote>
</aside>
<p>You’re accidentally avoiding problems with mut() because x[mask] in it copies values from x to a new memory. Note that this behaviour is different from “x[mask] = …” in-place assignments, handled by <code>__setitem__</code>.</p>

username: marioo
cooked: <p>Understood. Thank you very much for your explanation <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>