Leaf variable was used in an inplace operation

OswinLG: <p>I code a function which implements some operations including <a href="http://torch.mm" rel="nofollow noopener">torch.mm</a>, torch.index_select and <a href="http://torch.cc" rel="nofollow noopener">torch.cc</a>. However, there comes out an AssertionError, <em>leaf variable was used in an inplace operation</em>.</p>
<p>In the source code of <em><strong><a href="http://Variable.py" rel="nofollow noopener">Variable.py</a></strong></em>(line 199), I found the assertion, <em><strong>assert self.</strong>__<strong>version == 0</strong></em>. But it’s not clear to say what is going wrong here. Could anyone help me on this?</p>

colesbury: <p>Loosely, tensors you create directly are leaf variables. Tensors that are the result of a differentiable operation are not leaf variables</p>
<p>For example:</p>
<pre><code class="lang-auto">w = torch.tensor([1.0, 2.0, 3.0]) # leaf variable
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True) # also leaf variable
y = x + 1  # not a leaf variable
</code></pre>
<p>(The PyTorch documentation for <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.is_leaf"><code>is_leaf</code></a> contains a more precise definition.)</p>
<p>An in-place operation is something which modifies the data of a variable. For example:</p>
<pre><code class="lang-auto">x += 1  # in-place
y = x + 1 # not in place
</code></pre>
<p>PyTorch doesn’t allow in-place operations on leaf variables  that have <code>requires_grad=True</code> (such as parameters of your model) because the developers could not decide how such an operation should behave. If you want the operation to be differentiable, you can work around the limitation by cloning the leaf variable (or use a non-inplace version of the operator).</p>
<pre><code class="lang-auto">x2 = x.clone()  # clone the variable
x2 += 1  # in-place operation
</code></pre>
<p>If you don’t intend for the operation to be differentiable, you can use <code>torch.no_grad</code>:</p>
<pre><code class="lang-auto">with torch.no_grad():
    x += 1
</code></pre>

henrye: <p>Is it bad practice to get around pytorch disallowing in-place operations by assigning to Variable().data?</p>

tom: <p>That depends on the situation.<br>
For example to initialize or update parameters, assigning to <code>.data</code> is the way to go. Usually, you cannot backprop when changing Variables’ <code>.data</code> in the middle of a forward pass…<br>
I’d summerise my experience as “don’t do it unless you have reason to believe its the rigght thing”.</p>
<p>Best regards</p>
<p>Thomas</p>

henrye: <p>Yes, I remember now, messing with backprop and autograd was why I was running into problems with in-place assignment before. Using .data as I am currently for initialising word embeddings seems ok then.</p>

Yoni_Keren: <p>Hi,</p>
<p>Trying to read the code for optim (I want to implement something a bit differently) and your previous example/explanation of what is a leaf Variable doesn’t seem to be valid anymore.</p>
<p>In particular,<br>
you wrote</p>
<pre><code class="lang-auto">y = x + 1  # not a leaf variable
</code></pre>
<p>Well, here’s output from my termial for the code which you have mentioned:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]))
&gt;&gt;&gt; x.is_leaf
True
&gt;&gt;&gt; y = x + 1
&gt;&gt;&gt; y.is_leaf
True
&gt;&gt;&gt; y
Variable containing:
 2
 3
 4
 5
[torch.FloatTensor of size (4,)]
</code></pre>
<p>So, can someone please explain what is a leaf Variable, and what is not a leaf variable? Clearly a non-leaf-variable cannot be optimized, but what is it?</p>

Ed_Beeching: <pre><code>Came across a similar issue. Reason is requires grad.

x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]), requires_grad=True)
x.is_leaf    
#True
y = x + 1
y.is_leaf
#False</code></pre>

11185: <p>Hi, I used to create leaf variable like:</p>
<p><code>y = torch.autograd.Variable(torch.zeros([batch_size, c, h, w]), requires_grad=True)</code></p>
<p>Then I want to assign value to indexed parts of y like below,(y_local is a Variable computed based on other variables and I want to assign the value of y_local to part of the y and ensure that the gradients from y can flow to the y_local.)</p>
<p><code>y.data[:,:,local_x[i]:local_x[i+1],local_y[i]:local_y[i+1]] = y_local.data</code></p>
<p>I am wondering such operation supports the normal gradient backward for the y_local varible?</p>

shivangi: <p>I am also facing similar issue. Please let me know how were you able to resolve it</p>

dreamyun: <p>leaf variable, in essence, is a variable, or a tensor with requires_grad=True. So, if a tensor with requires_grad=False, it does not belong to the variable, let alone leaf variable.</p>

klory: <p>But <code>y</code> is a variable in this case, it’s not a leaf node.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c8b628ff34426df9d06b2ad2af04573ca0a14f25.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8b628ff34426df9d06b2ad2af04573ca0a14f25" title="50%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c8b628ff34426df9d06b2ad2af04573ca0a14f25_2_690x302.png" alt="50%20PM" data-base62-sha1="sDzLLpyUb2znaOzNPo7Lf6JKcBv" width="690" height="302" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c8b628ff34426df9d06b2ad2af04573ca0a14f25_2_690x302.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c8b628ff34426df9d06b2ad2af04573ca0a14f25_2_1035x453.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/c8b628ff34426df9d06b2ad2af04573ca0a14f25.png 2x" data-dominant-color="030303"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">50%20PM</span><span class="informations">1166×512 28.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

klory: <aside class="quote no-group quote-modified" data-username="colesbury" data-post="2" data-topic="308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colesbury/48/26_2.png" class="avatar"> colesbury:</div>
<blockquote>
<p>because that usually isn’t correct.</p>
</blockquote>
</aside>
<p>Could you please explain why it’s not correct?</p>

danmoller: <p>That usually isn’t corret… well… what if it IS correct? What should I do?<br>
I’m pretty sure of what I’m doing, but can’t do it on Pytorch.</p>

nima_rafiee: <p>I had a similar case. I used:</p>
<pre><code class="lang-auto">y = torch.zeros([batch_size, c, h, w]), requires_grad=False)
</code></pre>
<p>then I update the value of y according to the value of the network output and then apply a loss function on y and it worked for me.</p>

pinocchio: <aside class="quote no-group quote-modified" data-username="colesbury" data-post="2" data-topic="308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colesbury/48/26_2.png" class="avatar"> colesbury:</div>
<blockquote>
<p>PyTorch doesn’t allow in-place operations on variables you create directly (such as parameters of your model) because that usually isn’t correct</p>
</blockquote>
</aside>
<p>Is the reason this is not “usually correct” because we could have just initialized it directly with the data that we wanted in the first place instead of doing a in-place op?</p>

pinocchio: <aside class="quote no-group" data-username="Yoni_Keren" data-post="6" data-topic="308" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yoni_keren/48/827_2.png" class="avatar"> Yoni_Keren:</div>
<blockquote>
<p>Hi,</p>
<p>Trying to read the code for optim (I want to implement something a bit differently) and your previous example/explanation of what is a leaf Variable doesn’t seem to be valid anymore.</p>
<p>In particular,<br>
you wrote</p>
<pre><code class="lang-auto">y = x + 1  # not a leaf variable
</code></pre>
<p>Well, here’s output from my termial for the code which you have mentioned:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]))
&gt;&gt;&gt; x.is_leaf
True
&gt;&gt;&gt; y = x + 1
&gt;&gt;&gt; y.is_leaf
True
&gt;&gt;&gt; y
Variable containing:
 2
 3
 4
 5
[torch.FloatTensor of size (4,)]
</code></pre>
<p>So, can someone please explain what is a leaf Variable, and what is not a leaf variable? Clearly a non-leaf-variable cannot be optimized, but what is it?</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/colesbury">@colesbury</a> can you address this question please?</p>
<p>Why is <code>y</code> a leaf when you claim it should not?</p>
<p>I also agree it shouldn’t be a leaf but Pytorch disagrees with us…why?</p>

colesbury: <p><a class="mention" href="/u/pinocchio">@pinocchio</a>, I’m updating my reply and correcting the example. The not “usually correct” wasn’t a good explanation. The actual reason is that the PyTorch developers could not come to a consensus on reasonable semantics for such an operation.</p>

pinocchio: <p>I think you are wrong, <code>y</code> is indeed <strong>not a leaf</strong>. Maybe you had a weird version of Pytorch?</p>
<pre><code class="lang-auto">def inplace_playground():
    import torch

    x = torch.tensor([1,2,3.], requires_grad=True)
    y = x + 1
    print(f'x.is_leaf = {x.is_leaf}')
    print(f'y.is_leaf = {y.is_leaf}')
    x += 1
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">x.is_leaf = True
y.is_leaf = False
</code></pre>
<hr>
<p><a class="mention" href="/u/colesbury">@colesbury</a> I think you were correct. Not sure what you corrected but I tried the leaf thing and it seems your right that y is not a leaf (as expected).</p>
<p>Thanks!</p>

pinocchio: <aside class="quote no-group" data-username="colesbury" data-post="17" data-topic="308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/colesbury/48/26_2.png" class="avatar"> colesbury:</div>
<blockquote>
<p>The actual reason is that the PyTorch developers could not come to a consensus on reasonable semantics for such an operation.</p>
</blockquote>
</aside>
<p>Why not? What were the competing semantics? What’s the difficulty in defining the semantics for leafs + in-place ops?</p>

jeremysun1224: <p>Perhaps, you could try adding the following code before updating the gradient:</p>
<pre><code class="lang-auto">with torch.no_grad():
</code></pre>
<p>and it works:</p>
<pre><code class="lang-auto">N, D_in, H, D_out = 64, 1000, 100, 10

X = torch.randn(N, D_in).cuda()
y = torch.randn(N, D_out).cuda()

# device = torch.cuda.device('cuda:0')
device = torch.device('cuda:0')
w1 = torch.randn(D_in, H, requires_grad=True, device=device)
w2 = torch.randn(H, D_out, requires_grad=True, device=device)

learning_rate = 1e-6
for it in range(10):
    # 1.forward pass
    y_pred = X.mm(w1).clamp(min=0).mm(w2).cuda()
    # 2.compute loss
    loss = (y_pred - y).pow(2).sum().cuda()
    print(f'iter {it}, loss {loss}')
    # Backward pass
    loss.backward()
    # update weights of w1 and w2
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad
        w1.grad.zero_()
        w2.grad.zero_()
</code></pre>