In-place operation when indexing target Variable with ByteTensor

username: dpernes
cooked: <p>I have a problem where I need to index a Variable that requires gradient using a ByteTensor. However, this seems to be an in-place operation, and so PyTorch throws an error:</p>
<pre><code class="lang-auto">RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>
<p>The following piece of code reproduces the error in a very simple way:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

x = torch.LongTensor([0,1,0,1,1,0])
y = Variable(torch.zeros(x.size()), requires_grad=True)

y[x == 0] = Variable(torch.Tensor([-5]))

print('y',y)
</code></pre>
<p>Clearly, the intended output would be:</p>
<pre><code class="lang-auto">y Variable containing:
-5
 0
-5
 0
 0
-5
[torch.FloatTensor of size (6,)]
</code></pre>
<p>As a workaround, we can use an explicit for-loop over the elements of x:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

x = torch.LongTensor([0,1,0,1,1,0])
y = Variable(torch.zeros(x.size()), requires_grad=True)

for i in range(x.size(0)):
  if x[i] == 0:
    y[i] = Variable(torch.Tensor([-5]))

print('y',y)
</code></pre>
<p>Here, <code>y[i] = Variable(torch.Tensor([-5]))</code> is not an in-place operation (don’t know why) and therefore it works. However, this for-loop is very inefficient, taking a lot of time if <code>x</code> has a large dimension.</p>
<p>Are there any alternatives? Thanks in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Since gradients are needed for <code>y</code> itself, your can’t modify it inplace.<br>
What you can do is clone it before, that way the cloned version will not be a leaf Variable and can thus be changed inplace:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable

x = torch.LongTensor([0,1,0,1,1,0])
y = Variable(torch.zeros(x.size()), requires_grad=True)

tmp = y.clone()
tmp[x == 0] = Variable(torch.Tensor([-5]))

print('y',y)
print('tmp',tmp)
</code></pre>

username: dpernes
cooked: <p>Hi,</p>
<p>Thank you for your reply!<br>
That’s a pretty weird behavior… Just for curiosity, why can’t the leaf variables be changed in-place?</p>

username: albanD
cooked: <p>Because a gradient update for a leaf Variable <code>a</code> is (with extra terms if you use fancy optimizer) <code>a = a + lambda * a.grad</code>. As you can see, you need the original value of <code>a</code> to be able to perform this update. This is why you cannot change this value inplace, otherwise, you couldn’t do this operation anymore since you would have lost the original a.</p>

username: dpernes
cooked: <p>I thought PyTorch would not perform gradient updates on leaf Variables, but only on model Parameters. Isn’t that right?</p>

username: albanD
cooked: <p><code>Parameter</code> comes from the <code>nn</code> package and is just a convenient tool to define parameters in the context of neural networks.<br>
From the <code>autograd</code> point of view, they are <code>Variable</code> with <code>requires_grad=True</code> (hence they are leaf Variables).</p>
<p>The <code>optim</code> package works with any <code>Variable</code> that has <code>requires_grad=True</code>, so you can use it without using <code>nn</code>  (and thus without using <code>Parameter</code>s), but it is less convenient for neural network usage.<br>
But for example, you can use pytorch to get numerical value of any mathematical function (even though this is not it’s original purpose). The sample below for example show you how to implement a function that computes numerical gradients for any 1D function:</p>
<pre><code class="lang-auto">import torch
from torch import autograd
from torch.autograd import Variable

def square(x):
    return x ** 2
def cube(x):
    return x ** 3

# Implement derivative of fn at point y (fn has to be R -&gt; R)
def derivative(fn, y):
    y.requires_grad = True
    out = fn(y)
    return autograd.grad(out, y)[0]


x = Variable(torch.Tensor([3]))

print("Evaluate square at 3")
print(square(x))
print("Evaluate derivative of square at 3 (should be 2*3)")
print(derivative(square, x))
print("Evaluate derivative of cube at 3 (should be 3*(3)**2")
print(derivative(cube, x))
</code></pre>

username: dpernes
cooked: <p>Yeah, it makes total sense <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"> Thank you!</p>

username: BruceShakeham
cooked: <p>Hi Aldan,</p>
<p>Just curious. The same rule would apply also to non-leaf variables. Then how come it is okay to do in-place operation for non-leaf variables? Or, are we running into any danger for doing in-place operations for non-leaf variables, even if there was no error message? Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The difference for non-leafs, is that you don’t mind loosing the reference to the original value. Because changing it inplace means that you actually just want the new version.<br>
We do extensive checks to make sure that if you don’t get any error, the gradients are correct. Note that in some cases, you get an error even though we could compute the gradients in theory, but we prefer to be over-restrictive here.<br>
So if you don’t get any error, it works fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>