How to perform indexing with autograd?

username: smani
cooked: <p>I need to set all the values which are greater than 0 to those values itself and all the values less than zeros to exp(a*x), where a is a constant. I tried as follows:</p>
<pre><code>   x[x&lt;=0] = torch.exp(a*x[x&lt;=0])
</code></pre>
<p>Is it correct?<br>
I am getting a run time error: a leaf variable that requires grad has been used in an in-place operation.<br>
How to overcome this error?</p>

username: sbelharbi
cooked: <p>The instruction seems right; and it does what you want.<br>
<a href="https://discuss.pytorch.org/t/leaf-variable-was-used-in-an-inplace-operation/308/2">However</a>, in term of autograd, Pytorch does not like it when performing in-place operations on leaf-variable that require grad.</p>
<p>Here some code, in Pytroch 1.0.0, Python 3.7.0, to reproduce your error:</p>
<pre><code class="lang-python">import torch
from torch.distributions import normal

# Reproducibility
torch.manual_seed(0)

# Create data
dist = normal.Normal(0., 1.)
y = dist.sample((5, 2))

# This is your leaf-variable.
x = y.data
# Make it requires grad.
x.requires_grad = True

print("X BEFORE:\n {}".format(x))
# Make it non-leaf-variable!!!
# x = x + 0.  # if you remove this you will obtain the error: x[x &lt;= 0] = torch.exp(a*x[x &lt;= 0])
# RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

a = 10.
x[x &lt;= 0] = torch.exp(a*x[x &lt;= 0])

print("X AFTER:\n {}".format(x))
</code></pre>
<p>Running the above code will generate the error:</p>
<pre><code class="lang-bash">Traceback (most recent call last):
  File "xxxxx.py", line 21, in &lt;module&gt;
    x[x &lt;= 0] = torch.exp(a*x[x &lt;= 0])
RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>
<p>When making <code>x</code> non-leaf variable, autograd works:</p>
<pre><code class="lang-bash">X BEFORE: 
tensor([[ 1.5410, -0.2934],
        [-2.1788,  0.5684],
        [-1.0845, -1.3986],
        [ 0.4033,  0.8380],
        [-0.7193, -0.4033]], requires_grad=True)
X AFTER:
 tensor([[1.5410e+00, 5.3169e-02],
        [3.4486e-10, 5.6843e-01],
        [1.9498e-05, 8.4329e-07],
        [4.0335e-01, 8.3803e-01],
        [7.5215e-04, 1.7713e-02]], grad_fn=&lt;IndexPutBackward&gt;)
</code></pre>
<p>The trick here is to apply a non-in-place operation over a leaf variable to make it non-leaf variable.  In this example, we added <code>0.</code> to <code>x</code>. Probably, there are better and canonical ways to do it in Pytorch.</p>
<p>Setting <code>x.requires_grad = False</code> works fine, without the need to make the leaf variable non-leaf:</p>
<pre><code class="lang-python">import torch
from torch.distributions import normal

# Reproducibility
torch.manual_seed(0)

# Create data
dist = normal.Normal(0., 1.)
y = dist.sample((5, 2))

# This is your leaf-variable.
x = y.data
# Make it requires grad.
x.requires_grad = False

print("X BEFORE:\n {}".format(x))
# Make it non-leaf-variable!!!
# x = x + 0.  # if you remove this you will obtain the error: x[x &lt;= 0] = torch.exp(a*x[x &lt;= 0])
# RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

a = 10.
x[x &lt;= 0] = torch.exp(a*x[x &lt;= 0])

print("X AFTER:\n {}".format(x))
</code></pre>
<p>Running gives:</p>
<pre><code class="lang-bash">X BEFORE:
 tensor([[ 1.5410, -0.2934],
        [-2.1788,  0.5684],
        [-1.0845, -1.3986],
        [ 0.4033,  0.8380],
        [-0.7193, -0.4033]])
X AFTER:
 tensor([[1.5410e+00, 5.3169e-02],
        [3.4486e-10, 5.6843e-01],
        [1.9498e-05, 8.4329e-07],
        [4.0335e-01, 8.3803e-01],
        [7.5215e-04, 1.7713e-02]])
</code></pre>
<p>I hope this helps.</p>