Understanding Autograd + ReLU(inplace = True)

username: vadimkantorov
cooked: <p>How do I compute/check/understand gradients of inplace ReLU? I did <code>clone()</code> to bypass “a leaf Variable that requires grad has been used in an in-place operation”. Gradient should obviously be 0, but I get 1.</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

# not in place
x = torch.tensor([-1.0]).requires_grad_().clone()
print(torch.autograd.grad(F.relu(x), (x, ))[0])
# tensor([0.])

# inplace
x = torch.tensor([-1.0]).requires_grad_().clone()
print(torch.autograd.grad(F.relu_(x), (x, ))[0])
# tensor([1.])
</code></pre>
<p>My final usecase is testing some ResNet-like code like following:</p>
<pre><code class="lang-python">def residual_activation_inplace(x, residual, activation, inplace):
  for r in residual:
    x += r
  return activation(x, inplace = inplace)

x = torch.rand(3, 4)
residual = [torch.rand(3, 4), torch.rand(3, 4)]
y = residual_activation_inplace(x, residual, F.relu, inplace = True)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Since you apply the relu inplace in the second case, <code>x</code> now points to the output of the relu. And so you actually do <code>dx/dx = 1</code>.</p>
<p>If you do the following to have access to the gradient of the original x (before the inplace), it will work.</p>
<pre><code class="lang-auto">x = torch.tensor([-1.0]).requires_grad_()
x2 = x.clone()
print(torch.autograd.grad(F.relu_(x2), (x, ))[0])
</code></pre>
<p>In spirit, you can replace <code>F.relu_(x)</code> by <code>y = F.relu(x); x = y</code>. Where <code>x</code> is the actual python variable and thus will affect everything that uses x.</p>

username: vadimkantorov
cooked: <p><code>d(x)/dx = 1</code> - makes sense <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: singleroc
cooked: <p>Hi, guys,</p>
<p>I am confused by the gradient of inplace version ReLU until I found this thread. However, it is still unclear how the backward performs in inplace ReLU.</p>
<p>Base on <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml</a></p>
<blockquote>
<p>name: relu_(Tensor(a!) self) → Tensor(a!)<br>
self: threshold_backward(grad, result, 0)</p>
</blockquote>
<p>we can see the inplace ReLU (relu_) replies on the output tensor (result) for its back-propogation, however, this tensor has already been modified to non-negative range. It is confusing how to passing back gradient as we lost the sign of the tensor.</p>
<p>See example below, for inplace ReLU, the saved_tensors in the backward function is non-negative</p>
<blockquote>
<p>class MyReLU(torch.autograd.Function):<br>
“”"<br>
We can implement our own custom autograd Functions by subclassing<br>
torch.autograd.Function and implementing the forward and backward passes<br>
which operate on Tensors.<br>
“”"<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, input):<br>
“”"<br>
In the forward pass we receive a Tensor containing the input and return<br>
a Tensor containing the output. ctx is a context object that can be used<br>
to stash information for backward computation. You can cache arbitrary<br>
objects for use in the backward pass using the ctx.save_for_backward method.<br>
“”"<br>
ctx.save_for_backward(input)<br>
return input.clamp(min=0)<br>
<span class="mention">@staticmethod</span><br>
def backward(ctx, grad_output):<br>
“”"<br>
In the backward pass we receive a Tensor containing the gradient of the loss<br>
with respect to the output, and we need to compute the gradient of the loss<br>
with respect to the input.<br>
“”"<br>
input, = ctx.saved_tensors<br>
grad_input = grad_output.clone()<br>
grad_input[input &lt; 0] = 0    # <em>for inplace version, grad_input = grad_output, as input is modified into non-negative range?</em><br>
return grad_input</p>
</blockquote>
<p>Thus, the only way for correct inplace ReLU backward propogration is to save another tensor which indicate the sign of the input (flag = input &lt; 0)?</p>
<p>In other word, for inplace ReLU, a flag (e. g. flag = input &lt; 0) tensor is saved for backward, rather than the input/output tensor?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The gradient we want to compute here is indeed: <code>1 if input &gt; 0</code> and <code>0 if inputs &lt;= 0</code>.<br>
The nice thing is that <code>inputs &lt;= 0</code> &lt;=&gt; <code>relu(inputs) = 0</code>.<br>
So we can actually compute the gradient based on the result with <code>grad_input[result == 0] = 0</code> (or with &lt;=, that would give the same result as result &gt;=0).</p>

username: singleroc
cooked: <p>Thank you for the explain. With  grad_input[result == 0] = 0, no necessary for extra flag stored. Clear.</p>