When inplace operation are allowed and when not

jie_yang: <p>I’m trying to understand how the shortcuts work in neural networks models (e.g., ResNet models) implemented using Pytorch.</p>
<p>I’m trying the following ResNet block found on a github repository:</p>
<pre><code class="lang-auto">class BasicBlock(nn.Module):
    expansion = 1
    def __init__(self, in_planes, planes, stride=1,
                 shortcut_enabled = True, 
                 device='cuda'
                ):
        super(BasicBlock, self).__init__()
        
        self.shortcut_enabled = shortcut_enabled
        
        
        self.conv1    = nn.Conv2d(in_planes, planes, 
                         kernel_size=3, 
                         stride=stride, padding=1, bias=False,
                                  device=device)
        self.bn1      = nn.BatchNorm2d(planes, device=device)
        self.conv2    = nn.Conv2d(planes, planes, 
                        kernel_size=3,
                        stride=1, padding=1, bias=False,
                        device=device)
        self.bn2      = nn.BatchNorm2d(planes, device=device)

        if self.shortcut_enabled:
            self.shortcut = nn.Sequential()
            if stride != 1 or in_planes != self.expansion*planes:
                self.shortcut = nn.Sequential(nn.Conv2d(in_planes, self.expansion*planes,kernel_size=1, stride=stride, bias=False, device=device),nn.BatchNorm2d(self.expansion*planes,device=device)
)

    def forward(self, x, verbose=0):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        if self.shortcut_enabled:
            out += self.shortcut(x)
        out = F.relu(out)
        return out
</code></pre>
<p>This block of code runs (both in learning and evaluation mode) without any problem. However, since it makes an inplace operation on a Tensor, i.e. <code>out += self.shortcut(x)</code> , I expected an error like <code>one of the variables needed for gradient computation has been modified by an inplace operation</code>, as in other similar models that I tested in the past.</p>
<p>I would understand when, in Pytorch, an inplace operation can be made and when it is not allowed.<br>
why can Resnet Block use in-place operation and It works when training.</p>

srishti-git1110: <p>Hi,<br>
In-place operations produce the error you mentioned when a variable used in a computation graph (and hence will be involved in the backward call for gradient computation) is modified in-place and hence, during the time the backward call is made, has its value changed/at a different version from what it was when this variable was used in the graph (forward pass).</p>
<p>Since the value is now changed, it would lead to wrong computation of gradients - PyTorch flags this as an error once the backward pass is started.</p>
<p>TLDR - if there’s a mismatch in the tensor versions between when the tensor was used to construct the computation graph vs when the backward call is made, an error is produced.</p>
<p>In your case, if there’s no such error, there must be no conflicting tensor versions and you should be fine using the in-place ops.</p>

KFrank: <p>Hi Jie (and Srishti)!</p>
<aside class="quote no-group" data-username="jie_yang" data-post="1" data-topic="169583" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jie_yang/48/56139_2.png" class="avatar"> jie_yang:</div>
<blockquote>
<p>I would understand when, in Pytorch, an inplace operation can be made and when it is not allowed.</p>
</blockquote>
</aside>
<p>The part that can be confusing is that a given tensor used in the forward<br>
pass may or may not be used in the backward pass.  It depends on the math<br>
of the gradient computation (and the details of how the relevant <code>backward()</code><br>
function is implemented).  It’s not always obvious.</p>
<p>Here are some explanatory examples:</p>
<p>The derivative of <code>exp (x)</code> is <code>exp (x)</code>.  So it makes sense that pytorch saves<br>
the <em>output</em> of <code>exp (x)</code> when it is computed during the forward pass so that it<br>
can reuse it – rather than recompute it – during the backward pass.  If you<br>
modify the output of <code>exp (x)</code> (inplace), you will trigger an inplace error.</p>
<p>On the other hand, the <em>input</em> to <code>exp (x)</code> (that is, <code>x</code> itself) isn’t of any use in<br>
computing the derivative of <code>exp (x)</code> (assuming you have saved <code>exp (x)</code>),<br>
so pytorch doesn’t save <code>x</code> and you can modify it.</p>
<p>Conversely, the derivative of <code>x**2</code> is <code>2 * x</code>, so pytorch saves the input<br>
to the <code>x**2</code> computation, but not its output, for use in the backward pass.<br>
Modifying <code>x</code> itself will now trigger an inplace error, but modifying the output<br>
of the <code>x**2</code> computation won’t.</p>
<p>Lastly, the derivative of <code>10 * x</code> is just the scalar <code>10</code>.  Neither the (tensor)<br>
input nor output of the <code>10 * x</code> computation is useful for computing the<br>
derivative during the backward pass, so pytorch saves neither, and you<br>
can modifying either one without triggering an inplace error.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.arange (3.0, requires_grad = True)
&gt;&gt;&gt; u = 2 * t                             # (so it won't be a leaf variable)
&gt;&gt;&gt; uexp = u.exp()                        # uexp is the gradient of u.exp()
&gt;&gt;&gt; loss = uexp.sum()                     # (some dummy loss)
&gt;&gt;&gt; u[0] = 666                            # inplace modification of u
&gt;&gt;&gt; loss.backward (retain_graph = True)   # okay
&gt;&gt;&gt; uexp[0] = 666                         # inplace modification of uexp
&gt;&gt;&gt; loss.backward()                       # inplace modification error
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]], which is output 0 of ExpBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.arange (3.0, requires_grad = True)
&gt;&gt;&gt; u = 2 * t                             # (so it won't be a leaf variable)
&gt;&gt;&gt; usq = u**2                            # 2 * u is the gradient of u**2
&gt;&gt;&gt; loss = usq.sum()                      # (some dummy loss)
&gt;&gt;&gt; usq[0] = 666                          # inplace modification of usq
&gt;&gt;&gt; loss.backward (retain_graph = True)   # okay
&gt;&gt;&gt; u[0] = 666                            # inplace modification of u
&gt;&gt;&gt; loss.backward()                       # inplace modification error
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]], which is output 0 of struct torch::autograd::CopySlices, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.arange (3.0, requires_grad = True)
&gt;&gt;&gt; u = 2 * t                             # (so it won't be a leaf variable)
&gt;&gt;&gt; uten = 10 * u                         # gradient of 10 * u is just 10
&gt;&gt;&gt; loss = uten.sum()                     # (some dummy loss)
&gt;&gt;&gt; u[0] = 666                            # inplace modification of u
&gt;&gt;&gt; loss.backward (retain_graph = True)   # okay
&gt;&gt;&gt; uten[0] = 666                         # inplace modification of uten
&gt;&gt;&gt; loss.backward()                       # also okay
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

jie_yang: <p>Very clear explaination. Thank you !!!<br>
In my case, because the equation “out += self.shortcut(x)” is like “10*x”, so pytorch doesn’t save any input and output. Also, Batchnorm save input, I can change its output.</p>

bpfrd: <p>In pixelCNN code (here for example <a href="https://github.com/axeloh/pixelcnn-pytorch/blob/master/masked_cnn_layer.py#:~:text=self%2C%20x" class="inline-onebox" rel="noopener nofollow ugc">pixelcnn-pytorch/masked_cnn_layer.py at master · axeloh/pixelcnn-pytorch · GitHub</a>) one can multiply the network parameter (which is a leaf node with requires_grad=True)  with a mask tensor but still doesn’t get any error! how is this possible?</p>

Sam-gege: <p>are you referring to</p>
<pre><code class="lang-python">class MaskedConv2d(...):
    def __init__(self, ...):
        ...
    def forward(self, x):
        self.weight.data *= self.mask
</code></pre>
<p>I remember reading from somewhere that Tensor.data creates a new tensor, so in-place operation is performed on that tensor, that’s why you don’t get <code>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</code>.</p>
<p>Also, version counter doesn’t increase after in-place operation on <code>Tensor.data</code>:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

x=nn.Parameter(torch.tensor([1.,2.,3.]))
print(f'version before inplace operation: {x._version}')
x.data*=torch.tensor([True,False,True])
print(f'version after inplace operation: {x._version}')
x.sum().backward()
print(f'gradient at x: {x.grad}')

#version before inplace operation: 0
#version after inplace operation: 0
#gradient at x: tensor([1., 1., 1.])
</code></pre>
<p>As shown above, the version counter doesn’t increase. What’s more, the mask has no effect during backpropagation, not sure if this is what they desired. Normally if you applies a mask, this mask should also act on gradient:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

x=nn.Parameter(torch.tensor([1.,2.,3.]))
y=x*1 # make it non-leaf
print(f'version before inplace operation: {y._version}')
y*=torch.tensor([True,False,True])
print(f'version after inplace operation: {y._version}')
y.sum().backward()
print(f'gradient at x: {x.grad}')

#version before inplace operation: 0
#version after inplace operation: 1
#gradient at x: tensor([1., 0., 1.])
</code></pre>