What's the difference between nn.ReLU() and nn.ReLU(inplace=True)?

yunjey: <p>I implemented generative adversarial network using both <code>nn.ReLU()</code> and <code>nn.ReLU(inplace=True)</code>. It seems that <code>nn.ReLU(inplace=True)</code> saved very small amount of memory.</p>
<p>What’s the purpose of the using <code>inplace=True</code>?<br>
Is the behavior different in backpropagation?</p>

apaszke: <p><code>inplace=True</code> means that it will modify the input directly, without allocating any additional output. It can sometimes slightly decrease the memory usage, but may not always be a valid operation (because the original input is destroyed). However, if you don’t see an error, it means that your use case is valid.</p>

laoreja: <p>In this <a href="http://pytorch.org/docs/master/notes/autograd.html#in-place-operations-on-variables" rel="nofollow noopener">http://pytorch.org/docs/master/notes/autograd.html#in-place-operations-on-variables</a> document, in place operation is not encouraged. But why did most official example in torchvision (e.g. Resnet) use nn.ReLU(inplace=True).</p>
<p>Also, does<br>
x = self.conv1(x)<br>
x = self.conv2(x)<br>
be considered as in place operation (as they use the same variable name) or not?</p>

cdancette: <p>x = self.conv1(x)<br>
x = self.conv2(x)</p>
<p>is not an in place operation, because you use the same variable name, but it’s not the same variable underneath. You just point your x name to a new variable, the old one is still in memory (because it’s referenced by the pytorch graph)</p>

laoreja: <p>Thanks! It makes sense!<br>
But as inplace operation is not encouraged, why most official examples use nn.ReLU(inplace=True)?</p>

harryhan618: <p>I’m a newbie of pytorch. So I wonder whether nn.ReLI(inplace=True) would do harm to backprop? And what about F.ReLU(inplace)?</p>

rasbt: <aside class="quote no-group" data-username="harryhan618" data-post="6" data-topic="948">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harryhan618/48/47760_2.png" class="avatar"> harryhan618:</div>
<blockquote>
<p>nn.ReLI(inplace=True)</p>
</blockquote>
</aside>
<p>That’s a good question! I think that this was an initial limitation based on the PyTorch (whitepaper) manuscript on arxiv. But based on what I’ve seen (e.g., by <span class="mention">@soumith</span> <a href="https://gist.github.com/soumith/71995cecc5b99cda38106ad64503cee3" class="inline-onebox" rel="noopener nofollow ugc">dcgan.py · GitHub</a>) it seems that inplace ops like <code> nn.ReLU(inplace=True)</code> are supported in the autodiff engine now. Not sure, but I guess the same should be true for the functional one as it is referencing the nn. one.</p>

yxchng: <p><a class="mention" href="/u/cdancette">@cdancette</a> In that case, the relu(inplace=True) in vision/resnet.py is actually not inplace since it is used with <code>x = relu(x)</code> in forward()?</p>

streaver91: <p>For relu, when input is negative, both the grad and output should be zero, grads should stop propagating from there, so inplace doesn’t hurt anything while saves memory.</p>

Jay_Timbadia: <p>Is this an in-place operation?</p>
<p>b = torch.tensor(5)<br>
y = torch.sigmoid_(torch.tensor(4)) &amp; y = torch.sigmoid(b)</p>
<p>Thanks!</p>

cdancette: <p>Even if you use <code>x = relu(x)</code>, it is still inplace, reassining x to the output of relu(x) does nothing here.</p>
<p>You can check :after <code>relu(x)</code> and <code>x = relu(x)</code>, x has the same value in both cases.</p>

cdancette: <p><code>torch.sigmoid_</code> is an inplace operation</p>
<p><code>torch.sigmoid</code> is not.</p>
<p>You can check on pytorch:</p>
<pre><code class="lang-python">&gt;&gt;&gt; a = torch.tensor(1.0)
&gt;&gt;&gt; torch.sigmoid(a)
tensor(0.7311)
&gt;&gt;&gt; print(a)
tensor(1.0)
</code></pre>
<pre><code class="lang-python">&gt;&gt;&gt; a = torch.tensor(1.0)
&gt;&gt;&gt; torch.sigmoid_(a)
tensor(0.7311)
&gt;&gt;&gt; print(a)
tensor(0.7311)
</code></pre>
<p>In the first case, a still has its original value, while in the second case, a is different.</p>

helson73: <p>In case y = F.relu(x, inplace=True), it won’t hurt anything if value of x should  always be positive in your computational graph. However, some other node that shares x as input while it requires x has both positive and negative value, then your network may malfunction.</p>
<p>For example, in the following situation,</p>
<p>y = F.relu(x, inplace=True)   (1)<br>
z = network(x)                      (2)</p>
<p>If (1) is declared first and execuated first, then value of x is changed, then z may have incorrect expected value.</p>

cwinkler: <p>Hi all,</p>
<p>Even though there are multiple answers, I will explain my problem here. I am facing the following error message:</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 64, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>It occurs when trying to compute the gradients for the second backward pass. I have the suspicion it has something to do with the fact that I am keeping state variables of an LSTM feature map creator as self objects inside an <em>nn.Module</em>. Here is the code snippet of my forward loop:</p>
<pre><code class="lang-auto">    def forward(self, z, xlr=None, logdet=0, logpz=0, eps=None, reverse=False,
                use_stored=False):

        self.h_new, self.c_next = self.conv_lstm(z, (self.h, self.c))

        # Encode
        if not reverse:
            for i in range(self.L):
                print("Level", i)
                for layer in self.level_modules[i]:

                    if isinstance(layer, modules.Squeeze):
                        z = layer(z, reverse=False)
                        self.h_new = layer(self.h_new, reverse=False)

                    elif isinstance(layer, FlowStep):
                        z, logdet = layer(z, lr_feat_map=self.h_new, # lr_downsampled_feats[i + 1], # TODO: change this part
                                          x_lr=xlr, logdet=logdet, reverse=False)

                    elif isinstance(layer, modules.GaussianPrior):
                        z, logdet, logpz = layer(z, logdet=logdet, logpz=logpz,
                                                 lr_feat_map=self.h_new, #lr_downsampled_feats[i + 1],
                                                 eps=eps, reverse=False)
                        self.h = self.last_squeezer(self.h_new, reverse=True)
                        self.c = self.c_next
</code></pre>
<p>Do you think it would be the best to just pass on the hidden and context states through the function outputs? I have already set loss.mean().backward(retain_graph=True) and skimmed the code for other inplace operations.</p>
<p>Any help would be much appreciated !! Please let me know if further code snippets are required.</p>