What is `in-place operation`?

Rickyim: <p>I came across the term <code>in-place operation</code> in the<br>
documentation <a href="http://pytorch.org/docs/master/notes/autograd.html" rel="nofollow noopener">http://pytorch.org/docs/master/notes/autograd.html</a> What does it mean?</p>

albanD: <p>Hi,</p>
<p>An in-place operation is an operation that changes directly the content of a given Tensor without making a copy. Inplace operations in pytorch are always postfixed with a <code>_</code>, like <code>.add_()</code> or <code>.scatter_()</code>. Python operations like <code>+=</code> or <code>*=</code> are also inplace operations.</p>

bkowshik: <p>I initially found in-place operations in the following PyTorch tutorial:</p>
<ul>
<li><a href="http://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#operations" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#operations</a></li>
</ul>
<h3>Adding two tensors</h3>
<pre><code class="lang-python">import torch

&gt;&gt;&gt; x = torch.rand(1)
&gt;&gt;&gt; x

 0.2362
[torch.FloatTensor of size 1]


&gt;&gt;&gt; y = torch.rand(1)
&gt;&gt;&gt; y

 0.7030
[torch.FloatTensor of size 1]
</code></pre>
<h3>Normal addition</h3>
<pre><code class="lang-python"># Addition of two tensors creates a new tensor.
&gt;&gt;&gt; x + y

 0.9392
[torch.FloatTensor of size 1]


# The value of x is unchanged.
&gt;&gt;&gt; x

 0.2362
[torch.FloatTensor of size 1]
</code></pre>
<h3>In-place addition</h3>
<pre><code class="lang-python"># An in-place addition modifies one of the tensors itself, here the value of x.
&gt;&gt;&gt; x.add_(y)

 0.9392
[torch.FloatTensor of size 1]


&gt;&gt;&gt; x

 0.9392
[torch.FloatTensor of size 1]
</code></pre>

Rickyim: <p>So in this tutorial, the way the network constructed in the forward method is in-place operation right?<br>
<a href="http://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py</a></p>

sjin: <p>I understand that  <code>x.add_(y)</code>  is an in-place operation.<br>
Is  <code>x = x + y</code>  in-place and will it cause any problem for autograd?</p>

albanD: <p>Hi,</p>
<p>No <code>x = x + y</code> is not inplace. <code>x += y</code> is inplace.</p>

Ibrahim_El-Fayoumi: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="16244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>x = x + y</code> i</p>
</blockquote>
</aside>
<p>What is the difference? they both will modify x?</p>

Deepali: <p>Yes true they both modify x. But in-place operation does not allocate new memory for x.</p>
<p>Eg. Normal operation vs In place operation</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.rand(1)
&gt;&gt;&gt; y = torch.rand(1)
&gt;&gt;&gt; x
tensor([0.2738])
&gt;&gt;&gt; id(x)
140736259305336
&gt;&gt;&gt; x = x + y   # Normal operation
&gt;&gt;&gt; id(x)
140726604827672 # New location
&gt;&gt;&gt; x += y
&gt;&gt;&gt; id(x)
140726604827672 # Existing location used (in-place)
</code></pre>

Ibrahim_El-Fayoumi: <p>Thanks, that make sense</p>

nguyenhoa93: <p>Thanks, this is a good explanation.</p>

amirg: <p>Which one is faster? in-place or normal operation?</p>

albanD: <p>That depends what you mean by faster <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
It does the same amount of computations. So that does not change that.<br>
But since there is less memory accesses, this can lead to speed up if your task is bound by memory bandwidth (which is quite often the case on GPU).</p>

pinocchio: <p>Are in-place operations added to the computation graph/tracked by autograd?</p>
<p>This experiment makes me think yes since <code>a_clone</code> has <code>&lt;MulBackward0&gt;</code> as meta-data.</p>
<p>Let me know if this conclusion is correct:</p>
<pre><code class="lang-auto">def clone_playground():
    import torch

    a = torch.tensor([1,2,3.], requires_grad=True)
    a_clone = a.clone()
    print(f'a is a_clone = {a is a_clone}')
    print(f'a == a_clone = {a == a_clone}')
    print(f'a = {a}')
    print(f'a_clone = {a_clone}')
    #a_clone.fill_(2)
    a_clone.mul_(2)
    print(f'a = {a}')
    print(f'a_clone = {a_clone}')
    a_clone.sum().backward()
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">a is a_clone = False
a == a_clone = tensor([True, True, True])
a = tensor([1., 2., 3.], requires_grad=True)
a_clone = tensor([1., 2., 3.], grad_fn=&lt;CloneBackward&gt;)
a = tensor([1., 2., 3.], requires_grad=True)
a_clone = tensor([2., 4., 6.], grad_fn=&lt;MulBackward0&gt;)
</code></pre>
<p>from here: <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/33" class="inline-onebox">Clone and detach in v0.4.0</a></p>

albanD: <p>I’ll answer there to keep all the discussion in a single place.</p>

Alex_Fann: <p>Made a summary here. Hope it is helpful:</p>
<blockquote>
<p>Best practice: Avoid inplace operations if it is not necessary as it changes the state of tensors silently. Non-inplace operations will make a copy before doing the operation. Thus, if an operation is inplace within a function, it affects the tensor’s state outside of the function while the non-inplace operation does not change the state unless you reassign it outside of the function.</p>
</blockquote>
<p>e.g.</p>
<pre><code class="lang-auto">def inplace_op(X):
	X += 1
	return X

X = torch.rand(4, 2)
inplace_op(X) # X is changed without re-asigned to X
</code></pre>
<p>Summary of inplace operations:</p>
<ul>
<li>x *= 3</li>
<li>X[…]  = …</li>
<li>X.add_(1)</li>
</ul>
<p>Common examples of inplace operations:</p>
<ul>
<li>x += 1, x *= 3, …</li>
<li>x[2] = 2, X[0, 0] = 2</li>
<li>x[:, 3] = 3</li>
<li>X[2] /= 5</li>
<li>X[:, 3] /= 4</li>
<li>X[:, 3] = X[:, 3] / 4</li>
</ul>
<p>These are not inplace operations:</p>
<ul>
<li>x = x + 1</li>
<li>y = x.clone; y[0] += 100</li>
</ul>

Computer-CGuy: <p>Your examples saved my days of debugging. Thnx <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>