How does clone interact with backpropagation?

username: Brando_Miranda
cooked: <p>I was wondering, how does <code>.clone()</code> function interact with backpropagation?</p>
<p>For example I had to pieces of code and I don’t understand what their difference is, can someone explain me the difference (maybe so I understand why clone is even needed for backprop to work “properly”, whatever properly means):</p>
<p>E.g1:</p>
<pre><code>import torch
from torch.autograd import Variable
x = Variable(torch.rand(2,1),requires_grad=False) #data set
w = Variable(torch.rand(2,2),requires_grad=True) #first layer
v = Variable(torch.rand(2,2),requires_grad=True) #last layer
w[:,1] = v[:,0]
y = torch.matmul(v, torch.matmul(w,x) )
</code></pre>
<p>vs</p>
<pre><code>import torch
from torch.autograd import Variable
x = Variable(torch.rand(2,1),requires_grad=False) #data set
w = Variable(torch.rand(2,2),requires_grad=True) #first layer
v = Variable(torch.rand(2,2),requires_grad=True) #last layer
w[:,1] = v[:,0]
y = torch.matmul(v, torch.matmul(w.clone(),x) )
</code></pre>
<p>why do we need clone in the second one? What does “backprop work properly” mean?</p>

username: fmassa
cooked: <p>Both of your scripts fail for me.</p>
<p>Short reason:</p>
<pre><code class="lang-auto">RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>
<p>If you want to perform in-place operations (like <code>w[:, 1] = v[:, 0]</code>), you can clone the variable before the in-place operation. This makes it not be a leaf variable anymore.</p>
<pre><code class="lang-auto">w1 = w.clone()
w1[:,1] = v[:,0]
y = torch.matmul(v, torch.matmul(w1,x) )
</code></pre>
<p>Using in-place Variables is very tricky in many cases. A lot of frameworks doesn’t support them, so they just perform copies instead. PyTorch support in-place operations, but because other operations can require the content of the Variable to perform backpropagation, you can’t modify it inplace or else you will have wrong gradients.</p>

username: brisker
cooked: <p>If I get a copy of a Variable and extract a slice of the copy, then the gradient only back prop through the sliced dimension?<br>
<strong>what if this Variable is the model parameters?</strong></p>

username: pinocchio
cooked: <p>Funny, I was wondering exactly what the title of the question is asking. How does backprop interact to clone?</p>
<p>I build this simple script:</p>
<pre><code class="lang-auto">## print clone backward
import torch
​
a = torch.tensor([1,2,3.], requires_grad=True)
c = a.sigmoid()
c_cloned = c.clone()
​
print(f'c_cloned: {c_cloned}')
</code></pre>
<p>and got this:</p>
<pre><code class="lang-auto">c_cloned: tensor([0.7311, 0.8808, 0.9526], grad_fn=&lt;CloneBackward&gt;)
</code></pre>
<p>but notice that the gradient recorded the clone operation (i.e. <code>grad_fn=&lt;CloneBackward&gt;</code>) which I thought was very weird.</p>
<p>So my question is…what does backprop react to a clone operation? What does it even mean backprop through a clone operation?</p>

username: ptrblck
cooked: <p>All incoming gradients to the cloned tensor will be propagated to the original tensor as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 2, requires_grad=True)
y = x.clone()
y.retain_grad()

z = y**2
z.mean().backward()

print(y.grad)
print(x.grad)
</code></pre>

username: keqiao_li
cooked: <p>Hi Ptrblck,</p>
<p>Just want to double check, it will be the same for “SliceBackward”?</p>

username: ptrblck
cooked: <p>Yes, the backward function of the slicing operation will propagate the gradient to the slices values in the original tensor as seen here:</p>
<pre><code class="lang-python">x = torch.randn(5, 5, requires_grad=True)
y = x[3:4, 3:5]
print(y)
# tensor([[-0.0479,  0.8982]], grad_fn=&lt;SliceBackward0&gt;)

y.mean().backward()
print(x.grad)
# tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
#         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
#         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
#         [0.0000, 0.0000, 0.0000, 0.5000, 0.5000],
#         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000]])
</code></pre>

username: keqiao_li
cooked: <p>Thank you for the help! One follow-up question: is there a way slice matrix in batch-wise?<br>
For example:</p>
<p><code>output = model(input)</code> , where output has a shape of <code>10 x 1 x 32 x 32 (batch size x channel x height x width)</code></p>
<p>Slicing operation will be different for each sample:<br>
sample 1:  <code>output[0, :, :16, :16]</code><br>
sample 2: <code>output[0, :, :15, :15]</code><br>
sample 3: <code>output[0, :, :26, :26]</code>…something like that.</p>
<p>Does PyTorch have a way to do the above operation efficiently? without using for loop.</p>

username: sajjad_mozaffari
cooked: <p>I guess you can use advanced indexing only if the size of different slices matches. e.g.:<br>
sample 1: <code>output[0, :, 1:16, 1:16]</code><br>
sample 2: <code>output[0, :, :15, :15]</code><br>
sample 3: <code>output[0, :, 11:26, 11:26]</code></p>
<p>Here you can find documentation in numpy (works similarly in pytorch):<br>
<a href="https://numpy.org/devdocs/user/basics.indexing.html#advanced-indexing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://numpy.org/devdocs/user/basics.indexing.html#advanced-indexing</a></p>