Confusion about using .clone

isalirezag: <p>considering these two nets:</p>
<pre><code class="lang-auto">
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.conv1 = nn.Conv2d(3, 6, kernel_size=1, stride=1, bias=False)
        self.conv2 = nn.Conv2d(6, 6, kernel_size=1, stride=1, bias=False)
        self.conv3 = nn.Conv2d(6, 6, kernel_size=1, stride=1, bias=False)
        
    def forward(self,input):
        input = F.relu(self.conv1(input))
        input2 = input
        
        output = F.relu(self.conv2(input))
        output2 = F.relu(self.conv3(input2))
        
        return output, output2
</code></pre>
<pre><code class="lang-auto">        
class Net2(nn.Module):
    def __init__(self):
        super(Net2, self).__init__()
        
        self.conv1 = nn.Conv2d(3, 6, kernel_size=1, stride=1, bias=False)
        self.conv2 = nn.Conv2d(6, 6, kernel_size=1, stride=1, bias=False)
        self.conv3 = nn.Conv2d(6, 6, kernel_size=1, stride=1, bias=False)
        
    def forward(self,input):
        input = F.relu(self.conv1(input))
        input2 = input.clone()
        
        output = F.relu(self.conv2(input))
        output2 = F.relu(self.conv3(input2))
        
        return output, output2
</code></pre>
<p>What is the difference between Net and Net2? is the Net model wrong?</p>
<p>When I use clone in Net2, if i do back-propagation (via backward on the loss between output2 and target)<br>
will weights in conv1 get updated as well or it just updates the conv3 weights?</p>

ptrblck: <p>Both models work identically as seen here:</p>
<pre><code class="lang-python">model1 = Net()
model2 = Net2()
model2.load_state_dict(model1.state_dict())

x = torch.randn(1, 3, 24, 24)

outputs1 = model1(x)
outputs2 = model2(x)

# Compare outputs
print((outputs1[0] == outputs2[0]).all())
print((outputs1[1] == outputs2[1]).all())

# Compare gradients
outputs1[0].mean().backward(retain_graph=True)
outputs1[1].mean().backward()
outputs2[0].mean().backward(retain_graph=True)
outputs2[1].mean().backward()

for p1, p2 in zip(model1.parameters(), model2.parameters()):
    print((p1.grad == p2.grad).all())
</code></pre>

isalirezag: <p>so to clarify, clone will copy the data to another memory but it has no interfere with gradient back propagation. in other words, when i use clone it will back propagate till the input unless i use detach …</p>

ptrblck: <p>Yes. <code>.clone()</code> is recognized by Autograd and the new tensor will get the grad function as <code>grad_fn=&lt;CloneBackward&gt;</code>.</p>

txytju: <p>Why retain_graph=True is used when outputs1[0].mean().backward() is called? Will this accelerate training?</p>

MariosOreo: <p>After calling <code>output1[0].mean().backward()</code>, it frees the computation graph which still need in <code>output1[1].mean().backward()</code>. So it needs ‘retain_graph’.</p>

Ahmed_m: <p>Does this main that there is no difference between “Clone()” and “=” in terms of the gradient copying?</p>

ptrblck: <p>I’m not sure I understand your question correctly.<br>
If you are referring to the original question, neither <code>input</code> nor <code>input2</code> were manipulated after the assignment, so that both model architectures yield the same results.</p>

Ahmed_m: <p>I mean: when we save a differentiable variable into another with “=”, will this also move the gradients and all the computation graph to the newly defined variable? Or we should use “clone()” for explicitly doing this?</p>

ptrblck: <p>The new variable will reference the old one, so that both will see the value changes and gradients:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1)
w = lin.weight

lin(torch.randn(1, 1)).backward()

print(lin.weight.grad)
print(w.grad)
print(id(w)==id(lin.weight))
</code></pre>

pinocchio: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="39673" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Yes. <code>.clone()</code> is recognized by Autograd and the new tensor will get the grad function as <code>grad_fn=&lt;CloneBackward&gt;</code> .</p>
</blockquote>
</aside>
<p>I never understood this, what is the point of recording <code>.clone()</code> as an operation? It’s extremely unintuitive to me. When I see clone I expect something like deep copy and getting a fresh new version (copy) of the old tensor. Having copying as an operation in a forward pass (like using the identity) but not calling it the identity is extremely confusion.</p>
<p>Am I correct or do I not understand how <code>.clone()</code> works?</p>
<p>I don’t understand what the use case for it is really.</p>

ptrblck: <p><code>clone</code> can be used e.g. on activations, which should be passed to multiple modules, where each module might manipulate the activation in-place.<br>
Here is a small example:</p>
<pre><code class="lang-python"># Setup
module1 = nn.Sequential(
    nn.ReLU(inplace=True),
    nn.Linear(10, 1))

module2 = nn.Sequential(
    nn.Linear(10, 2))


torch.manual_seed(2809)
act = torch.randn(1, 10)
print(act)

# Wrong, since act will be modified inplace
out1 = module1(act)
print(act)
out2 = module2(act)

# Right
torch.manual_seed(2809)
act = torch.randn(1, 10)
print(act)

# Wrong, since act will be modified inplace
out1 = module1(act.clone())
print(act)
out2 = module2(act)
</code></pre>
<p>If you don’t <code>clone</code> the activation, the first module would apply the relu on it and the call to <code>module2</code> would get the wrong tensor.</p>
<p>If <code>act</code> was created by previous operations (layers), Autograd will properly calculate all gradients.</p>

pinocchio: <p>Let me see if I can paraphrase to see if I got it.</p>
<p><code>.clone()</code> is useful to create a copy of the original variable that doesn’t forget the history of ops so to allow gradient flow and avoid errors with inlace ops. The main error of in-place ops is overwriting data needed for the backward pass (or writing an in-place op to leaf node, in this case there would be no error message).</p>

ptrblck: <aside class="quote no-group" data-username="pinocchio" data-post="13" data-topic="39673">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p><code>.clone()</code> is useful to create a copy of the original variable that doesn’t forget the history of ops so to allow gradient flow</p>
</blockquote>
</aside>
<p>Yes, that is correct.<br>
While my example stressed out the inplace operation, <code>clone()</code> might of course be useful in other use cases and specific model architectures (the inplace ops were just the first use case that came to my mind).</p>
<aside class="quote no-group" data-username="pinocchio" data-post="13" data-topic="39673">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p>or writing an in-place op to leaf node, in this case there would be no error message</p>
</blockquote>
</aside>
<p>If the inplace op is not allowed, PyTorch should raise an error and should not silently fail.<br>
If you encounter such a silent error, please let us know.</p>

Alex-Fabbri: <p>Thanks for the explanation! I was wondering if you could give an example of other use cases for cloning besides the inplace operation example. I was trying to think of other cases where it may be necessary/beneficial but couldn’t think of any.</p>

ptrblck: <p>Generally, <code>clone</code> is useful whenever you are dealing with references and would like to use the current value without any potential future changes.<br>
E.g. if you would like to compare values pulled from a <code>state_dict</code>, you would have to use <code>clone()</code> to create the reference values. Otherwise they would be updated in the next <code>optimizer.step()</code> call and your initially stored values would change as well.</p>
<p>That being said, in a “standard CNN training routine” you probably wouldn’t need to call it.</p>

SEM: <p>Thank you for your detailed explanation about .clone() method.<br>
I face an issue during training stacked of 2 unet module. the input to the network is a batch of single image and some geometric transformation on it. Then I apply inverse of augmentation to the images (F <sup>-1</sup>) and compute a loss <span class="hashtag">#1</span> function. Also, I have loss <span class="hashtag">#2</span> at the end of the network.</p>
<p>(x + F(x)) --&gt; Unet1 --&gt;  loss#1(y+ F<sup>-1</sup>(y)) --&gt; Unet2 --&gt; loss#2(z+ F<sup>-1</sup>(z))</p>
<p>In general, I got this error:<br>
<em>“one of the variables needed for gradient computation has been modified by an inplace operation”.</em><br>
I resolved this error by utilizing y.clone() and then apply inverse geometric transformation.</p>
<p>(x + F(x)) --&gt; Unet1 --&gt;  loss#1(y+ F<sup>-1</sup>(y.clone())) --&gt; Unet2 --&gt; loss#2(z+ F<sup>-1</sup>(z.clone()))</p>
<p>Now, it seems that it is working well, yet I am suspicious about it. Is it Ok or not? does the gradient flow back to the Unet1 and does the gradient from Unet2 flow back to Unet1?</p>
<p>Note; I use pytorch functions for some geometric transformation, but for others not. I have converted them to numpy to do that.</p>

ptrblck: <p>Based on the error message it seems that <code>F^(-1)</code> might apply some inplace operations on the input tensor, which replace values needed to calculate the gradients. A <code>.clone()</code> operation might solve this issue.<br>
Nevertheless, you could check, if the desired gradients are calculated by calling <code>backward()</code> on intermediate outputs or just the final loss and checking the <code>.grad</code> attributes of all parameters, which should get gradients.<br>
If some of these <code>.grad</code> attributes are returning a <code>None</code> value, Autograd didn’t calculate any gradiets for them and your computation graph might have been broken.</p>
<aside class="quote no-group" data-username="SEM" data-post="17" data-topic="39673">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sem/48/23066_2.png" class="avatar"> SEM:</div>
<blockquote>
<p>I have converted them to numpy to do that.</p>
</blockquote>
</aside>
<p>That sounds risky, as this would cut the computation graph, since Autograd isn’t able to backpropagate through numpy operations. You would either need to stick to PyTorch methods or implement a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>

SEM: <p>I think in spite of the fact A.clone() have resolved the issue, some of the gradients may be None. In fact, the network does not train properly and the loss#2 just ringing about a certain value. Maybe the computational graph is broken and network does not actually trained.<br>
I will try to stick to PyTorch for geometric transformations.</p>

behzadtabari: <p>I believe one of the instances that this notion really makes sense is the skip connection or residual network block where you do not want the gradients of the residuals being messed up with the gradients of the training block.</p>