Why Tensor.clone() is called clone and not copy?

username: 11158
cooked: <p>In much older library numpy method that copy ndarray is called copy. Why in torch the same method is called clone? Are there any specific reasons?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think this is mostly for historical reasons in particular <code>copy</code> (now <code>copy_</code>) was used a long time ago to copy into a tensor while <code>clone</code> is used to create an identical clone of a given Tensor.</p>

username: Sybil
cooked: <p>Hi, albanD.<br>
Could you explain the difference between <code>b = a.clone()</code> and <code>b.copy_(a)</code>?</p>
<p>The docs said that</p>
<blockquote>
<p>Unlike copy_(), clone() is recorded in the computation graph. Gradients propagating to the cloned tensor will propagate to the original tensor.</p>
</blockquote>
<p>However, in the example below, the gradient was also backpropagated to the original tensor:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.randn(2,2,requires_grad=True)
&gt;&gt;&gt; x
tensor([[0.5113, 0.3028],
        [0.7036, 1.4417]], requires_grad=True)
&gt;&gt;&gt; x.grad
&gt;&gt;&gt; y = x*2+3
&gt;&gt;&gt; y_copy = torch.zeros_like(y)
&gt;&gt;&gt; y_copy.copy_(y)
tensor([[4.0227, 3.6057],
        [4.4072, 5.8835]], grad_fn=&lt;CopyBackwards&gt;)
&gt;&gt;&gt; z = y_copy*3+3
&gt;&gt;&gt; z
tensor([[15.0681, 13.8171],
        [16.2216, 20.6504]], grad_fn=&lt;AddBackward0&gt;)
&gt;&gt;&gt; loss=torch.sum(z-15)
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; x.grad
tensor([[6., 6.],
        [6., 6.]])
</code></pre>
<p>And the <code>y_clone = y.clone()</code> operation showed the same behavior. Could you explain the difference?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think this is most likely misleading doc here <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">  The master doc has been updated and is clearer: <a href="https://pytorch.org/docs/master/generated/torch.clone.html?highlight=clone#torch.clone">https://pytorch.org/docs/master/generated/torch.clone.html?highlight=clone#torch.clone</a></p>
<p>The difference is that if you use copy_, the original value won’t get gradients. But for clone, there is no original value so not this issue.</p>
<pre><code class="lang-auto">y = torch.rand(10, requires_grad=True)

res = y.clone().copy_(x)
res.sum().backward()
assert (y.grad == 0).all()
</code></pre>

username: Sybil
cooked: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="34443">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>res.sum().backward()</code></p>
</blockquote>
</aside>
<p>Hi, thank you for your reply, but, sorry, I still didn’t get it.<br>
Please take a look at following examples:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.randn(2,2,requires_grad=True)
&gt;&gt;&gt; y = x.clone()
&gt;&gt;&gt; res=y.sum()
&gt;&gt;&gt; res.backward()
&gt;&gt;&gt; y.grad
&gt;&gt;&gt; x.grad
tensor([[1., 1.],
        [1., 1.]])
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.randn(2,2,requires_grad=True)
&gt;&gt;&gt; y = torch.randn(2,2)
&gt;&gt;&gt; y.copy_(x)
tensor([[ 0.4119, -0.7538],
        [-0.3020, -0.6225]], grad_fn=&lt;CopyBackwards&gt;)
&gt;&gt;&gt; res = y.sum()
&gt;&gt;&gt; res.backward()
&gt;&gt;&gt; y.grad
&gt;&gt;&gt; x.grad
tensor([[1., 1.],
        [1., 1.]])
</code></pre>
<p>As I see it, the two operations behave the same if used separately.</p>
<p>In your example, there were three tensors (x, y, and res) and you used <code>.clone().copy_(x)</code> together. To be honest, I got more confused. Could you explain the reason why did you use them together?</p>
<p>And what’s the difference between <code>&gt;&gt;y.grad &gt;&gt;nothing printed out</code> and <code>&gt;&gt;y.grad &gt;&gt;a tensor of zeros printed out</code>?</p>
<p>Thank you for your time!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You should ignore the note in the old doc as I think it is just confusing.<br>
The two actually propagate gradients.</p>
<p>In my example, I use clone to avoid changing the original Tensor because the copy is done inplace.</p>
<p>A gradient can be None for few reasons. Either because the Tensor does not require gradients, is not a leaf Tensor or is independent of the output that you backwarded on.</p>