No gradient on cuda?

username: Sam-gege
cooked: <p>It seems I can’t get a gradient when sending tensors to cuda:</p>
<pre><code class="lang-python">import torch
print(torch.__version__)

x = torch.tensor(4.2, requires_grad=True).cuda()
y = torch.tensor(5.2, requires_grad=True).cuda()
output = x * y
output.backward()

print(output)
print(x.grad)
print(y.grad)
</code></pre>
<p>and the output is:</p>
<pre><code class="lang-auto">1.10.1
tensor(21.8400, device='cuda:0', grad_fn=&lt;MulBackward0&gt;)
None
None
C:\Users\**\anaconda3\lib\site-packages\torch\_tensor.py:1013: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at  aten\src\ATen/core/TensorBody.h:417.)
  return self._grad
</code></pre>
<p>obviously the tensors <code>x</code> and <code>y</code> are leaf tensors, how come their gradients are not calculated? am I missing something? Thanks for any advice!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sam-gege" data-post="1" data-topic="144807">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam-gege/48/42338_2.png" class="avatar"> Sam-gege:</div>
<blockquote>
<p>obviously the tensors <code>x</code> and <code>y</code> are leaf tensors, how come their gradients are not calculated?</p>
</blockquote>
</aside>
<p>That’s not the case, since <code>cuda()</code> is a differentiable operation as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(4.2, requires_grad=True)
print(x.is_leaf)
# &gt; True

x = torch.tensor(4.2, requires_grad=True).cuda()
print(x.is_leaf)
# &gt; False
print(x.grad_fn)
# &gt; &lt;ToCopyBackward0 object at 0x7f2aa29adcd0&gt;
</code></pre>
<p>If you want to create a trainable tensor on the GPU, either use the <code>device</code> argument in its initialization or call <code>detach().requires_grad_()</code> on it.</p>

username: Sam-gege
cooked: <p>Thanks for your reply, that’s interesting to know! May I please ask how come <code>cuda()</code> is differentiable?? I thought it is just used to send data from cpu to gpu?</p>

username: eqy
cooked: <p>It should be differentiable in the same way that <a href="https://pytorch.org/docs/stable/generated/torch.nn.Identity.html" rel="noopener nofollow ugc">nn.Identity</a> is differentiable; it introduces another tensor in the computation graph even if it does not apply a transformation to the input to produce an output.</p>

username: ptrblck
cooked: <p>Additionally to what <a class="mention" href="/u/eqy">@eqy</a> said: it allows you to use different devices without detaching the computation graph. E.g. you could use CPU operations, push the data to the GPU, perform more operations etc. as seen here:</p>
<pre><code class="lang-python"># setup
x = torch.randn(1, 1, requires_grad=True)
lin_on_cpu = nn.Linear(1, 1)
lin_on_gpu = nn.Linear(1, 1).cuda()

# forward
out_on_cpu = lin_on_cpu(x)
out_on_gpu = out_on_cpu.to('cuda')
out_on_gpu = lin_on_gpu(out_on_gpu)

# backward
out_on_gpu.mean().backward()
print(x.grad) # gives a valid gradient
# &gt; tensor([[0.0911]])
</code></pre>
<p>and can be used for e.g. model sharing using multiple GPUs etc.</p>