Strange error from loss.backward()

username: viswanatha_reddy
cooked: <p>I got this error when doing the backward pass:</p>
<pre><code class="lang-auto">loss.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function DivBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.cuda.FloatTensor
</code></pre>
<p>This occurs when executing <code>loss.backward()</code>.<br>
Any idea is appreciated. Thanks in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is quite unexpected. Do you have custom <code>autograd.Function</code> that you defined? Do you use hooks to change some gradients during the backward?</p>

username: viswanatha_reddy
cooked: <p>I was able to solve the error. Some of the tensors are placed on cpu others on gpu. To solve the issue I have moved all the tensors to gpu.</p>
<pre><code class="lang-auto">torch.tensor(1, dtype=torch.float, requires_grad=True) 
#changed to 
torch.tensor(1, dtype=torch.float, requires_grad=True).cuda()
</code></pre>

username: rustagiadi95
cooked: <p>Hi there,</p>
<p>The solution adressed above is absolutely correct. Some of the tensors are at gpu and some are at cpu.</p>
<p>As suggested by the above solutions as you can move the tensor to gpu by .cuda() call, similarly you can move the tensor/model on the cpu by .cpu() call. It will come handy.</p>
<p>Either you can use the solution above, but defining the device on which the neural networks and tensors is the right and more informative approch. Define the device by :-<br>
Device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>torch.cuda.is_available() checks if cuda(or gpu) is available to you or not</p>
<p>Now I can already see that you have cuda on your system, but still its an additive line to check which device you can have while working with pytorch.</p>
<p>Then the model can be moved on the said device as :<br>
model = model.to(Device)</p>
<p>and the tensors(t) can be put on the device the same way :<br>
t = t.to(Device)</p>
<p>Hope this helps, this is just for further info.<br>
Thanks</p>

username: ptrblck
cooked: <p>Good to hear it’s working.<br>
However, note that you are creating a non-leaf tensor with the <code>cuda</code> call as described by <a class="mention" href="/u/alband">@albanD</a> <a href="https://discuss.pytorch.org/t/valueerror-cant-optimize-a-non-leaf-tensor/21751/2">here</a>, so you would probably want to create the tensor as:</p>
<pre><code class="lang-python">torch.tensor(1, dtype=torch.float, requires_grad=True, device='cuda:0')
</code></pre>

username: viswanatha_reddy
cooked: <aside class="quote no-group" data-username="rustagiadi95" data-post="4" data-topic="62344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/46a35a/48.png" class="avatar"> rustagiadi95:</div>
<blockquote>
<p>t = t.to(Device)</p>
</blockquote>
</aside>
<p>I have used this, but the error here is strange. Eventhough using the above statement (t.to(device)) some of the tensors are on gpu and others are on cpu. It is not showing any error during forward propagation. The error is occurring during backprop. I need some clarification why it is not throwing any error during forward prop. TIA.</p>

username: ptrblck
cooked: <p>Could you post a minimal code snippet to reproduce this error?</p>