RuntimeError: Expected isFloatingType(grads[i].scalar_type()) to be true, but got false. (Could this error message be improved?

ecdrid: <p>I am getting this weird error when i am using the following funcs in my training loop as soon as it hits <code>loss.backward()</code></p>
<pre><code class="lang-auto">def pos_weight(pred_tensor, pos_tensor, neg_weight=1, pos_weight=1):
    # neg_weight for when pred position &lt; target position
    # pos_weight for when pred position &gt; target position
    gap = torch.argmax(pred_tensor, dim=1) - pos_tensor
    gap = gap.type(torch.float32)
    return torch.where(gap &lt; 0, -neg_weight * gap, pos_weight * gap)

def loss_fn(start_logits, end_logits, start_positions, end_positions):
    # had verified all tensors are of same torch.float32 dtype, not sure where the error is :(
    loss_fct = nn.CrossEntropyLoss(reduction='none') # do reduction later
    
    start_loss = loss_fct(start_logits, start_positions) * pos_weight(start_logits, start_positions, 1, 1)
    end_loss = loss_fct(end_logits, end_positions) * pos_weight(end_logits, end_positions, 1, 1)
    
    start_loss = torch.mean(start_loss)
    end_loss = torch.mean(end_loss)
    total_loss = (start_loss + end_loss)
    return total_loss
</code></pre>
<pre><code class="lang-auto"># Dry Run Test on Dummy Values

start = torch.Tensor([[0.1, 0.1, 0.1, 0.8, 0.1]]).float()
start_target = torch.Tensor([1]).long()

end = torch.Tensor([[0.1, 0.1, 0.1, 0.8, 0.1]]).float()
end_target = torch.Tensor([3]).long()
loss_fn(start, end, start_target, end_target) # returns tensor(3.5881)
</code></pre>
<p>System Config -:</p>
<p>pytorch_version is <code>1.5.0</code><br>
on Ubuntu</p>
<p>cat /usr/local/cuda/version.txt returns the below,</p>
<blockquote>
<p>CUDA Version 9.2.148<br>
CUDA Patch Version 9.2.148.1</p>
</blockquote>
<p>nvidia-smi returns the below,</p>
<blockquote>
<p>NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0</p>
</blockquote>
<p>Complete Traceback,</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-20-fd990f79dfa6&gt; in &lt;module&gt;
      1 torch.cuda.empty_cache()
----&gt; 2 run(fold=0)

&lt;ipython-input-18-114e1832c8bf&gt; in run(fold)
     80 
     81     for epoch in range(config.EPOCHS):
---&gt; 82         train_fn(train_data_loader, model, optimizer, device, scheduler=scheduler)
     83         jaccard = eval_fn(valid_data_loader, model, device)
     84         print(f"Jaccard Score = {jaccard}")

&lt;ipython-input-13-aef75b9df068&gt; in train_fn(data_loader, model, optimizer, device, scheduler)
     40         loss = loss_fn(outputs_start, outputs_end, targets_start, targets_end, class_preds, class_label)
     41 
---&gt; 42         loss.backward()
     43         # model = model.float() found on SO, old discussion post here for this error;
     44         optimizer.step()

~/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    196                 products. Defaults to ``False``.
    197         """
--&gt; 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    199 
    200     def register_hook(self, hook):

~/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--&gt; 100         allow_unreachable=True)  # allow_unreachable flag
    101 
    102

RuntimeError: Expected isFloatingType(grads[i].scalar_type()) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.) (validate_outputs at /opt/conda/conda-bld/pytorch_1587428111115/work/torch/csrc/autograd/engine.cpp:476)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7fc24cc2db5e in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0x2ae32c7 (0x7fc2713782c7 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) + 0x548 (0x7fc271379368 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, bool) + 0x3d2 (0x7fc27137b2f2 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: torch::autograd::Engine::thread_init(int) + 0x39 (0x7fc271373969 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7fc2746ba548 in /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: &lt;unknown function&gt; + 0xc819d (0x7fc2a568819d in /root/anaconda3/lib/python3.7/site-packages/zmq/backend/cython/../../../../.././libstdc++.so.6)
frame #7: &lt;unknown function&gt; + 0x76ba (0x7fc2a887b6ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #8: clone + 0x6d (0x7fc2a85b141d in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<hr>
<p>Also, if i use this variant of the <code>loss_fn</code>, then no error as such happens during training loop;</p>
<pre><code class="lang-auto">def loss_fn(start_logits, end_logits, start_positions, end_positions):
    loss_fct = nn.CrossEntropyLoss()
    start_loss = loss_fct(start_logits, start_positions)
    end_loss   = loss_fct(end_logits, end_positions)
    total_loss = (start_loss + end_loss)
    return total_loss
</code></pre>

albanD: <p>Hi,</p>
<p>Do you get a nice python stack trace or just the cpp one?</p>

ecdrid: <p>Updated the traceback;</p>

albanD: <p>Do you have custom autograd Function in your code?<br>
It seems that one of them returned a gradient which is integer type while gradients should always be floating point types.<br>
Enabling anomaly mode might help you pinpoint the faulty Function.</p>

ecdrid: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="79298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Do you have custom autograd Function in your code?</p>
</blockquote>
</aside>
<p>No i don’t have this Alban;</p>
<blockquote>
<p>Enabling anomaly mode might help you pinpoint the faulty Function.</p>
</blockquote>
<p>Can you share how do we enable that? (Will post back what i found!)</p>
<p>Thanks;</p>

albanD: <blockquote>
<p>Can you share how do we enable that?</p>
</blockquote>
<p>Ho yes sure.<br>
<code>torch.autograd.set_detect_anomaly(True)</code></p>
<blockquote>
<p>No i don’t have this Alban;</p>
</blockquote>
<p>Ok, And do you manually provide grad_output to .backward() or autograd.grad() ?</p>

ecdrid: <blockquote>
<p>And do you manually provide grad_output to .backward() or autograd.grad() ?</p>
</blockquote>
<p>Well i don’t think i am even aware of this <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"> ; It’s a very simple training loop, nothing complicated at all except the loss_fn which i modified;</p>

ecdrid: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="79298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>torch.autograd.set_detect_anomaly(True)</p>
</blockquote>
</aside>
<p>Well, same traceback actually;</p>

albanD: <p>It is quite weird idd.<br>
Do you think you can get a small code sample (30 lines) that I could run?</p>

ecdrid: <p>I can send you in a private gist, [the whole training pipeline] because code is part of an on-going kaggle comp (hence it’s not small either but easily readable);</p>
<p>Apologies for the inconvenience;</p>
<p>Thanks;</p>

albanD: <p>I was wondering if you had something like the dry run example above;</p>
<pre><code class="lang-auto"># Dry Run Test on Dummy Values

start = torch.Tensor([[0.1, 0.1, 0.1, 0.8, 0.1]]).float()
start_target = torch.Tensor([1]).long()

end = torch.Tensor([[0.1, 0.1, 0.1, 0.8, 0.1]]).float()
end_target = torch.Tensor([3]).long()
loss_fn(start, end, start_target, end_target) # returns tensor(3.5881)
</code></pre>
<p>But <code>loss_fct</code> is missing. Is that the part you cannot share?</p>

ecdrid: <blockquote>
<p>But  <code>loss_fct</code>  is missing. Is that the part you cannot share?</p>
</blockquote>
<p><code>loss_fct</code> is <code>loss_fct = nn.CrossEntropyLoss(reduction='none')</code> (shared above in loss_fn)?<br>
Sorry if i didn’t understand your question.</p>

albanD: <p>The problem is that <code>pos_weight</code> is not actually differentiable because the argmax op is not differentiable.<br>
But due to a  bug on our side, this is detected too late and here is a minimal repro:</p>
<pre><code class="lang-auto">import torch
from torch import nn

t = torch.rand(10, requires_grad=True)
bad = torch.argmax(t)

res = bad + 2

res.sum().backward()
</code></pre>
<p>You can fix your code by adding a <code>.detach()</code> to the output of <code>torch.where</code> before returning it in <code>pos_weight</code>.</p>

ecdrid: <p>Thanks a bunch! <img src="https://discuss.pytorch.org/images/emoji/apple/smile_cat.png?v=9" title=":smile_cat:" class="emoji" alt=":smile_cat:"></p>
<p>Not opening an issue on github then <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ?</p>

albanD: <p>Well, still a problem. Opened the issue here: <a href="https://github.com/pytorch/pytorch/issues/37680">https://github.com/pytorch/pytorch/issues/37680</a></p>

ecdrid: <p>Glad i was able to break/find a potential bug in PyTorch! haha just kidding <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>