RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

abd: <p>hello,<br>
I am getting this error when I try this code:</p>
<pre><code class="lang-auto">source = source - torch.mean(source, dim=1, keepdim=True)
		template = template - torch.mean(template, dim=1, keepdim=True)

		output = model(template, source)
		loss_val = ChamferDistanceLoss()(template, output['transformed_source'])
		print(loss_val)
                #model.named_parameters()
		# forward + backward + optimize
		optimizer.zero_grad()
		loss_val.backward()
		optimizer.step()
</code></pre>
<p>print(loss_val)==&gt;tensor(0.2791, device=‘cuda:0’, grad_fn=DivBackward0)<br>
How can I solve this problem?<br>
Thanks in advance,</p>

albanD: <p>Hi,</p>
<p>Could you enable anomaly mode to see where it comes from? <code>torch.autograd.set_detect_anomaly(True)</code> at the begining of your script.</p>
<p>Also if you have custom autograd.Function, could you share their code here?</p>

abd: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,<br>
Anomaly mode:&lt;torch.autograd.anomaly_mode.set_detect_anomaly object at 0x7eff1bdba390&gt;<br>
loss_val tensor(0.2113, device=‘cuda:0’, grad_fn=DivBackward0)<br>
/opt/conda/conda-bld/pytorch_1573049304260/work/torch/csrc/autograd/python_anomaly_mode.cpp:40: UserWarning: No forward pass information available. Enable detect anomaly during forward pass for more information.</p>
<p>I don’t understand just why this error.<br>
loss_val:tensor(0.2113, device=‘cuda:0’, grad_fn=DivBackward0)<br>
<code>loss_val.backward()</code> did’nt work</p>
<p>thanks,</p>

albanD: <p>Interesting… The errors seems to be that some gradients are not floating point numbers.<br>
Do you have a custom autograd Function in your code?<br>
If not, can you share a small code sample (30 lines) that reproduces the issue)?</p>

abd: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<pre><code class="lang-auto">def train_one_epoch(device, model, train_loader, optimizer):
	model.train()
	train_loss = 0.0
	pred  = 0.0
	count = 0
	for i, data in enumerate(tqdm(train_loader)):
		template, source, igt = data

		template = template.to(device)
		source = source.to(device)
		igt = igt.to(device)

		# mean substraction
		source = source - torch.mean(source, dim=1, keepdim=True)
		template = template - torch.mean(template, dim=1, keepdim=True)

		output = model(template, source)
		loss_val = ChamferDistanceLoss()(template, output['transformed_source'])
		# print(loss_val.item())

		# forward + backward + optimize
		optimizer.zero_grad()
		loss_val.backward()
		optimizer.step()

		train_loss += loss_val.item()
		count += 1

	train_loss = float(train_loss)/count
	return train_loss
</code></pre>
<p>error in loss_val.backward()</p>

albanD: <p>Hi,</p>
<p>I am missing few key pieces here:<br>
What is <code>ChamferDistanceLoss</code> ?<br>
What is <code>model</code> ?</p>

abd: <p>Function chamfer_istance</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

def chamfer_distance(template: torch.Tensor, source: torch.Tensor):
	from .cuda.chamfer_distance import ChamferDistance
	cost_p0_p1, cost_p1_p0 = ChamferDistance()(template, source)
	cost_p0_p1 = torch.mean(torch.sqrt(cost_p0_p1))
	cost_p1_p0 = torch.mean(torch.sqrt(cost_p1_p0))
	chamfer_loss = (cost_p0_p1 + cost_p1_p0)/2.0
	return chamfer_loss


class ChamferDistanceLoss(nn.Module):
	def __init__(self):
		super(ChamferDistanceLoss, self).__init__()

	def forward(self, template, source):
		return chamfer_distance(template, source)
</code></pre>
<p>file train_pcrnet.py:line 99(loss_val.backward())<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/vinits5/pcrnet_pytorch/blob/master/train_pcrnet.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/vinits5/pcrnet_pytorch/blob/master/train_pcrnet.py" target="_blank" rel="nofollow noopener">vinits5/pcrnet_pytorch/blob/master/train_pcrnet.py</a></h4>
<pre><code class="lang-py">import argparse
import os
import sys
import logging
import numpy
import numpy as np
import torch
import torch.utils.data
import torchvision
from torch.utils.data import DataLoader
from tensorboardX import SummaryWriter
from tqdm import tqdm

# Only if the files are in example folder.
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
if BASE_DIR[-8:] == 'examples':
	sys.path.append(os.path.join(BASE_DIR, os.pardir))
	os.chdir(os.path.join(BASE_DIR, os.pardir))
	
from pcrnet.models import PointNet
</code></pre>

  This file has been truncated. <a href="https://github.com/vinits5/pcrnet_pytorch/blob/master/train_pcrnet.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


albanD: <p>And you’re sure that these outputs here: <a href="https://github.com/vinits5/pcrnet_pytorch/blob/5ee2a2b5c12faa9c43c4c7ae1e503e098ecd4b46/pcrnet/losses/cuda/chamfer_distance/chamfer_distance.py#L61">https://github.com/vinits5/pcrnet_pytorch/blob/5ee2a2b5c12faa9c43c4c7ae1e503e098ecd4b46/pcrnet/losses/cuda/chamfer_distance/chamfer_distance.py#L61</a> are all floats?</p>