`Expected self.scalar_type() == ScalarType::Float to be true, but got false.` when doing quantization aware training?

feiyuhuahuo: <p>I’m trying to do QAT for <code>yolov5</code>, but I got the following error:<br>
<code>RuntimeError: Expected self.scalar_type() == ScalarType::Float to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.) </code><br>
This is the total log:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 464, in &lt;module&gt;
    train(hyp, opt, device, tb_writer)
  File "train.py", line 269, in train
    pred = model(imgs)  # forward
  File "/home/feiyu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/feiyu/yolov5/models/yolo.py", line 132, in forward
    x = self.forward_once(x, profile)  # single-scale inference, train
  File "/home/feiyu/yolov5/models/yolo.py", line 150, in forward_once
    x = m(x)  # run
  File "/home/feiyu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/feiyu/yolov5/models/common.py", line 114, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "/home/feiyu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/feiyu/yolov5/models/common.py", line 48, in fuseforward
    return self.act(self.conv(x))
  File "/home/feiyu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 731, in _call_impl
    hook_result = hook(self, input, result)
  File "/home/feiyu/anaconda3/lib/python3.7/site-packages/torch/quantization/quantize.py", line 82, in _observer_forward_hook
    return self.activation_post_process(output)
  File "/home/feiyu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/feiyu/anaconda3/lib/python3.7/site-packages/torch/quantization/fake_quantize.py", line 104, in forward
    self.quant_max)
RuntimeError: Expected self.scalar_type() == ScalarType::Float to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

</code></pre>
<p>And here is the code where error occurs:</p>
<pre><code class="lang-auto">class Conv(nn.Module):
    # Standard convolution
    def __init__(self, c1, c2, k=1, s=1, p=None, g=1, act=True):  # ch_in, ch_out, kernel, stride, padding, groups
        super().__init__()
        self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p), groups=g, bias=False)
        self.bn = nn.BatchNorm2d(c2)
        self.act = nn.Hardswish() if act else nn.Identity()

    def forward(self, x):
        return self.act(self.bn(self.conv(x)))

    def fuseforward(self, x):
        return self.act(self.conv(x))
</code></pre>
<p>I tried use <code>nn.ReLU</code> to replace <code>nn.Hardswish</code>, but it didn’t work. And I checked the input <code>x</code> <code>dtype</code>, which is always <code>torch.float32</code>.</p>

feiyuhuahuo: <p>I directly add <code>X = X.float()</code> above where the error appears in <code>torch.quantization.fake_quantize.py</code>.  Because I found the <code>dtype</code>  of <code>X</code> is torch.float16. Surprisingly, it works, the model can be successfully trained.</p>
<pre><code class="lang-auto">        if self.fake_quant_enabled[0] == 1:
            if self.qscheme == torch.per_channel_symmetric or self.qscheme == torch.per_channel_affine:
                X = torch.fake_quantize_per_channel_affine(X, self.scale, self.zero_point,
                                                           self.ch_axis, self.quant_min, self.quant_max)
            else:
                import pdb
                # try:
                # I modified the code here. $$$$$$$$$$$$$$$$$
                X= X.float()
                X = torch.fake_quantize_per_tensor_affine(X, float(self.scale),
                                                          int(self.zero_point), self.quant_min,
                                                          self.quant_max)
                # except:
                #     pdb.set_trace()
        return X
</code></pre>
<p>Is this normal?<br>
I haven’t completed training now, thus I can’t tell whether I can get a useful model. My PyTorch version is 1.7.0</p>

feiyuhuahuo: <p>My bad. I didn’t notice that the forward progress is wrapped by <code>torch.cuda.amp.autocast</code>. It’s OK now.</p>

loveltyoic: <p>hi, I met the same problem. Could you please paste the updated code here, thanks.</p>

addisonklinke: <p><a class="mention" href="/u/loveltyoic">@loveltyoic</a> There’s more discussion on the related Github issue <a href="https://github.com/pytorch/pytorch/issues/50417" rel="noopener nofollow ugc">here</a></p>