RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.HalfTensor) should be the same

username: dulanga_heshan
cooked: <p>I created a custom object detection model using the yoloV5 PyTorch version, so I tried to expose it as flask rest API,  by referring to this <a href="https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html" rel="noopener nofollow ugc">pytorch guide</a><br>
I trained and developing flask applications in the same python environment which installed PyTorch 1.7 cuda version</p>
<pre><code class="lang-auto">def transform_image(image_bytes):
    my_transforms = transforms.Compose([transforms.Resize(255),
                                        transforms.CenterCrop(224),
                                        transforms.ToTensor(),
                                        transforms.Normalize(
                                            [0.485, 0.456, 0.406],
                                            [0.229, 0.224, 0.225])])
    image = Image.open(io.BytesIO(image_bytes)).convert('RGB')
    return my_transforms(image).unsqueeze(0)
</code></pre>
<pre><code class="lang-auto">model = torch.load("weights/best.pt", map_location="cuda")['model']
model.eval()

with open("BikesHelmets128.png", 'rb') as f:
    image_bytes = f.read()
    tensor = transform_image(image_bytes=image_bytes)
    print(tensor)

outputs = model.forward(tensor)
_, y_hat = outputs.max(1)
predicted_idx = str(y_hat.item())
 print(predicted_idx)
</code></pre>
<p><strong>Error stack</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "app.py", line 46, in &lt;module&gt;
    outputs = model.forward(tensor)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 121, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 137, in forward_once
    x = m(x)  # run
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 93, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 34, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same
</code></pre>
<p>I tried several ways by changing transforms, changing device type, and read images using open cv, etc.<br>
thank you</p>

username: Alexey_Demyanchuk
cooked: <p>Hi. It would be better if you provide error stack, to check from what point the problem is coming. Also, the general advice is to explicitly check if both your model and your input are on the same device.</p>

username: dulanga_heshan
cooked: <p><a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a> thanks for your quick response, I added my error stack, and both modal and image are in the same device</p>

username: Alexey_Demyanchuk
cooked: <p>It seems like the error is a bit different from you have in the topic title. In your code you explicitly put the model on the cuda device. But I never see you put your tensor on the same device as well. So, something like <code>tensor = tensor.cuda()</code> should work.</p>

username: dulanga_heshan
cooked: <p>Yes i tried both ways when i applied<code> tensor = tensor.cuda()</code> it throws error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "app.py", line 46, in &lt;module&gt;
    outputs = model.forward(tensor)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 121, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 137, in forward_once
    x = m(x)  # run
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 93, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 34, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same
</code></pre>
<p>and if i removed <code> tensor = tensor.cuda()</code></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "app.py", line 46, in &lt;module&gt;
    outputs = model.forward(tensor)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 121, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\yolo.py", line 137, in forward_once
    x = m(x)  # run
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 93, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\Documents\projects\python\fyp\helmet_covers_yolo\test\models\common.py", line 34, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\Users\D.ShaN\AppData\Local\conda\conda\envs\fyp\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same
</code></pre>
<p>Sorry I updated the question  with correct error stack</p>

username: Alexey_Demyanchuk
cooked: <p>From the first error you can see, that now both tensors are on the same device, but model want a tensor in half precision (HalfTensor). You should cast input to this type, I believe. Look <a href="https://discuss.pytorch.org/t/training-with-half-precision/11815/17">here</a> for some reasoning.<br>
In short (on modern version of pytorch) you need:</p>
<pre><code class="lang-auto">from  torch.cuda.amp import autocast
with autocast():
    outputs = model.forward(tensor)
</code></pre>
<p>Hope this helps.</p>

username: dulanga_heshan
cooked: <p>yeah it works i used <code>model.forward(tensor.half())</code> and now it works fine, thank you</p>

username: jinfagang
cooked: <p>Your works is not autocast(), it’s make your input to half match your moded cause your model is half. autocast() doesn;t work.</p>

username: Owais_Ansari
cooked: <p>You can also convert the input tensors into HalfTensor before passing them into a model</p>
<pre><code class="lang-auto">tensor = tensor.type(torch.cuda.HalfTensor)
outputs = model.forward(tensor)
</code></pre>