RuntimeError: expected type torch.cuda.FloatTensor but got torch.FloatTensor

username: aast226
cooked: <p>I am getting the error:<br>
<code>RuntimeError: expected type torch.cuda.FloatTensor but got torch.FloatTensor</code><br>
when running:<br>
<code>scores, classification, transformed_anchors = retinanet(batch)</code></p>
<p>See code below:</p>
<pre><code class="lang-auto">import torch
import model
from torchvision import transforms
from PIL import Image

torch.set_default_tensor_type(torch.cuda.FloatTensor)

def image_loader(loader, image_name):
    image = Image.open(image_name)
    image = loader(image).float()
    image = image.unsqueeze(0)
    return image

def main():

    retinanet = model.resnet50(num_classes=80, pretrained=True, device='cuda:0')
    state_dict_path = 'coco_resnet_50_map_0_335_state_dict.pt'
    retinanet.load_state_dict(torch.load(state_dict_path))
    retinanet = retinanet.cuda()
    retinanet.eval()

    for name, param in retinanet.named_parameters():
        if param.device.type != 'cuda':
            print('param {}, not on GPU'.format(name))

    image_path = 'image.jpg'

    data_transforms = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor()
    ])

    batch = image_loader(data_transforms, image_path).cuda()
    print(batch.type())
    print(batch.device)

    with torch.no_grad():
        scores, classification, transformed_anchors = retinanet(batch)

if __name__ == '__main__':
    main()
</code></pre>
<p>There is no output when running:</p>
<pre><code class="lang-auto">    for name, param in retinanet.named_parameters():
        if param.device.type != 'cuda':
            print('param {}, not on GPU'.format(name))
</code></pre>
<p>Implying that every parameter has device type ‘cuda’</p>
<p>And the output from running:</p>
<pre><code class="lang-auto">    print(batch.type())
    print(batch.device)
</code></pre>
<p>Is:</p>
<pre><code class="lang-auto">torch.cuda.FloatTensor
cuda:0
</code></pre>
<p>Given that the model and the input tensor are both on the GPU, I’m not sure why I getting this error.<br>
(Note that <code>import model</code> refers to the file <a href="https://github.com/AljoSt/pytorch-retinanet/blob/replace_nms/model.py" rel="nofollow noopener">https://github.com/AljoSt/pytorch-retinanet/blob/replace_nms/model.py</a> from the repo <a href="https://github.com/AljoSt/pytorch-retinanet/tree/replace_nms" rel="nofollow noopener">https://github.com/AljoSt/pytorch-retinanet/tree/replace_nms</a>)</p>

username: ptrblck
cooked: <p>Could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here, please?<br>
I cannot find any obvious error, so I hope the error might point to the line of code causing this issue.</p>

username: aast226
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="64474">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA_LAUNCH_BLOCKING=1</p>
</blockquote>
</aside>
<p>Here is the stack trace</p>
<pre><code class="lang-auto">/home/nvidia/.local/lib/python3.6/site-packages/torch/nn/modules/upsampling.py:129: UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.{} is deprecated. Use nn.functional.interpolate instead.".format(self.name))
Traceback (most recent call last):
  File "min.py", line 41, in &lt;module&gt;
    main()
  File "min.py", line 38, in main
    scores, classification, transformed_anchors = retinanet(batch)
  File "/home/nvidia/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nvidia/projects/pytorch-retinanet/model.py", line 268, in forward
    transformed_anchors = self.regressBoxes(anchors, regression)
  File "/home/nvidia/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nvidia/projects/pytorch-retinanet/utils.py", line 105, in forward
    pred_ctr_x = ctr_x + dx * widths
RuntimeError: expected type torch.cuda.FloatTensor but got torch.FloatTensor
</code></pre>

username: ptrblck
cooked: <p>Thanks for the information.<br>
Could you add a print statement to <code>BBoxTransform</code> and check the device of these tensors?<br>
I would assume that <code>self.mean</code> and <code>self.std</code> are not pushed to the device correctly (<a href="https://github.com/AljoSt/pytorch-retinanet/blob/303d6d3ed10d6408696bb49b78a8ec36f680d563/pytorch_retinanet/utils.py#L84-L89">line of code</a>), as they are not registered as a buffer or parameter.</p>

username: aast226
cooked: <p>I added the lines:</p>
<pre><code class="lang-auto">print(f'Mean is of type: {self.mean.type()}')
print(f'std is of type: {self.std.type()}')
</code></pre>
<p>And the output was:</p>
<pre><code class="lang-auto">Mean is of type: torch.FloatTensor
std is of type: torch.FloatTensor
</code></pre>
<p>So it confirms that they were not the correct type. I also added the following two lines:</p>
<pre><code class="lang-auto">self.std = self.std.cuda()
self.mean = self.mean.cuda()
</code></pre>
<p>Which yielded the output:</p>
<pre><code class="lang-auto">Mean is of type: torch.cuda.FloatTensor
std is of type: torch.cuda.FloatTensor
</code></pre>
<p>But, when I run the program I get the same error message as before. I went through and checked to see if any other variables were the wrong type and found that <code>ctr_x</code>, <code>ctr_y</code>, <code>widths</code>, and <code>heights</code> were also not the correct type. By changing the device for these variables I was able to run inference.</p>

username: ptrblck
cooked: <p>Good to hear it’s working!<br>
However, I couldn’t find related issues in the repository and would assume this issue would be common. <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_raised_eyebrow.png?v=9" title=":face_with_raised_eyebrow:" class="emoji" alt=":face_with_raised_eyebrow:"></p>

username: aast226
cooked: <p>That’s true, before I posted this I also looked through that repo for related issues and couldn’t find any. I tested this on multiple devices running different versions of pytorch and got effectively the same error on each device.</p>