RuntimeError during evaluation upsample_bilinear2d_out_frame" not implemented for 'Byte'

username: jitesh
cooked: <p>I have trained a custom object detection model using the steps described in this <a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=UYDb7PBw55b-" rel="nofollow noopener">link</a>. I am able to train my model but when I try to evaluate it at the end of an epoch, I get the following error</p>
<pre><code class="lang-auto">Epoch: [0] Total time: 0:00:06 (0.2223 s / it)
creating index...
index created!
Traceback (most recent call last):
  File "train.py", line 106, in &lt;module&gt;
    evaluate(model, data_loader_test, device=device)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/sarvani/Desktop/flir/test_frcnn/custom/engine.py", line 107, in evaluate
    outputs = model(image)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 47, in forward
    images, targets = self.transform(images, targets)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torchvision/models/detection/transform.py", line 41, in forward
    image, target = self.resize(image, target)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torchvision/models/detection/transform.py", line 70, in resize
    image[None], scale_factor=scale_factor, mode='bilinear', align_corners=False)[0]
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/nn/functional.py", line 2503, in interpolate
    return torch._C._nn.upsample_bilinear2d(input, _output_size(2), align_corners)
RuntimeError: "upsample_bilinear2d_out_frame" not implemented for 'Byte'
</code></pre>
<p>My code to load the data is as follows</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    
    def __init__(self, root_dir,transform=None):
        self.root = root_dir
        self.rgb_imgs = list(sorted(os.listdir(os.path.join(root_dir, "rgb/"))))
        self.annotations = list(sorted(os.listdir(os.path.join(root_dir, "annotations/"))))


        self._classes = ('__background__',  # always index 0
                         'car','person','bicycle','dog','other')

        self._class_to_ind = {'car':'1', 'person':'2', 'bicycle':'3', 'dog':'4','other':'5'}
	    

    def __len__(self):
        return len(self.rgb_imgs)

    def __getitem__(self, idx):
        self.num_classes = 6

        img_rgb_path = os.path.join(self.root, "rgb/", self.rgb_imgs[idx])   
        img = Image.open(img_rgb_path)
        img = np.array(img)
        img = img.transpose((2, 0, 1))
        img = torch.from_numpy(img)

        filename = os.path.join(self.root,'annotations',self.annotations[idx])
        tree = ET.parse(filename)
        objs = tree.findall('object')

        num_objs = len(objs)
        labels = np.zeros((num_objs), dtype=np.float32)
        seg_areas = np.zeros((num_objs), dtype=np.float32)
        
        boxes = []
        for ix, obj in enumerate(objs):
            bbox = obj.find('bndbox')
            x1 = float(bbox.find('xmin').text)
            y1 = float(bbox.find('ymin').text)
            x2 = float(bbox.find('xmax').text)
            y2 = float(bbox.find('ymax').text)

            cls = self._class_to_ind[obj.find('name').text.lower().strip()]
            boxes.append([x1, y1, x2, y2])
            labels[ix] = cls
        
        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        image_id = torch.tensor([idx])
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
        labels = torch.as_tensor(labels, dtype=torch.float32)
    
        target =  {'boxes': boxes,
                'labels': labels,
                'area': area,
                "image_id":image_id
                }
        target["iscrowd"] = iscrowd
        return img,target

</code></pre>
<p>My train.py is as follows</p>
<pre><code class="lang-auto">num_classes = 6
model = fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

device = torch.device('cuda')
model = model.cuda()

dataset_train = CustomDataset('FLIR/images/train')
dataset_val = CustomDataset('FLIR/images/val')


data_loader_train = torch.utils.data.DataLoader(
    dataset_train, batch_size=4, shuffle=True,collate_fn=utils.collate_fn)

data_loader_test = torch.utils.data.DataLoader(
    dataset_val, batch_size=4 shuffle=False,collate_fn=utils.collate_fn)

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.Adam(params,lr=0.05,weight_decay=0.0005)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

num_epochs = 30

for epoch in range(num_epochs):
    train_one_epoch(model, optimizer, data_loader_train, device, epoch, print_freq=1)
    lr_scheduler.step()
    evaluate(model, data_loader_test, device=device)
</code></pre>
<p>The evaluation function long with the necessary files used are at this <a href="https://github.com/pytorch/vision/tree/master/references/detection" rel="nofollow noopener">link</a>.</p>
<p>Can someone please help me out.</p>

username: ptrblck
cooked: <p>Could you check the <code>tensor.type()</code> of the input to your model during evaluation?<br>
It seems you are passing <code>uint8</code> tensors, while <code>FloatTensors</code> are expected.</p>

username: mapostig
cooked: <p>Hello!<br>
I am following the same tutorial (but using my own dataset)<br>
Within the transforms I do a cast in order to cast the image tensors to type(torch.FloatTensor)</p>
<pre><code class="lang-auto">class ToTensor(object):
    def __call__(self, image, target):

      image = image.transpose((2, 0, 1)) 
      return torch.from_numpy(np.flip(image,axis=0).copy()).type(torch.FloatTensor), target
</code></pre>
<p>Nevertheless, during training:</p>
<pre><code class="lang-auto">num_epochs = 10
for epoch in range(num_epochs):
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, train_loader, device, epoch, print_freq=10)
    # update the learning rate
    lr_scheduler.step()
</code></pre>
<p>I obtain the following error:</p>
<p>Loss is nan, stopping training<br>
{‘loss_classifier’: tensor(2.0606, device=‘cuda:0’, grad_fn=&lt;/NllLossBackward/&gt;), ‘loss_box_reg’: tensor(0., device=‘cuda:0’, grad_fn=&lt;/DivBackward0/&gt;), ‘loss_objectness’: tensor(213.0078, device=‘cuda:0’,<br>
grad_fn=&lt;/BinaryCrossEntropyWithLogitsBackward/&gt;), ‘loss_rpn_box_reg’: tensor(nan, device=‘cuda:0’, grad_fn=&lt;/DivBackward0/&gt;)}<br>
An exception has occurred, use %tb to see the full traceback.</p>
<p>SystemExit: 1</p>
<p>Why is this happening?<br>
Moreover the image visualization after the tensor cast is very poor.</p>
<p>Thankyou</p>

username: ptrblck
cooked: <p>You error message seems to raise a NaN loss or do you also see the “not implemented for Byte” error?<br>
In the former case, could you create a new topic with a description of your model and use case (e.g. are you using mixed-precision training, what type or model etc.) and tag me there please?</p>
<aside class="quote no-group" data-username="mapostig" data-post="3" data-topic="71625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mapostig/48/21099_2.png" class="avatar"> mapostig:</div>
<blockquote>
<p>Moreover the image visualization after the tensor cast is very poor.</p>
</blockquote>
</aside>
<p>If you are visualizing with <code>matplotlib</code>, you might have to check the current range, as <code>matplotlib</code> might try to normalize and clip the image.</p>

username: mapostig
cooked: <p>Hello!</p>
<p>Finally I was able to solve the problem.<br>
My boxes where in the format [xmin, ymin, w, h] whilst they should be as [xmin, ymin, xmax, ymax]<br>
Moreover, the image values to feed the network were between (0, 255) and they should be between (-1, 1) (isn’t it?)</p>
<p>Also, when I create the model I should pass the mean and std of the images set:</p>
<pre><code class="lang-auto">"""#Finetuning the model"""

num_classes  = 2
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True, image_mean = mean, image_std = std)
</code></pre>
<p>Otherwise, the fasterrcnn_resnet50_fpn would use some default mean and std (from ImageNet maybe?) -see part of fasterrcnn_resnet50_fpn  code below-</p>
<pre><code class="lang-auto">if image_mean is None:
            image_mean = [0.485, 0.456, 0.406]
        if image_std is None:
            image_std = [0.229, 0.224, 0.225]
        transform = GeneralizedRCNNTransform(min_size, max_size, image_mean, image_std)
</code></pre>
<p>Thankyou so much for your time!!</p>

username: mapostig
cooked: <p>Moreover, I saw the  “not implemented for Byte” error, and when I cast the images to <code>FloatTensors</code> as you suggested, the error was the NaN loss.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mapostig" data-post="5" data-topic="71625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mapostig/48/21099_2.png" class="avatar"> mapostig:</div>
<blockquote>
<p>Moreover, the image values to feed the network were between (0, 255) and they should be between (-1, 1) (isn’t it?)</p>
</blockquote>
</aside>
<p>Usually you would normalize the input images to have a zero mean and a unit variance via <code>transforms.Normalize</code>, which is beneficial for the training.<br>
The data type is <code>float32</code> by default, which would explain the <code>byte</code> dtype error.</p>
<aside class="quote no-group" data-username="mapostig" data-post="6" data-topic="71625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mapostig/48/21099_2.png" class="avatar"> mapostig:</div>
<blockquote>
<p>when I cast the images to <code>FloatTensors</code> as you suggested, the error was the NaN loss.</p>
</blockquote>
</aside>
<p>That’s not good. Could you post your complete code (if possible), so that I could take a look, please?</p>

username: mapostig
cooked: <p>Hello,</p>
<p>At the moment I cannot post the code since till I have my Lab permission…<br>
Nevetheless, at the end the problem wasn’t solved only by passing the mean and std to the backbone</p>
<p>Thanks a lot for your help</p>

username: ptrblck
cooked: <p>Sure, in case you want to debug a bit further:</p>
<ul>
<li>Add <code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of your script. This would yield a stack trace with the operation, which caused the first NaN output.</li>
<li>If you are using mixed-precision training (via native amp, apex, or your manual implementation), disable it for the sake of debugging.</li>
<li>Update to the latest PyTorch version, if not already done.</li>
<li>Check all <code>torch.log</code>, <code>torch.sqrt</code>, <code>torch.pow</code> calls etc. which might easily create Infs/NaNs.</li>
</ul>
<p>Let me know, if you can release the model or a smaller fake example, which would reproduce this issue.</p>

username: Daniel_Ajisafe
cooked: <p>I found the pre-trained model was saved in HalfTensor (fp16) using pytorch “autocast” while my input data was in fp32. When I used <code>with torch.inference_mode():</code> immediately after the dataloader, the problem was solved. I am guessing both input tensor and model are now in fp16. But interestingly, this is not the case when I print the <code>dtype</code> of the input tensor.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Daniel_Ajisafe" data-post="10" data-topic="71625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daniel_ajisafe/48/36236_2.png" class="avatar"> Daniel_Ajisafe:</div>
<blockquote>
<p>I found the pre-trained model was saved in HalfTensor (fp16) using pytorch “autocast”</p>
</blockquote>
</aside>
<p><code>torch.amp.autocast</code> is a context manager to apply mixed-precision training by transforming input activations to lower <code>dtype</code>s if possible and safe. It will not change any parameter <code>dtype</code> and thus <code>autocast</code> is not responsible for your model’s <code>float16</code> parameters.</p>