Training retinanet_resnet50_fpn_v2 with varying output sizes and num_classes=2

gerald-ftk: <p>I’ve been trying to train a RetinaNet on the SKU110K dataset. The first issue I encountered is that the <code>retinanet_resnet50_fpn_v2()</code> function throws the following error when num_classes=2:</p>
<pre><code class="lang-auto">model = retinanet_resnet50_fpn_v2(weights='DEFAULT', score_thresh=0.35, num_classes=2).to(device)

ValueError: The parameter 'num_classes' expected value 91 but got 2 instead.
</code></pre>
<p>My dataset only has 2 outputs (object and background), so why does the <code>retinanet_resnet50_fpn_v2</code> function require <code>num_classes=91</code>?</p>
<p>The other issue is that each input image has a differing number of bounding boxes associated with it. My Dataset loads items in the following way:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_name = self.image_names[idx]
        class_name, width, height = self.ds[self.ds.image_name == img_name].iloc[0, 5:8]
        boxes = np.array(self.ds[self.ds.image_name == img_name].iloc[:, 1:5])

        image = Image.open(f"{img_name}")

        image = image.resize((640, 640))

        # Rescale everything to 640
        Fx = image.width // 640
        Fy = image.height // 640

        boxes[:, 0] = boxes[:, 0] / Fx
        boxes[:, 2] = boxes[:, 2] / Fx
        boxes[:, 1] = boxes[:, 1] / Fy
        boxes[:, 3] = boxes[:, 3] / Fy

        image = self.transform(image) # [T.ToImageTensor(), T.ConvertImageDtype()]

        sample = {"img": image, "boxes": boxes, "label": 1}

        return sample
</code></pre>
<p>In this scenario, <code>sample['img']</code> is a Tensor of shape <code>(3, 640, 640)</code> and <code>sample['boxes']</code> is an array of shape (N, 4), where N is the number of bounding boxes in the image.</p>
<p>The other issue I have is that when I get a batch from the dataloader, I get</p>
<pre><code class="lang-auto">
stack expects each tensor to be equal size, but got [74, 4] at entry 0 and [128, 4] at entry 1
</code></pre>
<p>Are there any Retinanet examples out there that use torchvisions v2 model?</p>

gerald-ftk: <p>I also tried adding a collate function to the dataloader that filled the batch with <code>torch.ones</code> tensors, but that throws a <code>AssertionError: All bounding boxes should have positive height and width. Found invalid box </code> error because it has 0 width and height.</p>

ptrblck: <aside class="quote no-group" data-username="gerald-ftk" data-post="1" data-topic="178533">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gerald-ftk/48/59936_2.png" class="avatar"> gerald-ftk:</div>
<blockquote>
<p>My dataset only has 2 outputs (object and background), so why does the <code>retinanet_resnet50_fpn_v2</code> function require <code>num_classes=91</code>?</p>
</blockquote>
</aside>
<p>The pretrained model is trained on <code>91</code> classes. If you want to change the classifier to output logits for 2 classes only initialize the model in its original form, replace the classifier, and finetune it.</p>
<aside class="quote no-group" data-username="gerald-ftk" data-post="1" data-topic="178533">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gerald-ftk/48/59936_2.png" class="avatar"> gerald-ftk:</div>
<blockquote>
<p>Are there any Retinanet examples out there that use torchvisions v2 model?</p>
</blockquote>
</aside>
<p>Could you check if the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">Object detection finetuning tutorial</a> would work for this model, too?</p>
<aside class="quote no-group" data-username="gerald-ftk" data-post="2" data-topic="178533">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gerald-ftk/48/59936_2.png" class="avatar"> gerald-ftk:</div>
<blockquote>
<p>but that throws a <code>AssertionError: All bounding boxes should have positive height and width. Found invalid box </code> error because it has 0 width and height.</p>
</blockquote>
</aside>
<p>Try to filter out invalid bounding boxes which are empty.</p>

gerald-ftk: <p>Thanks for the response <a class="mention" href="/u/ptrblck">@ptrblck</a>, I’m understanding it now.</p>
<blockquote>
<p>The pretrained model is trained on <code>91</code> classes. If you want to change the classifier to output logits for 2 classes only initialize the model in its original form, replace the classifier, and finetune it.</p>
</blockquote>
<p>I understand the 91 class issue now, I should only be fine-tuning the regression head, rather than the entire backbone. Thanks for your help!</p>
<blockquote>
<p>Could you check if the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">Object detection finetuning tutorial</a> would work for this model, too?</p>
</blockquote>
<p>I’ve been using this tutorial, I think I was just having issues with the slightly different class names used in RetinaNet compared to FasterRCNN</p>
<blockquote>
<p>Try to filter out invalid bounding boxes which are empty.</p>
</blockquote>
<p>If I don’t add empty bounding boxes to the Dataset, then I get</p>
<p><code>RuntimeError: stack expects each tensor to be equal size, but got [150, 4] at entry 0 and [162, 4] at entry 1</code></p>
<p>How is one supposed to train the model since there are varying bounding boxes for each input image?</p>

ptrblck: <p>I would again refer to the linked tutorial since the <code>PennFudanDataset</code> also returns target with a variable number of bounding boxes:</p>
<pre><code class="lang-python">dataset = PennFudanDataset('PennFudanPed', transforms=None)

for img, target in dataset:
    print(target["boxes"].shape)

torch.Size([2, 4])
torch.Size([1, 4])
torch.Size([1, 4])
torch.Size([2, 4])
torch.Size([2, 4])
torch.Size([2, 4])
torch.Size([3, 4])
torch.Size([2, 4])
...
</code></pre>

gerald-ftk: <p>I’m bad at programming</p>