Finetuning FasterRCNN for multi-class examples

sree_harsha: <p>Hello,</p>
<p>I’m trying to fine-tune Faster RCNN with the Resnet-50 backend via Torchvision, using some references I have arrived at a <strong>getitem</strong> call which looks like the following</p>
<pre><code class="lang-auto">
def __getitem__(self, index: int):

        file_name = self.file_names[index]
        records = self.data[self.data['file_name'] == file_name]
        
        image = np.array(Image.open(file_name), dtype=np.float32)
        image /= 255.0

        if self.transform:
            image = self.transform(image)  
            
        if self.mode != "test":
            boxes = records[['xmin', 'ymin', 'xmax', 'ymax']].values
            
            area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
            area = torch.as_tensor(area, dtype=torch.float32)

            labels = torch.ones((records.shape[0],), dtype=torch.int64)
            
            iscrowd = torch.zeros((records.shape[0],), dtype=torch.int64)
            
            target = {}

            target['boxes'] = boxes
            target['labels'] = labels
            target['image_id'] = torch.tensor([index])
            target['area'] = area
            target['iscrowd'] = iscrowd 
            target['boxes'] = torch.stack(list((map(torch.tensor, target['boxes'])))).type(torch.float32)

            return image, target, file_name
        else:
            return image, file_name
</code></pre>
<p>This line <code>labels = torch.ones((records.shape[0],), dtype=torch.int64)</code> assumes that there is only one other class and 0 in the case of Faster RCNN is reserved for the Background.</p>
<p>I have a dataset with multiple classes, and I am unable to figure out how to modify and pass the one hot encoded version to train the Faster RCNN model for a a multi-class scenario</p>
<p>Any advice would be great,</p>
<p>Thank you!</p>

MD_ASIF_IQBAL_FAHIM: <p>Sir If I have 21 classes then what should I need to write?</p>

unicorn: <p>Is this issue resolved? How did you fix it for multi classes? Please let me know</p>

douglasrizzo: <p>Object classes should be labelled sequentially, <strong>starting from 1</strong>. So, if you have 4 object classes, they should be numbered 1, 2, 3, and 4 in the <code>labels</code> variable mentioned by OP.</p>
<p>However, when creating the <code>FasterRCNN</code> object, the documentation for the <code>num_classes</code> argument says that you have to account for the background, which, by default, is class number 0.</p>
<aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/850491eb9c3daff887468718bfa1f90f94042d9d/torchvision/models/detection/faster_rcnn.py#L54-L55" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/850491eb9c3daff887468718bfa1f90f94042d9d/torchvision/models/detection/faster_rcnn.py#L54-L55" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/850491eb9c3daff887468718bfa1f90f94042d9d/torchvision/models/detection/faster_rcnn.py#L54-L55</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="54" style="counter-reset: li-counter 53 ;">
          <li>num_classes (int): number of output classes of the model (including the background).</li>
          <li>    If box_predictor is specified, num_classes should be None.</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So, in our example with 4 object classes, we actually have to pass <code>num_classes=5</code> to <code>FasterRCNN</code>, even though we don’t have 4 classes of object annotations in our dataset.</p>