Do we have to count the background as an object class for SSD?

username: douglasrizzo
cooked: <p>The documentation for the <code>SSD</code> class mentions that we should not count the background as an object class when passing the number of classes as a parameter to instantiate an SSD object.</p>
<aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L144" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L144" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L144</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="134" style="counter-reset: li-counter 133 ;">
          <li>    - scores (Tensor[N]): the scores for each detection</li>
          <li>
          </li>
<li>Args:</li>
          <li>    backbone (nn.Module): the network used to compute the features for the model.</li>
          <li>        It should contain an out_channels attribute with the list of the output channels of</li>
          <li>        each feature map. The backbone should return a single Tensor or an OrderedDict[Tensor].</li>
          <li>    anchor_generator (DefaultBoxGenerator): module that generates the default boxes for a</li>
          <li>        set of feature maps.</li>
          <li>    size (Tuple[int, int]): the width and height to which images will be rescaled before feeding them</li>
          <li>        to the backbone.</li>
          <li class="selected">    num_classes (int): number of output classes of the model (excluding the background).</li>
          <li>    image_mean (Tuple[float, float, float]): mean values used for input normalization.</li>
          <li>        They are generally the mean values of the dataset on which the backbone has been trained</li>
          <li>        on</li>
          <li>    image_std (Tuple[float, float, float]): std values used for input normalization.</li>
          <li>        They are generally the std values of the dataset on which the backbone has been trained on</li>
          <li>    head (nn.Module, optional): Module run on top of the backbone features. Defaults to a module containing</li>
          <li>        a classification and regression module.</li>
          <li>    score_thresh (float): Score threshold used for postprocessing the detections.</li>
          <li>    nms_thresh (float): NMS threshold used for postprocessing the detections.</li>
          <li>    detections_per_img (int): Number of best detections to keep after NMS.</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>However, further down in the same file, an SSD object is instantiated in a function that explicitly says that the background <strong>should</strong> be counted as an object class, but this is not taken into account in the code (i.e. I did not see <code>num_classes</code> be decremented by one when creating the SSD object).</p>
<aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L589" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L589" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L589</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="579" style="counter-reset: li-counter 578 ;">
          <li>anchor_generator = DefaultBoxGenerator([[2], [2, 3], [2, 3], [2, 3], [2], [2]],</li>
          <li>                                       scales=[0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05],</li>
          <li>                                       steps=[8, 16, 32, 64, 100, 300])</li>
          <li>
          </li>
<li>defaults = {</li>
          <li>    # Rescale the input in a way compatible to the backbone</li>
          <li>    "image_mean": [0.48235, 0.45882, 0.40784],</li>
          <li>    "image_std": [1.0 / 255.0, 1.0 / 255.0, 1.0 / 255.0],  # undo the 0-1 scaling of toTensor</li>
          <li>}</li>
          <li>kwargs = {**defaults, **kwargs}</li>
          <li class="selected">model = SSD(backbone, anchor_generator, (300, 300), num_classes, **kwargs)</li>
          <li>if pretrained:</li>
          <li>    weights_name = 'ssd300_vgg16_coco'</li>
          <li>    if model_urls.get(weights_name, None) is None:</li>
          <li>        raise ValueError("No checkpoint is available for model {}".format(weights_name))</li>
          <li>    state_dict = load_state_dict_from_url(model_urls[weights_name], progress=progress)</li>
          <li>    model.load_state_dict(state_dict)</li>
          <li>return model</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Here is the documentation for this function, which says we should include the background in the number of classes.</p>
<aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L563" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L563" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/959666891589c35e6d225943253f523cffbae4cc/torchvision/models/detection/ssd.py#L563</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="553" style="counter-reset: li-counter 552 ;">
          <li>Example:</li>
          <li>
          </li>
<li>    &gt;&gt;&gt; model = torchvision.models.detection.ssd300_vgg16(pretrained=True)</li>
          <li>    &gt;&gt;&gt; model.eval()</li>
          <li>    &gt;&gt;&gt; x = [torch.rand(3, 300, 300), torch.rand(3, 500, 400)]</li>
          <li>    &gt;&gt;&gt; predictions = model(x)</li>
          <li>
          </li>
<li>Args:</li>
          <li>    pretrained (bool): If True, returns a model pre-trained on COCO train2017</li>
          <li>    progress (bool): If True, displays a progress bar of the download to stderr</li>
          <li class="selected">    num_classes (int): number of output classes of the model (including the background)</li>
          <li>    pretrained_backbone (bool): If True, returns a model with backbone pre-trained on Imagenet</li>
          <li>    trainable_backbone_layers (int): number of trainable (not frozen) resnet layers starting from final block.</li>
          <li>        Valid values are between 0 and 5, with 5 meaning all backbone layers are trainable.</li>
          <li>"""</li>
          <li>if "size" in kwargs:</li>
          <li>    warnings.warn("The size of the model is already fixed; ignoring the argument.")</li>
          <li>
          </li>
<li>trainable_backbone_layers = _validate_trainable_layers(</li>
          <li>    pretrained or pretrained_backbone, trainable_backbone_layers, 5, 5)</li>
          <li>
      </li>
</ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This is confusing. Should we or should we not count the background as an object class when instantiating the SSD? In either case, how should object classes be ID’d during training?</p>
<p>As an example, with Faster RCNN, the background is counted as an object class (with ID 0 reserved for it) and actual object classes are identified during training starting from ID 1. What should be the procedure for SSD?</p>

username: douglasrizzo
cooked: <p>This was solved here: <a href="https://github.com/pytorch/vision/issues/4106" class="inline-onebox" rel="noopener nofollow ugc">Documentation confusing on whether SSD and RetinaNet count background as class object · Issue #4106 · pytorch/vision · GitHub</a></p>