# Benchmarking Deep Learning Models for Instance Segmentation
## 4 Review Method
### Speed-Focused Models  
**CenterMask**[5] was proposed as an anchor-free instance segmentation model. It is an architecture with a spatial attention-guided mask (SAG-Mask) branch added after the anchor-free one-stage object detector (FCOS) [26] and uses an FPN backbone (Figure 9). When CenterMask was proposed, few studies had considered the inference speed, and the accuracy was lower than that of Mask R-CNN [4]; hence, CenterMask was designed to improve both accuracy and speed. The authors who presented CenterMask [5] argued that anchor-box-based object detectors depend strongly on predefined anchors; hence, when too many anchor boxes are created, they not only cause imbalances between positive and negative RoI samples, but also increase the computational costs. To overcome the shortcomings of these anchor boxes, an FCOS is used as an object detector in CenterMask in the first stage, which is anchor-free. In mask prediction, each RoI pooling operation should be allocated considering the RoI scale when extracting the features from each RoI. However, as RoAlign proposed for Mask R-CNN [4] does not consider the input scale, the authors [5] designed a scale-adaptive RoI assignment function suitable for an FCOS. When CenterMask was proposed, attention methods were widely used for object detection because they can help the model extract more important features while suppressing unimportant features. Channel attention focuses on _what_ important information is in the channels of the feature maps, whereas spatial attention focuses on _where_ the important information is located in the feature maps. The spatial attention module (SAM) inside the SAG-Mask helps to predict the mask by emphasizing meaningful pixels while reducing noise in Figure 9. The FCOS receives feature maps from the FPN backbone and predicts the bboxes. The bboxes received from the FCOS are used to extract features inside the predicted RoI through the scale-adaptive RoI assignment function, and the final masks are output through the later SAG-Mask, which consists of convolutional layers, a spatial attention module, upsampling processes, and other processes. Figure 8: Shifted window approach for computing self-attention (COCO-val2017-00000548267 [10]). Figure 9: CenterMask architecture. The box and mask results were graphically modified, not the actual inference result (COCO-val2017-000000015746 [10]). **YOLACT**[8] was proposed as a one-stage instance segmentation model separated by two parallel subprocesses: (1) the generation of a prototype mask set and (2) the prediction of mask coefficients per instance (Figure 10). The basic model is a modification of RetinaNet [27], a one-stage object detection model. In YOLACT, instance segmentation processes were parallelized with a ProtoNet and a prediction head to add mask branches without feature localization steps. In the prediction head, the feature maps from the FPN are input and sent to the class, box, and mask coefficient branches through convolution. ProtoNet receives the deepest feature map of the FPN, P3, as an input and generates a predefined number of prototype masks, but each mask does not activate (localize) one specific instance. Therefore, for each instance that is not erased via Fast-NMS, YOLACT performs a mask assembly that creates a mask by linearly combining the prototype mask and mask coefficient. The final mask is generated by cropping outputs of the mask assembly based on the predicted bbox and by thresholding. The author of [8] also proposed Fast NMS, which speeds up with slight degradation of AP, instead of the standard NMS used in Mask R-CNN. The Fast NMS is as follows: when a reference anchor box is compared to other anchor boxes with higher confidences than itself, if the compared anchor box with the largest intersection-over-union (IoU) value is less than the threshold, the reference anchor box is not erased. In standard NMS, when an anchor box is erased, it is no longer compared to any other anchor box.