# Benchmarking Deep Learning Models for Instance Segmentation
## 4 Review Method
### Accuracy-Focused Models  
**Mask R-CNN**[4] is a model for two-stage instance segmentation. It creates a region of interest (RoI) candidate in the first step and classifies and segments the RoI in the second step (Figure 3). As an extension of Faster R-CNN [23], branches for mask prediction in each RoI are added in parallel to the existing branch for classification and bbox regression. The structure of the head varies depending on the backbone, and when a ResNet-FPN backbone is used, the region proposal network (RPN) produces an RoI using a feature pyramid generated through the pre-processed input image. Faster R-CNN is a model for object detection, and the RoIPool of the RPN has a significant adverse effect on the pixels of the prediction mask due to the misalignment between the RoI and feature when quantizing the RoI into a feature map. To segment the pixel units, the RPN replaces the RoIPool of the existing Fast R-CNN RPN with RoIAlign, which contains location information using bilinear interpolation. Feature maps extracted through RoIAlign are delivered to the classification and bbox regression branches through the fully connected layer to obtain the final class scores and bbox, and to minimize the loss of spatial information in the mask branch, a small fully convolutional network [24] is used instead of an fc-layer to improve the mask performance. **SOTR**[3] has been proposed as a segmentation model combining an FPN backbone and a transformer network. SORR consists of a process that predicts the category of each instance through a transformer and a process that generates a dynamic segmentation mask by computing the dynamic convolution kernel produced by the kernel head and features generated by the multi-level upsampling module (Figure 4). Many instance segmentation methods based on CNNs have adopted a top-down approach that first detects objects and then predicts segmentation masks. The authors who presented SORR [3] noted that this approach has advantages in terms of translation equivariance and location information,  
Figure 3: Mask R-CNN architecture. The mask result was graphically modified, not the actual inference result (COCO-val2017-000000032901 [10]). but for large objects, the receptive field of a CNN may lead to poor results when connecting high-level features to instances. They also stated that the performance of the object detector determines the inference speed and mask quality of the image, which can lead to poor performance when processing complex images. To overcome these issues, SOTR adopts a bottom-up approach and dynamically creates masks while learning position-sensitive features regardless of the location and scale of the bbox and grouping post-processing. In the transformer network, to replace the original transformer layer using multi-head self-attention [20] (Figure 5a), researchers proposed the pure twin transformer layer and hybrid twin transformer layer, which are based on twin attention and shown in Figure 5b,c. Transformers receive some feature maps generated by the FPN backbone. After the transformer, there is a class head for predicting the instance class and a kernel head for creating the convolution kernel dynamically. The multi-level upsampling module receives some feature maps from the FPN and the P5 features reconstructed by the transformer to create a final mask through dynamic convolution operations. **Dual-Swin-L**[2] is a multi-backbone network that combines two same identical large Swin transformer networks [25] (Figure 6). It is designed to perform object detection efficiently by combining high- and low-level features (Figure 7). CBNetV2 [2] is a dense higher-level composition that efficiently combines high- and low-level features and an auxiliary supervision to optimize the training process of the model. Therefore, Dual-Swin-L  
Figure 4: SOTR architecture. “\(*\)” indicates the dynamic convolution operation. The mask result was graphically modified, not the actual inference result (COCO-val2017-000000032861 [10]). Figure 5: Three transformer layer designs.