# Panoptic Segmentation-Based Attention for Image Captioning
## 6 Experimental Results
### Evaluation
#### 6.3.1 Segmentation-Region Features versus Detection-Region Features  
We used Mask R-CNN [7] to perform instance segmentation and object detection, which resulted in equal numbers of attention regions in both tasks. Thus, the difference between segmentation regions and detection regions is that the segmentation regions are more fine-grained and better-matching to the shape of the instances. We compared InstanceSegAtt with DetectionAtt to demonstrate the impact of segmentation-region features. As shown in Table 1, comparing against DetectionAtt verifies the effectiveness of using segmentation-region features. Our InstanceSegAtt model improves the CIDEr score from 112.3 to 117.3 compared with the DetectionAtt model. The performance gap between InstanceSegAtt and DetectionAtt demonstrates that using features from more fine-grained regions is beneficial. The training curves in terms of CIDEr metric are shown in Figure 6. During 50 epochs' training, the InstanceSegAtt model consistently surpasses the DetectionAtt model. This result indicates that, since segmentation regions do not include irrelevant regions that have negative impact on captioning models during training, using segmentation-region features leads to better convergence and performance. **Dense Annotation Split. ** In order to better evaluate the effect of using features of fine-grained segmentation regions, we present a new split from the COCO dataset called dense annotation split. This is based on the intuition that the segmentation-based attention model ought to distinguish instances even if they are overlapped, while the detection-based attention model may be confused by the features of the rectangular regions which include irrelevant regions. The dense annotation split consists of images in which some of the instances are highly overlapped. We generated this split by selecting images for which the IoU (intersection-over-union) between any of their two instances was over 0.5, resulting in 12,967, 570, and 608 images in the training, validation, and testing sets of [PERSON] split [42], respectively. We then evaluated the performance of InstanceSegAtt and DetectionAtt on this split. Figure 6: Comparison of CIDEr scores on the validation set for InstanceSegAtt and DetectionAtt during 50 epochs’ training. As shown in Figure 7, compared with the DetectionAtt model that uses detection-region features, InstanceSegAtt is better at handling images with overlapped objects, as the segmentation-region features do not overlap with each other. For example, in the first column of Figure 7, the detection region of the person and snowboard are highly overlapped. InstanceSegAtt correctly generates the word \"snowboard\", while DetectionAtt cannot. Similarly, in the second column of Figure 7, the dense detection regions make DetectionAtt hard to generate \"a woman\" like InstanceSegAtt. In the third column of Figure 7, the detection regions of the two giraffes contain each other, which could confuse the DetectionAtt model. In the last column of Figure 7, the rectangle detected region of the boy in the second row contains the region of the man; therefore, DetectionAtt cannot correctly recognize their relationship. Table 3 shows the evaluation results on the dense annotation test split. The performance gap between InstanceSegAtt and DetectionAtt is larger than their gap in Table 1, which demonstrates that the segmentation-based attention model has the advantage in handling densely annotated images. We consider that the features of overlapped regions make it hard for DetectionAtt to distinguish the individual instances in overlapped regions. Thus, DetectionAtt has lower scores compared with InstanceSegAtt. It is also observable that the performance of PanoSegAtt is better than that of InstanceSegAtt. This suggests that the contextual information from stuff region features is of benefit for the model to recognize the partially occluded objects. Such a conclusion is widely acknowledged in object detection [56; 57; 58].