# Panoptic Segmentation-Based Attention for Image Captioning
## 6 Experimental Results
### Evaluation  
In this subsection, we first compare our results with state-of-the-art models on MSCOCO dataset. To demonstrate the effect of the segmentation-region features, we conducted qualitative and quantitative analyses of the difference between InstanceSegAtt and DetectionAtt. Moreover, to demonstrate the effect of the features of stuff regions, we also conducted qualitative and quantitative analyses of the difference between PanopticSegAtt and InstanceSegAtt. We report results using the COCO captioning evaluation tool [11], which reports the following metrics: BLEU [45], METEOR [46], ROUGE-L [47], and CIDEr [40]. Table 1 shows the overall results on the MSCOCO dataset. Compared with the method of [PERSON] et al. [26] which is most similar to our method, our PanopticSegAtt model surpasses their method in all metrics by a large margin. We consider that it is because of the more fine-grained attention regions and the combination of panoptic segmentation in our method. We also compared our PanopticSegAtt with PanopticSegAtt (w/o Dual-Attend). The full model improved the CIDEr from 118.2 to 119.4, which shows that with the dual-attention module, our model can generate more accurate captions. We then compared our method with the typical attention methods [13; 14; 2; 4; 15; 2; 13]. For example, SCA-CNN [14] uses spatial and channel-wise attention in the CNN. [PERSON] et al. [15] adaptively attends to the image and caption during decoding. Our PanopticSegAtt model significantly outperforms these methods in all metrics, which demonstrates the power of our panoptic segmentation-based attention mechanism. We also compared our method with state-of-the-art methods [48; 49; 50; 51; 52]. Our PanopticSegAtt model outperforms these methods in most of the metrics, especially on the CIDEr metric, which is considered to be the metric most aligned with human judgments. Note that Stack-Cap [53] has higher scores, as the model of this method has three LSTMs to perform coarse-to-fine decoding, which is more complex than our method. Since [PERSON] [6] used the extra Visual Genome [24] dataset to train the object detector, their attention regions are much richer than ours. Thus, we did not compare with these two methods directly. Figure 5 shows the statistical results of the CIDEr scores of the captions in the Karpathy test split for DetectionAtt, InstanceSegAtt, and PanopticSegAtt. Captions within the interval with a score from 0 to 1 are not accurate enough to describe the images. Among them, the number of DetectionAtt is the highest, InstanceSegAtt is second, and PanopticSegAtt is the smallest. In the interval with a score from 1 to 3, captions can accurately describe the images. Among them, the number of PanopticSegAtt is the highest, InstanceSegAtt is second, and DetectionAtt is the smallest. In the interval where the score is greater than 3, the number of captions of the three methods is almost the same. This is because the images in this interval are simple so that all the models work well for them. The above results indicate the advantages of using the segmentation features and the stuff regions in our method. We also evaluated our model on the online COCO test server in Table 2. Our PanopticSegAtt model achieves comparable scores compared to the state-of-the-art models.