# Panoptic Segmentation-Based Attention for Image Captioning
[PERSON]  
2 School of Computer Science and Informatics, Cardiff University, Cardiff CF10 3 AT, UK; [PERSON]L_ADDRESS] (X.S. ); [PERSON]ADDRESS] (P.L.R.) [PERSON]  
2 School of Computer Science and Informatics, Cardiff University, Cardiff CF10 3 AT, UK; [PERSON]L_ADDRESS] (X.S. ); [PERSON]ADDRESS] (P.L.R.) [PERSON]  
2 School of Computer Science and Informatics, Cardiff University, Cardiff CF10 3 AT, UK; [PERSON]L_ADDRESS] (X.S. ); [PERSON]ADDRESS] (P.  
the convolutional layer of the CNN, allowing the model to preserve the spatial information and dynamically attend to different regions when generating words. However, in the top-down attention mechanism, the attention regions correspond to a uniform grid of receptive fields. Since the sizes and shapes of these receptive fields are equal, they are independent of the content of the image. With the advances in object detection, detection-based attention mechanisms [3; 4; 5] were proposed to enable the model to attend to the detected salient image regions. Compared with the top-down attention mechanism, the detection-based attention mechanism has a better performance as it can generate variable numbers and sizes of rectangular attention regions. However, it is still not fine-grained enough because there may be other objects or background in the rectangular attention regions. In this paper, we introduce a novel attention mechanism called panoptic segmentation-based attention, as illustrated in Figure 1. This mechanism comes from the goal of finding more fine-grained attention regions. Naturally, image segmentation, a more fine-grained form of detection, was taken into consideration. We considered it first. Instance segmentation [7; 8] is a more challenging task than object detection, as it requires not only detecting all objects in an image but also segmenting the instances of each object class. The results from instance segmentation only contain the main part of the object and do not include the background or overlapped regions. Therefore, we can obtain more fine-grained attention regions with the aid of instance segmentation. However, instance segmentation only segments instances of \"things\" classes (countable objects with specific shapes; e.g., cars and persons), neglecting \"stuff\" classes [9] (amorphous background regions; e.g., sky and grass). Losing information of the stuff regions may weaken the model's ability to understand the scene. Recently, [PERSON] et al. [10] proposed panoptic segmentation that performs segmentation for all classes (things and stuff) in the image. Inspired by this work, we built our attention mechanism upon this idea. Based on the segmentation regions generated from panoptic segmentation, our method extracts image features based on the shape of the segmentation regions and generates captions based on the attention-weighted features. As shown in Figure 2, compared with the detection-based attention mechanisms that contain irrelevant regions around the objects in their attention regions, the attention regions of our approach contain one instance in each region with irrelevant regions masked out, which is more fine-grained and can avoid the negative impact of the background or overlapped regions. Moreover, while the detection-based attention mechanisms only detect things classes and extract scene information in a top-down way [3; 6], our approach processes things and stuff classes independently via a dual-attention module. Incorporating the features of stuff regions can provide richer context information and yield better performance. The main contributions of this paper are:  
Figure 1: An overview of our proposed method. Given an image, we first generate segmentation regions of the image and use a convolutional neural network to extract the segmentation-region features. For readability, we have applied a color map to each segment. The segmentation-region features are then fed to the LSTM to generate the captions.