# Boosted Transformer for Image Captioning
## 3 Methodology
### Boosted Transformer  
In this section, we introduce a boosted transformer to solve the problem of learning to generate a natural description given an image. We regard the combination of CNN, concept-guided attention, and the feed-forward network as the BT encoder to obtain the boosted visual features. As for the BT decoder, it is similar to transformer with the exception of adding a vision-guided attention module. The overall architecture of BT is shown in Figure 2.  
Image encoder: Traditional image captioning uses a CNN model pre-trained on the image classification task (e.g., Resnet101 [18]) as the feature extractor, but recent research showed that an object-based feature extractor can obtain more accurate information about the specific image regions. In this paper, we acquire the visual features of the image through the latter approach as the way adopted by faster RCNN [19]. For simplicity, given an image I, we can get image features V and semantic concepts Y, where V = \(\{v_{1},v_{2},...,v_{n}\}\in\mathbb{R}^{n\times 2048}\), Y = \(\{y_{1},y_{2},...,y_{n}\}\in\mathbb{R}^{n}\) indicate the n regional features and the corresponding semantic concepts extracted by the detection model. Especially, we adopt the linear-projected visual features as the visual features for the coherence with the model dimension in the following sections, where \(V\in\mathbb{R}^{n\times d_{model}}\). After that, CGA is used to contextualize the visual features and the semantic concepts for the boosted visual features as elaborated in Section 3.3.  
Caption decoder: Given a caption, word embedding [20] is first used to project each token into the word vector, forming a sequence matrix \(E^{cap}(Y)=\{w_{1},w_{2},...w_{L}\}\in\mathbb{R}^{L\times d_{model}}\), where L is the length of the caption and word embedding \(E^{cap}\in\mathbb{R}^{vocab\times d_{model}}\) is a trainable matrix. Then, we append the position information to sequence matrix \(E^{cap}(Y)\) by the positional encoding as Equations (1) and (2),  
Figure 2: An illustration of boosted transformer for image captioning. The overall architecture of the model is a transformer-based encoder-decoder framework. Faster R-CNN is first leveraged to detect a set of salient image regions and then to obtain the aligned visual features and semantic concepts. Next, a concept-guided attention module composed of two self-attention mechanisms and an integration module is used to get the boosted visual features. After that, The boosted visual features and the sequence representations are exploited by the caption decoder, which contains a sequence self-attention, vision-guided attention, sequence-to-image attention, and a feed-forward network to generate a caption. seq, sequence.  
obtaining the final sequence representations \(S\in\mathbb{R}^{L\times d_{model}}\). The BT decoder contains a feed-forward network and three kinds of attention modules, i.e., (1) sequence self-attention, (2) VGA, and (3) sequence-to-image attention. Sequence self-attention has three identical inputs, in which queries, keys, and values are all sequence representations. This attention mechanism not only allows each positional vector to reflect itself, but also attends to all the other positional vectors in the sequence, promoting the contextual information only with the sequence representations. Sequence-to-image attention imitates the distinctive attention mechanism [21] in which the queries come from the sequence information of the decoder, while the keys and values are from visual features. This allows every positional word representation to attend to all the regional features, so that different regional features can be treated discriminately for the final prediction. As for the VGA module, we elaborate it in Section 3.4.