# Boosted Transformer for Image Captioning
## 3 Methodology
### Concept-Guided Attention for the Encoder  
The proposed concept-guided attention is an integrated module containing two kind of inputs as shown in Figure 3a, used to develop the mutual effectiveness between visual features and semantic concepts for image captioning. It is the detail representation of the boosted transformer encoder layer in Figure 2. To obtain a concept representation associated with image content, we concatenate visual features \(V^{i}\) and embedding vectors of semantic concepts \(E^{\textit{se}}(Y)\) in the channel dimension and then project the results to hidden dimension \(d_{model}\) with a nonlinear function \(\phi_{o}\). Residual connection from the visual features to the output of projection helps to keep the visual peculiarity. After that, the concept representations \(C\) and visual features \(V^{i}\) are contextualized by two independent self-attention modules and then integrated into boosted visual features.  
\[C=\phi_{o}([V^{i};E^{\textit{se}}(Y)])+V^{i} \tag{7}\]  
\[V^{i+1}=FFN(CGA(V^{i},C)) \tag{8}\]  
where \(E^{\textit{se}}\in\mathbb{R}^{vocab\times d_{o}}\) is another word embedding for projecting the semantic concepts into the vector, independent of caption word embedding \(E^{\textit{cap}}\). The concept vocabulary is 472 and \(d_{o}=128\) for this smaller vocabulary, compared with the caption vocabulary of 9488 words. \(\phi_{o}\) is a nonlinear projection layer and implemented as the fully-connected network with a ReLU activation function and dropout. \(V^{i}\) is the visual features inputs of the \(i^{\text{th}}\) layer of the image encoder. \(V^{0}\) represents the initial features extracted from the image, and \(V^{i+1}\) is the boosted visual features integrated by \(V^{i}\) and \(C\). The outputs of the encoder layer \(V^{i+1}\) are regarded as the next encoder layer's inputs. FFN is the feed-forward network as shown in Equation (6).  
Figure 3: The overview of the BT encoder. Our proposed image encoder adopts a flexible architecture, which can decide whether to use the concept representations. (**a**) is an encoder layer with the visual features and the concept representations as inputs. (**b**) only uses the visual features as the inputs. The outputs of either (**a**) or (**b**) serve as the next layer’s visual feature inputs.  
Here, we devise two fusion strategies to integrate the different representations, e.g., gated sum and channel connect. For the gated sum strategy, we set a learnable parameter as the weight value, as shown in Equation (9).  
\[CGA(V^{i},C)=\lambda\times MultiHead(V^{i},V^{i},V^{j})+(1-\lambda)\times Multi Head (C,C,C) \tag{9}\]  
For the channel connect strategy, we concatenate the visual features and concept representations in the channel dimension and then reduce it to the model hidden dimension with a linear matrix, as shown in Equation (10).  
\[CGA(V^{i},C)=W\left(\left[MultiHead(V^{i},V^{i},V^{i});MultiHead(C,C,C) \right]\right) \tag{10}\]  
where \(W\in\mathbb{R}^{(d_{model}+d_{o})\times d_{model}}\), and \(\lambda\) is learnable with the training process. Concept representations \(C\) are obtained by Equation (7).