# Boosted Transformer for Image Captioning
## 4 Experiments
### Qualitative Analysis  
In order to further demonstrate the effectiveness of BT, we show some visualizations of the attention map, comparing the attention map changes with VGA, as shown in Figure 6. The result revealed that VGA can effectively attend to some new focuses digressing from the language prior. For example, in Fig ATT_s2s, the attention map was obtained by sequence self-attention, where the color of the position corresponding to \"standing\" and \"two\" or \"on\" and \"two\" are relatively dark, indicating higher focus. This is because the situation that those words co-appeared in happened frequently in the training set. After the processing of VGA, the attention distribution was changed and became more reasonable, as shown in ATT_VGA, where \"standing\" had a higher focus on \"people\". ATT_s2i is the attention map between caption tokens and the visual areas. In addition, \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**BLEU -1**} & \multicolumn{2}{c}{**BLEU-2**} & \multicolumn{2}{c}{**BLEU-3**} & \multicolumn{2}{c}{**BLEU-4**} & \multicolumn{2}{c}{**METEOR**} & \multicolumn{2}{c}{**ROUGE-L**} & \multicolumn{2}{c}{**CIDEr**} \\ \cline{2-13}  & **c5** & **c40** & **c5** & **c40** & **c5** & **c40** & **c5** & **c40** & **c5** & **c40** & **c5** & **c40** \\ \hline SCST [30] & 78.1 & 93.7 & 61.9 & 86.0 & 47.0 & 75.9 & 35.2 & 64.5 & 27.0 & 35.5 & 56.3 & 10.7 & 114.7 & 116.0 \\ LSTM-A [15] & 78.7 & 93.7 & 62.7 & 86.7 & 47.6 & 76.5 & 35.6 & 65.2 & 27.0 & 35.4 & 56.4 & 70.5 & 116.0 & 118.0 \\ StackCap [33] & 77.8 & 93.2 & 61.6 & 86.1 & 46.8 & 76.0 & 34.9 & 64.6 & 27.0 & 35.6 & 56.2 & 70.6 & 114.8 & 118.3 \\ Up-Down [29] & 80.2 & 95.2 & 64.1 & 88.8 & 49.1 & 79.4 & 36.9 & 68.5 & 27.6 & 36.7 & 57.1 & 72.4 & 17.9 & 120.5 \\ CAVP [35] & 80.1 & 94.9 & 64.7 & 88.8 & 50.0 & 79.7 & 37.9 & 69.0 & 28.1 & 37.0 & 58.2 & 73.1 & 121.6 & 123.8 \\ DA [34] & 79.4 & 94.4 & 63.5 & 88.0 & 48.7 & 78.4 & 36.8 & 67.4 & 28.2 & 37.0 & 57.7 & 72.2 & 120.5 & 122.0 \\ SGAE [7] & 80.6 & 95.0 & 65.0 & 88.9 & 50.1 & 79.6 & 37.8 & 68.7 & 28.1 & 37.0 & 58.2 & 73.1 & 122.7 & 125.5 \\ BT & 80.5 & 94.7 & **65.2** & 88.8 & **50.6** & **80.1** & **38.6** & **69.6** & **28.8** & **37.9** & **58.5** & **73.5** & **125.0** & **126.8** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Leaderboard of the published state-of-the-art image captioning models on the online COCO test server, where c5 and c40 denote using 5 and 40 references for testing, respectively. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **B@1** & **B@4** & **M** & **R** & **C** & **S** \\ \hline Google NICv2 [9] & - & 32.1 & 25.7 & - & 99.8 & - \\ AdaATT [11] & 74.2 & 33.2 & 26.6 & - & 108.5 & - \\ Att2 all [30] & - & 34.2 & 26.7 & 55.7 & 114.0 & - \\ StackCap [33] & 78.6 & 36.1 & 27.4 & 56.9 & 120.4 & 20.9 \\ Up-Down [29] & 79.8 & 36.3 & 27.7 & 56.9 & 120.1 & 21.4 \\ GCN-LSTM [16] & 80.5 & 38.2 & 28.5 & 58.3 & 127.6 & 22.0 \\ DA [34] & 79.9 & 37.5 & 28.5 & 58.2 & 125.6 & 22.3 \\ SGAE [7] & 80.8 & 38.4 & 28.4 & 58.6 & 127.8 & 22.1 \\ \hline CGA\_lst-GS & 81.0 & 39.5 & 29.3 & 58.9 & 130.9 & 23.1 \\ VGA\_4 th & 81.1 & 39.4 & 29.3 & 58.9 & 131.0 & 23.1 \\ BT & **81.2** & **39.7** & **29.4** & **59.1** & **131.5** & **23.2** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The performance comparison with the state-of-the-art methods on the COCO Karpathy test split. we present some examples of images and captions in Figure 7. We can see that the generated descriptions of BT contained more image-associated tokens compared with the baseline. Figure 6: Visualizations of the attention maps, including the seq-to-self attention map, i.e., ATT_s2s, the boosted seq-to-self attention map, i.e., ATT_VGA, and the seq-to-image attention map, i.e., ATT_s2i. For a better visualization effect, we only highlight some more caption-related objects (10-100 objects in total, dependent on the specific image). Figure 7: Examples generated by the BT model on the Microsoft COCOvalidation set.