# A Review of Transformer-Based Approaches for Image Captioning
## 6 Discussion and Analysis
### Results and Performance Metrics  
When discussing the results for image captioning, the offline performances based on the Karpathy splits [27] as well as the online performance on the MS COCO evaluation server [68] are often considered. The performance on the Karpathy splits is very useful for purposes of comparison since not all methods will appear on the online server. Table 8 shows the offline performance metrics of the VLP and non-VLP models. The results for Conceptual Captions were only reported for the online evaluation server and have therefore been left out of Table 8. The CIDEr results are all based on cross-entropy optimization with CIDEr optimization (SCST). A gradual improvement of the results is noted when moving towards the more recent models. COS-Net [91] achieves the best performance among the non-VLP models assessed. The performance of COS-Net is remarkable since, being a non-VLP model, it outperforms a number of other models that use VLP. This goes to show the effectiveness of performing grid feature extraction using an encoder based on CLIP as well the approach of comprehending and ordering the rich semantics in images. For the VLP models, the Rouge Metric has been omitted since many of the authors of these VLP models do not report it. The CIDEr performance is indicated in the case of cross-entropy loss and SCST CIDEr optimization. In all instances where there is CIDEr optimization, there is an improvement over plain cross-entropy loss, which shows the effectiveness of CIDEr optimization. The improvements gained from vision-language pre-training are evident when comparing the non-VLP and VLP results in Table 8. The OFA [106] and GIT2 [111] models have the best performance, achieving CIDEr scores of 154.9 and 152.7, respectively. Note that GIT2 is a bigger version of GIT. The Universal Captioner [112] also achieves a high CIDEr score of 150.2, which supports the effectiveness of their approach of taking the natural descriptive style of captions into account to generate more fluent and human-like captions. Table 9 shows the online performance of both the non-VLP and VLP models on the MS COCO evaluation server (for the cases in which those results are available). The \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Model** & **VLP** & **B** & **S** & **M** & **R** & **C** & **C\({}^{\ddagger}\)** \\ \hline Stacked attention [77] & ✗ & 33.3 & - & - & 54.8 & - & 108.1 \\ Object relation [79] & ✗ & 38.6 & 22.6 & 28.7 & 58.4 & - & 128.3 \\ AoA [84] & ✗ & 40.2 & 22.8 & 29.3 & 59.4 & 122.7 & 132.0 \\ Entangled [85] & ✗ & 39.9 & 22.6 & 28.9 & 59.0 & 119.3 & 127.6 \\ Meshed-Memory [86] & ✗ & 40.5 & 23.5 & 29.7 & 59.5 & - & 134.5 \\ X-Linear [87] & ✗ & 40.7 & 23.8 & 29.9 & 59.7 & 122.1 & 135.3 \\ Image Transformer [90] & ✗ & 39.5 & 22.8 & 29.1 & 59.0 & - & 130.8 \\ COS-Net [91] & ✗ & 42.9 & 24.7 & 30.8 & 61.0 & 129.5 & 143.0 \\ OSCAR [38] & ✓ & 41.7 & 24.5 & 30.6 & - & 127.8 & 140.0 \\ Unified VLP [109] & ✓ & 39.5 & 23.2 & 29.3 & - & - & 129.3 \\ VinVL [39] & ✓ & 41.0 & 25.2 & 31.1 & - & 130.8 & 140.9 \\ VIVO [40] & ✓ & 34.9 & 21.7 & 28.4 & - & 119.8 & - \\ LEMON [52] & ✓ & 42.6 & 25.5 & 31.4 & - & 139.1 & 145.5 \\ SimVLM [103] & ✓ & 40.6 & 25.4 & 33.7 & - & 143.3 & - \\ OFA [106] & ✓ & 44.9 & 26.6 & 32.5 & - & 145.3 & 154.9 \\ CoCa [110] & ✓ & 40.9 & 24.7 & 33.9 & - & 143.6 & - \\ GIT [111] & ✓ & 44.1 & 26.3 & 32.2 & - & 144.8 & 151.1 \\ GIT2 [111] & ✓ & 44.1 & 26.4 & 32.2 & - & 145.0 & 152.7 \\ Universal [112] & ✓ & 42.9 & 25.2 & 31.5 & - & - & 150.2 \\ PaLI [55] & ✓ & - & - & - & - & 149.1 & - \\ BLIP-2 [113] & ✓ & 43.7 & - & - & - & 145.8 & - \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Offline performance (Karpathy test split) of the VLP and non-VLP models. B refers to BLEU-4, S to SPICE, M to METEOR, R to ROUGE-L and C to CIDEr. C\({}^{\ddagger}\) gives values for CIDEr optimization. VLP methods continue to yield better performance than the non-VLP methods.