# A Review of Transformer-Based Approaches for Image Captioning
## 6 Discussion and Analysis
### Out-of-Domain Captioning  
A key challenge for captioning systems is that they largely perform well on test sets closely related to the training datasets whereas when gauged against other out-of-domain datasets, their performance is wanting. In trying to meet this challenge, [PERSON] et al. [122] proposed the Deep Compositional Captioner (DCC) to enable generation of captions relating to objects not present in paired image-sentence datasets used during training. Their image encoder is a CNN and the decoder for the language model is an LSTM. They developed their out-of-domain test dataset by holding out a subset of the MS COCO dataset, ensuring that the out-of-domain dataset has objects not seen during training. [PERSON] et al. [123] developed the Novel Object Captioner (NOC), which was an extension of DCC [122]. [PERSON] et al. [124] developed LSTM-C, which is a Long Short-Term Memory with a _Copying_ Mechanism. The copying mechanism gives the model the ability to select novel objects learned from external sources and insert them at the appropriate places in the generated captions. [PERSON] et al. [125] developed a similar model (Long Short-Term Memory with Pointing, LSTM-P), which enabled vocabulary expansion and incorporation of novel object categories via a _pointing_ mechanism. [PERSON] et al. [126] came up with Neural Baby Talk, an attention-based method for captioning that is based on using visual concepts from an image to fill in slots in a sentence template. The visual concepts are derived from object detectors. This allows the use of different object detectors to help fill in the template slots. This lends itself to generating sentences based on out-of-domain images entailing novel objects not seen during training. The Decoupled Novel Object Captioner (DNOC) [127] follows a similar approach of filling in slots, referred to as placeholders. The Switchable Novel Object Captioner [128] was an extension of the DNOC model in which the placeholder during sentence generation was replaced by a proxy visual word, which is meant to enable generation of better sentences by taking advantage of visual similarities between novel objects and seen objects. They used a _switchable_ LSTM which switches between standard LSTM sentence generation and retrieving object classes from a key-value object memory. [PERSON] et al. [129] developed an architecture-agnostic model, which uses a constrained beam search to enforce the inclusion of selected words (novel object classes) in the generated captions. Many of the works follow the approach of [122] of using a held-out subset of MS COCO for the out-of-domain dataset. The interest in this research area has brought about the creation of the Novel Object Captioning at Scale (Nocaps) dataset [57]. This dataset offers \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **B** & **M** & **R** & **C** \\ \hline Conceptual Captions [59] & - & 0.4 & 0.7 & 1.0 \\ AoA [84] & 71.2 & 38.5 & 74.5 & 129.6 \\ Entangled Transformer [85] & 70.2 & 38.0 & 73.9 & 124.4 \\ Meshed-Memory [86] & 72.8 & 39.0 & 74.8 & 132.1 \\ X-Linear [87] & 72.4 & 39.2 & 75.0 & 133.5 \\ Image Transformer [90] & 71.5 & 38.4 & 74.5 & 129.6 \\ VinVL [39] & 74.9 & 40.8 & 76.8 & 138.7 \\ COS-Net [91] & 74.7 & 40.1 & 76.4 & 138.3 \\ OFA [106] & 78.7 & 42.7 & 79.0 & 149.6 \\ GIT [111] & 78.3 & 42.0 & 78.4 & 148.8 \\ GIT2 [111] & 78.3 & 42.1 & 78.4 & 149.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Online performance based on the evaluation test server. CIDEr values are based on evaluation with 40 reference captions per image. a large-scale benchmark which is geared towards improving out-of-domain captioning. Nocaps draws from MS COCO and the Open Images dataset, which contains many object categories not present in MS COCO. It has been seen that models that perform well on the standard MS COCO do not necessarily perform very well on Nocaps or when exposed to test sets containing object classes not seen during training. Table 10 shows the CIDEr scores of the VLP models on the Nocaps dataset [57].