# A Review of Transformer-Based Approaches for Image Captioning
## 2 Datasets and Evaluation
### Evaluation Metrics  
In this subsection we take a look at the main evaluation metrics that are used to gauge and compare models. The metrics are summarized in Table 2. The evaluation metrics provide an avenue for different researchers to compare their works. Since captioning has the natural language component as a major output, the evaluation metrics are largely taken from the language domain, where they are for instance applicable in other tasks such as machine translation. The BiLingual Evaluation Understudy (BLEU) metric [70] was originally designed for sentences resulting from machine translations. It is precision-based and it analyzes the co-occurrence of n-grams between the generated sentence and the reference sentence and calculates an error metric. The matches are position-independent so that a test sentence with more matches with the reference sentence is deemed as a better match than one with fewer matches. The BLEU scores are usually reported in terms of the cumulative scores, which are calculated based on the individual n-gram scores at all orders from 1 to n and weighted using the geometric mean. Thus, if n is 4, cumulative scores for BLEU-1, BLEU-2, BLEU-3 and BLEU-4 are given. ROUGE [71] refers to Recall-Oriented Understudy for Gisting Evaluation. It is an evaluation algorithm that was originally designed to evaluate text summarization algorithms. ROUGE is recall-based and it counts the number of overlapping units such as word sequences, n-grams and word pairs between generated captions and ground truth captions. A number of variants or ROUGE exist, namely ROUGE-N, ROUGE-L, ROUGE-W and ROUGE-S. ROUGE-N measures the n-gram recall between a candidate text and a set of reference texts. ROUGE-L measures the longest common subsequence between the candidate and reference texts. ROUGE-W is a version of ROUGE-L that incorporates weighting. ROUGE-S measures the skip-bigram co-occurrence statistics. Skip-bigrams are all pairs of ordered words in a piece of text, sampled with an arbitrary distance between them. ROUGE-L is the variant commonly used for image captioning since it has a higher correlation with human judgements [71]. METEOR [72] stands for Metric for Evaluation of Translation with Explicit ORdering and is an automatic metric originally designed for machine translation. It was designed to overcome a number of weaknesses observed in the BLEU metric that preceded it such as the lack of recall and the lack of explicit word matching [72]. METEOR matches different aspects of n-grams from different sentences to find alignments and then evaluates a score. The matching of the n-grams can be based on aspects such as surface forms, stemmed forms and meanings. METEOR uses a combination of unigram-precision and unigram-recall. METEOR has shown better correlation than BLEU and ROUGE to judgments by human subjects. CIDEr [73] refers to Consensus-based Image Description Evaluation. CIDEr makes an effort to capture human judgment of consensus. CIDEr measures how similar a sentence is to the consensus description of an image, i.e., to how an image is described by most people. It is based on evaluations (average cosine similarity) performed on n-grams taken \begin{table}
\begin{tabular}{l l l l l} \hline
**Metric** & **Basis** & **Correlation** & **Original Domain** & **Operation** \\ \hline BLEU [70] & Precision-based & Low & Summarizing and translation & Co-occurrence of n-grams \\ ROUGE [71] & Recall-based & Moderate & Summarizing and translation & Longest common sequence (ROUGE-L) \\ METEOR [72] & Precision and recall & Moderate & Summarizing and translation & Matching and comparison of n-grams \\ CIDEr [73] & Precision and recall & High & Image captioning & Cosine similarity of n-grams \\ SPICE [74] & Precision and recall & High & Image captioning & Sentence comparison using scene graphs \\ \hline \end{tabular}
\end{table} 
Table 2: The main evaluation metrics used in image captioning. The ‘Correlation’ column refers to human-judgment correlation. from the generated captions and the ground truth captions.