# Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
## 6 Multimodal Benchmark
### Image Captioning  
The early image captioning tasks mainly used the Flickr30K [188] and Flickr8K datasets, and the images of these datasets came from the Flickr website. The most commonly used dataset is COCO Captions [189], which contains images of complex scenes between people, animals, and common daily objects. The annotations for image descriptions in the COCO Captions dataset are based on the entire image. Flickr30K Entities mark the nouns mentioned in the caption in Flickr30K, and mark the corresponding bbox. Localized Narratives [190] provides each word with a specific region in the image represented by its tracking segment, including nouns, verbs, adjectives, and prepositions. TextCaps [191] requires models to read and reason about text in images to generate descriptions about them. The images in VizWiz [192] are taken by visually impaired people using mobile phones, are of low quality and involve a wide variety of everyday activities, most of which require some text reading. The dataset aims to make more people aware of the needs of the blind, and to develop assistive technologies to solve the visual challenges in their daily life and to solve the vision problems of the blind. Nocaps [193] aims to evaluate whether a model can accurately describe objects of emerging classes in test images without corresponding training data. As for the evaluation metrics of image captioning, BLEU [194], METEOR [195], CIDEr [196], and SPICE [197] are some of the most used. **BLEU** is Bilingual Evaluation Understudy, which is calculated by modified n-gram precision. It is a precision-based metric but not recall, while short sentences always obtain a higher score. BLEU is calculated by the formula:  
\[BLEU=BP\cdot\exp\sum_{n=1}^{N}w_{n}\log p_{n} \tag{1}\]  
where \(n\) denotes n-gram, \(w_{n}\) denotes the weight and is usually set to \(\frac{1}{n}\), \(BP\) denotes the brevity penalty, and \(p_{n}\) denotes the n-gram-level precision. **METHOR** is Metric for Evaluation of Translation with Explicit ORdering. It claims to have a better correlation with human judgment when considering the order and matching of words. METHOD is a recall-based metric calculated by the formula:  
\[METHOR=(1-pen)\times F_{means} \tag{2}\]where \(F_{means}\) denotes the weighted F-score, \(pen\) denotes the penalty factor to penalize the word order in the candidate translation that differs from the word order in the reference translation. **CIDEr** is Consensus-based Image Description Evaluation. It calculates the cosine angle of its TF-IDF vector (note that each dimension of the vector represents an n-gram and not necessarily a word) to obtain the similarity between the candidate sentence and the reference sentence by the formula:  
\[CIDEr_{n}(c,S)=\frac{1}{M}\sum_{i=1}^{M}\frac{g^{n}(c)\cdot g^{n}(S_{i})}{||g^ {n}(c)||\times||g^{n}(S_{i})||} \tag{3}\]  
where \(c\) denotes the candidate caption, \(S\) denotes the set of reference captions, \(n\) denotes the n-gram, \(M\) denotes the number of reference captions, and \(g^{n}(\cdot)\) denotes the TF-IDF vector based on the n-gram. **SPICE** is Semantic Propositional Image Caption Evaluation. It uses a graph-based semantic representation to encode objects, attributes, and relationships in a caption. It first parses the caption to be evaluated and the reference captions into syntactic dependency trees using the Probabilistic Context-Free Grammar (PCFG) dependency parser and then maps the dependency trees into scene graphs using a rule-based approach. Finally, the F-score values of objects, attributes, and relationships in the caption to be evaluated are calculated. SPICE is calculated by a series of formulas:  
\[\left\{ \begin{aligned} SPICE(c,S)&=\frac{2\cdot P(c,S )\cdot R(c,S)}{P(c,S)+R(c,S)}\\ P(c,S)&=\frac{|T(G(c))\bigotimes T(G(S))|}{|T(G(c)) |}\\ R(c,S)&=\frac{|T(G(c))\bigotimes T(G(S))|}{|T(G(S))|} \end{aligned} \right.