SlowGuess/ACL_2020_OCR huggingface dataset:
Yanzhi Yi and Hangyu Deng and Jinglu Hu
Graduate School of Information, Production and Systems, Waseda University,
2-7 Hibikino, Wakamatsu, Kitakyushu-shi, Fukuoka, Japan, 808-0135
yiyanzhi@akane.waseda.jp, deng.hangyu@fuji.waseda.jp, jinglu@waseda.jp
Evaluating image captions is very challenging partially due to the fact that there are multiple correct captions for every single image. Most of the existing one-to-one metrics operate by penalizing mismatches between reference and generative caption without considering the intrinsic variance between ground truth captions. It usually leads to over-penalization and thus a bad correlation to human judgment. Recently, the latest one-to-one metric BERTScore can achieve high human correlation in system-level tasks while some issues can be fixed for better performance. In this paper, we propose a novel metric based on BERTScore that could handle such a challenge and extend BERTScore with a few new features appropriately for image captioning evaluation. The experimental results show that our metric achieves state-of-the-art human judgment correlation.
Image captioning is one of the key visual-linguistic tasks that asks for generated captions with specific images. Researchers look forward to inexpensive evaluation metrics that closely resemble human judgment, which remains a challenging task since most of the metrics can hardly get close to human judgment.
Image captioning is a one-to-many task since each image can correspond to many possible captions. Different captions may focus on different parts of the image; this not only creates a challenge for generating the captions (Dai et al., 2017; Venugopalan et al., 2017), but also for evaluating them. Most of the existing one-to-one evaluation metrics, however, overlook such a challenge. These one-to-one metrics (Lin, 2004; Vedantam et al., 2015; Zhang et al., 2019) ignore other reference captions since the score is computed by comparing the candidate capture with one single reference caption.
Figure 1: Intrinsic variance exists in a set of ground truth captions for an image. Differences between two references are commonly caused by two reasons: different concerns or different descriptions. Different concerns mean different expressions between references are caused by different regions of interest in an image, while different descriptions mean references focus on the same part but use different ways to explain it. One-to-one metrics can hardly deal with the cases caused by different concerns. For example, they may regard Cand as a good caption compared with Ref1; while regard Cand as a bad caption compared with Ref2 or Ref3.
When there are multiple reference captions, prior works compute individual scores for each reference caption and pool these scores together afterward. Intrinsic variance exists in a set of ground truth captions for an image, since different captions may have different concerns or descriptions. It's challenging to find a remedy for such over-penalization if the metric looks at only one single reference caption.
BERTScore (Zhang et al., 2019) is the latest one-to-one metric that computes token-level cosine
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 985-994 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
similarity between two sentences by contextual embeddings of pre-trained models, and greedily picks and adds up cosine values as a score. It reaches high performance in machine translation tasks and a system-level image captioning evaluation task.
In one-to-one evaluation, although it is hard to consider all references directly, it is possible to combine references into a single one using contextual embedding from the pre-trained language model. In this work, we propose a metric where all of the references are combined as a new comprehensive embedding by detecting the mismatches between two contextual embeddings. To achieve this goal, we add the concept of mismatch into cosine similarity by a threshold for mismatch detection and proper penalization. Also, our metric considers the importance of different words, and our research shows that adding a stop word list is an efficient way.
Using various image captioning evaluation datasets with human annotations like Microsoft COCO (Lin et al., 2014), Flickr8k (Hodosh et al., 2013), COMPOSITE (Aditya et al., 2015) and PASCAL-50S (Vedantam et al., 2015), the experimental results show that our metric achieves state-of-the-art correlation in several tasks, especially in caption-level tasks. Our main contribution is a novel metric that can detect mismatches among captions, build a combined caption with multi-references, and achieve high human correlation in image captioning evaluation tasks. The code for our metric is released at here<sup>1</sup>.
2.1 Automated caption evaluation
For captions evaluation, a traditional method is scoring by human experts, which is a precise but expensive way. Current image captioning models are evaluated by automatic metrics, which compute the similarity between generated captions and ground truth captions.
Currently, most widely used caption metrics are n-gram matching metrics such as BLEU, METEOR, ROUGE, CIDEr. BLEU (Papineni et al., 2002) is a precision-based n-gram overlap matching metric that counts the number of overlap n-grams among all of references and the candidate. Several modifications can be applied to improve BELU, such as different n-gram (e.g.  $n = 1,2,3,4$ ),
brevity penalty for a short candidate, and geometrical average. BLEU is a fast, low-cost metric but has a low correlation with human judgment. METEOR (Denkowski and Lavie, 2014) computes both precision and recall in unigram, and consider more factors such as word stems, synonyms, and paraphrases. ROUGE (Lin, 2004) is a package of measures for automatic text summaries evaluation: ROUGE-N uses n-gram co-occurrence statistics; ROUGE-L uses the longest common subsequence; ROUGE-W uses weighted longest common sub-sequence; ROUGE-S uses skip-bigram co-occurrence statistics. CIDEr (Vedantam et al., 2015) represents a sentence as an n-grams vector with tfidf (term frequency-inverse document frequency), and compute the cosine similarity between reference and candidate.
LEIC (Cui et al., 2018) uses a trained neural model to predict whether a caption is generated by humans. LEIC is trained with COCO images data and uses data augmentation, which helps to achieve a high human correlation. However, LEIC suffers from high computational cost to train in the COCO data. SPICE (Anderson et al., 2016) computes F1 score according to the scene graph created by captions. SPICE reaches a high correlation with human judgment while suffers from long repetitive sentence evaluation (Liu et al., 2017).
2.2 Pre-trained language models and BERTScore
Thanks to the development of a pre-trained language model, better sentence representation can be used in diverse kinds of NLP tasks. Previous works mainly focus on linguistic representation such as word embedding (Mikolov et al., 2013; Pennington et al., 2014; Goldberg and Levy, 2014), which are only word-level embedding without positional information. After the success of Transformer (Vaswani et al., 2017), a series of language model approaches are proposed such as GPT (Radford et al., 2018), BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2019), XLNET (Yang et al., 2019), XLM (Lample and Conneau, 2019), RoBERTa (Liu et al., 2019). These approaches learn from a huge number of unlabeled text data as a pre-trained process and can fine-tune in downstream tasks with a few epochs.
BERTScore is the latest one-to-one matching metric for text similarity. Benefiting from the contextual embedding of the pretrained language
$^{1}$ https://github.com/ck0123/improved-bertscore-for-image-captioning-evaluation
Figure 2: This figure explains the differences between an error and different concerns when mismatches occur. We show an image as nine parts grid chart  $(3 \times 3)$ . In error case, caption<sub>1</sub> focuses on three parts while  $b_{3}$  cannot represent the part that matches  $a_{1}$ . In different-concerns cases, two captions can represent well in different parts of the same image. However, a mismatch occurs since  $a_{3}$  and  $b_{3}$  attend to a different part of the image.
models, BERTScore measures two texts similarity by token-level cosine similarity computing and greedily pick strategy: (1) feed reference text and candidate text into pre-trained model, and extract two contextual embeddings  $r = [r_1,..,r_n]$ ,  $c = [c_1,..,c_m]$ ; (2) compute the cosine similarity matrix between  $r$  and  $c$  by  $\frac{\mathbf{r} \times \mathbf{c}}{\|\mathbf{r}\|_{\|\mathbf{c}\|}}$ ; (3) greedily pick the maximum value from cosine similarity matrix for each reference token as a matching value; (4) collect all the matching values with optional inverse document frequency weights.
Inverse document frequency (idf) computes a score for word frequency in the whole corpus. Given  $N$  documents  $[s_1,.., s_N]$  and each word  $w$ ,idf score is:
$$
\operatorname {i d f} (w) = - l g \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {I} [ w \in s ]
$$
where  $\mathbb{I}[\cdot ]$  is an indicator function and  $lg$  is the base 10 logarithm.
The recall of BERTScore (BS for short) is :
$$
\mathrm {B S} = \frac {\sum_ {r _ {i} \in r} \operatorname {i d f} (r _ {i}) \max _ {c _ {j} \in c} \mathbf {r} _ {i} ^ {\top} \mathbf {c} _ {j}}{\sum_ {r _ {i} \in r} \operatorname {i d f} (r _ {i})}
$$
BERTScore can adequately deal with different descriptions by knowledge from a pre-trained model and achieves high performance in both machine translation, image captioning evaluation tasks. However, as a one-to-one metric approach, it still suffers from different-concerns problems. Another pitfall in BERTScore comes from the strategy "greedy pick": when no candidate word attends to a specific reference word, this reference word
still gets value by picking a maximum cosine value greedily, which causes under-penalization.
Inspired by BERTScore, our metric treats the mismatches between captions carefully, and try to give a proper score for the similarity between captions.
Proper scoring for generated captions should consider the information about multi-references and avoid the wrong penalization. In this section, we provide the idea about references combination and fix some under or over penalization issues for cosine similarity-based metrics.
3.1 Preliminary concept of references combination
Token-level mismatches lead to two kinds of problems: different descriptions and different concerns. We introduce these two concepts in Figure 2. Some methods are available for description problems like thesaurus or similarity with contextual embedding, while few of methods handle the different-concerns problem in multi-references cases.
The common ways for one-to-one text metrics to deal with multi-references cases are pooling the results by some strategies like average or maximum. Maximum picks the maximum of results, which can get a higher score than average meanwhile ignores other references directly. Average merges all the results with each reference, which can consider all references. Although average slightly reduce the impact of different concerns, both of the two overpenalize the generated caption since they already
Figure 3: Combination of references comes from a phenomenon that: mismatches between two ground truth captions can't be errors but different concerns. After the greedy matching process, we collect all the mismatch tokens and create a combined contextual embedding as a combined caption. For example, the threshold value  $\beta$  is 0.4, and all the tokens in  $Ref_{2}$  can't match  $a_{3}$  with a value bigger than 0.4. After the combination of all references, our metric provides a better recall score between the combined caption and the candidate caption withidf weighted.
regard those mismatches from different concerns as errors during the one-to-one text evaluation process.
Different from average and maximum strategies, the strategy of our metric is to combine reference captions. The combination works based on a fact that: all of the reference captions are ground truth captions so that the mismatches between references should not be errors, but considering different concerns (cosine similarity with contextual embedding also ensures that mismatches are not from errors).
Once we choose a base reference caption and pick up all the mismatches among base and others, the combination among the base and mismatches contains all the information in references without duplicate.
After that, the evaluation between the candidate caption and the combined caption does not suffer from the problems from inter references variance any more.
3.2 Mismatch detection with overlap and cosine similarity
It is hard to define the "differences" between captions clearly. To simplify the problem, we regard mismatches in token-level between two embeddings as differences between two captions.
Mismatch is a concept from n-gram overlap matching metrics like BLEU. We find a mismatch when a word from one sentence cannot be found in
the other sentence. Although mismatch is a clear concept to word-level comparison, overlap-based mismatch results in some problems like synonyms. Meanwhile, cosine similarity-based metrics like BERTScore can address this problem quite well. BERTScore uses a pre-trained language model's contextual embedding and regard cosine similarity between two tokens as their similarity. Therefore, the match values change from overlap's discrete value (0 or 1) to cosine's continuous value (0 to 1) with semantic and positional information, which make similarity values more precise.
However, a weakness of cosine similarity is that we cannot distinguish match and mismatch directly since the concept of mismatch does not exist in cosine similarity. To achieve references combination, we simply set a threshold function  $\varphi$  for distinguish the mismatch: when the cosine value is bigger than the threshold, we keep it; otherwise, we set it to 0, which is shown as follows.
$$
\varphi (x, \beta) = \left\{ \begin{array}{l l} x & x > \beta \\ 0. 0 & x \leq \beta \end{array} \right. \tag {1}
$$
where  $x$  is the cosine value and  $\beta$  is the threshold value.
$\mathcal{S}$  is the improved "greedy pick" function for each  $r_i$  reference token with threshold:
$$
\mathcal {S} (r _ {i}, c, \beta) = \varphi \left(\max  _ {c _ {j} \in c} \mathbf {r} _ {i} ^ {\top} \mathbf {c} _ {j}, \beta\right) \tag {2}
$$
where  $r = [r_1,..,r_n]$  and  $c = [c_1,..,c_m]$  are contextual embedding. We call this process "cut" for
removing low cosine values. The standard "cosine & greedy pick" process is a case when the threshold value  $\beta$  equals to 0. Then we can get the greedy recall similarity with threshold indicator:
$$
R = \frac {\sum_ {r _ {i} \in r} \operatorname {i d f} (r _ {i}) \mathcal {S} (r _ {i} , c , \beta)}{\sum_ {r _ {i} \in r} \operatorname {i d f} (r _ {i}) \operatorname {s g n} (\mathcal {S} (r _ {i} , c , \beta))} \tag {3}
$$
where  $\mathrm{sgn}$  is the sign function.
With a threshold indicator, our metric acquires the ability to detect mismatches. Furthermore, since we cut all the low cosine value, the bad impact of greedy pick (mentioned in Section 2) will be eliminated, which means our metric provides a more reasonable similarity for each token pair.
Empirically for a pre-trained language model, the threshold value in different tasks is similar due to the same architecture and the same widely pretraining process in an ample amount of text data. In this work, we use the threshold value 0.4 for BERT (base) and 0.83 for RoBERTa (large) as the recommended settings.
3.3 The combination of references
Contextual embeddings are extracted from the pretrained model. Since the inputs of the model contain both token embedding and position embedding, contextual embedding for each token also contains its semantic and positional information. Therefore, the change of tokens' position does not change the inner positional information for each token. For example, [embed A, embed B] is the contextual embedding sentence generated from [word A, word B]. Both [embed A, embed B] and [embed B, embed A] (only switch tokens' position) still provide same positional information.
Using this characteristic, we can now easily combine all of the references with the following steps: (1) choose a random reference caption embedding as a base,  $A$ ; (2) compute the similarity between  $A$  and another reference  $B$  with a threshold; (3) collect those tokens from  $B$  that mismatch comparing with  $A$ ,  $B'$ ; (4) concatenate  $A$  and  $B'$  as a new base caption  $A$ ; (5) repeat steps above until used all the references;
$R_{comb}$  computes the recall score for combined reference and candidate. Figure 3 shows references combination  $Comb$  and the computation of  $R_{comb}$ .
$$
R _ {c o m b} = R (C o m b ([ r _ {1},.., r _ {M} ]), c) \tag {4}
$$
where  $M$  is the number of references.
3.4 Importance of different words
For proper scoring, our metric also focuses on a problem that token-level matching sometimes does not mean similarity between captions. A bird standing on the blue handrail and A bird flying on the blue sky are describing different images with only two words different but five words the same. The meaning of a caption is sensitive to the replacement of essential components like subject, predicate, object, while some replacement (like  $\mathbf{a} \rightarrow \mathbf{the}$ ) are not.
The problem is: in matching metric, we only focus on the match and mismatch while ignoring the importance of each word in the sentence. It is hard to provide optimal importance with each word and pick the important ones; in contrast, the removal of unimportant words is more comfortable to achieve.
In this work, our metric removes all the stop words and computes an another greedy cosine score as an additional score withoutidf weight,  $R_{rm}$ :
$$
R _ {r m} = \frac {\sum_ {r _ {i} \in r ^ {\prime}} \mathcal {S} \left(r _ {i} , c ^ {\prime} , \beta\right)}{\left| r ^ {\prime} \right|} \tag {5}
$$
where  $r'$  and  $c'$  are embeddings without stop words and  $|r'|$  means the length of sentence  $r'$ .
Although takingidf weight into consideration is convenient, using the stop word removal additionally is still necessary. The definition ofidf points out thatidf is an indicator of frequency, while frequency does not equate to importance. Take COCO caption corpus as an example: all theidf weights of common subjects are low such as man, dog, girl, etc; while those of playfully, sleepy are high. However, there is no doubt that mismatches occur in these common subjects will change the meaning dramatically.
3.5 Summary and metric formula
In this section, we discussed the mismatches between references, under-penalization of "greedy pick", and the importance of words. Moreover, we showed our idea about captions combination, greedy recall similarity with threshold indicator, and stop word removal. Including all of formulas above, the final expression of our metric is the product of  $R_{comb}$  and  $R_{rm}$ :
$$
\text {S c o r e} = R _ {\text {c o m b}} \times R _ {\text {r m}} \tag {6}
$$
Type	Metric	M1	M2
Task-agnostic	ROUGE-L	0.062 (0.846)	0.215 (0.503)
BLEU-1	0.029 (0.927)	0.165 (0.607)
BLEU-4	0.236 (0.459)	0.380 (0.222)
CIDEr	0.440 (0.151)	0.539 (0.071)
METEOR	0.703 (0.011)	0.701 (0.011)
BS (BERT-base)	0.807 (0.001)	0.735 (0.006)
BS (RoBERTa-large)	0.873 (0.000)	0.841 (0.000)
Ours (BERT)	0.875 (0.000)	0.797 (0.002)
Ours (RoBERTa)	0.932 (0.000)	0.869 (0.000)
Task-specific	SPICE	0.715 (0.009)	0.688 (0.013)
LEIC	0.939* (0.000)	0.949* (0.000)
Table 1: Pearson correlation of system level metrics scores with human judgment in 2015 COCO Captioning Challenge. We use 12 teams results on validation set with "Karpathy split". M1: the percentage of captions that are evaluated as better or equal to human captions; M2: the percentage of captions that are indistinguishable from human caption. BS means BERTScore and score with * are cited from (Cui et al., 2018).
The most convincing way for metric evaluation is the human correlation in caption-level and system-level tasks. In this section, we evaluate our metric in four typical image captioning evaluation datasets with standard metrics. We also consider the impact of each part in our metric by ablation experiment and key part replacements.
Microsoft COCO 2014 COCO dataset contains 123,293 images with 82,783 images in training set, 40,504 images in the validation set and 40,775 images in the test set. Each image has five human-annotated captions as ground truth captions.
In 2015 COCO Captioning Challenge (Chen et al., 2015), submissions of the challenge are evaluated by human judgments with five kinds of metrics: M1, percentage of captions that are evaluated as better or equal to human caption; M2, percentage of captions that pass the Turing Test; M3, average correctness of the captions on a scale 1-5 (incorrect - correct); M4, the average amount of detail of the captions on a scale 1-5 (lack of details - very detailed); M5, percentage of captions that are similar to human description.
Flickr 8K Flickr 8K dataset contains 8,092 images with five human-generated captions for each image. Flickr 8K provides an annotation called Expert Annotation, and each row contains one image, one candidate caption from Flickr 8K dataset (it may matches this image or not), and three expert
scores for the image-caption pair. Scores range from 1: indicating that the caption does not describe the image at all to 4: indicating that the caption describes the image.
COMPOSITE The COMPOSITE dataset contains 11985 human judgments from Flickr 8K, Flickr 30K, and COCO captions re-coined. Candidate captions come from human and two caption models scoring by Amazon Mechanical Turk (AMT) workers. All the captions score a 5-point scale from 1 (The description has no relevance to the image) to 5 (The description relates perfectly to the image).
PASCAL-50S PASCAL-50S dataset has 1000 images from UIUC PASCAL Sentence Dataset, and each image has 50 reference captions annotated by AMT worker. PASCAL-50S includes over 4000 candidate captions pair with human judgments. Different from COCO and Flickr format, PASCAL-50S consists of the triplet:  $\langle A,B,C\rangle$ .  $A$  is the reference sentence from an image, and  $B, C$  are two candidate sentences. AMT workers are asked Which of the two sentences,  $B$  or  $C$ , is more similar to  $A$ ? This kind of question is more accessible for workers to judge than provide correct scores. Candidate sentences come from human-written, or model generated, and four kinds of paired ways: human-correct (HC), human-incorrect (HI), human-model (HM), and model-model (MM).
Metric	M1	M2
BLEU-1	0.307	0.338
ROUGE-L	0.062	0.215
ROUGE-L (cat)	0.096	0.180
BLEU-1 (cat)	0.351	0.175
METEOR (cat)	0.662	0.571
METEOR	0.703	0.701
BS(BERT-base)	0.807	0.735
Ours (Unigram)	0.809	0.714
Ours (BERT+T)	0.822	0.741
Ours (BERT+T+cat+R)	0.857	0.783
Ours (BERT+TB)	0.867	0.755
Ours (BERT+TBR)	0.875	0.797
Table 2: We provide the ablation study and the replacement study in 2015 COCO Captioning dataset. As an additional experiment, we also compare concatenation with average in some standard metrics like: BLEU-1, ROUGE-L and METEOR. Abbreviation: BS means BERTScore, T means cut, B means combine, R means remove, cat means concatenation of references.
4.2 Compared Metrics
For comparison, we use common standard metrics in our scoring tasks, such as BLEU-1, ROUGE-L, METEOR, CIDEr, and SPICE. All these metrics are implemented in MS COCO evaluation tool. We also use the original BERTscore to check the improvement of our metrics. To be more convincing, we compare with the current SOTA training-based approach LEIC in COCO captioning 2015 and Flickr 8K.
Two metrics are implemented as baselines: (1) unigram overlap matching metric and (2) references concatenation metric with BERT. Unigram overlap matching metric is implemented for verifying the importance of contextual embedding from the pretrained language model. References concatenation metric with BERT is implemented for verifying the importance of references combination.