SlowGuess/NAACL_2022_OCR huggingface dataset:
Jungo Kasai $^{1*}$  Keisuke Sakaguchi $^{1}$  Lavinia Dunagan $^{1}$  Jacob Morrison $^{1*}$  Ronan Le Bras $^{4}$  Yejin Choi $^{1*}$  Noah A. Smith $^{1*}$
$^{\text{©}}$ Paul G. Allen School of Computer Science & Engineering, University of Washington \*Allen Institute for AI \*Department of Linguistics, University of Washington {jkasai, laviniad, jacobm00, yejin, nasmith}@cs.washington.edu {keisukes, ronanlb} @allenai.org
We establish THUMB, a rubric-based human evaluation protocol for image captioning models. Our scoring rubrics and their definitions are carefully developed based on machine- and human-generated captions on the MSCOCO dataset. Each caption is evaluated along two main dimensions in a tradeoff (precision and recall) as well as other aspects that measure the text quality (fluency, conciseness, and inclusive language). Our evaluations demonstrate several critical problems of the current evaluation practice. Human-generated captions show substantially higher quality than machine-generated ones, especially in coverage of salient information (i.e., recall), while most automatic metrics say the opposite. Our rubric-based results reveal that CLIPScore, a recent metric that uses image features, better correlates with human judgments than conventional text-only metrics because it is more sensitive to recall. We hope that this work will promote a more transparent evaluation protocol for image captioning and its automatic metrics.<sup>1</sup>
Recent progress in large-scale training has pushed the state of the art in vision-language tasks (Li et al., 2020; Zhang et al., 2021, inter alia). One of these tasks is image captioning, whose objective is to generate a caption that describes the given image. The performance in image captioning has been primarily measured in automatic metrics (e.g., CIDEr, Vedantam et al., 2015; SPICE, Anderson et al., 2016) on popular benchmarks, such as MSCOCO (Lin et al., 2014) and Flickr8k (Hodosh et al., 2013). Use of these metrics is justified based on their correlation with human judgments collected in previous work (Hodosh et al., 2013; Elliott and Keller, 2014; Kilickaya et al., 2017, inter alia).
Machines P R CIDEr A red fire hydrant spewing water on a street. 5 3 139.2 A red fire hydrant spraying water on a street. 5 3 205.2 Human A busted red fire hydrant spewing water all over a street creating a rainbow. 5 5 120.5
Figure 1: These machine captions are precise (in the scale of 1-5) but lose points in recall (i.e., coverage of salient information); they both ignore the rainbow in the picture. Automatic metrics, such as CIDEr, do not capture this failure.
Continuous use of these previous human judgments, however, raises significant concerns for development of both captioning models and automatic metrics because of their lack of transparency. In previous work, annotators (crowdworkers, typically) rate image captions directly (Hodosh et al., 2013), pairwise (Vedantam et al., 2015), or along multiple dimensions such as thoroughness (Aditya et al., 2015) and truthfulness (Yatskar et al., 2014). These scoring judgments depend highly on individual annotators' discretion and understanding of the annotation scheme (Freitag et al., 2021; Clark et al., 2021), making it difficult to decompose, interpret, and validate annotations. This lack of transparency also makes it difficult to interpret evaluation results for downstream applications where some aspects are particularly important (e.g., accessibility for people with visual impairments; Gleason et al., 2019, 2020). Further, these annotations were done only on relatively old models (e.g., MSCOCO leaderboard submissions in 2015; Anderson et al.,
*Work was done during an internship at AI2.
'All data are available at https://github.com/jungokasai/THumB.
Human Language Technologies, pages 3464 - 3478
2016). Correlations of automatic metrics with human judgments can break down especially when model types change (Callison-Burch et al., 2006), or generation models become increasingly powerful (Ma et al., 2019; Edunov et al., 2020). We thus develop an up-to-date, transparent human evaluation protocol to better understand how current models perform and how automatic metrics are correlated when applied to current models.
At the core of our rubrics are two main scores in a tradeoff: precision and recall (Fig. 1). The former measures accuracy of the information in a caption, and the latter assesses how much of the salient information in the image is covered. We then penalize a caption if we find a problem in fluency, conciseness, or inclusive language. Two or more authors evaluate every instance and collaborate to resolve disagreements, ensuring high quality of the annotations. We assess outputs from four strong models as well as human-generated reference captions from MSCOCO. We call our scores THUMB 1.0 (Transparent Human Benchmark), and release them publicly.
Key Findings We made several key observations from the evaluations.
- Machine-generated captions from recent models have been claimed to achieve superhuman performance using popular automatic metrics (human performance is ranked at the 250th place in the MSCOCO leaderboard), but they still show substantially lower quality than human-generated ones.
- Machines fall short of humans, especially in recall (Fig. 1), but most automatic metrics say the opposite. This finding is consistent with prior work that showed that machines tend to produce less diverse captions than humans (van Miltenburg et al., 2018).
- Human performance is underestimated in the current leaderboard paradigm, and there is still much room for improvement on MSCOCO captioning.
- CLIPScore and RefCLIPScore (Hessel et al., 2021), recently proposed metrics that use image features, improve correlations particularly in recall. While they fail to score human generation much higher than machine one, they capture an
aspect that is less reflected in text-only metrics.
- Currently available strong captioning models generate highly fluent captions. Fluency evaluation is thus no longer crucial in ranking these models.
2 Evaluation Protocol
We establish a transparent evaluation protocol for English image captioning models. Our rubrics and rules are developed through discussions among all annotators (first four authors of this paper) and designed to increase the reliability of evaluation (Jonsson and Svingby, 2007)
2.1 Evaluation Setups and Quality Control
We used images from the test data in the standard Karpathy split (Karpathy and Fei-Fei, 2015) of the MSCOCO dataset (Lin et al., 2014). The dataset consists of  $113\mathrm{K}$ ,  $5\mathrm{K}$ , and  $5\mathrm{K}$  train/dev./test everyday-scene photos sampled from Flickr. We randomly sampled 500 test images and prepared one human- and four machine-generated captions for every image ( $\S 2.3$ ). We first performed developmental evaluations of 250 captions for 50 images and created rubrics. We then proceeded with the rest of the captions. For every image, captions were shuffled, and thus annotators did not know which caption corresponded to which model, thereby avoiding a potential bias from knowledge about the models. We conducted two-stage annotations: the first annotator scores all captions for given images, and the second annotator checks and modifies the scores when necessary. After the developmental phase, the  $\kappa$  coefficient (Cohen, 1960) was 0.86 in precision and 0.82 in recall for the rest of the evaluated captions ( $\S 2.2.1$ ).<sup>3</sup> The first four authors of this paper conducted all evaluations; none of them are color blind or low vision, two are native English speakers, and one is a graduate student in linguistics. We finally ensured that at least one native speaker evaluated the fluency of every caption ( $\S 2.2.2$ ), meaning that if a caption is annotated by the two non-native speakers, one native speaker checks the fluency in an additional round.
$^{2}$ https://competitions.codalab.org/competitions/3221#results.
Furthermore, we found that a third annotator did not change the scores for all 100 captions randomly sampled for meta-evaluations, confirming the sufficiently high quality of our two-stage annotations. Disagreement in ratings can also result from a certain degree of subjectivity (Misra et al., 2016).
2.2 THUMB 1.0
Similar to the framework of the automatic SPICE metric (Anderson et al., 2016), we base our manual evaluations on two main scores: precision and recall. We also consider three types of penalty: fluency, conciseness, and inclusive language. The overall score is computed by averaging precision and recall and deducting penalty points.
2.2.1 Main Scores
The two main scores are assessed in the scale of 1-5. They balance information accuracy and coverage.
Precision Precision (P) measures how precise the caption is given the image. For instance, Caption 1-B in Table 1 is perfectly precise, while 1-A (dog vs. otter, one vs. two frisbees) and 1-C (three vs. two frisbees) are not precise. Precision guards against hallucinations from the language model (table in 2-B) that are known to be common failures of image captioning models (Rohrbach et al., 2018). The score of 4 is reserved for relatively minor issues, such as attributes that are almost correct (e.g., pink vs. red in 4-C, Table 1) or cases where the caption does not contradict the image but is not guaranteed to be true (e.g., it is unclear whether the girl is sitting on a couch in 3-B). In addition to objects themselves, precision deals with information like properties, attributes, occasions, locations, and relations between objects (e.g., in a red suitcase vs. on a red suitcase in 4-A).
Recall Recall (R) measures how much of the salient information (e.g., objects, attributes, and relations) from the image is covered by the caption. This includes color (e.g., color of the frisbees in 1-A, 1-B, and 1-C) and guards against generic, uninformative captions that machines tend to produce (Wang et al., 2020). For instance, an otter is a small animal, and thus small animal is precise (1-C); however, it is much less informative (and less natural; Ordonez et al., 2013) than saying an otter. Similarly, Caption 5-B only says a woman is standing behind a counter at a donut shop, but she is selling donuts, not buying or looking at donuts, which is salient information from the picture. We do not take a point off if missing information is already expected from the caption (e.g., a double-decker bus is typically red). We often find it useful to take a generative approach when evaluating recall: what image does the caption lead us to imagine? When
the caption entails many potential images that substantially diverge from the given image, the recall score should be low.4
2.2.2 Penalties
Fluency Fluency (Flu.) measures the quality of captions as English text regardless of the given image. Initially, we scored fluency in the scale of 1–5, similar to P and R, but we found most captions from modern neural network models were highly fluent. Thus, we instead decided to take points off from the average of P and R if there's a fluency problem to account for minor issues that are much less problematic than losing one P/R point. The four annotators had extensive discussions and developed rubrics for fluency. Similar to recent work on professional evaluations for machine translation (Freitag et al., 2021), we evaluated under the following principle: if a fluency problem is expected to be easily corrected by a text postprocessing algorithm (e.g., grammatical error correction: Yuan and Briscoe, 2016; Sakaguchi et al., 2017), the penalty should be 0.1. This includes obvious misspellings or grammatical errors (e.g., A otter in 1-B) and missing determiners/hyphens (multi colored in 2-C).  $0.5+$  points were subtracted for more severe problems, such as duplication (e.g., A display case of donuts and doughnuts), ambiguity (e.g., A cat is on a table with a cloth on it), and broken sentences (e.g., A large concrete sign small buildings behind it.). See Table 6 in §A.1 for more extensive fluency rubrics. Note that the average fluency penalty was 0.01; this confirms that fluency is no longer crucial in ranking models for MSCOCO captioning and contrasts with human evaluations previously done for older captioning models.
Conciseness The scores so far do not take into account conciseness of captions. Specifically, a model could simply increase all scores by describing every detail in a picture. For instance, the following caption is overly repetitive: a woman lying on her back with knees bent on a beach towel under a multicolored, striped beach umbrella, surrounded by sand, and with clear blue sky above. We subtract 0.5 points for these captions. Note that most machine captions were short, and this penalty was only applied to two human-generated captions. It might become more crucial for future models
Prior work found recall (or specificity) can vary across cultures or languages (van Miltenburg et al., 2017). We focus on the English language in this work.
Image	Caption	P	R	Flu.	Total
1-A: Up-DownA dog playing with a frisbee on the ground.	3	4	0	3.5
1-B: VinVL-baseA otter is laying on the sand next to two frisbees.	5	4	0.1	4.4
1-C: VinVL-largeA small animal laying on a rock with three frisbees.	4	3	0	3.5
2-A: Up-DownA close up of a plate of broccoli.	5	3	0	4
2-B: Unified-VLP, VinVL-base, VinVL-largeA plate of pasta and broccoli on a table.	4	4	0	4
2-C: HumanA multi colored dish with broccoli and white twisted pasta in it.	5	5	0.1	4.9
3-A: Unified-VLPA little girl holding a video game controller.	3	4	0	3.5
3-B: VinVL-largeA little girl is blow drying her hair on a couch.	4	5	0	4.5
3-C: HumanA little girl holding a blow dryer next to her head.	5	5	0	5
4-A: Up-DownA black cat laying in a red suitcase.	3	5	0	4
4-B: Unified-VLP, VinVL-base, VinVL-largeA black cat sitting on top of a red suitcase.	5	5	0	5
4-C: HumanA large black cat laying on top of a pink piece of luggage.	4	5	0	4.5
5-A: Up-Down, Unified-VLPA man standing in front of a display of donuts.	3	2	0	2.5
5-B: VinVL-largeA woman standing behind a counter at a donut shop.	5	3	0	4
5-C: HumanWoman selling doughnuts with doughnut stock in the background.	5	5	0.3	4.7
Table 1: Example evaluations of machine- and human-generated captions. None of these captions get penalties in conciseness and inclusive language. Evaluated captioning models are described in §2.3. All MSCOCO images are provided under a Creative Commons Attribution 4.0 License (Lin et al., 2014).
with a more powerful object detection module that catches many objects in the picture.
Inclusive Language We found that some instances substantially diverge from inclusive language when humans are described (van Miltenburg, 2020), raising a concern for downstream applications. In these cases, we added a penalty: 0.5 points were deducted for a subjective comment about appearance (e.g., very pretty girl), and 2 points for more severe problems (e.g., beautiful breasts).
2.2.3 Rules of THUMB
In our development phase, we established the following additional rules to clarify our annotation scheme.
Avoiding Double Penalties When an error is accounted for in precision, we correct the error be
fore scoring the recall, thereby avoiding penalizing the precision and recall for the same mistake. For example,  $\mathrm{P} = 3$  is given to Caption 1-A in Table 1 because of its wrong detection (dog vs. otter; one vs. two frisbees), but we score the recall assuming that the caption is now an otter playing with two frisbees on the ground. This ensures that a generic, useless caption, such as there is something on something  $(\mathrm{P} = 5, \mathrm{R} = 1)$ , would be ranked considerably lower than a dog on the beach with two pink and yellow frisbees  $(\mathrm{P} = 3, \mathrm{R} = 5)$ . Similarly, the wrong detection in 5-A (man vs. woman) is handled only in precision. Note that such error correction is not applicable to hallucinations because there is no alignment between a part of the image and a hallucinated object (e.g., table in 2-B). This rule departs from the definition of recall in SPICE (Anderson et al., 2016), an automatic metric that measures the
$F_{1}$  score in scene graphs predicted from reference and generated captions; their alignment is limited to WordNet synonyms (Miller, 1995). This means that classifying an otter as a dog or even a small animal would result in cascading errors both in precision and recall, overrating captions that completely overlook the otter or ones that make a more severe classification error (e.g., miscategorize the otter as a car, compared to a dog).
Object Counts as Attributes All counts are considered as object attributes, and wrong counts are handled in precision. This simplifies the distinction between precision and recall. For instance, both a frisbee (1-A) and three frisbees (1-C) are precision problems, while saying some frisbees would be a recall problem when it is clear that there are exactly two frisbees. Note that this is in line with SPICE, which treats object counts as attributes in a scene graph, rather than duplicating a scene graph for every instance of an object (Anderson et al., 2016).
Black and White Photo MSCOCO contains black and white or gray-scale pictures. Some captions explicitly mention that they are black and white, but we disregard this difference in our evaluations. The crowdsourced instructions for creating reference captions do not specify such cases (Chen et al., 2015). Further, we can potentially run post-processing to determine whether it is black and white to modify the caption accordingly, depending on the downstream usage.
Text Processing Image captioning models often differ slightly in text preprocessing. As a result, we found that generated captions were sometimes slightly different in format (e.g., tokenized or tokenized; lowercased or not). For better reproducibility, we follow the spirit of SACREBLEU (Post, 2018), which has become the standard package to compute BLEU scores for machine translation: all evaluations, including automatic metrics, should be done on clean, untokenized text, independently of preprocessing design choices. We apply the following minimal postprocessing to the model outputs and human captions.
- Remove unnecessary spaces at the start or end of every caption.
- Uppercase the first letter.
- Add a period at the end if it doesn't exist, and remove a space before a period if any.
We keep the postprocessing minimal for this work and encourage future model developers to follow
the standard practice in machine translation: every model has to output clean, truecased, untokenized text that is ready to be used in downstream modules. This also improves the transparency and reproducibility of automated evaluations (Post, 2018).
2.3 Evaluated Captions
We evaluated the following four strong models from the literature as well as human-generated captions. They share similar pipeline structure: object detection followed by crossmodal caption generation. They vary in model architecture, (pre)training data, model size, and (pre)training objective. Evaluating captions from them will enable us to better understand what has been improved and what is still left to future captioning models.
- Up-Down (Anderson et al., 2018) trains Faster R-CNN (Ren et al., 2015) on the Visual Genome dataset (Krishna et al., 2016) for object detection. It then uses an LSTM-based crossmodal generation model.
- Unified-VLP (Zhou et al., 2020) uses the same object detection model as Up-Down. The transformer-based generation model is initialized with base-sized BERT (Devlin et al., 2019) and further pretrained with 3M images from Conceptual Captions (Sharma et al., 2018).
- VinVL-base and VinVL-large (Zhang et al., 2021) train a larger-scale object detection model with the ResNeXt-152 C4 architecture (Xie et al., 2017) on ImageNet (Deng et al., 2009). The transformer generation model is initialized with BERT and pretrained with 5.7M images.
- Human randomly selects one from the five human-generated reference captions in MSCOCO. Those captions were created by crowdworkers on Amazon Mechanical Turk (Chen et al., 2015).
Further details are described in  $\S A.3$  of Appendix.
We present results and analysis from our evaluations. Our transparent evaluations facilitate assessments and analysis of both captioning models (§3.1) and automatic metrics (§3.2).
3.1 Comparing Models
Seen in Table 2 (left section) is the model performance that is averaged over the 500 test images and broken down by the rubric categories. Overall, Human substantially outperforms all machines in the
Model	THUMB 1.0	Automatic Metrics
P↑	R↑	Flu.↓	Con.↓	Inc.↓	Total↑	BLEU	ROUGE	BERT-S	SPICE	CIDEr	CLIP-S	RefCLIP-S
Human	4.82	4.35	0.019	0.02	0.00	4.56+0.03	26.2	50.4	0.938	23.7	111.5	0.791	0.834
VinVL-large	4.54	3.97	0.005	0.00	0.00	4.25+0.04	33.3	56.5	0.946	26.4	141.8	0.784	0.834
VinVL-base	4.47	3.95	0.001	0.00	0.00	4.21+0.04	32.3	55.9	0.945	25.6	138.4	0.779	0.830
Unified-VLP	4.35	3.77	0.004	0.00	0.00	4.06+0.04	31.6	55.8	0.945	24.3	128.5	0.771	0.821
Up-Down	4.29	3.50	0.014	0.00	0.00	3.88+0.05	28.4	52.2	0.939	21.0	110.7	0.746	0.803
P, R, and total scores. In particular, we see a large gap between Human and the machines in recall (e.g., Human 4.35 vs. VinVL-large 3.97). This contrasts with the automatic metric-based ranking of the MSCOCO leaderboard, where Human is ranked at the 250th place. This result questions claims about human parity or superhuman performance on MSCOCO image captioning. The four machine captioning models are ranked in the expected order, though the small difference between VinVL-large and VinVL-base suggests that simply scaling up models would not lead to a substantial improvement. We see that the three models that are initialized with pretrained BERT (VinVL-large/base, Unified-VLP) are particularly fluent, but the problem is small in the other models as well.
While we compute representative, total scores, our transparent rubrics allow for adjusting weighting of the categories depending on the application of interest. For instance, in the social media domain, recall can be more important than precision to make captions engaging to users (Shuster et al., 2019). To assess the models independently of these aggregation decisions, we count the number of times when each model outperforms/underperforms all the others both in P and R (strictly best/worst, Table 3). We see patterns consistent with Table 2. For example, Human is most likely to be strictly best and least likely to be strictly worst. This suggests that machine captioning models would still fall short of crowdworkers in a wide range of downstream scenarios.
Table 2: Performance of image captioning models with respect to THUMB 1.0 (left) and automatic metrics (right). All scores are averaged over 500 images randomly sampled from the Karpathy test split. P: precision; R: recall; Flu.: fluency; Con.: conciseness; Inc.: inclusive language.  $90\%$  confidence intervals for total scores are calculated by bootstrapping (Koehn, 2004). All reference-based metrics take as input the same four crowdsourced captions that are not used in Human for fair comparisons. THUMB 1.0 scores Human substantially higher than the machines, unlike all automatic metrics.
Model	Human	Vin-large	Vin-base	U-VLP	Up-Down
# Best ↑	327	180	161	112	74
# Worst ↓	65	128	150	190	269
Table 3: # times when each captioning model is strictly best/worst in the caption set (i.e., best/worst both in precision and recall).
3.2 Comparing Automatic Metrics
While carefully-designed human judgments like ours should be considered more reliable, automatic metrics allow for faster development cycles. Our transparent evaluations can also be used to analyze how these automatic metrics correlate with different aspects of image captioning. Table 2 (right section) shows automatic scores of the captioning models over 7 popular metrics for image captioning. CLIP-S (Hessel et al., 2021) is a referenceless metric that uses image features from CLIP (Radford et al., 2021), a crossmodal retrieval model trained on 400M image-caption pairs from the web. RefCLIP-S augments CLIP-S with similarities between the generated and reference captions. All other metrics, such as SPICE (Anderson et al., 2016) and CIDEr (Vedantam et al., 2015), only use reference captions without image features.