SlowGuess/NAACL_2022_OCR huggingface dataset:
SPICE SPICE (Anderson et al., 2016) predicts scene graphs from the generated and reference captions using the Stanford scene graph parser (Schuster et al., 2015). It then measures the  $F_{1}$  score between scene graphs from the generated and reference captions. WordNet Synsets are used to cluster synonyms (Miller, 1995). We again use the implementation from the pycocoevalcap package.
BERTScore BERTScore (Zhang et al., 2020) aligns tokens between the generated and reference captions using contextual word representa
https://github.com/mjpost/sacreBLEU/ blob/v1.2.12/sacrebleu.py#L999. https://pypi.org/project/rouge-score/. 10https://github.com/salaniz/ pycocoevalcap.
Fluency Error Type	Penalty	Example
Obvious spelling error, one vs. two words	0.1	cel phone, surf board
Grammatical error that can be easily fixed	0.1	a otter
Casing issue	0.1	tv, christmas
Hyphenation	0.1	horse drawn carriage
Interpretable but unnatural wording	0.1	double decked bus
Non-trivial punctuation	0.2	A bird standing in the wooded area with leaves all around.
Misleading spelling error	0.5	A good stands in the grass next to the water. (good→goose)
Duplication	0.5	A display case of donuts and doughnuts.
Ambiguity	0.5	A cat is on a table with a cloth on it.
Awkward construction	0.1–0.5	There is a freshly made pizza out of the oven.
Broken sentence	0.5+	A large concrete sign small buildings behind it.
Table 6: Fluency penalty rubrics.
tions from BERT (Devlin et al., 2019). We use the HuggingFace implementation (Wolf et al., 2020) and compute the  $F_{1}$  score. As in Zhang et al. (2020), we take the maximum score over all reference captions.
CLIPScore CLIPScore (Hessel et al., 2021) is the only referenceless metric out of the 7 metrics. It measures the cosine similarity between the generated caption and given image using the representations from CLIP. It is shown to correlate better with human judgments from prior work, compared to previous reference-based metrics (Hessel et al., 2021). We use the official implementation by the authors.[11]
RefCLIPScore RefCLIPScore augments CLIP-Score with the maximum similarity between the generated and reference captions. We again use the official implementation.
A.3 Evaluated Captions
We evaluated the following four strong models from the literature as well as human-generated captions. They share similar pipeline structure but vary in model architecture, (pre)training data, model size, and (pre)training objective. Evaluating captions from them will enable us to better understand what has been improved and what is still left to future captioning models.
Up-Down The bottom-up and top-down attention model (Up-Down, Anderson et al., 2018) performs pipelined image captioning: object detection that finds objects and their corresponding image regions and crossmodal generation that predicts a
caption based on the features from object detection. The bottom-up attention finds salient image regions during object detection, and the top-down one attends to these regions during crossmodal generation. Up-Down uses Faster R-CNN (Ren et al., 2015) and LSTMs (Hochreiter and Schmidhuber, 1997) for object detection and crossmodal generation respectively. Faster R-CNN is trained with the Visual Genome dataset (Krishna et al., 2016), and the crossmodal generation model is trained on the MSCOCO dataset. We generate captions for the test data with a model optimized with crossentropy. $^{12}$
Unified-VLP Unified-VLP (Zhou et al., 2020) also runs a pipeline of object detection and crossmodal generation. Faster R-CNN and the transformer architecture (Vaswani et al., 2017) are used for object detection and crossmodal generation respectively. Similar to Up-Down, the Faster R-CNN object detection model is trained with the Visual Genome dataset. The transformer generation model, on the other hand, is initialized with base-sized BERT (Devlin et al., 2019) and pretrained on the Conceptual Captions dataset (3M images, Sharma et al., 2018) with the masked and left-to-right language modeling objectives for the captions. The crossmodal generation model is then finetuned on the MSCOCO dataset. We apply beam search of size 5 to the model with CIDEr optimization.
VinVL-base, VinVL-large VinVL with Oscar (Li et al., 2020; Zhang et al., 2021) performs a similar pipeline of object detection, followed by crossmodal generation. The crossmodal model is initialized with BERT (Devlin et al., 2019) as in
https://github.com/jmhessel/pycocoevalcap.
12https://vision-explorer.allenai.org/ image Captioning.
Unified-VLP but uses detected object tags to encourage alignments between image features and word representations. The object detection model with the ResNeXt-152 C4 architecture (Xie et al., 2017) is pretrained with ImageNet (Deng et al., 2009) and trained on 2.5M images from various datasets. The transformer-based crossmodal generator is initialized with BERT, pretrained with 5.7M images, and finetuned for MSCOCO captioning. We use VinVL-base and VinVL-large that are both finetuned with CIDEr optimization $^{13}$  and generate captions with beam search of size 5.
Human In addition to machine-generated captions from the four models, we assessed the quality of human-generated reference captions from MSCOCO. This will allow us to understand the performance gap between machines and humans, as well as the quality of crowdsourced captions. Human-generated captions were created using Amazon Mechanical Turk (Chen et al., 2015). Crowdworkers were only given the following instructions (Chen et al., 2015):
- Describe all the important parts of the scene.
- Do not start the sentences with "There is."
- Do not describe unimportant details.
- Do not describe things that might have happened in the future or past.
- Do not describe what a person might say.
- Do not give people proper names.
- The sentences should contain at least 8 words.
Every image has five human-generated captions, and we randomly selected one for each to evaluate. We found, however, a non-negligible number of noisy captions in the MSCCOCO dataset from annotation spammers. We often find subjective adjectives (e.g., very nice/clean/cute) or words that diverge from inclusive language in reference captions, probably because crowdworkers increased the number of words in captions effortlessly (see the last instruction item that says captions have to have  $8+$  words). To better estimate the performance of a human that invests reasonable effort into the captioning task, we resampled a caption for  $13\%$  of the test images, which would have been given a total score lower than 4.0.
A.4 Additional Machine vs. Human Examples
Table 7 provides an additional example that contrasts machine- and human-generated captions. All machines generate generic captions and ignore the most important information that a traditional Thanksgiving dinner is being served on the table.
$^{13}$ https://github.com.microsoft/Oscar/blob/master/VinVL_MODEL_ZOO.md#Image-Captioning-on-COCO.
Image	Caption	P	R	Total
10-A: Up-Down	5	2	3.5
A table that has some food on it.
10-B: Unified-VLP	5	2	3.5
A table with plates of food on a table.
10-C: VinVL-base	4	2	3
A red table topped with plates of food and bowls of food.
10-D: VinVL-large	5	3	4
A table with a turkey and other food on it.
10-E: Human	5	5	5
A table set for a traditional Thanksgiving dinner.
Table 7: Additional example that contrasts machine- and human-generated captions. Similar to Table 5, machine-generated captions ignore the most salient information: Thanksgiving dinner. Note that this case is specific to North America; such salient information can vary across cultures or languages (van Miltenburg et al., 2017). None of these captions are penalized for fluency, conciseness, or inclusive language.