SlowGuess/NAACL_2021_OCR huggingface dataset:
Tomer Levinboim
Venice, CA 90291
{tomrl,asht,piyushsharma,rsoricut}@google.com
Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and without access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.
Image captioning technology produces automatic image descriptions using natural language with the goal of being consumed by end-users that may not be able to directly access the images. This need arises either because the user has a permanent condition (accessibility for visually impaired people), or due to a temporary situation where the user cannot use the visual modality (such as limited bandwidth, or smart voice-assistant). In any of these situations, exposing the end-users to a generated caption that is incorrect negatively impacts user-trust, as it can have undesirable consequences for how they act next (for example, how they comment on a social-media site based on their misguided understanding).
In this paper, we propose to mitigate such risks through Quality Estimation (QE) of image captions. That is, we propose to automatically compute a quality estimation score  $QE(\text{image}, \text{caption})$  for a generated caption, and use it to control the quality of the captions presented to the user. For example, by filtering out captions with a low QE score (below a carefully chosen threshold), only high scoring captions would be served thereby minimizing the risks associated with low-quality captions.
We emphasize two aspects of QE that have guided us in our design choices: First, the QE task is distinct from the model selection task: model selection measures output similarity to a fixed, ground-truth annotated dataset during training time (with traditional offline solutions such as CIDEr and SPICE). In contrast, a QE model estimates the caption quality with respect to the input image only and does so on previously unseen samples at prediction time where ground-truth captions are unavailable. Second, a QE model's goal is to assess the caption as a whole and relate it to the image content in a way that QE(image, caption) aligns with human understanding of language and their perception of visual information.
To address these aspects we develop an image-caption evaluation process for collecting vast amounts of human judgements. Specifically, we design the process to elicit only the type of human signal that is required for quality estimation – human annotators are shown the image and asked to evaluate the caption as a whole by simply answering whether it is good or not. This type of high level feedback trades away the ability to understand in what way the caption is wrong, but its simplicity enables scaling up human evaluations to cover many more images, which promotes the generalization of the QE model to unseen images.
The dataset resulting from the evaluation process includes captions generated by various image captioning model over 16,000 unique images from
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3157-3166 June 6-11, 2021. ©2021 Association for Computational Linguistics
the Open Image Dataset (Kuznetsova et al., 2018) for a total of 55,000 unique <image, caption> pairs, over which we collected approximately 600,000 binary human ratings. We denote this dataset as Caption-Quality, provide extensive details on its generation process as well as make it publicly available<sup>1</sup>, available.
The following summarizes our contributions:
1. We release the Caption-Quality dataset of roughly 65k human rated image-caption pairs, obtained by collecting approximately 600k binary human ratings in total. By analyzing the collected ratings, we show that they encode a stable and consistent signal about the caption.
2. We establish baseline results on the QE task and demonstrate that the signal encoded in the collected ratings is learnable, yet, cannot be trivially captured by an image-text similarity model trained over a large scale image-captioning dataset.
3. We further test our QE models, trained over the Caption-Quality dataset, and show that they can successfully rank correct-and-helpful captions higher than incorrect or unhelpful ones, even though they were never exposed to such a fine-grained signal. This is done by collecting additional fine-grained caption annotations from trained human raters, over images that are out-of-domain for the QE model.
Our paper is most similar to work done on evaluation metrics of image captions, where the main difference is that QE does not have access to the ground truth captions.
Quality estimation has more than a decade long history in the Machine Translation (MT) field, from the early work based on feature engineering (Specia et al., 2009; Soricut and Echihabi, 2010), to more recent neural-network-based approaches (Kreutzer et al., 2015; Kim and Lee, 2016; Kim et al., 2017). The QE track at the WMT conference (Specia et al., 2019) has been running for several years, with multiple participants and notable improvements in model performance over the years. However, there are significant differences in the formulation of the QE task between MT and image captioning, most notably the fact that the MT formulation is
uni-modal (text-only alignment). As a result, solutions for QE in the MT context tend to focus on feature-engineing that exploits this aspect (Specia et al., 2013; Kreutzer et al., 2015; Martins et al., 2017; Wang et al., 2018). In contrast, QE for Image Captioning is a bi-modal problem (image-and-text alignment), and therefore better suited to approaches based primarily on deep feature representations and multi-modal feature integration, as we present in this paper.
Beyond quality estimation modeling, the issue of effectively using quality estimators to improve the accessibility use-case for Blind or Visually Impaired (BVI) people has been previously studied (MacLeod et al., 2017). The main question of their study is how to best inform the BVI user about the uncertainty around the generated captions, experimenting with framing the captions using phrases like "I'm not really sure but I think it's $CAPTION" or "I'm 98% sure that's $CAPTION". The findings are relevant in that BVI users of this technology have difficulties calibrating themselves into trusting or distrusting $CAPTION, mostly because there is no alternative form of reference for the image content. Therefore, if the caption provided to them (even accompanied by "I'm not really sure but ..." ) is in dissonance with the rest of the context (as it may be available in text form, e.g., as part of a tweet thread as in the study cited above), they tend to resolve this dissonance not by believing that the caption is wrong, but by constructing scenarios or explanations that would somehow connect the two sources of information. To mitigate this problem, we propose a thresholding-based approach that simply decides whether to show a caption or not based on a QE model's prediction (See section 6.2).
3 Building the Caption-Quality Dataset
The key contribution of this paper is the Caption-Quality dataset, a large collection of binary human judgments on the quality of machine-generated image captions (in English). Below, we describe the dataset generation process, as well as the rating collection process with which we collect approximately 600,000 binary ratings via crowdsourcing. We then provide an analysis of the ratings which shows that they contain a consistent signal about the captions. Note that in the experiments (section 6.2), we further verify that indeed this signal captures the quality of the caption as perceived by
$^{1}$ https://github.com/google-research-datasets/image-caption-quality-dataset
trained humans annotators.
3.1 Image-Caption Generation
The starting point for our dataset is the Open Images Dataset (OID) (Kuznetsova et al., 2018) from which we randomly sample 16,000 images and then, for legal and privacy concerns, filter out those which contain faces<sup>2</sup>. The choice for OID images is driven by their image copyright status (CC BY) and the fact that they are out-of-domain for popular image captioning datasets such as COCO and Conceptual Captions.
To generate a diverse set of captions for annotation, we used several variants of Transformer-based (Vaswani et al., 2017) image-captioning models, trained on the Conceptual Captions dataset (Sharma et al., 2018), which consists of  $3.3\mathrm{M}$  training and  $\sim 15,000$  validation imagesc caption pairs. As previous work indicates (Sharma et al., 2018), for out-of-domain images (OID), captions produced by Conceptual Captions trained models tend to have higher quality compared to captions produced by COCO-trained models.
All of the models are trained to minimize the ground-truth caption perplexity; however, they differ on several important aspects (which contributes to caption diversity): the image feature representations, the number of object detection results they use, and the caption decoding procedure. We briefly discuss these differences below; for further details, see (Sharma et al., 2018; Changpinyo et al., 2019).
Global Image Representation Our captioning models use one of the following pretrained image encoders: (1) The Inception-ResNet-v2 model (Szegedy et al., 2016), (2) The Picturebook image encoder (Kiros et al., 2018), or, (3) The GraphRISE model (Juan et al., 2019), a ResNet-101 model (He et al., 2016) trained for an image classification task at ultra-fine granularity levels.
Object Representations The identification of objects in an image is done using a Faster R-CNN model, training it to predict both 1,600 object and 400 attribute labels in Visual Genome (Krishna et al., 2017), following the Bottom-Up Top-Down setting (Anderson et al., 2018). In terms of featurization for the identified bounding boxes, we use variants that include a ResNet-101 model pretrained on ImageNet (Russakovsky et al., 2015)
Figure 1: Our caption evaluation interface. Raters indicate whether the caption is good/bad, or, they can skip.
and one pre-trained using the Graph-RISE model (Juan et al., 2019).
Object Labels In addition to object-level representations, we detect object labels over the entire image, using a ResNet object-detection classifier trained on the JFT dataset (Hinton et al., 2015). The classifier produces a list of detected object-label identifiers, sorted in decreasing order by the classifier's confidence score. These identifiers are then mapped to embeddings  $o_j$  using an object-label embedding layer which is pre-trained to predict label co-occurrences in web documents using a word2vec approach (Mikolov et al., 2013).
Decoding To further increase caption variance, we use either greedy decoding or beam search with beam width 5.
3.2 Fast&Simple Human Annotation
Traditional approaches for human evaluation of automatically generated text, such as for image captioning (Vinyals et al., 2016) and machine translation (Banchs et al., 2015), approach the task by collecting human ratings across multiple evaluation dimensions, such as correctness, informativeness and fluency. Such fine-grained evaluations are typically used to expose model deficiencies during
Set	Samples	Unique Images	Unique Captions	Unique Models
Train	58354	11027	34532	11
Dev	2392	654	1832	4
Test	4592	1237	3359	4
Table 1: The Caption-Quality dataset statistics
Detected using the Google Cloud Vision API, https://cloud.google.com/vision/
development and can also assist during model selection. However, obtaining fine-grained rating on a large scale is a slow and costly process because it requires extensive manual labor by professionally trained human annotators. Furthermore, it is not immediately clear how the resulting multidimensional ratings can be combined to estimate the overall caption quality in a human-like manner.
To avoid these complications we develop an evaluation process that asks the human evaluators to rate the generated text not per dimension, but as a whole. The benefits of our approach are threefold: (1) the collected ratings better align with our end goal of quality estimation from a human perspective (2) having a single question accelerates caption evaluation, and (3) it substantially reduces the training and qualification requirements from the raters, which further contributes to the scalability of the evaluation process.
Specifically, we formulate the quality of an image-caption as the binomial probability  $p = P(GOOD|image, caption)$  that can be estimated from the Bernoulli process in which every trial corresponds to a different rater. We then leverage Google's crowdsourcing platform on which we present (image, caption) pairs and ask volunteer raters the following coarse binary question,
"Is this a good caption for the image?"
The raters can then select YES/NO, or skip to the next sample (SKIP) (see Fig. 1). In adopting this approach we take into account the fact that the platform's community consists of passionate volunteer raters, who may not have the linguistic background to provide fine-grained annotations. Furthermore, allowing the raters to skip captions reduces the risk of an undecided rater arbitrarily picking YES/NO just to move to the next image.
In order to reliably estimate the quality  $p$  we collect a high number of 10 ratings per image-caption sample. Once collected, the human ratings are further processed by: (1) filtering out (image, caption) entries that received more than 2 SKIP ratings (practically, the vast majority of images were kept), and (2) estimating  $p$  by averaging the 8 to 10 ratings  $r_i$  for each of the remaining (image, caption) pairs, and rounding to the closest score in  $\{0, \frac{1}{8}, \dots, \frac{7}{8}, 1\}$ , using the equation
$$
\hat {p} = r o u n d (m e a n (r _ {i}) * 8) / 8,
$$
Figure 2: A histogram of the dev, test and train  $p^*$ . The train set values were divided by 10 (for scale).
where  $r_i$  is 0 for NO answers and 1 for YES.
The resulting dataset, which we call the Caption-Quality v1.0 dataset, is then split into three image-disjoint subsets, used as train, dev and test folds in our experiments. We provide statistics for these subsets in Table 1, as well as histograms of  $\hat{p}$  in Fig. 2. Finally, we provide examples from the dev set in Table 2.
3.3 Stability Analysis
As described above, the interpretation of what a "GOOD" caption means is left up to the raters, which could lead to unstable or inconsistent human ratings (Graham et al., 2013). In order to verify the stability of the quality ratings  $\hat{p}$ , we study
Figure 3: A set of 509 captions were evaluated twice by different sets of 10 raters and 4 weeks apart. The figure shows a histogram of average human score differences  $(\hat{p}_1 - \hat{p}_2) \in [-1, 1]$ , with scores  $\hat{p}_i$  ( $i \in \{1, 2\}$ ) collected during the i-th evaluation.  $85\%$  of pairs are within a 0.25 distance, indicating that the evaluation setup produces a consistent and reliable signal.
<sup>3</sup>https://crowdsourced.google.com
the degree of agreement between different sets of 10 raters. We ran an evaluation over the same set of 509 image-captions twice, but 4 weeks apart<sup>4</sup>. An analysis of the difference of scores  $(\hat{p}_1 - \hat{p}_2)$  over these 509 pairs results in an almost zero mean (mean=0.015) as well as low variance (std=0.212). Figure 3 provides a histogram of the differences  $(\hat{p}_1 - \hat{p}_2)$  which clearly shows a concentration of the difference about 0. Furthermore, repeating this analysis over a different set of image-captions results in similar statistics.
In conclusion, the stability analysis shows that by collecting and averaging 8-10 coarse binary ratings, we obtain consistent and reproducible P(GOOD) estimates  $\hat{p}$  that are well-concentrated on a sample-level.
4 A Fine-Grained Caption Evaluation
We further collect fine-grained human annotations of image-captions to ascertain that the signal in the Caption-Quality dataset is beneficial for estimating the quality of image captions and filtering out low-quality ones. Specifically, we ask professional human annotators to evaluate image-captions across two specific dimensions: helpfulness and correctness<sup>5</sup>. Fig 4 shows the evaluation interface.
Distinguishing between correctness and helpfulness is particularly crucial for quality estimation, as it helps diagnose models that produce abstract
Figure 4: Fine-grained evaluation interface presented to professional raters. The raters determine whether (1) the caption provides a helpful description for a person who cannot see the image, (2) the information in the caption is correct.
or irrelevant captions which, while correct, do not provide useful image descriptions (specifically, for a person who is unable to see the image). For example, consider the correct yet abstract caption "Person in a sport event" compared to the more descriptive caption "Ice hockey player celebrates his goal against sports team" (See Fig 4). Another example of a correct but unhelpful caption is "A view of the game from my living room" because it conveys more information about the camera position rather than the actual image content. While the previously discussed Fast&Simple evaluation may assign all these captions with similar scores, the fine-grained evaluation is capable of capturing such nuanced differences.
We posit that the large-scale annotations obtained by the Fast&Simple approach will enable a model to distinguish between correct-and-helpful captions, and those that are not. We ran the fine-grained evaluation once over 2,700 images, collecting 3 ratings per image. The resulting dataset, denoted Caption-Ext is used for our extrinsic QE evaluations (Sec. 6.2).
This section presents a simple bilinear QE model which learns to combine the image and caption features to arrive at a quality estimate  $QE(image, caption)$ . To construct the bilinear model we rely on expressive image and text representations that are produced by pretrained models that were themselves trained on vast amounts of uni-modal data. Note that aside from building on top of pretrained models, we restrict further modeling to a simple architecture. This was done in order to establish a baseline for our new QE task, as well as to remain focused on providing evidence that the signal in the Caption - Quality dataset is both learnable and beneficial for quality estimation of image captions.
5.1 A Bilinear QE model
Our bilinear neural network model relies on three input types: caption, image and object labels. These representations are produced by the following pretrained models:
Global Image Embedding For a global image representation, we used the latest Graph-RISE model version (Juan et al., 2019) which produces a compact image embedding  $i$  of dimension  $D_{i} = 64$ .
4The evaluation platform roughly guarantees that the ratings are provided by different subsets of raters.
We also evaluate along a fluency dimension, but current captioning models tend to produce overall fluent outputs, which makes this dimension non-discriminative.
Image	Generated captions	Human rating
a general view of atmosphere .	0.375
this is a picture of a yacht .	0.75
the yacht is a great place to take a rest .	0.875
person and her husband take a walk .	0.125
people walking along the beach	0.25
people walking along the beach	0.5
cat in the grass with a dog	0.25
a tiger in the grass	0.25
cat lying on the grass	0.75
a police car in the middle of the road	0.125
automobile model in the rain .	0.5
vehicles drive through a flooded street	0.875
plants for sale at the local market	0.625
a selection of plants in the flower market	0.875
flowers for sale at the market	1.0
the team at the opening .	0.375
the cast performs on stage .	0.5
the cast of musical film	0.75
Table 2: Samples from the Caption-Quality dataset (dev fold). Images are paired with 3 captions and their corresponding mean human ratings. Repeated captions (which were generated by different captioning models) were rated by different sets of 10 raters, and tend to have similar scores (See stability analysis, cf. Figure 3). As can be seen, the higher scoring captions tend to include more information or contain fewer mistakes.
Using this model enables transfer learning for QE with respect to image representation.
Object Labels Embeddings Objects present in the image (e.g. "cat", "vehicle", "flower") can help assess the correctness and helpfulness of a candidate caption, where the intuition is that the caption should likely mention the more salient objects. We use the object label model mentioned in Sec. 3.1, whose resulting embedding sequence is  $O = (o_{1}, \ldots, o_{|O|})$ , where each  $o_{j}$  has dimension  $D_{o} = 256$ .
Caption Universal Sentence Embedding The caption text is embedded using a pretrained version of the Universal Sentence Encoder (USE) (Cer et al., 2018) into a  $D_{s} = 512$  dimensional vector  $s$ . The USE model itself is trained on large amounts of English sources (Wikipedia, web news, discussion forums, etc.) and fine-tuned using supervised labels from the SNLI corpus (Bowman et al., 2015). We have alternatively tried a BERT (Devlin et al., 2019) model as an encoder, but observed it provides no additional gains (Alikhani et al., 2020)
Given these features, the bilinear QE model (illustrated in Figure 5) processes each individual feature using a dense layer with a leaky-ReLU acti
vation (Xu et al., 2015), and then combines each of the resulting vector pairs using bilinear layers (see below). All bilinear outputs are then concatenated and fed to a dense layer with a sigmoid activation, to produce the quality estimation  $\hat{y}$
5.1.1 Bilinear Layers
A bilinear layer models the inner product of its two inputs after applying a linear transformation to the second input. This layer is defined as:
$$
b (x, y; B) = x ^ {T} B y = \langle x, B y \rangle \tag {1}
$$
where  $x \in R^{D_x}$  and  $y \in R^{D_y}$  are input features, and  $B \in R^{D_x \times D_y}$  is the learned parameter matrix. Linear and bias terms can be added by appending a constant 1 to each of  $x$  and  $y$ .
We use three such parameter matrices to capture the interaction between each pair of input-types:
1.  $B_{o,i} \in R^{D_o \times D_i}$ , applied to each of the object-label embeddings  $[o_1, \ldots, o_{|O|}]$  and the image embedding  $i$ .
2.  $B_{o,s} \in R^{\bar{D}_o \times D_s}$ , applied to each of the object-label embeddings  $[o_1, \ldots, o_{|O|}]$  and the sentence embedding  $s$
3.  $B_{i,s} \in R^{D_i \times D_s}$ , for the image embedding  $i$  and sentence embedding  $s$ .
Figure 5: The bilinear QE model: Each input-modality pair has its own dedicated bilinear layer. The inputs to the model are pre-trained embeddings (blue) for the image, caption and object-label input types. The model parameters (pink) may further be warm-started by pretraining the model on an image-text similarity task (see Section 5.2).
5.2 An Image-Text Similarity Baseline