SlowGuess/NAACL_2021_OCR huggingface dataset:
Having the large scale Conceptual Captions dataset (Sharma et al., 2018) opens up the option to pretrain a QE model on an image-text similarity task (Cui et al., 2018) before fine-tuning on the Caption-Quality dataset. We exercise this option by setting up a classification task whose goal is to match each image within a mini-batch with its corresponding ground truth caption. Specifically, we feed the bilinear QE model mini-batches of size 256 and train it to detect the ground-truth caption of each image among the other ground-truth captions in the batch (along the lines of noise-contrastive estimation (Gutmann and Hyvarinen, 2010)). The pretrained model achieves  $62\%$  accuracy over the Conceptual Captions dev set and serves as an image-text similarity baseline. In addition, its parameters serve as a fine-tuning initialization point that is better informed about the relationship between image and text compared to random initialization.
All QE models are trained on the Caption-Quality training set (Section 3). We use Mean Squared Error  $(MSE = \sum_{j=1}^{B} \frac{1}{N} (y_j - \hat{y}_j)^2)$  as the loss function, where  $\hat{y}_j$  are the predicted scores and  $y_j$  the ground-truth human scores. For optimization, we use Adam (Kingma and Ba, 2015) with batch size  $B = 256$  and tune the learning rate  $lr \in \{1e-4, 1e-5, 1e-6\}$ . Dropout rate is set to 0.2, and applied on the inputs of all trainable layers. The following pretrained models are fixed during optimization: the image encoder, the USE caption encoder, and object-label encoder. The number of object-labels is tuned over  $\{0, 5, 10, 20\}$ , while the pretrained variants were fixed to 16.
Model selection is done by picking the checkpoint that maximizes the dev set Spearman's correlation  $\rho^{\mathrm{S}}(y,\hat{y})$ . Specifically, compared to MSE (the
objective), the Spearman-based selection criterion better matches the intended use of the QE model, where at inference time, only images whose QE scores pass some threshold will be served. Since this threshold can be tuned, the absolute value of the predicted scores  $\hat{y}$  is not as critical as obtaining a monotonic relationship between the predicted and ground truth scores (using  $\rho^{\mathrm{S}}$  as the loss function is less feasible due to non-differentiability).
6.1 Spearman's  $\rho$  Analysis
We present in Table 3 our dev and test Spearman results based on selecting the best-performing model configurations over the dev set.
Rows 1 and 2 show the bilinear model achieves minor improvements given additional 20 object labels. The poor Spearman scores in row 3, which were obtained without fine tuning over the Caption-Quality dataset, demonstrate that predicting the human ratings cannot be trivially achieved with an image-text similarity model, even when trained on a large dataset as Conceptual Captions. On the other hand, after fine-tuning it for the QE task (row 4), both dev and test Spearman scores increase substantially by 6-7 Spearman points over the best non-pretrained variant, which demonstrates the effectiveness of bi-modal pretraining for the QE task.
6.2 Extrinsic Evaluation
So far we have shown that the signal in CaptionQuality is both consistent and learnable. In this section, we further show that the collected signal is effective for filtering out low-quality image captions. To do so, we evaluate the performance of CaptionQuality trained QE models over the Caption-Ext dataset, a more challenging setting which contains out-of-domain images (non-OID) and where each caption is annotated by three trained raters for its correctness and helpfulness (Sec. 4). Our analysis
Model	QE training features	learning rate	ρSdev	ρSTest	MSEdev	MSEtest
Bilinear	image, caption	1e-5	0.49	0.47	0.055	0.056
Bilinear	+ 20 object labels	1e-5	0.50	0.47	0.055	0.058
Bilinear (Pretrained)	-	1e-5	0.26	0.25	0.075	0.073
Bilinear (Pretrained)	image, caption, 16 labels	1e-5	0.57	0.53	0.053	0.053
Table 3: Spearman's  $\rho^{\mathrm{S}}$  scores on the Caption-Quality dev and test dataset (higher is better). The pretrained and fine-tuned bilinear model exhibits the best Spearman results on the QE task. MSE results show the same trend and are included for completeness.
reveals that QE models trained over the Caption-Quality dataset generalize well to this harder task, having the ability to distinguish between correct- and-helpful image-captions and those that are not, even though these models were never exposed to such fine-grained signal.
Specifically, for a given image, we define a caption as Ext-Good (extrinsically good) if a majority of raters agreed that it is at least partially-correct, and, a majority of raters agreed it is at least somewhat-useful. With this definition, we compute the Ext-Good precision and recall statistics of a QE model  $Q$  for each threshold  $th \in [0,1]$  using the following equations:
$$
p r e c i s i o n _ {t h} ^ {Q} = \frac {\sum_ {s} 1 _ {E x t - G o o d} ^ {s} \cdot 1 _ {Q E (s) > t h}}{\sum_ {s} 1 _ {Q E (s) > t h}} \tag {2}
$$
$$
r e c a l l _ {t h} ^ {Q} = \frac {\sum_ {s} 1 _ {E x t - G o o d} ^ {s} \cdot 1 _ {Q E (s) > t h}}{\sum_ {s} 1 _ {E x t - G o o d} ^ {s}} \tag {3}
$$
where the indicator variable  $1_{Ext-Good}^s$  is on only when  $s$  is Ext-Good, and similarly the indicator
Figure 6: Precision-Recall curves for the various Bilinear models. AUC values are reported in the legend. The pretrained and fine-tuned model (black) attains the highest precision values across almost all recall values.
variable  $1_{QE(s) > th}$  is on only when the QE score of sample  $s$  is higher than the threshold  $th$ .
Figure 6 shows the precision-recall curves and AUC scores for the same models analyzed in the previous section. A visual inspection of this figure shows that the precision of the pretrained and fine-tuned bilinear model (black) dominates the other models across almost all recall values. Indeed, in terms of AUC, the worst performing model is the image-text similarity baseline (blue; AUC=0.76) which has no access to the Caption-Quality dataset and its human ratings. On the other hand, the pretrained and fine-tuned model (which is also the Spearman maximizing model) attains the highest AUC score (AUC=0.84).
Put differently, to achieve precision=0.8 (i.e., 80% of served captions are both correct and helpful), the image-text similarity model would be thresholded to serve only its top 21% scoring image-captions (recall=0.21) while the pretrained and fine-tuned model would serve its top 71% scoring image-captions (recall=0.71, or x3.4 improvement). This analysis clearly demonstrates the usefulness of the Caption-Quality dataset for filtering out image-captions of low quality (where quality is determined by professional human raters).
Beyond its relevance for the QE task, we expect that the collected signal in the Caption-Quality dataset will find usage in other image captioning tasks, such as (1) fine-grained caption evaluation (that is, caption classifiers that evaluate captions across multiple dimensions) for example, by way of pretraining against our dataset, as well as (2) improving caption generation itself, for example, by means of QE-based caption re-ranking, or by using the ratings in a reinforcement learning setup, as has recently been done by (Seo et al., 2020).
In this paper we discussed how low-quality image-captions can negatively impact end-users and proposed a thresholding solution that relies on quality estimation of image captions, where caption quality is defined from a human perspective. To make this solution feasible we developed a scalable human evaluation process with which we annotated a large number of image-captions with their human estimated quality. We provided supporting evidence that the resulting dataset contains a consistent and reliable signal, as well as reported experimental results over professionally labeled fine-grained caption annotations, which verify that QE models trained over the Caption-Quality dataset are effective at filtering out low-quality image captions.
To encourage further research in automatic evaluation of image-captions, we make available our large-scale dataset of human judgments at https://github.com/google-research-datasets/Image-Caption-Quality-Dataset.
Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, and Matthew Stone. 2020. Cross-modal coherence modeling for caption generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6525-6535, Online. Association for Computational Linguistics.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of CVPR.
R. E. Banchs, L. F. D'Haro, and H. Li. 2015. Adequacy-fluency metrics: Evaluating mt in the continuous space model framework. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):472-482.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Soravit Changpinyo, Bo Pang, Piyush Sharma, and Radu Soricut. 2019. Decoupled box proposal and featurization with ultrafine-grained semantic labels improve image captioning and visual question answering. In EMNLP-IJCNLP.
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge J. Belongie. 2018. Learning to evaluate image captioning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5804-5812.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Crowd-sourcing of human judgments of machine translation fluency. In Proceedings of the Australasian Language Technology Association Workshop 2013 (ALTA 2013), pages 16-24, Brisbane, Australia.
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy. PMLR.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Da-Cheng Juan, Chun-Ta Lu, Zhen Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, and Sujith Ravi. 2019. Graph-rise: Graph-regularized image semantic embedding. CoRR, abs/1902.10814.
Hyun Kim, Hun-Young Jung, Hongseok Kwon, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-estimator: Neural quality estimation based on target word prediction for machine translation. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 17(1):3:1-3:22.
Hyun Kim and Jong-Hyeok Lee. 2016. A recurrent neural networks approach for estimating the quality of machine translation output. In Proceedings of the North American Chapter of the Association of Computational Linguistics, pages 494-498.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. *ICLR*.
Jamie Kiros, William Chan, and Geoffrey Hinton. 2018. Illustrative language understanding: Large-scale visual grounding with image search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 922-933, Melbourne, Australia. Association for Computational Linguistics.
Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. Quality estimation from scratch (quetch): Deep learning for word-level translation quality estimation. In Proceedings of the 10th Workshop on Statistical Machine Translation (WMT).
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and Li Fei-Fei. 2017. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32-73.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. 2018. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. CoRR, abs/1811.00982.
Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people's experiences with computer-generated captions of social media images. In CHI.
Andre F. T. Martins, Marcin Junczys-Dowmunt, Fabio Kepler, Ramón Fernández Astudillo, Chris Hokamp, and Roman Grundkiewicz. 2017. Pushing the limits of translation quality estimation. Transactions of the Association for Computational Linguistics, 5:205-218.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet large scale visual recognition challenge. IJCV, 115(3):211-252.
Paul Hongsuck Seo, Piyush Sharma, Tomer Levinboim, and Radu Soricut. 2020. Reinforcing an image caption generator using off-line human feedback. In Proceedings of AAAI.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL.
Radu Soricut and Abdessamad Echihabi. 2010. Trustrank: Inducing trust in automatic translations via ranking. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 612-621, Stroudsburg, PA, USA. Association for Computational Linguistics.
Lucia Specia, Frederic Blain, Varvara Logacheva, Ramon Astudillo, and Andre Martins. 2019. Findings of the wmt 2018 shared task on quality estimation. In Proceedings of the Third Conference on Machine Translation (WMT), Volume 2: Shared Task Papers.
Lucia Specia, Nicola Cancedda, Marc Dymetman, Marco Turchi, and Nello Cristianini. 2009. Estimating the sentence-level quality of machine translation systems. pages 28-37.
Lucia Specia, Kashif Shah, Jose Guilherme Camargo de Souza, and Trevor Cohn. 2013. QuEst - A Translation Quality Estimation Framework. In Proceedings of the 51th Conference of the Association for Computational Linguistics (ACL), Demo Session, Sofia, Bulgaria. Association for Computational Linguistics.
Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR, abs/1602.07261.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:1-1.
Jiayi Wang, Kai Fan, Bo Li, Fengming Zhou, Boxing Chen, Yangbin Shi, and Luo Si. 2018. Alibaba submission for wmt18 quality estimation task. In Proceedings of the Third Conference on Machine Translation (WMT).
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. Deep Learning Workshop, ICML.