SlowGuess/ACL_2020_OCR huggingface dataset:
Unigram overlap matching metric. In our unigram overlap matching metric, we remove contextual embedding from the pre-trained language model and only use unigram overlap matching. Different from continuous value methods like BERTScore, it is easy for overlap matching to distinguish the match and mismatch (1 or 0). In com
Flickr 8K	COMPOSITE
BLEU-1	0.318	0.282
BLEU-4	0.140	0.199
ROUGE-L	0.323	0.313
BS (RoBERTa)	0.367	0.392
BS (BERT)	0.393	0.399
METEOR	0.436	0.381
CIDEr	0.447	0.387
SPICE	0.458	0.418
LEIC	0.466*	-
Ours (RoBERTa)	0.451	0.449
Ours (Unigram)	0.471	0.420
Ours (BERT)	0.481	0.423
Inter-human	0.736*	-
Table 3: In caption-level experiments, we compute the Kendall correlation between human judgments and scores of metrics. Two dataset results are given: Flickr 8K and COMPOSITE. Both our unigram metric and BERT based metric outperform other metrics. Scores with * are cited from (Cui et al., 2018)
bination part, we collect all the mismatch words and combine them with the base reference caption. To reduce the impact of unimportant words, we remove stop words from the combined caption directly.
References concatenation. We also regard the concatenation of references as another baseline comparing with our combination method. The concatenation of references combines all the information from references as well. The difference between concatenation and our combination is the duplicate tokens in majority references. In this metric, we follow all the steps of our metric with BERT, except the combination.
4.4 System-Level Correlation
In system-level evaluation, we use twelve teams of human judgment results from COCO 2015 Captioning Challenge. We use data from "Karpathy splits" (Karpathy and Fei-Fei, 2015), which contains 113,287 train images, 5000 test images, and 5000 validation images. Each image has 5 references human captions. Following prior works (Anderson et al., 2016; Cui et al., 2018), we compute the Pearson correlation with human judgment. In the pre-trained model selection for BERTScore, we choose BERT (base), which is the most common model in the set of transformer language models, and RoBERTa (large), which is an optimized ver
$^{2}$ https://github.com/tylin/coco-caption
HC	HI	HM	MM	All
BLEU-1	53.1	94.7	90.9	56.9	73.9
BLEU-4	53.3	92.8	85.2	60.5	73.0
ROUGE-L	55.6	95.1	93.3	57.7	75.4
METEOR	61.4	97.2	94.9	64.5	79.5
CIDEr	55.0	98.0	91.0	64.6	77.2
SPICE	57.7	96.1	88.3	65.3	76.9
Ours (RBT)	62.5	97.7	95.0	59.4	78.7
BS (BERT)	64.4	97.9	96.6	59.0	79.5
Ours (BERT)	65.4	98.1	96.4	60.3	80.1
Table 4: In PASCAL-50S, candidate sentences come from human written or model generated. There are 4 kinds of paired ways: human-correct (HC), human-incorrect (HI), human-model (HM), and model-model (MM). Ours (BERT) outperforms in HC, HI and HM. Abbreviation: RBT means RoBERTa.
sion of BERT.
The experimental results in Table 1 show that our metrics with both BERT and with RoBERTa perform better than BERTScore and other standard metrics. What is more, our metric with RoBERTa can reach a high correlation of 0.932 with human judgment, which is even close to the training-based task-specific metric LEIC with image features.
4.5 Ablation and replacement
To check the influence of each part, we provide both ablation study and replacement study in 2015 COCO Captioning dataset. The results are showed in Table 2.
In ablation study, we use our metric with BERT and remove remove, combine and cut one by one. The result shows that each part of our metric is useful, and combine is the most influential part.
In the replacement study, we compare our metric with the unigram metric and concatenation metric to check the influence of contextual embedding and combination. The comparison between Ours (Unigram) and Ours (BERT+TBR) shows that contextual embedding is better than unigram matching in the system-level correlation task. The comparison between Ours (BERT+T+cat+R) and Ours (BERT+TBR) shows that the combination process is better than concatenation directly. Furthermore, we also show the comparison between concatenation and average in some standard metrics.
Model	Ours (BERT)	CIDEr-D
AoAnet	0.3529	1.296
M2-Transformer	0.3481	1.321
SAT	0.3296	0.893
CNN+LSTM	0.3055	0.946
NeuralTalk	0.2845	0.692
Table 5: We present some results on current state-of-the-art models (M2-Transformer and AoAnet) for image captioning models with respect to CIDEr-D. The experimental results show that: on both our metric and CIDEr-D, current models perform better. Abbreviation: SAT means Show, Attend and Tell.
4.6 Caption-Level Correlation
In caption-level evaluation tasks, we compute Kendall's correlation (Kendall, 1938) between metrics results and expert judgments.
In Flickr 8K, we use Expert Annotation with 5822 samples, including 158 correct image-caption pairs where the candidate caption equals one of captions in references set. Following the prior work (Anderson et al., 2016), we use 5664 samples and exclude those correct image-caption pairs. In COMPOSITE, captions are estimated by two kinds of standards: correctness and throughness, and we only focus on correctnesss in this work.
The experimental results in Table 3 show that our metric is quite suitable for caption-level evaluation in image captioning. Our metric outperforms other metrics (including training-based metric LEIC in Flickr 8K). Another interesting fact is that the unigram metric also has high performance in caption-level correlation tasks. In COMPOSITE, our unigram metric is comparable to our metric with BERT.
In PASCAL-50S, we use five references for metrics computation, which is comparable with previous experiments. The results in Table 4 show that in four kinds of caption pairs, our metric performs better than others in human-correct (HC), human-incorrect (HI), human-model (HM) classification.
5 More model results on our metric
In table 5, We evaluate some current state-of-the-art image captioning models reported from Codalab competition: Meshed-Memory-Transformer (Cornia et al., 2020), AoAnet (Huang et al., 2019). Some of models in 2015 COCO Captioning Chal
<sup>3</sup>https://competitions.codalab.org/competitions/3221
lenge are listed for comparison: (1) Show, Attend and Tell (Xu et al., 2015); (2) CNN+LSTM (Vinyals et al., 2015); (3) NeuralTalk (Karpathy and Fei-Fei, 2015). The result shows that: on our metric, current models perform better than previous models. It is worth noting that different judgments exist between AoAnet and M2-Transformer on our metric and CIDEr-D. According to our observation, several captions (1558/5000) generated by M2-Transformer are incomplete, like a bedroom with a bed and a tv in a or a wooden door with a skateboard on  $a$ . It may explain why M2-Transformer is a little worse than AoAnet on our metric.
In this work, we study the intrinsic variance among ground truth captions in image captioning evaluation. We propose an improved matching metrics based on BERTScore, which can combine all of the references for taking full advantage of multi-references. Our metric also benefits from stop word removal by reducing the impact of stop words. The experimental results show that our metric can reach state-of-the-art human correlation in several evaluation tasks.
Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, and Yiannis Aloimonos. 2015. From images to sentences through scene description graphs using commonsense reasoning and knowledge. arXiv preprint arXiv:1511.03292.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, pages 382-398. Springer.
X. Chen, H. Fang, TY Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv:1504.00325.
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-Memory Transformer for Image Captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5804-5812.
Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. 2017. Towards diverse and natural image descriptions via a conditional gan. In Proceedings of the IEEE International Conference on Computer Vision, pages 2970-2979.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.'s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722.
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853-899.
Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019. Attention on attention for image captioning. In International Conference on Computer Vision.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137.
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81-93.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2017. Improved image captioning via policy gradient optimization of spider. In Proceedings of the IEEE international conference on computer vision, pages 873-881.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assetss/researchcovers/languageeunsupervised/language understanding paper.pdf.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575.
Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2017. Captioning images with diverse objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5753-5761.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le.
2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.