SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3 presents the CIDEr scores on the validation set of the Conceptual Captions v1.1 (CC-1.1). The CIDEr metric fails to capture any meaningful correlation between its scores and the results of the SxS human evaluations.
6.3 Multilingual Models
We further explore the hypothesis that adding more languages inside one single model may perform
even better, as a result of both translation noise canceling out and the languages reinforcing each other in a common representation space. In this vein, we rename the bilingual version as PLuGS-2L, and train several additional models: a TTG-5L model, which uses a LangId token as input and uses for training all translated captions for all five languages and English; a  $\mathrm{TTG}_{\mathrm{large}}$ -5L model, for which we simply increased the capacity of the Transformer network (see Sec. 5.2); and a PLuGS-5L model, which is trained using groundtruth labels that are concatenations (using the LangId token as separator) between golden groundtruth En labels and their translated versions, for all five target languages.
Results using CIDEr are shown in Table 4. Across all languages, the TTG-5L models show a large gap in the CIDEr scores as compared to the TTG monolingual models. Using more capacity in the  $\mathrm{TTG}_{\mathrm{large}}$ -5L model closes the gap only slightly. However, the effect of using pivot-language stabilizers tends to be consistently larger, in terms of CIDEr improvements, than the ones obtained by increasing the model capacity.
To accurately evaluate the impact of multilinguality, we also perform SxS evaluations between the PLuGS-2L (as the base condition) vs.
Lang	TTG	PLuGS-2L	TTG-5L	TTGlarge-5L	PLuGS-5L
Fr	0.7932	0.7820	0.6834	0.7064	0.7264
It	0.7760	0.7813	0.6538	0.6885	0.6978
De	0.6079	0.6170	0.4992	0.5367	0.5503
Es	0.7907	0.7854	0.7093	0.7203	0.7284
Hi	0.7149	0.7155	0.5891	0.6201	0.6641
Table 4: CIDEr scores on CC-1.1 validation set for bilingual and multilingual models.
Lang	Wins	Losses	Gainsxs	BAccept	AAccept	GainAccept
Fr	21.3	18.3	3.0	69.8	68.7	1.1
It	22.2	18.2	4.0	56.4	55.5	0.9
Hi	26.8	27.0	-0.2	75.6	79.5	-3.9
Table 5: SxS performance of PLuGS-5L vs. PLuGS-2L models for three languages.
PLuGS-5L (as the test condition) models, over three languages (French, German, and Hindi). As shown in Table 5, the PLuGS-5L model performs better on French and Italian (3% and 4% better on  $Gain_{sxs}$ ), while performing worse on Hindi compared to the bilingual PLuGS Hindi model (-0.2% on  $Gain_{sxs}$ , -3.9% on  $Gain_{Accept}$ ). The results are encouraging, and indeed support the hypothesis that similar languages are reinforcing each other in the common representation space, explaining the gain observed for the Romance languages and the detrimental impact on Hindi.
We also note here that the human evaluation results, except for Hindi, come in direct contradiction to the CIDEr metric results, which indicate a large performance hit for PLuGS-5L vs. PLuGS-2L, across all languages. This reflects again the extreme care needed when judging the outcome of such experiments based on the existing automatic metrics.
6.4 Stabilizers Used as English Captions
As already mentioned, the PLuGS models generate outputs of the form Stabilizer +  $\langle \mathrm{LangId} \rangle +$  Caption. We therefore ask the following question: how does the quality of the Stabilizer output compare to the quality of captions produced by the baseline English model (that is, the same model whose captions are translated to the target languages in the TGT approach)?
We perform SxS human evaluations over Stabilizer captions (English) for three different PLuGS-2L models (trained for French, German, and Spanish). As shown in Table 6, the somewhat unexpected answer is that these Stabilizer outputs are consistently better, as English captions, compared
to the ones produced by the original monolingual English captioning model. The  $Gain_{sxs}$  are between  $5\%$  and  $6\%$  absolute percentage improvements, while  $Gain_{Accept}$  also improves up to  $3.4\%$  absolute for the PLuGS-Fr model.
We again note that the CIDEr metric is not able to correctly capture this trend, as shown by the results in Table 7, which indicate a flat/reverse trend.
6.5 Caption is Translation of Stabilizer
So far, we have verified that both the target-language Caption and the Stabilizer English outputs for the PLuGS-2L models are better compared to the alternative ways of producing them. Additionally, we want to check whether the Stabilizer and the target-language Caption are actually translations of each other, and not just independently good captions associated with the input image. In Table 9, we show the BLEU-4 score of the translation of the Stabilizer output for the PLuGS-2L models, compared to the corresponding PLuGS-2L Caption treated as a reference, using the images in the OID1k test set. The high BLEU scores are indeed confirming that the Caption outputs are close translations of the Stabilizer English outputs. This allows us to conclude that PLuGS models are indeed performing the double-duty of captioning and translation.
6.6 Stabilizers Used for Quality Estimation
Finally, we perform an experiment to understand the extent to which the quality of the Stabilizer outputs is correlated with the quality of the target-language Captions, so that a QE model (Levinboim et al., 2019) trained for English can be applied directly on PLuGS model outputs (more specifically,
Model	Wins	Losses	Gainsxs	BAccept	AAccept	GainAccept
PLuGS-Fr	26.9	21.8	5.1	70.4	67.0	3.4
PLuGS-De	26.6	21.3	5.3	70.4	69.7	0.7
PLuGS-Es	28.0	21.8	6.2	69.7	67.8	1.9
Table 6: Performance of Stabilizers used as captions from PLuGS models for three languages vs the captions produced by the baseline English model. The PLuGS Stabilizer outputs are better captions across all three languages.
Model	PLuGS	Baseline	Diff
PLuGS-Fr	0.8663	0.8772	-0.0139
PLuGS-De	0.8680	0.8772	-0.0092
PLuGS-Es	0.8590	0.8772	-0.0182
Table 7: CIDEr scores on CC-1.1 validation set for Baseline and PLuGS-Stabilizer outputs (English captions).
Model	Spearman ρ
TGT	TTG	PLuGS
PLuGS-Fr	0.3017	0.3318	0.5982
PLuGS-De	0.3246	0.2900	0.5862
PLuGS-Es	0.2928	0.3201	0.5566
on the Stabilizer outputs). To that end, we perform human evaluations of stand-alone captions.
In this type of evaluation, the raters are shown an image along with a single caption, and are asked to provide an absolute rating for the caption on a 4-point scale. As before, we define the metric Accept = Percent of images where majority of raters (2 of 3) marked Caption as Acceptable, Good or Excellent. Since these ratings are obtained individually for captions, we can use them to measure crosslingual quality correlations.
6.6.1 Quality Correlation between Stabilizer and Caption
We use the stand-alone caption evaluation results to compute quality correlations. Table 8 shows the correlation between the median human rating for the Stabilizer (English caption) vs Caption (target-language caption) for the PLuGS models considered. We see that the correlation is much higher compared to the baselines, calculated by computing the correlation of the median rating for the Stabilizer vs Caption (target-language) generated by the TGT and TTG approaches.
These results confirm that the PLuGS approach appears to be best suited for leveraging an existing
Table 8: Spearman correlation of Stabilizer vs TGT, TTG and PLuGS Captions across three languages.
Fr	It	De	Es	Hi
BLEU	93.3	92.9	88.2	93.9	88.2
Table 9: The BLEU-4 score of the translation of the stabilizer against the caption treated as the reference.
En QE model, due to the availability of the generated Stabilizer output that tends to maintain consistency between the English and the target-language caption, with respect to content accuracy.
We present a cross-modal language generation approach called PLuGS, which successfully combines the availability of an existing gold annotation (usually in English) with the availability of translation engines that automatically produce silver-data annotations. The result is a multilingual engine capable of generating high-quality outputs in the target languages, with no gold annotations needed for these languages.
We show that, for image captioning, the PLuGS approach out-performs other alternatives, while also providing the ability to pack multiple languages in a single model for increased performance. Surprisingly, by considering the generated outputs in the original language of the annotation (Stabilizer outputs), we find that the quality of the Stabilizers is higher compared to the outputs of a model trained on the original annotated data.
Overall, our results can be understood as a successful instance of transfer learning from a unimodal task (text-to-text translation) to a cross-modal task (image-to-text generation), which allows us to indirectly leverage the abundance of text-only parallel data annotations across many languages to improve the quality of an annotation-poor cross-modal setup.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. In ECCV.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632-3642, Brussels, Belgium. Association for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.
Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loic Barrault. 2019. Probing the need for visual context in multimodal machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4159-4170.
Soravit Changpinyo, Bo Pang, Piyush Sharma, and Radu Soricut. 2019. Decoupled box proposal and featurization with ultrafine-grained semantic labels improve image captioning and visual question answering. In EMNLP-IJCNLP.
Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2014. Long-term recurrent convolutional networks for visual recognition and description. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 215-233.
Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, hosted by the 54th Annual Meeting of the Association for Computational Linguistics, VL@ACL 2016, August 12, Berlin, Germany.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. JAIR.
Alan Jaffe. 2017. Generating image descriptions using multilingual data. In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 458-464.
Da-Cheng Juan, Chun-Ta Lu, Zhen Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, and Sujith Ravi. 2019. Graph-rise: Graph-regularized image semantic embedding. CoRR, abs/1902.10814.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. *ICLR*.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2015. Unifying visual-semantic embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. 2018. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. CoRR, abs/1811.00982.
T. Levinboim, A. Thapliyal, P. Sharma, and R. Soricut. 2019. Quality estimation for image captions based on large-scale human evaluations. arXiv preprint arXiv:1909.03396.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In Proceedings of ECCV.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the ACL.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of CVPR.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of ICML.
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. 2019. Fast deep neural network training on distributed systems and cloud tpus. IEEE Trans. Parallel Distrib. Syst., 30(11):2449-2462.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014a. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014b. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78.
Sanqiang Zhao, Piyush Sharma, Tomer Levinboim, and Radu Soricut. 2019. Informative image captioning with external sources of information. In ACL.