SlowGuess/NAACL_2021_OCR huggingface dataset:
Model	En⇒De	En⇒Fr
Ambiguous COCO	Ambiguous COCO
BLEU	METEOR	BLEU	METEOR
Multimodal Neural Machine Translation Systems
fusion-conv (Caglayan et al., 2017)	25.1	46.0	43.2	63.1
trg-mul (Caglayan et al., 2017)	26.4	47.4	43.5	63.2
VAG-NMT (Zhou et al., 2018)	28.3	48.0	45.0	64.7
ImagiT + ground truth	28.8	48.9	45.3	65.1
Text-only Neural Machine Translation Systems
Transformer baseline (Vaswani et al., 2017)	27.9	47.8	44.9	64.2
ImagiT	28.7	48.8	45.3	65.0
the translation is the core task. The batch size is 64, and the learning rate is initialized to be  $2e^{-4}$  and decayed to half of its previous value every 100 epochs. A similar learning schedule is adopted in Zhang et al. (2017). The margin size  $\gamma$  is set to 0.1, the balance weight  $\lambda_{1} = 20$ ,  $\lambda_{2} = 40$ .
Table 1 illustrates the results for the En-De Test2016, En-De Test2017, En-Fr Test2016 and En-Fr Test2017 tasks. Our text-only Transformer baseline (Vaswani et al., 2017) has similar results compared to most prior MNMT works, which is consistent with the previous findings (Caglayan et al., 2019), that is, textual modality is good enough to translate for Multi30K dataset. This finding helps to explain that it is already tricky for a MNMT model to ground visual modality even with the presence of annotated images. However, Our ImagiT gains improvements over the text-only Transformer baseline on four evaluation datasets, demonstrating that our model can effectively embed the visual semantics during the training time and guide the translation through imagination with the absence of annotated images during the inference time. We assume much of the performance improvement is due to ImagiT's strong ability to capture the interaction between text and image, generate semantic-consistent visual representations, and incorporate information from visual modality properly.
We also observe that our approach surpasses the results of most MNMT systems by a noticeable margin in terms of BLEU score and METEOR score on four evaluation datasets. Our ImagiT is also competitive with ImagiT + ground truth, which is our translation decoder taking ground truth visual representations instead of imagined ones, and can be regarded as the upper boundary of imagiT. This proves imaginative ability of ImagiT.
Table 2 shows results for the En-De En-Fr Am
ambiguous COCO. For Ambiguous COCO, which was purposely curated such that verbs have ambiguous meaning, demands more visual contribution for guiding the translation and selecting correct words. Our ImagiT benefits from visual imagination and substantially outperforms previous works on ambiguous COCO. and even gets the same performance as ImagiT + ground truth (45.3 BLEU).
The hyper-parameter  $\lambda_{1}$  in equation 15 is important. When  $\lambda_{1} = 0$ , there is no image captioning component, the BLEU score drops from 38.5 to 37.9, while this variant still outperforms the Transformer baseline. This indicates the effectiveness of image captioning module, since it will potentially prevent visual-textual mismatching, thus helps generator achieve better performance. When  $\lambda_{1}$  increases from 5 to 20, the BLEU and METEOR increase accordingly. Whereas  $\lambda_{1}$  is set to equal to  $\lambda_{2}$ , the BLEU score falls to 38.3. That's reasonable because  $\lambda_{2}\mathcal{L}_{trans}$  is the main task of the whole model.
Table 2: Experimental results on the Ambiguous COCO  $\mathrm{{En}} \Rightarrow  \mathrm{{De}}$  and  $\mathrm{{En}} \Rightarrow  \mathrm{{Fr}}$  translation task.
Evaluation metric	BLEU	METEOR
ImagiT, λ1=0	37.9	55.3
ImagiT, λ1=5	38.2	55.5
ImagiT, λ1=10	38.4	55.7
ImagiT, λ1=20	38.5	55.7
ImagiT, λ1=40	38.3	55.6
Table 3: Ablation studies of ImagiT with different weight settings
5.1 Can ImagiT generate visual grounded representations?
Since the proposed model does not require images as input, one may ask how it uses visual informa
tion and where the information comes? We claim that ImagiT has already been embedded with visual semantics during the training phase, and in this section, we validate that ImagiT is able to generate visual grounded representation by performing the image retrieval task.
For each source sentence, we generate the intermediate visual representation. Furthermore, we query the ground truth image features for each generated representation to find the closest image vectors around it based on the cosine similarity. Then we can measure the  $R@K$  score, which computes the recall rate of the matched image in the top K nearest neighborhoods.
R@1	R@5	R@10
ImagiT on Multi30K	64.7	88.7	94.2
ImagiT on MS COCO	64.3	89.5	94.7
Some previous studies on VSE perform sentence-to-image retrieval and image-to-sentence retrieval, but their results can not be directly compared with ours, since we are performing image-to-image retrieval in practical. However, from Table 4, especially for  $R@10$ , the results demonstrate that our generated representation has excellent quality of shared semantics and have been grounded with visual semantic-consistency.
5.2 How does the imagination help the translation?
Although we have validated the effectiveness of ImagiT on three widely used MNMT evaluation datasets. A natural question to ask is that how does the imagination guide the translation, and to which extent? When human beings confronting with complicate sentences and obscure words, we often resort to mind-picturing and mental visualization to assist us to auto-complete and fill the whole imagination. Thus we hypothesis that imagination could help recover and retrieve the missing and implicate textual information.
Inspired by Ive et al. (2019); Caglayan et al. (2019), we apply degradation strategy to the input source language, and feed to the trained Transformer baseline, MNMT baseline, and ImagiT respectively, to validate if our proposed approach could recover the missing information and obtain better performance. And we conduct the analysing
experiments on En-De Test2016 evaluation set.
Color deprivation is to mask the source tokens that refers to colors, and replace them with a special token [M]. Under this circumstance, text-only NMT model have to rely on source-side contextual information and biases, while for MNMT model, it can directly utilize the paired color-related information-rich images. But for ImagiT, the model will turn to imagination and visualization.
Table 4: Image retrieval task. We evaluate on Multi30K and MS COCO.
Model	S	S
text-only Transformer	37.6	36.3
MNMT	38.2	37.7
ImagiT	38.4	37.9
Table 5: Color deprivation.  $s$  represents the original source sentence,while  $\bar{s}$  is the degraded sentence.
Table 5 demonstrates the results of color deprivation. We implement a simple transformer-based MNMT baseline model using the multimodal self-attention approach (Yao and Wan, 2020). Thus the illustrated three models in Table 5 can be compared directly. We can observe that the BLEU score of text-only NMT decreases 1.3, whereas MNMT and ImagiT system only decreases 0.5. This result corroborates that our ImagiT has a similar ability to recover color compared to MNMT, but our ImagiT achieves the same effect through its own efforts, i.e., imagination. One possible explanation is that ImagiT could learn the correlation and cooccurrence of the color and specific entities during the training phase, thus imagiT could infer the color from the context and recover it by visualization.
Visually depictable entity masking. Plummer et al. (2015) extend Flickr30K with cereference chains to tag mentions of visually depictable entities. Similar to color deprivation, we randomly replace  $0\%$ ,  $15\%$ ,  $30\%$ ,  $45\%$ ,  $60\%$  visually depictable entities with a special token [M].
Figure 4 is the result of visually depictable entity masking. We observe a large BLEU score drop of text-only Transformer baseline with the increasing of masking proportion, while MNMT and ImagiT are relatively smaller. This result demonstrates that our ImagiT model can much more effectively infer and imagine missing entities compared to text-only Transformer, and have comparable capability over the MNMT model.
Figure 4: Visually depictable entity masking. From top to bottom is MNMT, ImagiT, text-only transformer.
5.3 Will better imagination with external data render better translation?
Our ImagiT model also accepts external parallel text data or non-parallel image captioning data, and we can easily modify the objective function to train with out-of-domain non-triple data. To train with text-image paired image captioning data, we can pre-train our imagination model by ignoring  $\mathcal{L}_{\text {trans }}$  term (Yang et al., 2020). In other words, the T2I synthesis module can be solely trained with MS COCO dataset. We randomly split MS COCO in half, and use  $\mathrm{COCO}_{\text {half }}$  and  $\mathrm{COCO}_{\text {full }}$  to pretrain ImagiT. The MS COCO is processed using the same pipeline as in Zhang et al. (2017). Furthermore, the training setting of  $\mathrm{COCO}_{\text {half }}$  and  $\mathrm{COCO}_{\text {full }}$  are the same with batch size 64 and maximum epoch 600. The results are:
BLEU	METEOR
ImagiT	38.4	55.7
ImagiT + COCOhalf	38.6	56.3
ImagiT + COCOfull	38.7	56.7
Table 6: Translation results when using out-of-domain non-parallel image captioning data.
As is shown in Table 6, our ImagiT model pretrained with half MS COCO gain 0.6 METEOR increase, and the improvement becomes more apparent when training with the whole MS COCO. We can contemplate that large-scale external data may further improve the performance of ImagiT, and we have not utilized parallel text data (e.g., WMT), even image-only and monolingual text data can also be adopted to enhance the model capability, and we leave this for future work.
This work presents generative imagination-based machine translation model (ImagiT), which can
effectively capture the source semantics and generate semantic-consistent visual representations for imagination-guided translation. Without annotated images as input, out model gains significant improvements over text-only NMT baselines and is comparable with the SOTA MNMT model. We analyze how imagination elevates machine translation and show improvement using external image captioning data. Further work may center around introducing more parallel and non-parallel, text, and image data for different training schemes.
This work brings together text-to-image synthesis, image captioning, and neural machine translation (NMT) for an adversarial learning setup, advancing the traditional NMT to utilize visual information. For multimodal neural machine translation (MNMT), which possesses annotated images and can gain better performance, manual image annotation is costly, so MNMT is only applied on a small and specific dataset. This work tries to extend the applicability of MNMT techniques and visual information in NMT by imagining a semantic equivalent picture and making it appropriately utilized by visual-guided decoder. Compared to the previous multimodal machine translation approaches, this technique takes only sentences in the source languages as the usual machine translation task, making it an appealing method in low-resource scenarios. However, the goal is still far from being achieved, and more efforts from the community are needed for us to get there. One pitfall of our proposed model is that trained ImagiT is not applicable to larger-scale text-only NMT tasks, such as WMT'14, which is mainly related to economies and politics, since those texts are not easy to be visualized, containing fewer objects and visually depictable entities. We advise practitioners who apply visual information in large-scale text-to-text translation to be aware of this issue. In addition, the effectiveness of MNMT model largely depends on the quantity and quality of annotated images, likewise, our model performance also depends on the quality of generated visual representations. We will need to carefully study how the model balance the contribution of different modality and response to ambiguity and bias to avoid undesired behaviors of the learned models.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and S. Frank. 2018. Findings of the third shared task on multimodal machine translation. In WMT.
Ozan Caglayan, Walid Aransa, Adrien Bardet, M. García-Martínez, Fethi Bougares, Loic Barrault, Marc Masana, L. Herranz, and Joost van de Weijer. 2017. Lium-cvc submissions for wmt17 multimodal translation task. *ArXiv*, abs/1707.04481.
Ozan Caglayan, Loic Barrault, and Fethi Bouguares. 2016. Multimodal attention for neural machine translation. ArXiv, abs/1609.03976.
Ozan Caglayan, P. Madhyastha, Lucia Specia, and Loic Barrault. 2019. Probing the need for visual context in multimodal machine translation. In NAACL-HLT.
Iacer Calixto, Desmond Elliott, and S. Frank. 2016. Dcu-uva multimodal mt system report. In WMT.
Iacer Calixto and Qun Liu. 2017. Incorporating global visual features into attention-based neural machine translation. In EMNLP.
Iacer Calixto, Qun Liu, and N. Campbell. 2017. Doubly-attentive decoder for multi-modal neural machine translation. ArXiv, abs/1702.01287.
Iacer Calixto, Miguel Rios, and W. Aziz. 2019. Latent variable model for multi-modal translation. In ACL.
Michael J. Denkowski and A. Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In WMT@ACL.
Desmond Elliott, S. Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. ArXiv, abs/1710.07177.
Desmond Elliott, S. Frank, K. Sima'an, and Lucia Spe- cia. 2016. Multi30k: Multilingual english-german image descriptions. ArXiv, abs/1605.00459.
Desmond Elliott and Ákos Kádár. 2017. Imagination improves multimodal translation. In IJCNLP.
Ian J. Goodfellow, Jean Pouget-Abadie, M. Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS.
Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. Attention-based multimodal neural machine translation. In WMT.
J. Ive, P. Madhyastha, and Lucia Specia. 2019. Distilling translations with visual awareness. In ACL.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Jindrich Libovický and Jindrich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. ArXiv, abs/1704.06567.
Tsung-Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollar, and C. L. Zitnick. 2014. Microsoft coco: Common objects in context. ArXiv, abs/1405.0312.
Hyeonwoo Noh, Seunghoon Hong, and B. Han. 2015. Learning deconvolution network for semantic segmentation. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1520-1528.
Kishore Papineni, S. Roukos, T. Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
Bryan A. Plummer, Liwei Wang, C. Cervantes, Juan C. Caicedo, J. Hockenmaier, and S. Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74-93.
Tingting Qiao, J. Zhang, Duanqing Xu, and D. Tao. 2019. Mirrorgan: Learning text-to-image generation by redescription. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1505-1514.
S. Reed, Zeynep Akata, Xinchen Yan, L. Logeswaran, B. Schiele, and H. Lee. 2016. Generative adversarial text to image synthesis. In ICML.
Lucia Specia, S. Frank, K. Sima'an, and Desmond Eliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In WMT.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Oriol Vinyals, A. Toshev, S. Bengio, and D. Erhan. 2015. Show and tell: A neural image caption generator. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156-3164.
T. Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and X. He. 2018. Attengan: Fine-grained text to image generation with attentional generative adversarial networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1316-1324.
Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Weinan Zhang, Yong Yu, and Lei Li. 2020. Towards making the most of bert in neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9378-9385.
Shaowei Yao and Xiaojun Wan. 2020. Multimodal transformer for multimodal machine translation. In ACL.
Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, J. Zhou, and Jiebo Luo. 2020. A novel graph-based multi-modal fusion encoder for neural machine translation. In ACL.
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Han Zhang, Tao Xu, and Hongsheng Li. 2017. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pages 5908-5916.
Zhuosheng Zhang, Kehai Chen, Rui Wang, M. Utiyama, Eiichiro Sumita, Z. Li, and Hai Zhao. 2020. Neural machine translation with universal visual representation. In ICLR.
Mingyang Zhou, Runxiang Cheng, Y. Lee, and Zhou Yu. 2018. A visual attention grounding neural model for multimodal machine translation. In EMNLP.
Xiaojin Zhu, A. Goldberg, M. Eldawy, C. Dyer, and Bradley Strock. 2007. A text-to-picture synthesis system for augmenting communication. In AAAI.