SlowGuess/ACL_2020_OCR huggingface dataset:
We use Transformer as the underlying architecture for the translation and captioning modules. Each encoder/decoder of the translator is with 6-layer stacked Transformer network, 8 heads, 1024 hidden units, and 4096 feed-forward filter size. The captioner is a 6-layer Transformer decoder with the same configuration. The visual encoder is a 1-layer MLP which maps visual feature to the shared 1,024-dim embedding space then adds the positional encoding to encode spatial locations (normalized top-left and bottom-right coordinates) of visual objects. Our implementation is based on the codebase of XLM and MASS.
We respectively conduct unsupervised MMT experiments on Multi30K-half for two language pairs: English-French and English-German.
Pre-Training Pre-training is a critical step for unsupervised MT. We follow the setup in UMMT (Su et al., 2019) for a fair comparison. For each language, we create a text-only pre-training set by combining the shuffled first 10 million sentences of the WMT News Crawl datasets from 2007 to 2017 with 10 times of M30k-half, resulting in a text-only dataset with 10.145 million unparalleled sentences in English, French, German respectively.
For text pre-training, we leverage the script and the masked seq-to-seq objective proposed in
8230
MASS, which randomly masks a span in a sentence then encourages the model to decode and reconstruct the masked sequence as the monolingual language model pre-training. More details can be found in the original paper. Note that there is no fine-tuning (back-translation) on WMT for a fair comparison with other baselines.
For multimodal pre-training of the captioning modules, we use the out-of-domain MS-COCO (Lin et al., 2014) dataset. We randomly split the training set into two disjoint subsets. Each set contains 56,643 images and 283,215 sentences. We use the translate-train strategy as in XNLI (Connieu et al., 2018b). We leverage Google Translate to translate one set of English sentences into French and German. We pre-train the captioning modules with Eq. 9 and fix them during fine-tuning to avoid overfitting. Note that the captioning modules are trained on non-parallel sentences with disjoint image subsets, which implies no overlap between English-German or English-French sentences.
Fine-tuning on Multi30K-half We fine-tune on the training set of Multi30K-half for 18 epochs. We train our model with the Adam optimizer (Kingma and Ba, 2014) with a linear warm-up and a learning rate varying from  $10^{-7}$  to  $10^{-5}$ . We apply a linearly decreasing weight from 1.0 to 0.1 at 10-th epoch for  $w^{CPT}$  as we empirically observe that the generated captions are relatively too noisy to serve as good pseudo pairs in the later stage of training. The margin  $\gamma$  in VSE is set to 0.1. Other hyper-parameters in Transformer follow the default setting in MASS. We use 4 Titan Xp GPUs with 1,000 tokens in each mini-batch for training.
Evaluation and Model selection For evaluation, we report BLEU scores by multi-bleu.pl in Moses and METEOR² scorea on the Multi30K testing set.
For model selection without a parallel validation corpus, we consider the unsupervised criterion proposed in (Lample et al., 2018a) based on the BLEU scores of "round-trip" translations (source  $\rightarrow$  target  $\rightarrow$  source and target  $\rightarrow$  source  $\rightarrow$  target) which have been empirically shown to correlate well with the testing metrics.
4.4 Baseline Models
We compare recent unsupervised text-only and multimodal MT baselines listed in the following: (1) MUSE (Conneau et al., 2018a) is a word-to-word
MT model with pre-trained Wikipedia embeddings. (2) UNMT (Lample et al., 2018a) sets the tone of using denoising autoencoder and back-translation for unsupervised MT. (3) XLM (Conneau and Lample, 2019) deploys masked language model from BERT. (4) MASS (Song et al., 2019) uses a masked seq-to-seq pre-training objective, achieves the current state-of-the-art performance in text-only unsupervised MT. (5) Game-MMT (Chen et al., 2018) is a reinforcement learning-based unsupervised MMT. (6) UMMT (Su et al., 2019) use visual feature for denoising autoencoder and back-translation. UMMT is the current state of the art in unsupervised MMT. We either use the reported scores in the original papers or use their best scripts with their pre-trained language models publicly available for fine-tuning on Multi30K-half.
4.5 Main Results: Unsupervised MMT
4.5.1 Comparison with the Baseline Models
Table 1 presents the benchmark results with other state-of-the-art unsupervised MT and MMT models on the Multi30K testing set. The first four rows show the results of the recent text-only MT models. Game-MMT and UMMT are MMT models using both image and text inputs. Our full model  $(\mathrm{T} + \mathrm{V} + \mathrm{VSE} + \mathrm{CBT} + \mathrm{CPT})$  yields new state-of-the-art performance in BLEU and METEOR, outperforming the text-only and multimodal baseline model by a large margin. Notably, our full model outperforms UMMT by  $+5.5\sim 12.5$  BLEU scores, sets a new state of the art in unsupervised MMT.
Although pre-training plays a vital role in unsupervised MT, comparing Ours-Text only and Ours-Full, the results suggest that multimodal content can further boost the performance for unsupervised MT. Images provide  $+2.7\sim 3.7$  BLEU score improvement across four tasks. Note that our model uses different monolingual pre-training corpora to MASS and XLM for the fair comparison with UMMT. With a similar pre-training objective, our text-only model is worse than MASS, while Ours-Full outperforms MASS by  $+2.3\sim 3.7$  in BLEU.
Comparing the multimodal models trained with and without visual content (UMMT-T vs. UMMT-Full and Ours-T vs. Ours-Full), our model achieves  $+2.5\sim 3.7$  improvements in BLEU while  $+1.4\sim 2.5$  for UMMT. The results imply that, even with a higher text-only baseline (e.g. 49.5 vs. 37.2 in en  $\rightarrow$  fr), the proposed model incorporates visual
<sup>1</sup>https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl
$^{2}$ https://github.com/cmu-mtlab/metror
8231
en→fr	fr→en	en→de	de→en
Model	BLEU	METEOR	BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
MUSE† (Conneau et al., 2018a)	8.5	-	16.8	-	15.7	-	5.4	-
UNMT† (Lample et al., 2018a)	32.8	-	32.1	-	22.7	-	26.3	-
XLM† (Conneau and Lample, 2019)	46.3	64.3	42.0	38.1	27.4	48.7	30.7	31.0
MASS† (Song et al., 2019)	49.8	65.8	43.7	38.7	30.2	51.3	32.5	33.4
Game-MMT (Chen et al., 2018)	-	-	-	-	16.6	-	19.6	-
UMMT-T† (Su et al., 2019)	37.2	33.7*	38.5	36.4	21.0	25.4*	25.0	28.4
UMMT-Full (Su et al., 2019)	39.8	35.5*	40.5	37.2	23.5	26.1*	26.4	29.7
Ours-Text only†	49.5	65.7	43.5	38.5	30.1	51.5	32.4	33.0
Ours-Full	52.3	67.6	46.0	39.8	33.9	54.1	36.1	34.7
content more effectively.
In Figure 3, we provide some qualitative results on the Multi30K testing set. We observe a consistent improvement of unsupervised translation quality with our full model to the text-only one. Without parallel translation pairs as the vital supervision, the proposed pseudo visual pivoting successfully disambiguates the word semantics in the similar syntactic category and results in improved cross-lingual word alignment; for instance, "cafe" vs. "soda" machine in the third French example, and "felsigen" (rocky) vs. "verschneiten" (snowy) in the first German example.
4.5.2 Ablation Studies
To quantify module-wise contribution in pseudo visual pivoting, we summarize our ablation studies in Table 2. Comparing the performance improvement from text-only to the model with regional visual features  $(\mathrm{T} + \mathrm{V})$ , the features of salient visual objects contribute  $+0.6 \sim 0.9$  BLEU score over a much higher text-only baseline compared to UMMT.
In pseudo visual pivoting, +VSE promotes the alignments in the monolingual VSE spaces and results in an additional  $+1.3\sim 2.0$  gain in BLEU. This improvement validates our hypothesis that the visual space can effectively serve as the bridge connecting the source and target language latent spaces. Also, synthesizing image-pivoted pseudo caption pairs effectively provides weak supervision for aligning the cross-lingual latent space in unsupervised MMT. We observe that the pivoted captions for paired translation (CPT) is more effective than treating them as back-translation pairs (CBT). Utilizing generated image-pivoted captions is shown to be a promising approach for weakly supervised
Table 1: Results on unsupervised MT. Comparison with benchmarks on the Multi30K testing set. Our full model is with  $\mathrm{T + V + VSE + CBT + CPT}$ . The best score is marked bold.  $\dagger$  means text-only. \* is the METEOR score shown in the UMMT paper.
Model (Ours)	en→fr	fr→en	en→de	de→en
Text only	49.52	43.48	30.10	32.35
T+V	50.43	44.10	31.01	32.95
T+V+VSE	51.72	45.73	32.67	34.94
T+V+CPT	51.64	45.55	33.04	35.02
T+V+CBT	51.23	45.21	32.51	33.87
T+V+VSE+CBT	51.81	45.83	33.01	34.38
T+V+CPT+CBT	51.85	45.65	33.61	35.85
T+V+VSE+CPT	52.19	46.10	33.73	35.60
Full Model	52.29	45.98	33.85	36.07
Table 2: Ablation studies. BLEU comparison of different training objectives.
or unsupervised MMT. The full model which employs VSE, CBT, and CPT achieves  $+1.9 \sim 3.1$  improvements compared to our multimodal baseline (row two, visual feature only).
4.5.3 Generalizability
How does our unsupervised MMT model generalize when images are not available at the testing time? Table 3 shows the testing results without images. As can be observed, our model generalizes well. The differences are mostly less than 1.0 in BLEU. As our model, when being tested without visual content, still outperforms other unsupervised text-only or multimodal MT models listed in Table 1, the minor drop in BLEU implies that the improved cross-lingual latent space alignment via pseudo visual pivoting is likely to be more critical than using images as an input feature for decoding. Luckily, such alignment is already preserved in the training phase with the proposed approach.
An interesting question is: How much does the visual content (as a feature) contribute? As in leave-one-feature-out cross-validation, we compare
T: un jeune garcon se tient sur un chariot de vêtements.
T+V: un jeune garcon s'apos accroche à un poteau de vêtements
GT: un jeune garçon s'apos accroche à un portant.
SRC: a young boy is hanging onto a clothing rack.
(a) English  $\rightarrow$  French
T: un chat assis sur le sommet d'apos un magasin de vêtements
T+V: un chat est assis sur un panneau de magasin.
GT: un chat est assis sur une enseigne de magasin.
SRC: a cat sits on top of a store sign.
Figure 3: Qualitative results of the proposed model. GT: ground truth. T+V: Our full model.
T: deux garçons en train de faire une machine à café.
T+V: deux garçons devant une machine à soda
GT: deux garçons devant une machine à soda.
SRC: two boys in front of a soda machine.
T: ein mann und eine junge auf einem verschreiben strand.
T+V: ein mann und ein junge auf einem felsigen strand
GT: ein mann und ein junge auf einem felsigen strand
SRC: a man and a boy on a rocky beach.
(b) English  $\rightarrow$  German
T: mann springt mit einem felsbrocken im hintergrund.
T+V: mann springt vor einer felsformation im hintergrund in die luft
GT: mann springt vor einer felsformation im hintergrund.
SRC man jumping with a rock formation in background.
T: zwei manner spielten gitarre im freien.
T+V: zwei Männer spielten gitarre vor einem große publikum.
GT: zwei Männer玩游戏 gitarre vor einem großen publikum.
SRC: two men playing guitar in front of a large audience.
Model	en→fr	fr→en	en→de	de→en
UMMT	39.44-0.35	40.30-0.23	23.18-0.34	25.47-0.92
Ours-no VSE	51.60-0.25	45.39-0.26	33.25-0.36	35.15-0.70
Ours-Full	51.64-0.65	45.48-0.50	33.32-0.53	35.04-1.03
the difference of performance between referencing with and without images. The larger the difference (the subscripts in Table 3) implies a model better utilizes visual content. Compared with UMMT, our model has better utilization. We observe that the key to such difference is the VSE objective. Our model trained without the VSE objective results in worse utilization (smaller difference at the testing time), possibly because the source text-image pairs are distant in the multilingual VSE space.
4.5.4 Real-pivoting & Low-resource Corpora
Will our model benefit from "real" pivoting (src-img<sub>1</sub>, img<sub>1</sub>-tgt, overall src-img<sub>1</sub>-tgt)? We train our models with overlapped images while leaving sentences in the source and target languages unparalleled (use no translation pairs). From the first three rows in Table 4, the performance is improved when training with the overlapped images and their corresponding sentences. Comparing the improvement from  $0\%$  to  $100\%$  of the text-only model and the full model, a larger gain is observed with the proposed pseudo visual pivoting which aligns and reduces uncertainty in the language latent spaces.
Furthermore, under the low-resource setting (3.0K non-parallel data, row six and seven), a substantial improvement over the text-only model is still observed. These results suggest that the proposed pseudo visual pivoting is likely to generalize to the semi-supervised and the low-resource setting, which we consider as our future work.
Table 3: BLEU with text-only inputs at the testing time. Subscripts are the differences to testing with  $\mathrm{T + V}$
Img overlap % (# imgs/sents)	en→fr	fr→en	en→de	de→en
0% (14.5K/14.5K)	52.29	45.98	33.85	36.07
50% (22K/22K)	55.13	47.54	34.61	37.01
100% (29K/29K)	58.34	50.57	35.45	38.55
0% (T only/14.5K)	49.52	43.48	30.10	32.35
100% (T only/29K)	53.35	46.27	31.35	34.06
0% (3.0K/3.0K)	31.48	27.91	23.94	26.60
0% (T only/3.0K)	30.33	26.95	21.65	23.47
Table 4: Testing BLEU of the full T+V model and the text-only model trained with overlapped images or low-resource unpaired corpora.
4.5.5 Supervised Case
Although the proposed pseudo visual pivoting targets unsupervised MMT, we are also interested in its performance under the fully supervised setup. To gain insights, we conduct supervised MMT experiments by changing the back-translation objective for unsupervised MT (Eq. 6) to the supervised MT objective (Eq. 2) with additional visual inputs. We benchmark with recent supervised MMT models, including Imagination (Elliott and Kádár, 2017), LIUM-CVC (Caglayan et al., 2017), and VAG (Zhou et al., 2018) on Multi30K.
Table 5 shows the testing results. Our model significantly outperforms other baselines and achieves state-of-the-art performance. Comparing to the unsupervised model trained with full Multi30K (Table  $4,100\%$  (29K/29K)), the direct supervision from parallel translation pairs results in a  $+6.5\sim7.1$  gain in BLEU. Notably, images provide a minor improvement with full supervision from translation pairs. This result implies that, compared to serving as a complementary feature, visual information likely contributes more to improving crosslingual alignment via pseudo visual pivoting for MMT with limited supervision.
8233
en→fr	en→de
Model	BLEU	METEOR	BLEU	METEOR
Imagination	-	-	30.2	51.2
LIUM-CVC	52.7	69.5	30.7	52.2
VAG	53.8	70.3	31.6	52.2
Ours (T)	65.2	79.3	42.0	60.5
Ours (T+V)	65.5	79.1	42.3	60.6
Table 5: Supervised MMT results on Multi30K
Unsupervised MT For pivoting with a third language, First et al. (2016) pre-train a multi-way multilingual model to generate pseudo pairs to improve zero-shot translation. Chen et al. (2017) use a teacher-student framework and assume parallel sentences share a similar likelihood for generating sentences in the third language while Cheng et al. (2017) maximize the expected likelihood. Our model does not rely on a third language. Our framework is along the line of research in (Lample et al., 2018a,b; Conneau and Lample, 2019), which aims at learning an aligned latent space between the two languages to translate by reconstruction. Nevertheless, we focus on the multimodal setup where the visual space is dissimilar to the language spaces with challenging asymmetric interactions between modalities.
Supervised MMT Supervised MMT is introduced in (Specia et al., 2016) as a multi-encoder single-decoder framework with additional image inputs. Huang et al. (2016) encode word sequences with regional visual objects while Calixto and Liu (2017) leverage global visual feature. LIUM-CVC (Caglayan et al., 2017) uses element-wise multiplication to model the image-text interaction. Imagination (Elliott and Kádár, 2017) and VAG (Zhou et al., 2018) learns with the auxiliary image reconstruction and source-image-target triplet alignment tasks, respectively. While these methods achieve improvements, their advantage over the text-only models is still minor under the supervised scenario. As analyzed in (Caglayan et al., 2019), visual content is more critical when the textual content is limited or uncertain in MMT. We study the more challenging unsupervised MMT.
Unsupervised MMT To our best knowledge, three recent works have generalized MMT to the unsupervised setting. Nakayama and Nishida (2017) learn modal-agnostic fixed length image/sentence embeddings. In contrast, our model promotes fine-grained (object-token) varying-length embedding,
which better aligns VSE space. Game-MMT (Chen et al., 2018) use a captioning and a translation model maximizing the likelihood of translated captions to original sentences. We synthesize captions for symmetric back-translation and considers no ground truth image annotation in the loop. Empirically, it is preferred to separate real and generated captions. UMMT (Su et al., 2019) uses Transformers, autoencoder loss, and multimodal back-translation. We do not use autoencoder. Our model leverages object detection for multimodal back-translation and equips pseudo visual pivoting. Image Captioning and VSE Our method draws inspiration from captioning and cross-modal retrieval. Recent progress in captioning aims at using reinforcement learning to improve diversity (Dai et al., 2017) or maximize metric (Rennie et al., 2017). We use a vanilla MLE objective. For learning VSE, we leverage the contrastive loss (Kiros et al., 2014) from cross-modal retrieval, which is shown more robust than maximizing canonical correlation among modalities as in (Andrew et al., 2013; Huang et al., 2018). For encoding image and text, we generalize the cross-modality attention from SCAN (Lee et al., 2018) to the multilingual scenario for learning a multilingual VSE space (Gella et al., 2017; Huang et al., 2019a).
We have presented a novel approach: pseudo visual pivoting for unsupervised multimodal MT. Beyond features, we use visual content to improve the cross-lingual alignments in the shared latent space. Precisely, our model utilizes the visual space as the approximate pivot for aligning the multilingual multimodal embedding space. Besides, it synthesizes image-pivoted pseudo sentences in two languages and pairs them to translate by reconstruction without parallel corpora. The experiments on Multi30K show that the proposed model generalizes well and yields new state-of-the-art performance.
This work is supported by the DARPA grants funded under the AIDA program (FA8750-18-2-0018), the LWLL program (FA8750-18-2-0501), and the GAILA program (award HR0011990063). Xiaojun Chang is supported by Australian Research Council Discovery Early Career Award (DE190100626). The authors would like to thank the anonymous reviewers for their suggestions and Google Cloud for providing the research credits.
8234
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013. Deep canonical correlation analysis. In International Conference on Machine Learning, pages 1247-1255.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes Garcia-Martinez, Fethi Bougares, Loic Barrault, Marc Masana, Luis Herranz, and Joost Van de Weijer. 2017. Lium-cvc submissions for wmt17 multimodal translation task. In SECOND CONFERENCE ON MACHINE TRANSLATION, volume 2, pages 432-439.
Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loic Barrault. 2019. Probing the need for visual context in multimodal machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4159-4170, Minneapolis, Minnesota. Association for Computational Linguistics.
Iacer Calixto and Qun Liu. 2017. Incorporating global visual features into attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 992-1003.
Yun Chen, Yang Liu, Yong Cheng, and Victor O.K. Li. 2017. A teacher-student framework for zero-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1925-1935, Vancouver, Canada. Association for Computational Linguistics.
Yun Chen, Yang Liu, and Victor OK Li. 2018. Zeroresource neural machine translation with multiagent communication game. In Thirty-Second AAAI Conference on Artificial Intelligence.
Yong Cheng, Qian Yang, Yang Liu, Maosong Sun, and Wei Xu. 2017. Joint training for pivot-based neural