Venue: NAACL 2019

# Probing the Need for Visual Context in Multimodal Machine Translation

Ozan Caglayan LIUM, Le Mans University ozan.caglayan@univ-lemans.fr &Pranava Madhyastha Imperial College London pranava@imperial.ac.uk Lucia Specia Imperial College London l.specia@imperial.ac.uk &Loic Barrault LIUM, Le Mans University loic.barrault@univ-lemans.fr

###### Abstract

Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial. We posit that this is a consequence of the very simple, short and repetitive sentences used in the only available dataset for the task (Multi30K), rendering the source text sufficient as context. In the general case, however, we believe that it is possible to combine visual and textual information in order to ground translations. In this paper we probe the contribution of the visual modality to state-of-the-art MMT models by conducting a systematic analysis where we partially derive the models from source-side textual context. Our results show that under limited textual context, models are capable of leveraging the visual input to generate better translations. This contradicts the current belief that MMT models disregard the visual modality because of either the quality of the image features or the way they are integrated into the model.

## 1 Introduction

Multimodal Machine Translation (MMT) aims at designing better translation systems which take into account auxiliary inputs such as images. Initially organized as a shared task within the First Conference on Machine Translation (WMT16) [25], MMT has so far been studied using the Multi30K dataset [8], a multilingual extension of Flickr30K [47] with translations of the English image descriptions into German, French and Czech [8, 9].

The three editions of the shared task have seen many exciting approaches that can be broadly categorized as follows: (i) multimodal attention using convolutional features [1, 10, 11, 12] (ii) cross-modal interactions with spatially-unaware global features [10, 11, 12, 13] and (iii) the integration of regional features from object detection networks [14, 15]. Nevertheless, the conclusion about the contribution of the visual modality is still unclear: Gronroos et al. [16] consider their multimodal gains "modest" and attribute the largest gain to the usage of external parallel corpora. Lala et al. [9] observe that their multimodal word-sense disambiguation approach is not significantly different than the monomodal counterpart. The organizers of the latest edition of the shared task concluded that the multimodal integration schemes explored so far resulted in marginal changes in terms of automatic metrics and human evaluation [9]. In a similar vein, Elliott [16] demonstrated that MMT models can translate without significant performance losses even in the presence of features from unrelated images.

These empirical findings seem to indicate that images are ignored by the models and hint at the fact that this is due to representation or modeling limitations. We conjecture that the most plausible reason for the linguistic dominance is that - at least in Multi30K - the source text is sufficient to perform the translation, eventually preventing the visual information from intervening in the learning process. To investigate this hypothesis, we introduce several input degradation regimes (Section 2) and revisit state-of-the-art MMT models (Section 3) to assess their behavior under degraded regimes. We further probe the visual sensitivity by deliberately feeding features from unrelated images. Our results (Section 4) show that MMT models successfully exploit the visual modality when the linguistic context is scarce, but indeed tend to be less sensitive to this modality when exposed to complete sentences

Input Degradation

In this section we propose several degradations to the input language modality to simulate conditions where sentences may miss crucial information. We denote a set of translation pairs by \({\cal D}\) and indicate degraded variants with subscripts. Both the training and the test sets are degraded.

Color Deprivation.We consistently replace _source_ words that refer to colors with a special token [v] (\({\cal D}_{\text{C}}\) in Table 1). Our hypothesis is that a monomodal system will have to rely on source-side contextual information and biases, while a multimodal architecture could potentially capitalize on color information extracted by exploiting the image and thus obtain better performance. This affects 3.3% and 3.1% of the words in the training and the test set, respectively.

Entity Masking.The Flickr30K dataset, from which Multi30K is derived, has also been extended with coreference chains to tag mentions of _visually depictable_ entities in image descriptions Plummer et al. (2015). We use these to mask out the head nouns in the _source_ sentences (\({\cal D}_{\text{N}}\) in Table 1). This affects 26.2% of the words in both the training and the test set. We hypothesize that a multimodal system should heavily rely on the images to infer the missing parts.

Progressive Masking.A _progressively_ degraded variant \({\cal D}_{k}\) replaces all but the first \(k\) tokens of _source_ sentences with [v]. Unlike the color deprivation and entity masking, masking out suffixes does not guarantee systematic removal of visual context, but rather simulates an increasingly low-resource scenario. Overall, we form 16 degraded variants \({\cal D}_{k}\) (Table 1) where \(k\in\{0,2,\dots,30\}\). We stop at \({\cal D}_{30}\) since 99.8% of the sentences in Multi30K are shorter than 30 words with an average sentence length of 12 words. \({\cal D}_{0}\) - where the only remaining information is the source sentence length - is an interesting case from two perspectives: a neural machine translation (NMT) model trained on it resembles a target language model, while an MMT model becomes an image captioner with access to "expected length information".

Visual Sensitivity.Inspired by Elliott (2018), we experiment with _incongruent decoding_ in order to understand how sensitive the multimodal systems are to the visual modality. This is achieved by explicitly violating the test-time semantic congruence across modalities. Specifically, we feed the visual features in _reverse_ sample order to break image-sentence alignments. Consequently, a model capable of integrating the visual modality would likely deteriorate in terms of metrics.

## 3 Experimental Setup

Dataset.We conduct experiments on the English\(\rightarrow\)French part of Multi30K. The models are trained on the concatenation of the _train_ and _val_ sets (30K sentences) whereas _test2016 (dev)_ and _test2017 (test)_ are used for early-stopping and model evaluation, respectively. For _entity masking_, we revert to the default Flickr30K splits and perform the model evaluation on _test2016_, since _test2017_ is not annotated for entities. We use word-level vocabularies of 9,951 English and 11,216 French words. We use Moses Koehn et al. (2007) scripts to lowercase, normalize and tokenize the sentences with hyphen splitting. The hyphens are stitched back prior to evaluation.

Visual Features.We use a ResNet-50 CNN He et al. (2016) trained on ImageNet Deng et al. (2009) as image encoder. Prior to feature extraction, we center and standardize the images using ImageNet statistics, resize the shortest edge to 256 pixels and take a center crop of size 256x256. We extract spatial features of size 2048x8x8 from the final convolutional layer and apply L\({}_{2}\) normalization along the depth dimension Caglayan et al. (2018). For the non-attentive model, we use the 2048-dimensional global average pooled version pool5) of the above convolutional features.

Models.Our baseline NMT is an attentive model Bahdanau et al. (2014) with a 2-layer bidirectional GRU encoder Cho et al. (2014) and a 2-layer conditional GRU decoder Sennrich et al. (2017). The second layer of the decoder receives the output of the attention layer as input.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \({\cal D}\) & **a** & **lady** & **in** & **a** & **blue** & **dress** & **singing** \\ \hline \({\cal D}_{\text{C}}\) & a & lady & in & a & [v] & dress & singing \\ \({\cal D}_{\text{N}}\) & a & [v] & in & a & blue & [v] & singing \\ \hline \({\cal D}_{4}\) & a & lady & in & a & [v] & [v] & [v] \\ \({\cal D}_{2}\) & a & lady & [v] & [v] & [v] & [v] & [v] \\ \({\cal D}_{0}\) & [v] & [v] & [v] & [v] & [v] & [v] & [v] \\ \hline \hline \end{tabular}
\end{table}
Table 1: An example of the proposed input degradation schemes: \({\cal D}\) is the original sentence.

 

For the MMT model, we explore the basic multimodal attention (DIRECT) Caglayan et al. (2016) and its hierarchical (HIER) extension Libovicky and Helcl (2017). The former linearly projects the concatenation of textual and visual context vectors to obtain the multimodal context vector, while the latter replaces the concatenation with another attention layer. Finally, we also experiment with encoder-decoder initialization (INIT) Calixto and Liu (2017); Caglayan et al. (2017) where we initialize both the encoder and the decoder using a non-linear transformation of the pool5 features.

Hyperparameters.The encoder and decoder GRUs have 400 hidden units and are initialized with 0 except the multimodal INIT system. All embeddings are 200-dimensional and the decoder embeddings are tied Press and Wolf (2016). A dropout of 0.4 and 0.5 is applied on source embeddings and encoder/decoder outputs, respectively Srivastava et al. (2014). The weights are decayed with a factor of \(1e{-}5\). We use ADAM Kingma and Ba (2014) with a learning rate of \(4e{-}4\) and mini-batches of 64 samples. The gradients are clipped if the total norm exceeds 1 Pascanu et al. (2013). The training is early-stopped if _dev_ set METEOR Denkowski and Lavie (2014) does not improve for ten epochs. All experiments are conducted with _nmtpytorch1_Caglayan et al. (2017).

Footnote 1: github.com/lium-lst/nmtpytorch

## 4 Results

We train all systems three times each with different random initialization in order to perform significance testing with _multeval_Clark et al. (2011). Throughout the section, we always report the mean over three runs (and the standard deviation) of the considered metrics. We decode the translations with a beam size of 12.

We first present _test2017_ METEOR scores for the baseline NMT and MMT systems, when trained on the full dataset \(\mathcal{D}\) (Table 2). The first column indicates that, although MMT models perform slightly better on average, they are not significantly better than the baseline NMT. We now introduce and discuss the results obtained under the proposed degradation schemes. Please refer to Table 5 and the appendix for qualitative examples.

### Color Deprivation

Unlike the inconclusive results for \(\mathcal{D}\), we observe that all MMT models are significantly better than NMT when color deprivation is applied (\(\mathcal{D}_{\text{C}}\) in Table 2). If we further focus on the subset of the test set subjected to color deprivation (247 sentences), the gain increases to 1.6 METEOR for HIER. For the latter subset, we also computed the average color accuracy per sentence and found that the attentive models are 12% better than the NMT (32.5\(\rightarrow\)44.5) whereas the INIT model only brings 4% (32.5\(\rightarrow\)36.5) improvement. This shows that more complex MMT models are better at integrating visual information to perform better.

### Entity Masking

The gains are much more prominent with entity masking, where the degradation occurs at a larger scale: Attentive MMT models show up to 4.2 METEOR improvement over NMT (Figure 1). We observed a large performance drop with _incongruent decoding_, suggesting that the visual modality is

\begin{table}
\begin{tabular}{c c c} \hline \hline  & \(\mathcal{D}\) & \(\mathcal{D}_{C}\) \\ \cline{2-3} NMT & 70.6 \(\pm\) 0.5 & 68.4 \(\pm\) 0.1 \\ INIT & 70.7 \(\pm\) 0.2 & **68.9**\(\pm\) 0.1 \\ HIER & 70.9 \(\pm\) 0.3 & **69.0**\(\pm\) 0.3 \\ DIRECT & 70.9 \(\pm\) 0.2 & **68.8**\(\pm\) 0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Baseline and color-deprivation METEOR scores: bold systems are significantly different from the NMT system within the same column (\(p\)-value \(\leq 0.03\)).

Figure 1: Entity masking: all masked MMT models are significantly better than the masked NMT (dashed). Incongruent decoding severely worsens all systems. The vanilla NMT baseline is 75.9\({}^{2}\).

 

now much more important than previously demonstrated (Elliott, 2018). A comparison of attention maps produced by the baseline and masked MMT models reveals that the attention weights are more consistent in the latter. An interesting example is given in Figure 2 where the masked MMT model attends to the correct region of the image and successfully translates a dropped word that was otherwise a spelling mistake ("son"\(\rightarrow\)"song").

**Czech and German.** In order to understand whether the above observations are also consistent across different languages, we extend the _entity masking_ experiments to German and Czech parts of Multi30K. Table 3 shows the gain of each MMT system with respect to the NMT model and the subsequent drop caused by incongruent decoding3. First, we see that the multimodal benefits clearly hold for German and Czech, although the gains are lower than for French4. Second, when we compute the average drop from using incongruent images across all languages, we see how conservative the INIT system is (\(\downarrow\) 4.7) compared to HIER (\(\downarrow\) 6.1) and DIRECT (\(\downarrow\) 6.8). This raises a follow-up question as to whether the hidden state initialization eventually loses its impact throughout the recurrence where, as a consequence, the only modality processed is the text.

Footnote 3: For example, the INIT system for French (Figure 1) surpasses the baseline (50.5) by reaching 53.9 (+3.4), which ends up at 47.4 (\(\downarrow\) 6.5) after incongruent decoding.

Footnote 4: This is probably due to the morphological richness of DE and CS which is suboptimally handled by word-level MT.

### Progressive Masking

Finally, we discuss the results of the progressive masking experiments for French. Figure 3 clearly shows that as the sentences are progressively degraded, all MMT systems are able to leverage the visual modality. When the multimodal task becomes image captioning at \(k{=}0\), MMT models improve over the language-model counterpart by \(\sim\)7 METERO. Further qualitative examples show that the systems perform surprisingly well by producing visually plausible sentences (see Table 5 and the Appendix).

To get a sense of the visual sensitivity, we pick the DIRECT models trained on four degraded variants and perform _incongruent decoding_. We notice that as the amount of linguistic information increases, the gap narrows down: the MMT system gradually becomes less perplexed by the incongruence or, put in other words, less sensitive to the visual modality (Table 4).

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{3}{c}{\(\mathcal{D}_{4}\)} & \(\mathcal{D}_{6}\) & \(\mathcal{D}_{12}\) & \(\mathcal{D}_{20}\) & \(\mathcal{D}\) \\ \cline{2-6} DIRECT & **32.3** & **42.2** & **64.5** & **70.1** & **70.9** \\ Incongruent Dec. & \(\downarrow\) 6.4 & \(\downarrow\) 5.5 & \(\downarrow\) 1.4 & \(\downarrow\) 0.7 & \(\downarrow\) 0.7 \\ \hline Blinding & \(\downarrow\) 3.9 & \(\downarrow\) 2.9 & \(\downarrow\) 0.4 & \(\downarrow\) 0.5 & \(\downarrow\) 0.3 \\ NMT & \(\downarrow\) 3.7 & \(\downarrow\) 2.6 & \(\downarrow\) 0.6 & \(\downarrow\) 0.2 & \(\downarrow\) 0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The impact of incongruent decoding for _progressive masking_: all METEOR differences are against the DIRECT model. The blinded systems are both trained and decoded using incongruent features.

Figure 3: Multimodal gain in absolute METEOR for _progressive masking_: the dashed gray curve indicates the percentage of non-masked words in the training set.

\begin{table}
\begin{tabular}{r r l l} \hline \hline  & \multicolumn{3}{c}{\(+\) Gain (\(\downarrow\) Incongruence Drop)} \\ \cline{2-4}  & INIT & HIER & DIRECT \\ Czech & \(+\)1.4 (\(\downarrow\) 2.9) & \(+\)1.7 (\(\downarrow\) 3.5) & \(+\)1.7 (\(\downarrow\) 4.1) \\ German & \(+\)2.1 (\(\downarrow\) 4.7) & \(+\)2.5 (\(\downarrow\) 5.9) & \(+\)2.7 (\(\downarrow\) 6.5) \\ French & \(+\)3.4 (\(\downarrow\) 6.5) & \(+\)3.9 (\(\downarrow\) 9.0) & \(+\)4.2 (\(\downarrow\) 9.7) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Entity masking results across three languages: all MMT models perform significantly better than their NMT counterparts (\(p\)-value \(\leq 0.01\)). The incongruence drop applies on top of the MMT score.

 

We then conduct a contrastive "blinding" experiment where the DIRECT models are not only fed with incongruent features at decoding time but also trained with them from scratch. The results suggest that the blinded models learn to ignore the visual modality. In fact, their performance is equivalent to NMT models.

## 5 Discussion and Conclusions

We presented an in-depth study on the potential contribution of images for multimodal machine translation. Specifically, we analysed the behavior of state-of-the-art MMT models under several degradation schemes in the Multi30K dataset, in order to reveal and understand the impact of textual predominance. Our results show that the models explored are able to integrate the visual modality if the available modalities are complementary rather than redundant. In the latter case, the primary modality (text) sufficient to accomplish the task. This dominance effect corroborates the seminal work of Colavita (1974) in Psychophysics where it has been demonstrated that visual stimuli dominate over the auditory stimuli when humans are asked to perform a simple audiovisual discrimination task. Our investigation using source degradation also suggests that visual grounding can increase the robustness of machine translation systems by mitigating input noise such as errors in the source text. In the future, we would like to devise models that can learn when and how to integrate multiple modalities by taking care of the complementary and redundant aspects of them in an intelligent way.

## Acknowledgments

This work is a follow-up on the research efforts conducted within the "Grounded sequence-to-sequence transduction" team of the JSALT 2018 Workshop. We would like to thank Jindrich Libovicky for contributing the hierarchical attention to _nmtpytorch_ during the workshop. We also thank the reviewers for their valuable comments. Ozan Caglayan and Loic Barrault received funding from the French National Research Agency (ANR) through the CHIST-ERA M2CR project under the contract ANR-15-CHR2-0006-01. Lucia Specia and Pranava Madhyastha received funding from the MultiMT (H2020 ERC Starting Grant No. 678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects.

## References

* Bahdanau et al. [2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _Computing Research Repository_, arXiv:1409.0473. Version 7.
* Brendel et al. [2015]

\begin{table} SRC: an older woman in [ 

Loic Barrault, Fethi Bougares, Lucia Specia, Chiaraz Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 308-327, Belgium, Brussels. Association for Computational Linguistics.
* Caglayan et al. (2017) Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes Garcia-Martinez, Fethi Bougares, Loic Barrault, Marc Masana, Luis Herranz, and Joost van de Weijer. 2017a. LIUM-CVC submissions for WMT17 multimodal translation task. In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 432-439, Copenhagen, Denmark. Association for Computational Linguistics.
* Caglayan et al. (2018) Ozan Caglayan, Adrien Bardet, Fethi Bougares, Loic Barrault, Kai Wang, Marc Masana, Luis Herranz, and Joost van de Weijer. 2018. LIUM-CVC submissions for WMT18 multimodal translation task. In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 603-608, Belgium, Brussels. Association for Computational Linguistics.
* Caglayan et al. (2016) Ozan Caglayan, Loic Barrault, and Fethi Bougares. 2016. Multimodal attention for neural machine translation. _Computing Research Repository_, arXiv:1609.03976.
* Caglayan et al. (2017b) Ozan Caglayan, Mercedes Garcia-Martinez, Adrien Bardet, Walid Aransa, Fethi Bougares, and Loic Barrault. 2017b. NMTPY: A flexible toolkit for advanced neural machine translation systems. _Prague Bull. Math. Linguistics_, 109:15-28.
* Calixto et al. (2016) Iacer Calixto, Desmond Elliott, and Stella Frank. 2016. DCU-UvA multimodal MT system report. In _Proceedings of the First Conference on Machine Translation_, pages 634-638, Berlin, Germany. Association for Computational Linguistics.
* Calixto and Liu (2017) Iacer Calixto and Qun Liu. 2017. Incorporating global visual features into attention-based neural machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 992-1003, Copenhagen, Denmark. Association for Computational Linguistics.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
* Volume 2_, HLT '11, pages 176-181, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Colavita (1974) Francis B. Colavita. 1974. Human sensory dominance. _Perception & Psychophysics_, 16 (2):409-412.
* Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In _2009 IEEE Conference on Computer Vision and Pattern Recognition_, pages 248-255.
* Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 376-380. Association for Computational Linguistics.
* Elliott (2018) Desmond Elliott. 2018. Adversarial evaluation of multimodal machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2974-2978, Brussels, Belgium. Association for Computational Linguistics.
* Elliott et al. (2017) Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 215-233, Copenhagen, Denmark. Association for Computational Linguistics.
* Elliot et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. Multi30K: Multilingual english-german image descriptions. In _Proceedings of the 5th Workshop on Vision and Language_, pages 70-74, Berlin, Germany. Association for Computational Linguistics.
* Gronroos et al. (2018) Stig-Arne Gronroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Meraldo, Phu Pham, Mats Sjoberg, Umut Sulubacak, Jorg Tiedemann, Raphael Torney, and Raul Vazquez. 2018. The MeMAD submission to the WMT18 multimodal translation task. In _Proceedings of the Third Conference on Machine Translation_, pages 609-617, Belgium, Brussels. Association for Computational Linguistics.
* He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 770-778.
* Helcl et al. (2018) Jingfrich Helcl, Jindrich Libovicky, and Dusan Varis. 2018. CUNI system for the WMT18 multimodal translation task. In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 622-629, Belgium, Brussels. Association for Computational Linguistics.

 

* Huang et al. (2016) Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016. Attention-based multimodal neural machine translation. In _Proceedings of the First Conference on Machine Translation_, pages 639-645, Berlin, Germany. Association for Computational Linguistics.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _Computing Research Repository_, arXiv:1412.6980.
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions_, ACL '07, pages 177-180, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Lala et al. (2018) Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, and Lucia Specia. 2018. Sheffield submissions for WMT18 multimodal translation shared task. In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 630-637, Belgium, Brussels. Association for Computational Linguistics.
* Libovicky and Helcl (2017) Jindrich Libovicky and Jindrich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 196-202, Vancouver, Canada. Association for Computational Linguistics.
* Ma et al. (2017) Mingbo Ma, Dapeng Li, Kai Zhao, and Liang Huang. 2017. OSU multimodal machine translation system report. In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 465-469, Copenhagen, Denmark. Association for Computational Linguistics.
* Madhyastha et al. (2017) Pranava Swaroop Madhyastha, Josiah Wang, and Lucia Specia. 2017. Sheffield MultiMT: Using object posterior predictions for multimodal machine translation. In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 470-476, Copenhagen, Denmark. Association for Computational Linguistics.
* Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In _Proceedings of the 30th International Conference on Machine Learning_, volume 28 of _Proceedings of Machine Learning Research_, pages 1310-1318, Atlanta, Georgia, USA. PMLR.
* Plummer et al. (2015) B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In _2015 IEEE International Conference on Computer Vision (ICCV)_, pages 2641-2649.
* Press and Wolf (2016) Ofir Press and Lior Wolf. 2016. Using the output embedding to improve language models. _Computing Research Repository_, arXiv:1608.05859.
* Sennrich et al. (2017) Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Hadlow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Laubli, Antonio Valerio Miceli Barone, Jozer Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In _Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics_, pages 65-68. Association for Computational Linguistics.
* Specia et al. (2016) Lucia Specia, Stella Frank, Khalil Sima'an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In _Proceedings of the First Conference on Machine Translation_, pages 543-553, Berlin, Germany. Association for Computational Linguistics.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _J. Mach. Learn. Res._, 15(1):1929-1958.
* Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. _Transactions of the Association for Computational Linguistics_, 2:67-78.

## Appendix A Qualitative Examples

In this appendix, we provide further translation examples for color deprivation (Table 6), entity masking (Table 7) and progressive masking (Table 8). Specifically for the entity masking experiments, we also give further examples to showcase the behavior of the visual attention in Figure 4 and Figure 5 



\begin{table}
\begin{tabular}{l l l}  & SRC: & a girl in [v] is sitting on a bench \\ NMT: & pink \\ Init: & pink \\ Hier: & black \\ Direct: & black \\ \end{tabular}
\end{table}
Table 6: Color deprivation examples from the English\(\rightarrow\)French models: bold indicates correctly predicted cases. The colors generated by the models are shown in English for the sake of clarity.



* [v]: a [v] in a red [v] plays in the [v] NMT: un gargcon en t-shirt rouge joue dans la neige (a boy in a red t-shirt plays in the snow) MMT: un garcon en **maillot de bain** rouge joue dans l'eau** REF: un garcon en **maillot de bain** rouge joue dans l'eau** (a boy in a red swimsuit plays in the water)
* SRC: a [v] drinks [v] outside on the [ v] NMT: un homme boit du vin dehors sur le trottoir (a man drinks wine outside on the sidewalk) MMT: un **chien** boit de l'eau** debors sur **l'herbe** REF: un **chien** boit de l'eau** dehors sur **l'herbe** (a dog drinks water outside on the grass)
* SRC: two [v] are driving on a [v] NMT: deux hommes font du velo sur une route (two men riding bicycles on a road) MMT: deux **volures roulent sur une piste** (two cars driving on a track/circuit) REF: deux **voitness roulent sur** un circuit
* SRC: a [v] turns on the [v] to pursue a flying [v] NMT: un homme tourne sur la plage pour attriber un frisbee volant (a man turns on the beach to catch a flying frisbee) MMT: un **chien** tournre sur **l'herbe** pour attraper un frisbee valant (a dog turns on the grass to catch a flying frisbee) REF: un **chien** tourne sur **l'herbe** pour posuivreve une balle en l'air (a dog turns on the grass to chase a ball in the air)
* SRC: a [v] jumping [v] on a [v] near a parking [v] NMT: un homme sautant a cheval sur une plage pres d'un parking (a man jumping on a beach near a parking lot) MMT: une **fille** sautant a la **corde** sur un **trottoir** pres d'un parking REF: une **fille** sautant a la **corde** sur un **trottoir** pres d'un parking (a girl jumping rope on a sidewalk near a parking lot)

\begin{table} \end{table}
Table 7: Entity masking examples from the English\(\rightarrow\)French models: underlined and bold words highlight bad and good lexical choices, respectively. English translations are provided in parentheses. MMT is an attentive model.

 

* [25] M. C. Bader, J. D. Kwiatkowski, and A. S. Zebrook. The role of the mr-based mr- 


* [145] M. C. Bader, J.-M. Lee, and S.-J. Kim, "A new approach to the design of a 

* [v][v][v][v][v][v]
* [v][v][v]
* [v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]
* [v][v][v][v]