Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
Year: 2023

Abstract:
One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images. However, recent work in multimodal MT (MMT) has shown that obtaining improvements from images is challenging, limited not only by the difficulty of building effective cross-modal representations, but also by the lack of specific evaluation and training data. We present a new MMT approach based on a strong text-only MT model, which uses neural adapters, a novel guided self-attention mechanism and which is jointly trained on both visually-conditioned masking and MMT. We also introduce CoMMuTE, a Contrastive Multilingual Multimodal Translation Evaluation set of ambiguous sentences and their possible translations, accompanied by disambiguating images corresponding to each translation. Our approach obtains competitive results compared to strong text-only models on standard English→French, English→German and English→Czech benchmarks and outperforms baselines and state-of-the-art MMT systems by a large margin on our contrastive test set. Our code1 and CoMMuTE2 are freely available.

heading: 1 Introduction
text: Multimodal machine translation (MMT) typically refers to the use of additional non-textual data in text-based machine translation (MT). Here, we focus on the case where source texts are accompanied by images, the idea being to exploit visual data to improve the translation of ambiguous sentences. For example, in Figure 1, the English word glasses can either be translated as French verres ‘drinking vessels’ or lunettes ‘spectacles’, an ambiguity which is resolved using the image.
A main research direction of MMT has been how to best exploit image representations and combine
1https://github.com/MatthieuFP/VGAMT 2https://github.com/MatthieuFP/CoMMuTE
the image and text modalities (Yin et al., 2020; Caglayan et al., 2021; Calixto et al., 2017; Li et al., 2022). It has typically been difficult to surpass strong text-only baselines, the image modality often being ignored (Wu et al., 2021). A major issue holding back progress is that most current stateof-the-art MMT models (Yin et al., 2020; Elliott and Kádár, 2017; Wu et al., 2021; Li et al., 2022) are trained solely on the ∼30k examples of the Multi30k dataset (Elliott et al., 2016), comprising image captions and their translations. This causes two issues: (i) the models do not exploit the large amount of text-only data available and therefore perform poorly in comparison to state-of-the-art text-only MT systems, and (ii) we show that very few examples require images to be correctly translated, which means that the datasets are ill-adapted to evaluating the use of the image modality.
In this article, we aim to overcome these problems by proposing (i) a new MMT approach that is able to exploit (text-only) monolingual and parallel data as well as (multimodal) captioning data, and that reaches a good balance between maintaining high MT quality and effectively exploiting images, and (ii) a test set, CoMMuTE, containing contrastive evaluation pairs, where images provide the necessary context to disambiguate between multiple meanings of the same source sentence.
Our suggested model is inspired by work on adapting frozen language models (LMs) to multimodal inputs (Sung et al., 2022; Yang et al., 2022; Eichenberg et al., 2021; Pfeiffer et al., 2022); we
5394
propose to adapt a strong MT model to multimodal inputs with lightweight modules (Houlsby et al., 2019) to exploit the large amount of textual data it was trained on. We also propose to better exploit the image by introducing guided self-attention and by combining the standard MMT objective with a visually-conditioned masked language modelling (VMLM) objective (Li et al., 2019; Lu et al., 2019; Su et al., 2020). Our model obtains competitive results compared to strong text-only baselines on standard En→{Fr,De,Cs} MMT benchmarks (Elliott et al., 2016, 2017; Barrault et al., 2018) and outperforms them and state-of-the-art MMT models on our lexically ambiguous contrastive test set.3

heading: 2 Related Work
text: Multimodal MT data. The reference dataset to train and evaluate MMT models is Multi30k (Elliott et al., 2016). However, recent work has shown that most MMT systems trained and evaluated on it do not effectively exploit the image information; Elliott (2018) showed that replacing the ground truth image with a random one does not lead to the drop in performance that would be expected, while Wu et al. (2021) argued that the observed gain in performance was due to a regularisation effect. It is also notoriously difficult to beat text-only baselines on this benchmark (Barrault et al., 2018). This may be due to (i) some subsets of Multi30k having been translated independently from the images (Elliott et al., 2016) and (ii) most of the time, the source text being sufficient in theory to produce a perfect translation (i.e. the image is not necessary; see Section 5.2 for our own analysis).
Based on this, alternative test sets and evaluation methods have been proposed. Caglayan et al. (2019) proposed to probe the use of images in MMT models, while Li et al. (2021) proposed another training corpus and evaluation benchmark to evaluate MMT systems, but their work is only based on gender ambiguity and requires specific training data to train MMT models. Lala and Specia (2018) released a lexically ambiguous MMT evaluation dataset to evaluate models ability to disambiguate source sentences, but we found that text context is generally sufficient to translate the evaluation dataset correctly.
3CoMMuTE initially contained 50 lexically ambiguous sentences in a previous version of the paper. Results are now computed on the updated version of CoMMuTE comprising 155 ambiguous sentences. Conclusions remain the same.
Contrastive MT datasets. Another means of evaluating (and the one we adopt here) is to target specific phenomena through the use of contrastive test sets. They involve evaluating models based on their ability to rank pairs of translations, where one is correct and the other incorrect. They have been used for the evaluation of different linguistic phenomena, including grammaticality (Sennrich, 2017), multi-sense word disambiguation (Rios Gonzales et al., 2017; Raganato et al., 2019), pronoun translation (Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019) and lexical coherence/consistency (Bawden et al., 2018; Voita et al., 2019). Bawden et al. (2018) introduced the idea of conditioning which of the translations is correct depending on linguistic context, and we adopt the same strategy here with our CoMMuTE dataset, composed of lexically ambiguous sentences whose translations are determined by the visual context.
Adapting pretrained LMs to multimodal inputs. A lot of progress has been made through the use of pretrained LMs (Devlin et al., 2019; Conneau and Lample, 2019; Liu et al., 2020), often trained on raw text for text-only models or image captioning data for multimodal ones (Radford et al., 2021; Alayrac et al., 2022; Chen et al., 2022). One of the most efficient ways to learn multimodal LMs is the visually-conditioned masked language modelling (VMLM) objective (Chen et al., 2020; Lu et al., 2019; Su et al., 2020; Li et al., 2020; Zhou et al., 2021; Huang et al., 2021a; Li et al., 2019). Inspired by the masked language modelling (MLM) objective (Devlin et al., 2019), it consists in randomly masking input text tokens and predicting them conditionally based on the visual features. A lot of interest has also been shown in lightweight modules such as adapters (Houlsby et al., 2019) to adapt large frozen LMs to multimodal tasks (Eichenberg et al., 2021; Yang et al., 2022; Pfeiffer et al., 2022; Tsimpoukelli et al., 2021; Sung et al., 2022) in order to avoid catastrophic forgetting (De Lange et al., 2021). Based on these approaches, we propose to adapt a strong text-only MT model with lightweight modules in order to exploit the large amount of data it previously learned.
Which type of visual features in MMT systems? In terms of how images are represented in multimodal models, different strategies exist. Many works first proposed to incorporate global visual features from object recognition models pretrained
on ImageNet (Deng et al., 2009), such as ResNet50 (He et al., 2016), either in the form of a single vector or a set of features (Calixto et al., 2017; Elliott and Kádár, 2017; Calixto and Liu, 2017; Yao and Wan, 2020; Helcl et al., 2018). More recent global features extractor such as CLIP (Radford et al., 2021) exist, but to our knowledge have not been used in MMT models. Extending this idea, other works focused on entities in the image and extracted bounding boxes using a pretrained Faster R-CNN (Ren et al., 2015) in order to introduce more semantic visual information into MT (Grönroos et al., 2018; Ive et al., 2019; Caglayan et al., 2021). Recent efforts have been made to only select parts of the image that are relevant to the translation of the sentence. Some proposed to use a more selective attention mechanism between modalities (Liu et al., 2021; Ye et al., 2022), while others suggested extracting other types of visual features (Huang et al., 2021b; Fang and Feng, 2022). Based on this, Yin et al. (2020) decided to exploit local image-text correspondences in their model GraphMMT. Similar to their approach, we use a simpler method to extract relevant visual features, using the output queries from a state-of-the-art free-form text object detector MDETR (Kamath et al., 2021) as our local visual features (in addition to global features from CLIP).

heading: 3 Our approach: VGAMT
text: The two main aims of our approach are to (i) exploit a maximum available data (not just multimodal parallel text data) and to (ii) provide an effective way to combine image and text modalities. Our approach, shown in Figure 2, consists in taking a strong text-only MT model4 and adapting it to multimodal MT. To adapt this strong text-only model to multimodal inputs, we add several lightweight modules—bottleneck adapters (Houlsby et al., 2019) and linear visual projection layers—to the otherwise frozen initial model. The bottleneck adapters are lightweight linear layers introduced after each attention block and each feedforward layer to project embeddings down before projecting them up.
In terms of representing visual information, we choose to use two types of representation. We concatenate local (MDETR) features and global
4In practice, our starting point is mBART, which we finetune on a large parallel corpus (see Section 5 for more details).
(CLIP) features to the text inputs. We choose to use global features too, since the source sentence can describe more general aspects of the image than mere objects (such as scenes). We jointly train the non-frozen parts of our model on two distinct objectives: multimodal MT (MMT) and visuallyconditioned masked language modelling (VMLM), as described in Section 3.1. We also introduce a guided self-attention to exploit image information in a straightforward manner (see Section 3.2) in the encoder (while the decoder uses regular self- and cross-attentions and can only attend to embeddings related to text positions). We call our approach Visually Guided and Adapted Machine Translation (VGAMT).

heading: 3.1 Combining training objectives
text: As shown in Figure 2, we jointly train VGAMT on two objectives: visual masked language modelling (VMLM) and multimodal MT (MMT). VMLM (resp. MMT) consists in predicting masked tokens (resp. translating the sentence) conditioned on the image.5 The use of the VMLM objective in addition to MMT ensures that the model does not learn to ignore the visual inputs when translating (since Multi30k is mainly composed of very standard and unambiguous parallel sentences). We make sure to mask a high percentage (25%) of the text inputs so that the model is forced to attend to the image when producing translations.

heading: 3.2 Guided self-attention
text: The backbone of VGAMT is an encoder-decoder MT model, in which image features are concatenated to textual input embeddings and shared selfattention is used over the two input modalities (see Figure 2). Instead of using full self-attention (Caglayan et al., 2021) (connections between all image parts and all text tokens), we introduce guided self-attention. Guided self-attention consists in masking irrelevant connections between text and image representations; each text (resp. image) embedding can attend to itself and all other text (resp. image) positions, but can only attend to image (resp. text) positions conditioned on pre-extracted textimage alignments. We obtain these alignments (in the form of a cross-modal correspondence matrix) using MDETR (Kamath et al., 2021), which detects image regions and corresponding text spans based
5During training, we randomly draw batches from a parallel multimodal dataset (for MMT) and a monolingual multimodal one (for VMLM) with equal probability.
on a free-form text (see Figure 3 and Appendix B for more details).
Concretely, let Q, K and V denote the learnable query, key and value parameters of a standard self-attention mechanism. Attention can be defined as Attention(Q,K, V ) = A · V , where the attention matrix A = (aij) is defined as A = softmax ( QKT / √ dk ) , where dk is the dimension of the key vector, i.e.:
aij = eQiK
T j / √ dk
∑ l e QiKTl / √ dk
(1)
The idea behind our guided self-attention mechanism is that we want to allow subwords to attend to all subwords, all bounding boxes to attend to all bounding boxes, but to only allow cross-modal attention between a subword and bounding boxes that are linked by MDETR (see Figure 3). We therefore define a binary masking matrix C = (cij) where (i) cij = 1 if indices i and j correspond to embeddings coming from the same modality, and (ii) cij is provided by the MDETR matrix otherwise: it is 1 if MDETR has created a link between subword (resp. bounding box) i and bounding box (resp. subword) j. Once this guiding matrix C is defined, we can replace the standard attention (1) with our guided attention:
aij = cije
QiK T j / √ dk
∑ l cile QiKTl / √ dk . (2)
The main advantage of guided self-attention over full self-attention is that the model does not have to learn to ignore irrelevant text-image correspondences since alignments are introduced as a prior.

heading: 4 Contrastive Multilingual Multimodal Translation Evaluation (CoMMuTE)
text: To overcome the flaws of existing benchmarks (see Section 5.2), we introduce CoMMuTE, a Contrastive Multilingual Multimodal Translation Evaluation dataset6. It is composed of 155 lexically ambiguous sentences in English, each associated with two translations corresponding to two of the possible meanings of each sentence and two images that determine which of the translations is correct. It covers English→French, English→German and English→Czech. An example is given in Figure 4.
6CoMMuTE is distributed under Creative Commons Attribution Share Alike 4.0 International license.
Data collection. The test set contains 155 ambiguous sentences constructed around 155 lexically ambiguous words: 29 of the examples are from Bawden et al. (2018), and we created the remaining ones.7 We collected two images for each sentence under Creative Commons license (either Google Images or our own photos), so that the image illustrates without ambiguity one of the two meanings of the sentence. We do not restrict the image-text relation to be strictly descriptive (as for image captions) in order to have a more general evaluation dataset. Each sentence was translated into two possible translations (each corresponding to one of the images) by a native speaker of the target language. Appendix A provides some basic statistics.
The idea of CoMMuTE is to use MMT models to rank each of the two translations based on image information. The perplexity of a sentence for a given model is defined as: PPLq(y) = ∏N i=1 q(yi) − 1 N , where q is the probability distribution output by the model, N is the sequence length and y1, . . . , yN is the sequence of tokens. Now, let y1, . . . , yN1 be the sequence of tokens of the correct translation and y′1, . . . , y ′ N2
the sequence of tokens of the incorrect translation, a model makes a correct prediction if PPLq(y) ≤ PPLq(y′). i.e. the model considers the correct translation more likely than the incorrect one. For each example, we rank each of the translations based on each of the images (2 comparisons per example), and report the accuracy over all the examples. As CoMMuTE is perfectly balanced, a text-only model will get exactly 50% accuracy on this task.

heading: 5 Experiments

heading: 5.1 Text-only data
text: All our experiments are based on the strong MT model mBART8 (Liu et al., 2020), which we finetune on parallel text (see Table 1). We use Open-
7We could not take the entirety of the examples in (Bawden et al., 2018) as some examples were not adapted to disambiguation using visual (as opposed to linguistic) context.
8mBART is pretrained on CC25 (Wenzek et al., 2020).
Subtitles20189 (Lison et al., 2018), Wikipedia (Wołk and Marasek, 2014), Ted Talks (Reimers and Gurevych, 2020) and the Books datasets (Tiedemann, 2012). We preprocess the data using Moses scripts (Koehn et al., 2007).10

heading: 5.2 Multimodal data
text: Multi30k. We train our frozen MT model on the Multi30k dataset (Specia et al., 2016; Elliott et al., 2016) composed of English sentences, each accompanied by an image and French, German and Czech translations. It contains 29k train, 1014 dev and 1000 test examples (Test2016). Elliott et al. (2017) and Barrault et al. (2018) released two additional related test sets (Test2017 and Ambiguous Coco). However, on analysis of these sets and as shown in Table 2, we found that very few examples are image-dependent (i.e. the source sentence is ambiguous and the image is required to solve the ambiguity in the target language),11 meaning that an MMT system is unlikely to perform better than a text-only system. Moreover, most of these ambiguities are semantically similar and they only cover a few multi-sense words. Although Ambiguous Coco (Elliott et al., 2017) is designed to be an
9http://www.opensubtitles.org 10remove-non-printing-char.pl, normalization-
punctuation.pl, and clean-corpus-n.pl (4-100 tokens). 11We queried WordNet (Fellbaum, 1998) for all nouns and verbs in the English sentences. An example was considered image-dependent if, on manual assessment, there were multiple meanings, which were (i) compatible with the text context and (ii) could be disambiguated using the image.
ambiguous test set as it is built around multi-sense verbs, it was automatically created from sentences from MSCOCO (Lin et al., 2014) for which the textual context is often sufficient for disambiguation. These benchmarks remain useful to make sure MMT systems do not perform worse than text-only MT models on examples where images are not necessary to translate correctly. However, we consider them insufficient to assess how well MMT systems exploit images to improve translation.
Monolingual multimodal data. For the VMLM objective, we train our model on the Conceptual Captions (CC) dataset (Sharma et al., 2018) composed of 3.3M12 images aligned with English text.

heading: 5.3 Implementation details
text: For all our experiments, we use the mBART implementation from Hugging Face (Wolf et al., 2020). Experiments with adapters used bottleneck adapters (Houlsby et al., 2019) with a reduction factor of 8 and ReLU activation (Agarap, 2018). We use the implementation provided by adaptertransformers (Pfeiffer et al., 2020). We use a batch size of 512, the Adam optimiser (Kingma and Ba, 2014) with β1 = 0.9, β2 = 0.99 and a learning rate of 10−4 for En→Fr and 10−5 for En→{De,Cs}. We also applied 0.1 label smoothing (Szegedy et al., 2016) during training. We selected our final model according to the best BLEU score (Papineni et al., 2002) on the Multi30k dev set after at least one full pass over the Multi30k and Conceptual Captions training sets. We ran each experiment 3 times with different seeds and report the average BLEU13 (Papineni et al., 2002) and COMET (Rei et al., 2020) scores14 and the standard errors. We also report METEOR scores (Banerjee and Lavie, 2005) in Appendix E. All experiments were carried out on 8 NVIDIA V100 GPUs for ∼15h.

heading: 5.4 Baselines
text: We consider several text-only and multimodal baselines. All baselines except the MT models finetuned from mBART were trained from scratch with the original codebases and features released by the papers’ authors. Models trained on the (multimodal) MT objective only where trained on
12At the time of writing, we were able to collect ∼2M images and trained models on this subset.
13computed with the default parameters of sacreBLEU v2.0.0 from https://github.com/mjpost/sacrebleu.
14Using the wmt20-comet-da model.
Multi30k, while models jointly trained on the (multimodal) MT and (V)MLM objectives were trained on Multi30k and Conceptual Captions.
Text-only. We trained a text-only Seq2seq Transformer (Vaswani et al., 2017) from scratch and a text-only Seq2Seq Transformer initialised from TLM weights (Conneau and Lample, 2019). We refer to these models as Vanilla MT and TLM + MT respectively. We also trained several MT models initialised from pretrained mBART (Liu et al., 2020) and which we fine-tuned on parallel data (Lison et al., 2018; Wołk and Marasek, 2014). We refer to these models as mBART + MT. ‘w/ adapters’ specifies that the model’s weights are frozen except bottleneck adapters (Houlsby et al., 2019).
Multimodal. We trained several state-of-the-art multimodal MT models: Graph-MMT (Yin et al., 2020), Gated Fusion (Wu et al., 2021) and a Seq2Seq Transformer trained from VTLM weights (Caglayan et al., 2021) (hereafter VTLM + MMT).

heading: 5.5 Results and Analysis
text: Tables 3 and 4 show BLEU, COMET and accuracy scores for all models compared on several En→{Fr,De,Cs} test sets including CoMMuTE. An initial observation is that the text-only model is a strong baseline on the three standard benchmarks (Test2016, Test2017 and MSCOCO). As mentioned in Section 5.2, most of these evaluation datasets do not need visual context to be correctly translated. Our model VGAMT is on average on par with its counterpart text-only mBART+MT w/ adapters baseline for all Multi30k En→Fr test sets, while being on average just below this baseline on En→{De,Cs} Multi30k benchmarks. It outperforms other MMT models with a large margin due to both the effective use of textual knowledge from
the frozen MT model but also guided self-attention. Note that the scores reported for the baselines are lower than the ones reported in the original papers of the models for several reasons. First, we computed the scores on fully detokenised data to have a uniform evaluation between all models. We also report the average score from three different runs using different seeds and not the best score obtained over a single run.
More importantly, our VGAMT obtains strong improvements over both text-only baselines and state-of-the-art MMT systems on CoMMuTE; our model can use visual context to disambiguate sentences. This can be seen in Figure 5 (one of the
En→Cs Test2016 Test2018 CoMMuTE
Model BLEU COMET BLEU COMET Accuracy
Text-only Machine Translation
Vanilla MT* 31.3 ±0.0 0.593 ±0.008 26.0 ±0.2 0.379 ±0.008 50.0 TLM + MT* 32.6 ±0.1 0.642 ±0.002 26.8 ±0.2 0.432 ±0.006 50.0 mBART + MT* 32.1 0.865 29.6 0.747 50.0
w/ adapters 37.3 ±0.1 0.940 ±0.005 35.2 ±0.4 0.876 ±0.002 50.0
Multimodal Machine Translation
ambiguous examples from Multi30k), where in contrast to the baseline VGAMT produces the correct translation and Figure 6 (from CoMMuTE), where VGAMT correctly ranks the two translations. More examples are provided in Appendix D. We also propose to translate CoMMuTE source sentences and compare against the reference translations; the results are shown in Appendix F.

heading: 6 Ablation Study
text: To better understand the role of VGAMT’s components, we carry out several ablations for En→Fr and report all results in Table 5.
Adapters versus Fine-tuning. We compare the results of fine-tuning an unfrozen VGAMT model (w/o adapters) in comparison to our frozen model with adapters (VGAMT), all other things remaining equal. The unfrozen version faces a drop in
scores on all test sets except Test2017. Notably, the unfrozen model’s accuracy score of 60.5 on CoMMuTE is 6.6 points lower than our final VGAMT model. As well as providing a more lightweight solution that does not involve fine-tuning all parameters, using neural adapters and freezing other weights is useful in terms of performance.
Impact of the VMLM objective. To evaluate the impact of jointly training with MMT and VMLM objectives, we train a model on the MMT without VMLM (and therefore without monolingual multimodal data). The MMT model trained on MMT alone obtains 52.0 on CoMMuTE, compared to 67.1 for joint training, showing that VMLM helps our model to better exploit disambiguating images.
Guided self-attention. We study the impact of guided self-attention between modalities by comparing against classic full self-attention. Guided self-attention obtains better results than full selfattention, particularly on Test2017 and MSCOCO (+0.8 BLEU, +0.015 COMET on average). It also gets better results on CoMMuTE (+2.5 points). See Appendix C for analysis of guided attention scores.
VMLM and MMT joint training. We compare our VMLM and MMT joint training with disjoint training where VGAMT is first pretrained on VMLM then fine-tuned on MMT instead of cotraining on both VMLM and MMT. Table 5 shows that it results in a large drop of performance on all scores in average including 3.8 points on CoMMuTE.
MDETR. We examine the impact of MDETR features by training a model without them.15 The results without MDETR features are slightly lower
15More details are available in Appendix B.
than the full model on standard MMT benchmarks. However, the results are significantly lower on CoMMuTE (63.0±1.2 without MDETR features and 67.1±0.7 with MDETR features). This means that VGAMT benefits from MDETR features when disambiguating and translating sentences.
CLIP. We also study the impact of CLIP features by training a model without them.15 Including CLIP features gives slightly higher results on standard MMT benchmarks (+0.69 BLEU and +0.007 COMET scores on average on all benchmarks). VGAMT without CLIP features faces an extreme drop on CoMMuTE (50.3±0.00 w/o CLIP features vs. 67.1±0.7 w/ CLIP features), which shows that CLIP features are required for disambiguation.
VMLM sampling probability and degree of masking. We ran experiments to vary the VMLM sampling probability (see Section 3.1) and the percentage of masked text inputs (see Figure 7 for results on CoMMuTE). For the sampling between VMLM and MMT objectives, the maximum value is reached for p =50%, i.e. equal sampling between VMLM and MMT objectives (Figure 7a). Similar results are obtained for p = 75%, i.e. 3 VMLM batches for 1 MMT batch, but the translation quality is lower. For the percentage of masking, there is a peak at 25% masked text inputs and a constant decrease for higher values (Figure 7b).

heading: 7 Conclusion
text: We propose a new MMT approach (VGAMT) based on (i) adapting a strong text-only MT model with lightweight adapters and (ii) introducing better use of the text and image modalities through a novel guided self-attention mechanism and joint MMT and VMLM training. We also introduce
VMLM sampling probability
C oM
M uT
E a
cc ur
ac y
0.25 0.50 0.75
66.0
62.0
58.0
(a) Variation of VMLM sampling probability with fixed 25% masked text inputs.
% masked text 0.15 0.25 0.50 0.75
C oM
M uT
E a
cc ur
ac y 68.0
60.0
64.0
56.0
(b) Variation of masked text inputs with fixed 50% VMLM sampling probability.
Figure 7: CoMMuTE Results comparing multiple VMLM sampling probabilities and percentage of masked text inputs. 95% confidence interval in grey.
CoMMuTE, a contrastive test set designed to test the use of visual disambiguating context. Results for En→{Fr,De,Cs} show that VGAMT obtains competitive results compared with strong text-only baselines on standard benchmarks and widely outperforms these baselines and state-of-the-art MMT systems on CoMMuTE.
Limitations
In this work, we focused on En→{Fr,De,Cs} multimodal MT. At the time of writing, our method can only be applied for En→X MMT. It is indeed necessary to have access to a modulated object detector in the source language to extract the features and the image-text relationship exploited by our model. This type of modulated object detector is only available in English for the moment. We leave the extension of our method to non-English source languages to future work. Moreover, our method requires large amount of captioning data to perform well. It is therefore computationally expensive.

heading: Acknowledgements
text: This work was granted access to the HPC resources of IDRIS under the allocation 2022-AD011013908 and 2022-AD011012254 made by GENCI. It was also partly funded by the last four authors’ chairs in the PRAIRIE institute funded by the French national agency ANR as part of the “Investissements d’avenir” programme under the reference ANR-19P3IA-0001.

heading: A CoMMuTE statistics
text: Some basic statistics of the CoMMuTE dataset can be found in Table 6. The source side of the dataset is always English and two translations of each of the 155 English ambiguous sentences are provided in French, German and Czech.
B Visual features
We use MDETR (Kamath et al., 2021) features as our local visual features. Concretely, we extract the set of output queries features of size 64 from the MDETR decoder and introduce them as input.
In addition, we use CLIP (Radford et al., 2021) features as our global visual features. More specifically, we extract the output [CLS] features of size 512 from the ViT (Dosovitskiy et al., 2021) image encoder used by CLIP and introduced it as input.

heading: C Guided self-attention analysis
text: We studied the values of the cross-modal part of our guided self-attention. To do so, we followed the method proposed by Kobayashi et al. (2020) who showed that raw attention scores α are meaningless and instead proposed to conduct analysis on the normalised attention scores ∥αf∥, where α are the raw attention scores and f is the value vector in the attention mechanism. Figure 9b shows the cross-modal part of the guided self-attention map from the example displayed in Figure 9a where all the values have been averaged over all heads and all layers. In this example, the English word fans ‘cooling device or ardent admirer’ is ambiguous and the two meanings have different translations in
French, German and Czech. Given the region-text couples extracted by MDETR (Figure 9a), only the token fans can attend to the MDETR region features. The normalised attention scores of the embedding of the token fans on these regions are low in comparison to the scores on the text part and on the CLIP embedding. On the contrary, all embeddings can attend to CLIP embedding and the embedding of the token fans is the one with the highest normalised attention score with CLIP embedding.

heading: D Additional examples
text: Figure 12 shows examples from CoMMuTE and the perplexity scores obtained by VGAMT. It is able to choose the correct translations from English sentences with the ambiguous words chips, bugs, red light. However, it fails to choose the correct translation in the first case of Figure 12d; the picture shows a beam ‘ray of light’ and the perplexity of the correct (top) translation with the French translation rayon is higher than the incorrect (bottom) one with the French translation poutre. Nevertheless, the model gives a lower perplexity to the sentence with the correct image (1.847) in comparison to the same sentence with the incorrect image (2.616). So, even if VGAMT is not able to choose the correct translation in the first case of this example, it shows some evidence of being able to discriminate between the French translation with the correct image and the same French translation with the incorrect image. Figures 12e and 12f show two other similar examples in En→De MT.
In terms of translation (rather than reranking), Figure 8 shows an example from Multi30k where our model correctly translates the ambiguous word while the text-only baseline fails to do so.
Model Test2016 Test2017 MSCOCO
Text-only Machine Translation
mBART + MT* w/ adapters 79.9 ±0.3 76.0 ±0.2 69.5 ±0.6 w/o MLM objective 80.3 ±0.2 76.3 ±0.2 68.7 ±0.3
Multimodal Machine Translation
fans
The fans were still out of order.Cap:
Img:
MDETR
fans fans
(a) English sentence, its associated image and the region-text couples extracted by MDETR.
The
fans were still out
of order
.
Th e fa ns w er e
st ill
ou t of or de r . <e os > [E N ] C LI P
0.6
0.5
0.4
0.3
0.2
0.1
0.0
(b) Normalised attention scores from the cross-modal part of our guided self-attention mechanism. Only the embeddings of the token ‘fans’ can attend to MDETR features. All the embeddings can attend to CLIP features.
Figure 9: Guided self-attention map for the English sentence The fans were still out of order and its associated image. The sentence is ambiguous as English fan refers to ‘a cooling device’ or ‘an ardent admirer’. Values are averaged over all heads and all layers. <eos> refers to the end of sentence token.

heading: E METEOR scores
text: In order to compare to previous work, we also provide METEOR scores in Table 7 for En→{Fr,De,Cs} standard benchmarks. It confirms that VGAMT obtains competitive results over a strong text-only baseline on benchmarks where images are not necessary for translation. METEOR scores for the En→Fr ablations conducted in Section 6 are shown in Table 8.

heading: F Translating CoMMuTE
text: CoMMuTE is designed as a contrastive test set to be used for reranking. However, it is possible to translate the source sentences too and compare against the reference translations.
Table 9 shows the MT results on CoMMuTE comparing VGAMT and the strong text-only baseline. They may indicate that traditional metrics for MT task are ill-adapted to evaluating the use of visual information by MMT models. For instance, BLEU and METEOR scores for the text-only baseline are significantly higher than the scores for our model VGAMT on the En→Fr split whereas our
VGAMT obtains 67.10 accuracy on the contrastive evaluation (Table 3). It might be due to the fact that such metrics are less reliable on small datasets or that BLEU and METEOR are words matching metrics and therefore output low scores for synonyms or similar content described differently. On the other hand, COMET is an embedding-based metric, which outputs higher scores for synonyms which may be why VGAMT outperforms the textonly baseline with this metric; as illustrated by Figure 10 where VGAMT outputs noeud ‘bow’ which is a synonym of the reference translation ruban ‘bow’ in that case. That being said, the use of our contrastive dataset CoMMuTE therefore seems necessary to evaluate how well a MMT model exploits visual information in order to produce correct translations instead of relying only on standard metrics
for MT. Figure 10 illustrates how VGAMT can translate ambiguous words correctly by using images, while mBART + MT (our strong text-only baseline) cannot. In both cases, the baseline outputs French noeud papillon ‘bow tie’, while VGAMT produces the correct translations of bow. Figures 11a to 11f show the same effect for En→{Cs,De} translations. Even if VGAMT does not literally translate the ambiguous word as exemplified by Figure 11b, it produces a translation with the expected meaning based on the image; the text-only models were not able to do so.
1 2
ř
ř
1 2
(a) English word mole correctly translated in both cases (znaménko ‘skin blemish’ and krtka ‘burrowing mammal’).
1
2
1 2
(b) English word brush correctly translated in both cases (Bürste ‘cleaning tool’ and Pinsel ‘object used for painting’).
1 2
1
2
✓
✓
(a) The English word chips refers to ‘french fries’ or ‘poker chips’.
1
2
✓
✓
1 2
(b) The English word bugs refers to ‘a problem in a computer program’ or ‘a small insect’.
1
2 ✓
✓
21
(c) The English phrase red light refers to ‘a traffic signal that instructs moving vehicles to stop’ or ‘light that is red’.
1
2
X
✓
1 2
(d) The English word beam refers to ‘a ray of light’ or ‘a piece of timber or metal used to support the roof’.
1
2 ✓
✓
1 2
(e) The English word pen refers to ‘an instrument for writing or drawing with ink’ or ‘an area of land surrounded by a fence’.
1
2 ✓
1 2
✓
(f) The English word tube refers to ‘a long cylinder for holding liquids or gases.’ or ‘a railway system in some cities’.
Figure 12: Perplexity scores from VGAMT on different examples from CoMMuTE. It is possible to produce at least two different French translations from each source sentence in English, the correct translation therefore depends on the input image. For each sub-example, the correct (resp. incorrect) translation is the top (resp. bottom) one. The ambiguous parts of the sentences are highlighted in bold.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Last section ("Limitations", no number)
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 4

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 4

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4, Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: 4, 5.2

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 4
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We did human annotations ourselves.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We did human annotations ourselves.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No ethics concerns involved
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? As we did it ourselves, it could give information to reviewers.

heading: Future Work
text: We leave the extension of our method to non-English source languages to future work. Moreover, our method requires large amount of captioning data to perform well. It is therefore computationally expensive