Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation
Authors: Wenyu Guo
Qingkai Fang
Dong Yu
Yang Feng
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal Machine Translation
Text-to-image Generation

Abstract: Multimodal machine translation (MMT) simultaneously takes the source sentence and a relevant image as input for translation. Since there is no paired image available for the input sentence in most cases, recent studies suggest utilizing powerful text-to-image generation models to provide image inputs. Nevertheless, synthetic images generated by these models often follow different distributions compared to authentic images. Consequently, using authentic images for training and synthetic images for inference can introduce a distribution shift, resulting in performance degradation during inference. To tackle this challenge, in this paper, we feed synthetic and authentic images to the MMT model, respectively. Then we minimize the gap between the synthetic and authentic images by drawing close the input image representations of the Transformer Encoder and the output distributions of the Transformer Decoder. Therefore, we mitigate the distribution disparity introduced by the synthetic images during inference, thereby freeing the authentic images from the inference process. Experimental results show that our approach achieves state-of-the-art performance on the Multi30K En-De and En-Fr datasets, while remaining independent of authentic images during inference.

Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation
Wenyu Guo Qingkai Fang Dong Yu², Yang Fengl.2* 1.2+
Key Laboratory of Intelligent Information Processing
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
University of Chinese Academy of Sciences, Beijing, China Beijing Language and Culture University, China xk guuowenyuu 26. com, yudong@blcu edu. cn (fangqingkai21b, fengyang)@ict... ac.
Abstract
Multimodal machine translation (MMT) simul- taneously takes the source sentence and a rele- vant image as input for translation. Since there is paired image available for the input sen- tence in most cases, recent studies suggest uti- lizing powerful text-to-image generation mod- els to provide image inputs. Nevertheless, syn- thetic images generated by these models often follow different distributions compared thentic images. Consequently, using authentic images for training and synthetic images for inference can introduce distribution shift, re- sulting in performance degradation during in- ference, tackle this challenge, in this paper, we feed synthetic and authentic images the MMT model, respectively. Then we minimize the gap between the synthetic and authentic im- ages by drawing close the input image represen- lations of the Transformer Encoder and out- put distributions of the Transformer Decoder. Therefore, we mitigate the distribution dispar- ity introduced by the synthetic images during inference, thereby freeing the authentic images from the inference process. Experimental re- sults show that our approach achieves state-of- the-art performance on the Multi30K En-De and En-Fi Fr datasets, while remaining indepen- dent of authentic images during inference.
O
Cur
FLATE
Synthetic
Authentic
src: on horseback man attempls to rope young bull
example of the synthetic and authentic images of the same source sentence.
Figure I:
the dependence images at inference stands as pivotal challenge in MMT.
To tackle this challenge, some researchers have proposed to employ text-to-image generation mod- els (Long et al., 2021; Li et al., 2022b) to generate synthetic images to use as the associated image for the source text at inference, while during train- ing MMT models still use the available authentic images the visual context to generate transla- tion. Despite the remarkable capabilities of text- to-image generation models in generating highly realistic images from textual descriptions (Ramesh et al., 2022; Rombach et al., 2022), the synthetic images may exhibit different distribution patterns compared to the authentic (ground truth) images, As shown in Figure 1, the synthetic images may depict counterfactual scenes, omit information in the text, add information irrelevant from the text. Therefore, training with authentic images but infer- ring with synthetic images causing the distribution shift in the images, leading to a decline in transla- tion performance.
1 Introduction
Multimodal machine translation (MMT) integrates visual information into neural machine translation (NMT) to enhance language understanding with vi- sual context, leading to improvement translation quality (Li et al., 2022a; Guo et al., 2022; Fang and Feng, 2022). However, most existing MMT models require associated image with the input sentence, which is difficult satisfy inference and limits the usage scenarios. Hence, overcoming
In this paper, we embrace the utilization of text- to-image generation models in MMT and propose method to bridge the gap between synthetic and authentic images. In our method, also intro- duce synthetic images during training and feed syn- thetic images and authentic images to the MMT model, respectively. Then we minimize the gap
*Corresponding author: Yang Feng.
Code is publicly available at https: http:://gttttuuueee com/ ictn1p/SANMT.

---

between the images by drawing close the fol- lowing two terms of the MMT model when the two images are inputted respectively: the input image representations to the Transformer Encoder and the output distributions from the Transformer Decoder. Regarding the input representations, we leverage the Optimal Transport (OT) theory to mitigate the disparity of the representations. (KI.) divergence is utilized ensure the consis- tency of the output distributions. Consequently, we effectively eliminate the disparities introduced by synthetic images used during the inference pro- cess, thereby freeing the authentic images from the inference process.
zt refer the time step and the latent representa- tion at the t-th time step. & represents the VAE encoder. eo and т represent the U-Net and the CLIP text encoder, respectively. In this paper, we utilize the pre-trained Stable Diffusion model to generate images for the source sentence.
2.2 Multimodal Transformer
Multimodal Transformer (Yao and Wan, 2020) is powerful architecture designed for MMT. It re- places the self-attention layer in the Transformer (Vaswani et al., 2017) Encoder with a multimodal self-attention layer, which can learn multimodal representations from text representations under the guidance of the image- aware attention.
Experimental results show that approach achieves state-of-the-art performance on the Multi3KK En-De and En-Fr datasets, even in the ab- sence of authentic images. Further analysis demon- strates that our approach effectively enhances the representation and prediction consistency between synthetic and authentic images.
Formally, given the input text x and image a, the textual and visual representations can be denoted as H (hf: ....... (,)) and H (h.....,,h h2.). re- spectively. Here, N represents the length of the source sentence, and P denotes the size of visual features. In each multimodal self-attention layer, the textual and visual representations are concate- nated together as the query vectors:
2 Background
2.1 Stable Diffusion
H = H": H"W") C (N++)xxd
(2)
Stable Diffusson (Rombach al., 2022) is a text- to-image generation model based on the latent dif- fusion model (LDM), enabling the generation of
where W are learnable weights. The key and value vectors are the textual representations H highly realistic images from textual descriptions. mainly consists of VAE model, a U-Net model and CLIP text encoder.
Finally, the output of the multimodal self-attention layer is calculated as follows:
The training of the conditional LDM involves a forward diffusion process and backward de- noising process. During the diffusion process, the images are firstly compressed into the latent space using the VAE encoder, followed by the addition of Gaussian noise to the latent representations. In the denoising process, the U-Net is iteratively em- ployed to progressively eliminate noise from the noisy representations. To integrate textual guidance during the denoising process, the text description is encoded using the CLIP text encoder and incor- porated into the U-Net through the tross-attention mechanism. Finally, the VAE decoder reconstructs images from the latent representations. By leverag- ing mmage-text pairs, the conditional LDM can be optimized by:
c; Σ (ng
(3)
where ā. is the weight coefficient computed by the softmax function:
(
ā.g- = softmax
(4)
vdz
where Wº, W and WV are learnable weights, and de is the dimension of the key vector. The output of the last Multimodal Transformer Encoder layer fed into the Transformer Decoder to gen- erate the target translation. In this paper, we use the Multimodal Transformer as our base model architecture.
3 Method
Lism E [lle c0(2,,4, (o)))11 (I)
The original Multimodal Transformer typically re- quires a paired image the input sentence to pro- vide visual context, which is often impractical
where x and a represent the input text and image, respectively. e denotes the Gaussian noise. t and

---

many scenarios. To address this limitation, leverage the Stable Diffusion model during infer- ence to generate synthetic image based on the source sentence. However, as synthetic images may follow different distributions compared to the authentic (ground truth) images, using authentic images for training and synthetic images for in- ference could result in performance decline in translation quality due to the distribution shift in the images. Next, will introduce our proposed approach for bridging the gap between synthetic and authentic images.
Layn
Output Distributions
Output Distributions
Transformer Decoder
Transformer Decoder
Shared Weights
Multimodal
Multimodal
Transformer Encoder
Transformer Encoder
Shared Weights
Concat
Concat
Lot
3.1 Training with both Synthetic and Authentic Images
FFN
FFN
Weights
To address the issue of distribution shifts in images, we propose training the MMT model using a com- bination of synthetic and authentic images. Specifi- cally, we enhance the training set by augmenting each sentence with an additional synthetic image, and train the model with both synthetic and au- thentic images. By incorporating synthetic images during training, can alleviate the inconsistency between training and inference phases.
CLIP Image Encoder
CLIP Image Encoder
Synthetic
Text Embedding
Stable Diffusion
Authentic
Text Embedding
src: on horseback man attempts to rope a young bull
Technically, we utilize the Stable Diffusion model to generate an image s corresponding to the input text x. We employ the pre-trained CL.IP (Radford et al., 2021) image encoder to encode images into visual embeddings. Subsequently, incorporate feed-forward network (FFN) adjust the dimension of visual embeddings align with the dimension of word embeddings.
Figure Overview of proposed method.
The final training objective is as follows:
Ltrans
(Cagn + Laua).
(9)
2
3.2 Consistency Training
H FFN (CL.IImmeEEncoddr())) H FFX(CLLPImmgEnnnee((ss))
(5)
Incorporating synthetic images into the training process helps mitigate the distribution shift be- tween training and inference. However, the model's behaviour may still differ when handling synthetic and authentic images. To enhance the inherent consistency of the model when dealing with these two types of images, we introduce two consis- tency losses during training. Optimal transport loss is a common training object used to encour- age word-image alignment in vision-language pre- training tasks (Chen et al., 2020; Kim et al., 2021). The Kullback-Leibler divergence loss can also be used to improve visual representation consistency (7) in text-guided image inpainting tasks (Zhou and Long, 2023). Following previous works, the encoder side, we incorporate an optimal transport-
(6)
where H E Rlxd and H" E Ixd denote visual representations of the authentic and synthetic im- ages, respectively, and d is the dimension of word embeddings. Next, H and H are fed into the Mul- timodal Transformer to perform the MMT task.
Let denote the target sentence as y (y., .... VM). The loss functions for training with synthetic and authentic images can be formulated as follows:
M log (yll
Layn
j=1
M log logr(ygl=e),,,))
Loat
(8) based training objective to encourage consistency between representations synthetic and authentic

---

images. At the decoder side, we encourage con- sistency between the predicted target distributions based on both types images. Figure 2 illustrates the overview of our proposed method.
1 as an optimal transport problem, given by:
D (H H
min
Tay
s.t.
Tij m4 Vi E {I...,, d},
(12)
Representation Consisteney Intuitively, we hope the visual representations of synthetic and au- thentic images to be similar. However, directly opti- mizing the cosine similarity or L2 distance between H and H may not be optimal, as the visual repre- sentations of two images may exhibit different dis- tributions across the feature dimension. Therefore, we propose measuring the representation similarity between synthetic and authentic images based on the optimal transport distance.
j=1
> Tij = mjs bj E (l,...dd. We use the L2 distance as the cost function c, and the probability mass is defined as:
m, Σ [4g
(13)
119
my Σ, [hg]
The Optimal Transport (OT) problem is well- known mathematical problem that seeks to find the optimal mapping between two distributions with minimum cost. It has been widely employed loss function in various machine learning applica- tions. Following the principles of optimal transport theory (Villani, 2009), the two probability distribu- tions P and Q can be formulated as:
Following Kusner et al. (2015), we remove the second constraint to obtain a lower bound of the accurate OT solution. The relaxed OT improves the training speed without performance decline, which can be defined as:
D (H H min Tiye (hg,
(14)
P = (ww k s.t.) Σ m,
Em 1;
s.t. Σ
Túj TTg
m4 Vi E (1,,..., d]-
j=1
(10)
[(tū,, m m ) s.l. Σ mj =1
The final OT loss is defined as follows:
Lot
(D + D (H", H²), (15)
where each data point W/ has a probability mass my E oo). Given a transfer cost function c (wi, wz). and using Tij to represent the mass transported from w, to wj. the transport cost can be defined as:
Prediction Consistency In addition ensuring the consistency of representations between syn- thetic and authentic images, we also aim to enhance the model's 's consistency in the predicted probabil- ity distributions based on both two types of images. Inspired by previous works on speech translation (Fang et al., 2022; Zhou et al., 2023; Fang and Feng. 2023), we introduce a prediction consistency loss, defined as the Kullback-Leibler (KL) divergence between the two probability distributions:
D (P.Q) = min
T c (wi, Uj))
T20
ij
k
s.t.
To = m4 Vi E K},
(1I)
M
j=1
Cst
KI. [I.lll,,lue,,,r sl),u,,ssplppusss. a.a]
K
T m (1, KJ.
(16)
==1
Finally, the training objective is as follows:
L - -- + AL + Yoot
(17)
We regard two visual representations H³ (hi...., h2) and H (hq, hg) as two indepen- dent distributions. Each he and he here a scalar value. We formulate the distance between H and
where and y are the hyperparameters that control the contribution of the KL divergence loss and the optimal transport loss.

---

En-De
En-Fr
Models
Average
Test2016
Test2017
MSCOCO
Tesr2016
Test2017
MSCOCO
Text-only Transformer
TEXT-ONLY
40.69
34,26
30.52
62.84
54.35
44.81
44.58
Previous Image-must Systems
2 DCCN (Lin et al., 2020)
39.70
31.00
26.70
61.20
54,30
45.40
43.08
3 'RMMT (Wa 2021)
41.45
32,94
30.01
62.12
54.39
44.52
44.24
4
'Doubly-ATT (Calixto et al., 2017)
41,45
33,95
29.63
61.99
53.72
45.16
44.32
5
*Gated Fasion (Wu al., 2021)
41.96
33.59
29.04
61.69
54.85
44.86
44.33
6 Selective Attention (Li et al., 2022a)
41.84
34.32
30.22
62.24
54.52
44.82
44.66
7
Noisc-robust (Ye ct sl., 2022)
42.56
35.09
31,09
63.24
55.48
46.34
45.63
8
VALHALLA (Li et al., 2022b)
42.60
35.10
30.70
63.10
56.00
46.40
45.65
Our Image-must Systems
9 MUlTITRANS (A) (Yao and Wan, 2020)
41.46
34.36
30.07
62.57
$4.92
44.53
44.65
10 INTEGRATED (A)
42.21
33,94
31.05
62.76
54.73
45.18
44.98
11
OURS (A)
42.50
36.04
31.95
63.71
56.17
46.43
46.13
Previous Image-free Systems
12
JUVR-NMT (Zhang et al,, 2020)
40.79
32.16
29.02
61.00
53,20
43.71
43.31
13
'mmagination (Elliott and Kádár, 2017)
41.31
32.89
29.90
61.90
54.07
44.81
44.15
Distill (Peng al., 2022)
41.28
33.83
30.17
62.53
54.84
15
VALHALLA (Li et al., 2022b)
42,70
35.10
30.70
63.10
56.00
46,50
45.68
Our Image-free Systems
16 MULTITRANS (S) Yao and Wan, 2020)
41.46
34.36
30,07
62.54
$4.82
44.39
44.61
17
INTEGRATED (S)
42.21
33.94
31.05
62.76
54.73
45.14
44.97
18
OURS (S)
42.50
36.04
31.95
63.71
56.17
46.43
46.13
Table I: BLEU scores Multi30K Test2016, Test2017 and MSCOCO test sets of En-De and En-Fr tasks. "(A)" indicates using authentic images during the inference stage, while "(S)" indicates using synthetic images. indicates results under the Transformer- configuration, which are quoted from Wu al. (2021).
4 Experiments
The pre-trained CLIP (Radford et al., 2021) text encoder² is used to encode the input text. The num- ber of denoising steps is set to 50, The seed of the generator set to O. The scale of classifier-free guidance is 7.5 and the batch size is 1.
4.1 Dataset
We conduct experiments the Multi30K dataset (Elliott et al., 2016). Multi30K contains bilin- gual parallel sentence pairs with image annotations, where the English description of each image man- ually translated into German (De) and French (Fr). The training and validation sets consist of 29,000 and 1,014 instances, respectively. We report the results the Test2016, Test2017 and ambiguous MSCOCO test sets (Elliott et al., 2017), which con- tain 1,000, 1,000 and 461 instances respectively. We apply the byte-pair encoding (BPE) (Sennrich al., 2016) algorithm with 10K merge operations to segment words into subwords, resulting in a shared vocabulary of 9,712 and 9,544 entries for the En-De and En-Fr translation tasks, respectively.
Visual Features For both synthetic and authen- tic images, we use the pre-trained ViT-B/32 CLIP model to extract the visual features, which have dimension of [1 512].
Translation Model We follow Wu et al. (2021) to conduct experiments with the Transformer Tiny configuration, which proves to be more effective the small dataset like Multi30K. The translation model consists of 4 encoder and decoder layers, The hidden size is 128 and the filter size FFN is 256. There are heads in the multi-head self- attention module. The dropout set to 0.3 and the label smoothing is 0. Our implementation is based the open-source framework fairseq' (Ott et al., 2019). Each training batch contains 2,048 tokens and the update frequency is set to 4.
4.2 System Settings
Stable Diffusion We build our own inference pipeline with diffusers (von Platen al., 2022). We use the pre-trained VAE and U-Net !
"https://hugginggaace co/openai/
clp--vit-large-patchl
htttps: httts::/huggingface. httpss//hugggggaac...//oopps stableddiffusio---
hhttps://githbb com/oppenii/CLI http:////tttti.cccmm/eeee/ee//ss

---

En-De
Model
Similarity
Lu
Average
Tes:2016
Test2017
MSCOCO
MULTITRANS
39.43%
42.21
33.94
31.05
35.73
INTEGRATED
86.60%
41.71
33.57
29.42
34.90
OURS (S)
100.00%
3
x
42.31
35.76
31.04
36.37
42.50
36.04
31.95
36.83
Table 3: The average cosine similarity between visual representations the synthetic and authentic images on the Test2016 test set.
Table BLEU scores on three test sets with different training objectives in the image-free setting.
For evaluation, we average the last 10 checkpoints following previous works. The beam size is set to 5. We measure the results with BLEU (Papineni al., 2002) scores for all test sets. All models trained and evaluated using 1 Tesla V100 GPU. We set the KL loss weight 0.5. The OT loss weight y set to 0.1 for En-De and 0.9 for En-Fr.
Firstly, our method and the majority of the base- line systems demonstrate a substantial performance advantage over the TEXT-ONLY baseline, under- scoring the importance of the visual modality.
Secondly, for the MULTITRAN baseline, it is evident that the synthetic and authentic images yield different translation results. Overall, the au- thentic images outperform the synthetic images, Directly substituting authentic images with syn- thetic images results in performance decline due to the distribution shift in images.
4.3 Basline Systems
We conduct experiments in two different settings: image-must and image-free. In the image-must setting, authentic images from the test set are used to provide the visual context. In contrast, the au- thentic images are not used in the image-free set- ting. We implement the following three baseline systems for comparision:
Thirdly, for our INTEGRATED baseline, we in- corporate synthetic images into the training pro- cess. Experimental results show some improve- ments compared to the MULTITRANS bascline. The utilization of synthetic or authentic images during inference does significantly affect the re- sults, indicating the effectiveness of this approach in addressing the distribution shift between training and inference.
TEXT-ONLY Text-only Transformer is imple- mented under the Transformer- -inn configuration. It follows an encoder-decoder paradigm (Vaswani et al., 2017), taking only texts as input.
Finally, our method significantly outperforms all baseline systems in both image-must and image-free settings. It achieves state-of-the-art performance all test sets, while remaining in- dependent of the authentic images. The superior performance demonstrates the effectiveness of en- couraging the representation and prediction consis- tency between synthetic and authentic images the MMT training.
MULTITRANS Multimodal Transformer (Yao and Wan, 2020) trained on the original Multi30K dataset, as described in Section 2.2.
INTEGRATED Multimodal Transformer trained on our augmented Multi30K dataset, where each sentence is paired with an authentic image and synthetic image, as described in Section 3.1.
Besides, in the image-must must setting, we include DCCN (Lin al., 2020), RMMT (Wu et al., 2021), Doubly-ATT (Calixto et al., 2017), Gated Fusion (Wu et al., 2021), Selective Attention (Li et al., 2022a), Noise-robust (Ye et al., 2022), and VAL- HALLA (Li et al., 2022b) for comparison. In the image-free setting, include UVR-NMT (Zhang et al., 2020), Imagination (Elliott and Kádár, 2017), Distill (Peng et al.. 2022), and VAL- HALLA (Li et al., 2022b) for comparison.
Analysis
5.1 Effeets of Training Objectives
To investigate the impact of the KL divergence (KL) loss and the optimal transport (OT) loss, we conduct ablation experiments on the training objec- tives. The results are presented in Table 2. When only applying the OT loss (#2), we observe perfor- mance decline. When only using the KL loss (#3), observe an improvement of 0.64 BLEU, high- lighting the benefits of promoting prediction consis- tency. When both KL loss and OT loss are utilized (#4), we achieve a more significant improvement
4.4 Main Results on the Multi30K Dataset
Table summarizes the results both image-must and image-free settings. Each model is evaluated on three test sets for two language pairs.

---

En-De
En-De
Model
Tes12016
Test2017
MSCOCO
Average
Model
Tes12016
Test2017
MSCOCO
Average
(A)
41.64
35.15
OURS (S)
42.50
36.04
31.95
36.83
INTEGRATED (A)
41.21
34.31
30.10
35.21
COSINE
42.46
35.53
32.10
36.70
OURS (A)
42.42
35.44
31.88
36.58
L.
42.48
36.00
31.77
36.75
SELECTIVE (S)
41.64
33.88
29.92
35.15
INTEGRATED (S)
41.21
34.31
30.10
35.21
42.42
35.44
36.58
Table 6: BLEU scores of our approach and other loss functions.
Table 4: BLEU scores of three systems built upon the Selective Attention (Li et al., 2022a) model.
(SELECTIVE) (Li et al., 2022a), which incorporat- ing visual representations with a selective attention module and a gated fusion module. We apply the optimal transport loss before the selective attention module, and apply the KL loss in the decoder. The model settings and hyperparameters are the same as our experiments the Multimodal Transformer. Table 5: BLEU scores of our approach and other two regularization methods.
En-De
Model
Average
Test2016
Test2017
MSCOCO
OURS (S)
42.50
36.04
31.95
36.83
RANDOM
41.96
35.68
31.20
36,28
NOISE
41.86
35.72
30.77
36.12
The results are shown in Table 4. For the IN- TEGRATED baseline, integrating synthetic images into the training process can eliminate the gap and bring slight improvements. When we add the KL loss and OT loss during training, we observe a 1.43 BLEU improvements compared with the SE- LECTIVE baseline, demonstrating the effectiveness and generalizability of our method across different model architectures.
of 1.10 BLEU, demonstrating the benefits of si- multancously encouraging the representation and prediction consistency.
5.2 Representation Consistency
To investigate the effectiveness of method in aligning the representations of synthetic and au- thentic images, we measure the cosine similarity between their respective representations (HH and H on the test set. Specifically, we measure the cosine similarity of the visual representations af- ter the weighd- FFN network on Test2016 of En-De translation task during inference. Ta- ble 3 presents the results. For the MULTITRANS baseline, significant disparity exists between the visual representations of synthetic and authentic images. However, when the synthetic images are in- tegrated into the training process (INTEGRATED), the similarity between synthetic and authentic im- ages increases to 86.60%. In contrast, our method achieves an impressive similarity of 100.00%, high- lighting the effectiveness of our approach in bridg- ing the representation gap between synthetic and authentic images. The optimal transport loss plays a crucial role reducing the representation gap.
5.4 Comparison with Regularization Methods
We employ the Stable Diffusion model to generate images from textual descriptions as visual contexts. To verify the necessity of the rext-to-image genera- tion model, we compare its performance with other two regularization methods: RANDOM and NOISE. RANDOM means shuffling the correspondences be- tween synthetic images and textual descriptions in the training set. NOISE means using noise vectors as the visual representations. Results in Table 5 show that both two strategies perform worse than our approach, demonstrating that our superior per- formance is not solely attributed to regularization effeets but rather to the utilization of appropriate images and semantic information.
5.5 Comparison with Other Loss Functions
We measure the representation similarity between synthetic and authentic images, and minimize their distance during training. To assess the signifi- cance of the optimal transport loss, we conduct experiments using various loss functions. Table 6 presents the results when substituting the optimal transport loss with the COSINE embedding loss and L2 loss functions in order to improve repre- sentation consistency. The hyperparameters are the
5.3
Apply Our Method to Other Model Architecture
Our approach focuses on improving the training methodology for MMT, which is not restricted to specific model architecture. To validate the general- ity our method, we conduct experiments using an alternative model architecture: Selective Attention

---

En-Cs
Model
ABLEU
Model
Average
MSCTD
Test2016
Tesr2018
MULTITRANS(S)
0.23
TEXT-ONLY
34.01
29.46
31.73
21.37
MULNITRANS (S)
34.18
30.03
32.11
21.48
INTEGRATED(S)
0.43
INTEGRATED (S)
33,50
29.62
31.56
21.65
OURS(S)
0.56
OURS
35.23
31.19
33.21
24.40
Table 8: Results of the Incongruent Decoding on the Multi30K En-De test sets.
Table 7: BLEU scores of approach different languages and datasets.
case, the phrases "spielt" (play) and "auf einer roten gitarre" (on a red guitar) are correctly translated by our proposed model while other systems translate inaccurately. To verify that our system benefits from the incorporation of visual information, we also compute the BLEU score for cach sentence in the En-De 6est2016 test set and conduct a compar- ison between our model and the TEXT-ONLY base- line. Among 1,000 data instances, 34% achieve higher blue scores upon incorporating visual infor- mation, whereas 24.9% exhibit lower blue scores. We also manually annotated the En-De Test2016 test set, enlisting a German language expert who is certified at the professional eighth level for the annotation task. The annotation results reveal that 35.7% of the instances benefit from the visual fea- tures, while 30.8% show no improvement from visual features. These results demonstrate that our model enhances translation performance when vi- sual information incorporated, further confirming the improvement in translation results achieved by aligning synthetic and authentic images.
same as our experiments on the optimal transport loss. The drop of results indicates the effective- ness of the optimal transport loss in mitigating the representation disparity.
5.6 Results on Different Languages and Datasets
To verify that our method can be utilized in differ- ent language pairs and datasets, we conduct experi- ments the En-Cs task of the Multi30K dataset and the En-De translation task of the MSCTD dataset (Liang al., 2022). The MSCTD dataset is a multimodal chat translation dataset, including 20,240, 5,063 and 5,047 instances for the training, validation and test sets respectively. The results are shown in Table 7. Our method still achieves significant improvements among all the baselines on the En-Cs task and the MSCTD dataset, show- ing the generality of method among different languages and datasets.
5.7 Incongruent Decoding
We follow method (Elliott, 2018) to use an adversar- ial evaluation method to test if our method is more sensitive to the visual context. We set the congru- ent image's feature to all zeros. Then we observe the value ABLEU by calculating the difference be- tween the congruent data and the incongruent one. A larger value means the model is more sensitive the image context. As shown in Table 8, we conduct experiment on the three test sets of the En-De translation task and calculate the average incongruent results. Notably, our method achieves the highest score, providing substantial evidence of its sensitivity to visual information.
6
Related Work
6.1 Multimodal Machine Translation
Multimodal Machine translation (MMT) has drawn great attention in the research community. Existing methods mainly focus on integrating visual infor- mation into Neural Machine Translation (NMT) (Zhang et al., 2023). Multimodal Transformer (Yao and Wan, 2020) employs a graph perspective fuse visual information into the translation model. Gated and RMMT, introduced by Wu et al. (2021), are two widely-used architectures for integrating vi- sual and textual modalities. Selective Attention (Li et al., 2022a) is method that utilizes texts to select useful visual features. Additionally, Ji et al. (2022) proposes additional training objectives to enhance the visual awareness of the MMT model. Yang al. (2022) introduces cross-modal contrastive learning to enhance low-resource NMT.
5.8 Case Study
Table 9 presents two translation cases in the image-free scenario involving the four systems. In the first case, our model accurately translates the phrase "im maul" (in the mouth) and "rennt im schnce" (run in the snow), whereas the other sys- tems provide incomplete translations. In the second
In addition to methods that rely on authentic images, there is a growing interest in developing

---

SRC:
black dog with pink toy in its mouth is running in the snow ein schwarzer hund mit einem spielzeug im maul rennt im schnee
REF:
TEXT-ONLY:
ein schwarzer hund mit einem rosa spielzeug schnee rennt
MULTITRANS (S):
ein schwarzer hund mit einem rosa spielzeug im schnee
INTEGRATED (S):
ein schwarzer hund mit einem rosa spielzeug im schnce läuff
OURS (S):
ein schwarzer hund mit einem rosa spielzeug im maul rennt im schnee
SRC:
guitar player performs nightclub red guitar
REF:
gitarristin spielt in einem nachtklub auf einer roten giturre ein gitarrespieler tritt in einem nachtelub auf
TEXT-ONLY:
MULTITRANS (S):
ein gitarrespieler tritt in einem nachtelub auf
INTEGRATED (S): ein gitarrespieler spieit in einem nachtelub
OURS (S): ein gitarrespieler spielt in einem nachtclub auf einer roten gitarre
Table 9:
Two translation cases of four systems on the En-De task in the image- frree scenario. The red and blue tokens denote error and correct translations respectively.
MMT systems in image-free scenarios due to the difficulty of obtaining paired images during inference. The image retrieval-basse approaches (Zhang et al., 2020; Fang and Feng, 2022) utilize images retrieved based on textual information as visual representations. The proposed method (Li et al., 2022b) generates discrete visual representa- tions from texts and incorporate them the train- ing and inference process. Peng et al. (2022) pro- poses distillation techniques to transfer knowledge from image representations to text representations, enabling the acquisition of useful multimodal fea- tures during inference. In this paper, focus using synthetic visual representations generated from texts and aim bridge the gap between syn- thetic and authentic images.
generation, resulting in high-quality image synthe- sis. In this work, we employ the Stable Diffusion (Rombach et al., 2022) model, diffusion- based approach, our text-to-image generation model to generate content-rich images from texts.
Conclusion
In this paper, strive to address the disparity between synthetic and authentic images in multi- modal machine translation. Firstly, we feed syn- thetic and authentic images to the MMT model, respectively. Secondly, we minimize the gap be- tween the synthetic and authentic images by draw- ing close the input image representations of the Transformer Encoder and the output distributions of the Transformer Decoder through two loss func- tions. As result, we mitigate the distribution dis- parity introduced by the synthetic images during in- ference, thereby freeing the authentic images from the inference process. Through extensive exper- iments, we demonstrate the effectiveness of our proposed method in improving the quality and co- herence of the translation results.
6.2 Text-to-image Generation
Text-to-image generation models have made signif- icant progress in generating highly realistic images from textual descriptions. Early approaches (Kar- ras al., 2018; Xu et al., 2018) rely training Generative Adversarial Networks (GANs) (Good- fellow et al., 2014) to generate images. CLIP (Rad- ford et al., 2021), a powerful vision-and- -angugge model, has also been utilized to guide the image generation process in various methods (Ramesh et al., 2022; Abdal et al., 2022). Recently, re- searchers have achieved impressive results in zero- shot text-to-image generation scenarios, where im- ages are generated based on textual descriptions without specific training data (Ramesh al., 2021; Yu et al., 2022). The adoption of diffusion- based methods (Nichol et al., 2022; Saharia et al., 2022) has further pushed the boundaries of text-to-image
Limitations
The utilization of the text-to-image generation pro- cess may introduce additional computational over- head. Moreover, our method is currently limited to translation tasks from English to other languages, as the pre-trained text-to-image model lacks sup- port for languages other than English. The appli- cation of our method other tasks and languages remains unexplored, and we consider these limita- tions as potential areas for future research.

---

References
Qingkai Fang and Yang Feng. 2023. Understanding and bridging the modality gap for speech translation. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics.
Rameen Abdal, Peihao Zhu, John Femiani, Niloy J. Mitra, and Peter Wonka. 2022. Clip2stylegan: Un- supervised extraction of stylegan edit directions. In SIGGRAPH '22: Special Interest Group on Com- puler Graphics und Interactive Techniques Confer- ence, Vancouver, BC, Canada, August 7 11, 2022, pages 48:1-48:9. ACM.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingx- uan Wang. 2022. Stemm: Self-learning with speech- text manifold mixup for speech translation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics.
lacer Calixto. Qun Liu, and Nick Campbell. 2017. Doubly-attentive decoder for multi-modal neural ma- chine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017. Vancouver, Canada, July 30- August 4, Volume I: Long Papers, pages 1913-1924. Association for Computational Linguisties.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks.
Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang. Zhoujun Li, Dongdong Zhang, and Zheng Cui. 2022. LVP-M3: Language-ave awaarrr visual prompt for multilingual multimodal machine translation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2862- 2872, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: universal image-text representation learning. In Computer Vision ECCV 2020- 16th Eumpean Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 104-120, Springer.
Baijun Ji, Tong Zhang. Yicheng Zou, Bojie Hu, and Si Shen. 2022. Increasing visual awareness in multi- modal neural machine translation from informa- tion theoretic perspective. CoRR, abs/2210.00 10..2707.
Desmond Elliott. 2018. Adversarial evaluation of mul- timodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2974-2978. Association for Computational Linguistics.
Tero Karras, Samuli Laine, and Timo Aila. 2018. A style-based generator architecture for generative ad- versarial networks. CoRR, abs/18 182.04994.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine transla- tion and multilingual image description. In Proceed- ings of the Second Conference on Machine Transla- tion, WMT 2017 Copenhagen, Denmark, September 7-8, 2017, pages 215-233 1133. Association for Computa- tional Linguisties.
Wonjae Kim, Bokyung Son, and Iidoo Kim. 2021. Vilt: Vision-ani and lan-uag transformer without convolu- tion or region supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, PMLR.
pages
Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kil- ian Q Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd In- ternational Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 957-966. JMLR.org.
Desmond Elliott, Stella Frank. Khalil Sima'a an, and Lu- cia Specia. 2016. Multi30k: Multilingual english- german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, hosted by the 54th Annual Meeting of the Association for Computa- tional Linguistics, ACI 2016, August 12, Berlin. Germany. The Association for Computer Linguistics.
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma. and Jingbo Zhu. 2022a. On vi- sion features multimodal machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation Computational Linguistics (Volume 1: Long Pupers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 6327-6337. Association for Computa- tional Linguistics.
Desmond Elliott and Ákos Kádár. 2017, Imagination improves multimodal translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan. November 27- December 2017 Volume I: Long Papers. pages 130-141. Asian Federation of Natural Language Processing.
Qingkai Fang and Yang Feng, 2022. Neural machine translation with perase-level universal visual repre- sentations. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 5687-5698. Association for Computational Linguistics.
Yi Li, Rameswar Panda, Yoon Kim, Chun-Fu Richard Chen, Rogério Feris, David D. Cox. and Nuno Vas- concelos. 2022b. VALHALLA: visual hallucination for machine translation. In JEEF/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022. New Orleans. LA, USA, June 18-24. 2022. pages 5206-5216, IEEE

---

Yunlong Liang, Fandong Meng, Jinan Xu. Yufeng Chen, and Jie Zhou. 2022. MSCTD: A multimodal senti- ment chat translation dataset. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Treland, May 22-27, 2022, pages 2601- 2613. Association for Computational Linguisties.
of Machine Learning Research, pages 8748-8766 PMLR.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.06125.
Huan Lin, Fandong Meng. Jinsong Su, Yongjing Yin. Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jieho Luo. 2020. Dynamic context-guided capsule network for multimodal machine translation. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 1320-1328 1329. ACM.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image gen- eration. In Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8821 8821-8831. PMLR.
Quanyu Long, Mingxuan Wang, and Lei Li. 2021. Gen- erative imagination elevates machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5738 5748, Online. Association for Computa- tional Linguistics.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA. USA, June 18-24, 2022, pages 10674-10685. IEEE
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Me- Grew, Ilya Sutskever, and Mark Chen. 2022. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022. Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 16784-16804. PMLR.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I.. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes. Bureu Karagol Ayan, Tim Salimans, Jonathan Ho. David J. Fleet, and Mohammad Norouzi. 2022. Pho- torealistic text-to-image diffusion models with deep language understanding, In NeurlPS.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics, ACL 2016, August 7-12, 2016, Berlin, Ger- many, Volume I: Long Papers. The Association for Computer Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- serence of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLL 2019, Minneapo- lis, MN, USA, June 2-7, 2019, Demonstrations, pages 48-53. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 -6008.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania. USA. Association for Computational Linguistics.
Cédric Villani. 2009. Optimal transport: old and new. volume 338. Springer.
Ru Peng, Yawen Zeng, and Junbo Zhao. 2022. Distill the image to nowhere: Inversion knowledge distil- lation for multimodal machine translation. CoRR, abs/2210.04468.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pe- dro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. 2022. Diffusers: State- of-the-art diffusion models. https://githul: com/ huggingfaces waaaa/ddfffuuerrrrr
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Viriual Event, volume 139 of Proceedings
Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021. Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing,

---

ACL/IJCNLP 2021. (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. pages 6153-6166. Associa- tion for Computational Linguisties.
Yucheng Zhou and Guodong Long. 2023, Improving cross- modal alignment for text-guided image inpaint- ing. In Proceedings of the 17th Conference of the European Chapter of the Association for Computa- tional Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 3437-3448. Association for Computational Linguistics.
Tao Xu, Pengehuan Zhang. Qiuyuan Huang. Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In 2018 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22. 2018, pages 1316-1324. Computer Vi- sion Foundation IEEE Computer Society.
Zhe Yang, Qingkai Fang, and Yang Feng. 2022. Low- resource neural machine translation with cross-modal alignment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10134-10146, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Shaowei Yao and Xiaojun Wan. 2020, Multimodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the As- sociation for Compatational Linguisticx, ACI. 2020, Online, July 5-10, 2020, pages 4346 4350. Associa- tion for Computational Linguistics.
Junjie Ye, Junjun Guo, Yan Xiang. Kaiwen Tan, and Zhengtao Yu. 2022. Noise-robust cross-modal in- teractive learning with text2image mask for multi- modal neural machine translation. Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 5098-5108. Inter- national Committee on Computational Linguistics.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Lu- ong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson. Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022. Scaling autoregressive models for content-rich text- to-image generation. Trans. Mach. Learn. Res.. 2022.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhen- grui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023. Bayling: Bridging cross- tross-lingual align- ment and instruction following through interactive translation for large language models.
Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eicchiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal visual representation. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yan Zhou, Qingkai Fang, and Yang Feng. 2023, CMOT: Cross-mi modal mixup via optimal transport for speech translation. In Proceedings of the 61 Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 7873-7887, Toronto, Canada, Association for Computational Linguistics.


Reviews:
[{"review_id": "VMubi8jSbD", "reviewer": "Reviewer_JtfV", "summary": "", "questions": "As mentioned in \"Reasons To Reject\" above", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: The method proposed in the paper is a commonly used objective in visual-text models[1,2], and is not a new approach.\nThe Representation Consistency and Prediction Consistency proposed in the paper has already been used in other methods[3].**\n\nA: Thank you for your invaluable insights. While loss functions have previously been employed to generate multimodal representations, we have pioneered the application of these techniques in the realm of multimodal machine translation. We find the results indeed intriguing and these techniques highly beneficial for leveraging the synthetic images in multimodal machine translation. Furthermore, these techniques hold the potential to alleviate the reliance on authentic images during the inference phase in multimodal machine translation, which we believe is a worthwhile issue to address.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work proposes to reduce the distribution discrepancy between the training and inference phases by introducing two consistency losses.\n- The writing and organization of the paper are well done, making it easy to follow and understand.", "reasons_to_reject": "- The method proposed in the paper is a commonly used objective in visual-text models[1,2], and is not a new approach.\n- The Representation Consistency and Prediction Consistency proposed in the paper has already been used in other methods[3].\n\n[1] UNITER: UNiversal Image-TExt Representation Learning. ECCV 2020.\\\n[2] ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. ICML 2021.\\\n[3] Improving Cross-modal Alignment for Text-Guided Image Inpainting . EACL 2023.", "questions_to_authors": "As mentioned in \"Reasons To Reject\" above", "paper_topic_and_main_contributions": "This paper proposes to train the multimodal machine translation model with both synthetic and authentic images to mitigate the distribution shift caused by using different types of images for training and inference. To enhance the inherent consistency of the model when handling synthetic and authentic images, the paper introduces two consistency losses that align their representations at the encoder side and their predictions at the decoder side.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] UNITER: UNiversal Image-TExt Representation Learning. ECCV 2020.\\\n[2] ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. ICML 2021.\\\n[3] Improving Cross-modal Alignment for Text-Guided Image Inpainting. EACL 2023."}}, {"review_id": "iinoyTUXtq", "reviewer": "Reviewer_Ft2o", "summary": "", "questions": "1. Why does the average cosine similarity between visual representations of the synthetic and authentic images on the Test2016 test set reach 100%? It seems somewhat unbelievable, and I hope the authors can provide further explanation.\n\n2. Although the proposed method achieves state-of-the-art performance on multiple benchmarks, I have always been skeptical about the effectiveness of visual information in improving the translation of the source sentence. I believe that the MMT task lacks an effective evaluation metric for assessing the quality of translations based on the incorporation of image information.\n\n3. Using a single case to illustrate the improvement brought by visual information to the translation may not be sufficient. I recommend the authors to conduct a manual evaluation on a test set, for instance, to determine how many translations benefit from incorporating images and how many translations do not benefit from using visual information. This would provide a more comprehensive understanding of the impact of visual information on the quality of translations.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your precious advice. \n\n**Q: About the concern of the model's sensitivity of the quality and diversity of the synthetic images.**\n\nA: 1. The reliability of our method on the synthetic images is constrained, because we use the loss functions to enhance the consistency between synthetic and authentic visual representations during the training process. \n\n2. By configuring the generator parameters denoising step and manual seed of the Stable Diffusion model, the consistency of images generated from the same textual input is ensured. \n\n3. To validate the robustness of our approach, we conduct experiments using synthetic images generated by the Stable Diffusion model, varying in quality and diversity. We alter the denoising steps and manual seed settings to control these factors. The corresponding outcomes are presented in Table 1. Remarkably, with synthetic images of varying quality and diversity, our model consistently outperforms the baseline systems. Notably, shuffling the correspondences between synthetic images and textual descriptions in the training set, as shown in section 5.4, has minimal impact on translation quality reduction. We also measure the variance of the different results. These findings demonstrate that our method's efficacy is not contingent on the quality and diversity of the synthetic images.\n\n|**Model**  |**Test2016**  |**Test2017**  |**Test2018**  |**Average**|\n| :---: | :---: | :---: | :---: | :---: |\n|**OURS(S)**  |42.50  |36.04  |31.95  |**36.83** |\n|Denoising Step 40  | 42.24 | 35.68 | 32.17 |36.70 |\n|Denoising Step 60 | 42.37 | **36.05** | 31.80 |36.74 |\n|Manual Seed 1 | **42.74** | 34.94 |32.06  | 36.58 |\n|Manual Seed 2 |42.41 |35.41 |**32.41** |36.74 |\n|Variance |0.045 |0.218 |0.064 |0.006 |\n\nTable 1: BLEU scores of varying quality and diversity synthetic images upon En-De test sets.\n\n**Q: Why does the average cosine similarity between visual representations of the synthetic and authentic images on the Test2016 test set reach 100%? It seems somewhat unbelievable, and I hope the authors can provide further explanation.**\n\nA: We measure the cosine similarity of the visual representation after the shared weights FFN network during inference on Test 2016 of En-De translation task. We utilize global visual features instead of local visual features, concentrating more on the overall information of the images and are relatively easy to align. The results of the image-must and image-free system in our method are also the same. With the help of Optimal Transport Loss, we believe that it is feasible to completely eliminate disparities in visual representations.\n\n**Q: Although the proposed method achieves state-of-the-art performance on multiple benchmarks, I have always been skeptical about the effectiveness of visual information in improving the translation of the source sentence. I believe that the MMT task lacks an effective evaluation metric for assessing the quality of translations based on the incorporation of image information.**\n\nA: 1. Compared to the text-only Transformer baseline, the enhancements observed in our proposed method reflect the effectiveness of visual information. The resutls of the text-only neural machine translation baseline are shown in Table 2. These results will be added in the next version of our paper.\n\n|**Model** | **En-De Test2016** |**En-De Test2017** |**En-De MSCOCO**|**En-Fr Test2016** |**En-Fr Test2017** |**En-Fr MSCOCO** |**Average**|\n| --- | --- | --- | --- | --- | --- | --- |--- |\n|**TEXT-ONLY**|40.69 | 34.26 | 30.52 | 62.84 | 54.35 | 44.81 | 44.58 |\n\nTable 2: BLEU scores of the Text-only baseline model.\n\n2. We follow method [1] to use an adversarial evaluation method to test if our method is more sensitive to the visual context. We set the congruent image's feature to all zeros. Then we observe the value ΔBLEU by calculating the difference between the congruent data and the incongruent one. A larger value means the model is more sensitive to the image context. As shown in Table 3, we conduct experiment on the three test sets of the En-De translation task and calculate the average incongruent result on three test sets. The result of OURS(S) is the highest, which proves our method's sensitivity to visual information.\n\n|**Model**  |**ΔBLEU**  |\n| --- | :---: |\n| **MULTITRANS(S)** | 0.23 |\n| **INTEGRATED(S)** | 0.43 |\n| **OURS(S)** | 0.56 |\n\nTable 3: Results of the Incongruent Decoding upon En-De test sets.\n\n[1] Adversarial evaluation of multimodal machine translation. EMNLP 2018.\n\n**Q: Using a single case to illustrate the improvement brought by visual information to the translation may not be sufficient. I recommend the authors to conduct a manual evaluation on a test set, for instance, to determine how many translations benefit from incorporating images and how many translations do not benefit from using visual information. This would provide a more comprehensive understanding of the impact of visual information on the quality of translations.**\n\nA: We compute the BLEU score for each sentence in the En-De Test2016 test set and conduct a comparison between our model and the text-only baseline. Among 1000 data instances, 34% show improvement upon incorporating visual information, whereas 24.9% do not exhibit improvement from the utilization of visual information.\nWe also manually annotated the En-De Test2016 test set, enlisting a German language expert who is certified at the professional eighth level for the annotation task. By scoring both the text-only translation system and our translation system, we investigate whether the inclusion of visual information indeed enhances translation quality. The annotation results reveal that among 1000 data instances, 35.7% benefit from the utilization of visual features, while 30.8% show no improvement from visual features. Through the BLEU score and manual annotation, it becomes evident that the number of translation results benefiting from the use of visual features is greater than those not benefiting from them, signifying the capacity of visual features to enhance translation quality within our translation system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's strengths lie in its innovative approach to address the dependence on authentic images in multimodal machine translation. The proposed technique for minimizing the gap between synthetic and authentic images is novel and well-founded on principles like Optimal Transport and Kullback-Leibler divergence. The empirical results show significant performance improvements, making it relevant and impactful for the NLP community. The paper's contributions open up new possibilities for using text-to-image generation models in MMT, making it valuable for researchers and practitioners in the field.", "reasons_to_reject": "One potential weakness of the paper could be its heavy reliance on synthetic images during the training process. Although the proposed method effectively mitigates the distribution shift during inference, the extent of the dependence on synthetic images could raise concerns about the robustness of the approach. Additionally, the method's performance may be sensitive to the quality and diversity of the synthetic images generated. There might also be a risk of overfitting to specific synthetic image characteristics, limiting the generalizability of the proposed technique.", "questions_to_authors": "1. Why does the average cosine similarity between visual representations of the synthetic and authentic images on the Test2016 test set reach 100%? It seems somewhat unbelievable, and I hope the authors can provide further explanation.\n\n2. Although the proposed method achieves state-of-the-art performance on multiple benchmarks, I have always been skeptical about the effectiveness of visual information in improving the translation of the source sentence. I believe that the MMT task lacks an effective evaluation metric for assessing the quality of translations based on the incorporation of image information.\n\n3. Using a single case to illustrate the improvement brought by visual information to the translation may not be sufficient. I recommend the authors to conduct a manual evaluation on a test set, for instance, to determine how many translations benefit from incorporating images and how many translations do not benefit from using visual information. This would provide a more comprehensive understanding of the impact of visual information on the quality of translations.", "paper_topic_and_main_contributions": "This paper addresses the challenge of multimodal machine translation (MMT) by proposing a method to bridge the gap between synthetic and authentic images used during the inference process. The main contribution of the paper is the introduction of a technique that feeds both synthetic and authentic images to the MMT model, minimizing the disparity between their representations. This is achieved through the use of Optimal Transport (OT) theory to mitigate differences in input image representations and Kullback-Leibler (KL) divergence to ensure output distribution consistency. By doing so, the paper effectively eliminates the distribution shift caused by using synthetic images during inference, thereby freeing the authentic images from the inference process. The experimental results demonstrate state-of-the-art performance on the Multi30K En-De and En-Fr datasets, even without the need for authentic images.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wwy306QkOU", "reviewer": "Reviewer_ZYWp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We conduct experiments utilizing the MSCTD En-De dataset [2] to prove the validity of our proposed method. The training, validation and test sets consist of 20,240, 5,063 and 5,047 instances, respectively. The results are shown in Table 4:\n|**Model** |**BLEU** |\n| --- | :---: |\n| **TEXT-ONLY** | 21.37 |\n| **MULTITRANS(S)** | 21.48 |\n| **INTEGRATED(S)** | 21.65|\n| **OURS(S)** | **24.40** |\n\nTable 4: BLEU scores on the MSCTD En-De test sets.\n\nOur model significantly outperforms the baseline models, demonstrating the generality of our approach across different datasets. We will add these results in the next version.\n\n[2] MSCTD: A Multimodal Sentiment Chat Translation Dataset. ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - The paper proposes a new method to bridge the gap between synthetic and authentic images for multimodal machine translation. The proposed method feeds synthetic and authentic images to the MMT model, respectively, and minimizes the gap between the two images by drawing close the input image representations of the Transformer Encoder and the output distributions of the Transformer Decoder.\n- The paper provides a comprehensive evaluation against various baseline models and different datasets. While the main dataset is the Multi30K dataset, utilizing other datasets has increased the validity of the evaluation and ablation study.", "reasons_to_reject": "N/A\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method to bridge the gap between synthetic and authentic images for multimodal machine translation (MMT). The proposed method feeds synthetic and authentic images to the MMT model during training and inference, respectively, and minimizes the gap between the two types of images by drawing close the input image representations of the Transformer Encoder and the output distributions of the Transformer Decoder. \nThe main contributions of the paper are: \n- Introducing synthetic images during training to alleviate the inconsistency between training and inference phases. \n- Proposing an optimal transport-based training objective to encourage consistency between representations of synthetic and authentic images.\n - Introducing a prediction consistency loss to enhance the model's consistency in the predicted probability distributions based on both types of images.\n- Introducing synthetic images during training to alleviate the inconsistency between the training and inference phases.\n- Proposing an optimal transport-based training objective to encourage consistency between representations of synthetic and authentic images.\n- Introducing a prediction consistency loss to enhance the model's consistency in the predicted probability distributions based on both types of images.\n\nThree technical backgrounds and concepts required to understand the paper are:\n 1. Multimodal machine translation: The paper focuses on multimodal machine translation, which requires an understanding of both natural language processing and computer vision. \n2. Transformer model: The proposed method utilizes the Transformer model, which is a neural network architecture commonly used in natural language processing tasks. \n3. Optimal transport theory: The paper employs the Optimal Transport theory to mitigate the disparity of the representations between synthetic and authentic images. An understanding of this theory is required to comprehend the proposed approach.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the significant challenge of mitigating the distribution shift caused by using synthetic images during inference in multimodal machine translation (MMT). The authors propose a method that feeds both synthetic and authentic images to the MMT model, reducing the gap between their representations through Optimal Transport theory and ensuring output distribution consistency using KL divergence. The paper's contributions include the introduction of synthetic images during training, an optimal transport-based training objective, and a prediction consistency loss. The paper's contributions seem promising, but further exploration and addressing potential limitations may enhance this paper, some concerns regarding novelty and robustness have been raised.

Pros:

* Interesting Approach: The paper introduces an Interesting approach to addressing the dependence on authentic images in MMT. By leveraging principles like Optimal Transport and KL divergence, it proposes an Interesting technique to mitigate the distribution shift caused by synthetic images during inference.
		
* Positive Results: The paper provides a comprehensive evaluation against various baseline models and datasets, including the Multi30K dataset. This thorough evaluation enhances the validity of the approach and demonstrates its effectiveness.
		
* Clear Presentation: Reviewer 1 commends the clear writing and organization of the paper, making it easy to follow and understand.


Cons:
		
* Novelty is limited: Reviewer 1 raises concerns about the novelty of the proposed method, noting that the objectives and consistency concepts are not entirely new and have been used in other visual-text models.
		
* Potential Robustness problem: There are concerns about the robustness of the approach due to its heavy reliance on synthetic images during training. Reviewer 2 highlights potential sensitivity to the quality and diversity of synthetic images and the risk of overfitting to specific characteristics, limiting generalizability.
		
In summary, the paper offers an interesting approach to mitigate distribution shift in MMT caused by synthetic images during inference. It presents a thorough evaluation and demonstrates promising results. However, concerns regarding novelty and robustness have been raised.
Decision: Accept-Main