Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs
Authors: Yuxin Zuo
Bei Li
Chuanhao Lv
Tong Zheng
Tong Xiao
JingBo Zhu
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal machine translation
Large language models
VQA
Probing tasks

Abstract: This paper presents an in-depth study of multimodal machine translation (MMT), examining the prevailing understanding that MMT systems exhibit decreased sensitivity to visual information when text inputs are complete. Instead, we attribute this phenomenon to insufficient cross-modal interaction, rather than image information redundancy. A novel approach is proposed to generate parallel Visual Question-Answering (VQA) style pairs from the source text, fostering more robust cross-modal interaction. Using Large Language Models (LLMs), we explicitly model the probing signal in MMT to convert it into VQA-style data to create the Multi30K-VQA dataset. An MMT-VQA multitask learning framework is introduced to incorporate explicit probing signals from the dataset into the MMT training process. Experimental results on two widely-used benchmarks demonstrate the effectiveness of this novel approach. Our code and data would be available at: \url{https://github.com/libeineu/MMT-VQA}.

Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-
Pairs
Yuxin Zuo Bei Lift Chuanhao Lv², Tong Zheng?,
Tong Xiao and Jingho Zhu
School of Software, Northeastern University, Shenyang, China
School of Computer Science and Engineering, Northeastern University, Shenyang, China Nrans Research, Shenyang, China
truman yx. www.gmail....ooom com, (libei_neu, 1ct..nuuu...--yyyoootooooo. com eeeeettonnn336666e0 23560gmai com, (xiaotong, inghuiiinnoommmmill neu, edu. cn
zhengtong1
Abstract
lation process when the text is complete. Surpris- ingly, MMT systems can still behave well even the image imput is unrelated to the text (Grönroos et al., 2018; Lala et al., 2018; Elliott, 20. This raises questions about the actual role of visual input in the MMT framework. Aiming this, Caglayan et al. (2019) pointed out that vision features are helpful when the source sentence miss important patterns. Along this line, Li et al. (2022a) designed more spe- cific probing tasks evaluate how MMT behave a limited textual context. But a severe problem still remains, that MMT systems exhibit less sensitivity to the information conveyed by the vision modality when the text is complete.
This paper presents in-depth study of multi- modal machine translation (MMT), examining the prevailing understanding that MMT sys- tems exhibit decreased sensitivity to visual in- formation when text inputs are complete. In- stead, we attribute this phenomenon to insuffi- cient cross cross-modal interaction, rather than image information redundancy. novel approach is proposed to generate parallel Visual Question- Answering (VQA) style pairs from the source text, fostering more robust cross modal interac- tion. Using Large Language Models (LLMs), we explicitly model the probing signal in MMT convert into VQA-style data create the Muld30K- dataset. An MMT-VQA multitask learning framework is introduced to incorporate explicit probing signals from the dataset into the MMT training process. Experi- mental results on two widely-used benchmarks demonstrate the effectiveness of this novel ap- proach. Our code and data would be available at: https://gi thub. wtw.cocm///eeeeeeeeeeee wtt.///benne//MM--Y-
Our hypothesis posits that this may not result from the redundancy of visual data but rather from the lack of effective interaction between the text and visual modalities. It is a reasonable progres- sion to consider the probing task proposed by Li et al. (2022a), which explicitly measures the cross- modality ability. This task involves masking criti- cal entities and evaluating whether MMT can pro- duce accurate translation by attending to the image.
1 Introduction
Broadening the scope of traditional machine trans- lation, multimodal machine translation (MMT) en- hances the quality of text translation by factoring in an auxiliary visual modality (Specia et al., 2016). The key challenge of MMT is finding an effective way to integrate images into the translation model, thereby maximizing the utilization visual infor- mation (Caglayan et al.. 2016: Libovický and Helcl, 2017: Calixto and Liu, 2017; Qian et al., 2018). Meanwhile, it operates on the premise that the rele- vant visual context can help clarify or fill in gaps when the source sentence unclear or incomplete. While, rescarchers soon discovered that MMT systems do not always perform as expected: the vi- sual component often contributes little to the trans-
However, this process falls short in providing any actionable feedback to MMT it serves as a evaluation metric. Consequently, it is worth inves- tigating whether we can incorporate the probing signals into the training process, instead of using it solely for evaluation.
In this study, we focus on augmenting MMT systems by incorporating probing signals during training to fortify the interplay between text and visual data. This can actively stimulating the tex- tual representation to engage with visual data. To achieve this goal, we leverage Large Language Models (LLMs) to identify intricate contextual words and generate questions, thereby transform- ing original text into Visual Question-Answering (VQA) style pairs. This results in a new Multi30K- VQA dataset. On this basis, we introduce a MMT- VQA multi-task learning framework, prompting
Work done during an internship NiuTrans Research.
"Equal Contribution.
-Corresponding author.

---

the MMT model actively "probe" visual data guided by the textual context. Our main contributions can summarized as follows:
Feature
Test2016
Test2017
MSCOCO
ViT-base
40.68
33.35
28.61
MAF-hase
41.65
34.27
29.75
BLIP-basc
41.01
33.73
29.41
CLIP-hase
41,28
33.83
29.66
We first demonstrate that utilizing advanced pre-trained vision features, such as MAE and BLIP, yields consistent performance improve- ments across various scenarios.
Tabble 1: BLEU scores of Multi30K En-De task with Selective Attention applied.
incomplete text, CLIP showcased significant per- formance boost due its innate cross-modal mod- cling knowledge. Therefore, we can find that the cross-mis moddal knowledge of the pre-trained model does not seem to work when the text is complete. This leads us to fundamental question: So now that have better image represeniation, how can further enhance the cross-modal interaction ability of the model within the MMT context?
We release a Multi30K-VQA dataset via us- ing LLMs API, which consists of question- answering pairs originated from the source sentence.
We propose a MMT-VQA multi-task learning framework to explicitly model the probing signal in MMT to strengthen the interactions between text and visual modalities.
Experimental results on Multi30K En-De and En-Fr tasks demonstrate the effectiveness both in terms of BLEU&METEOR scores and spe-
MMT-VQA Multi-Task Learning Framework
cific testsets.
3.1 Probing Signal Modeling: Probing to VQA
2 An Empirical Study Advanced Vision Features
Indeed, a compelling proposition is to compel the MMT system to consider image representation even the absence of ambiguity. The Probing Task proposed by Li et al. (2022a) serves as an effective evaluation metric, determining the usefulness of image features in a limited textual context. Figure I shows how the probing task implicitly guides the model fill the masked position: "probing" the image information. Li et al. (2022a) demonstrated that the probing task efficiently modeling a cross- modal interaction scenario. An intriguing idea is, possible to use this task for training?
The use of stronger vision models as image en- coders in MMT has gained widespread attention. Li et al. (2022a) demonstrated that the application of more robust pre-trained vision models, such as the Vision Transformer (ViT) (Dosovitskiy al., 2021), considerably enhances MMT systems, as evidenced by the substantial improvements in the accuracy metrics of the proposed probing tasks. Continuing along this research trajectory, we ques- tion whether more recent pre-trained vision models can offer additional advantages to MMT systems, a topic yet to be fully explored. In this context, chose MAE (He al., 2022) due its notable impact the field of computer vision, well as two vision-language pre-trained models, namely CLIP (Radford et al., 2021) and BLIP (Li et al., 2022b), for their potential as vision encoders in a MMT system.
Probing task can spur and assess cross-modal interaction capabilities of a model simultaneously. making it effective method for capability en- hancement. However, in small datasets like Multi30K, the context information reduction and text coherence decrease during supervised training can significantly impact performance. Therefore, we should transform the Probing Task to other form. So how to extract the probing signal for training is necessary to explore.
Table shows comparisons on the Test2016 test set of the Multi30K En De task, based on the se- lective attention model (Li et al., 2022a). We see that the MAE-based MMT system delivers the best performance. This outcome was unexpected we anticipated models like CLIP, endowed with strong cross-modal modeling capabilities, to excel. How- ever, as shown in Table 2, probing tasks with
We propose model for the the explicit of prob- ing signals. Figure/ shows case of VQA data. The way to model to formulate questions based on the source, ensuring that all question content originates from the source, and the answer is the masked word in probing task (such as "yellow"

---

Test2016
Test2017
MSCOCO
Model Type
Feature
Restrict
Relaxed
BLEU
Restrict
Relaxed
BLEU
Restrict
Relaxed
BLEU
Color-based
ViT base
42.70
52.51
38.29
29.92
45.67
31.27
32.81
42.19
28.00
Vision PTM
MAE-base
49.46
61.00
39.75
34.12
53.54
32.69
37.50
56.25
29.02
BLIP-base
28.54
36.17
38.03
21.52
35.43
31.03
18.75
18.75
28.83
VL-PTM
CLIP-base
54.68
68.41
40.02
37.80
58.27
33.57
40.62
56.25
28.48
Character-basee
ViT-base
68.76
74.18
38.50
64.02
67.74
30.96
67.19
71.09
27.87
Vision PTM
MAE-base
63.77
69.33
38.58
62.03
67.00
31.86
65.62
69.53
28.07
BLIP-base
59.20
64.05
38.06
54.84
59.55
31.69
60.55
65.62
27.80
VL-PTM
CLIP-base
74.18
80.31
39.84
75.93
80.15
32.54
82.03
85.94
28.24
Table 2: The BLEU scores and the accuracy of Multi30K En-De task of Probing Tasks proposed by Li al. (2022a)
with Selective Attention applied.
Relationship between Probing Task and VQA
Source:
child yellow hoodie playing puddles.
Probing Task:
A child in a [MASK] hoodie playing in puddles.
VQA:
Query: color hoodie do child wear to play puddles?
Answer: Yellow.
Figure 1: An illustration of the probing task and the VQA task given the source- source-image pair.
in Figure 1). We thus constructed a VQA-style Duestion-Answeringg pair, transforming the origi- nal source (which required masking) into a parallel QA pair. Finally, we completed the explicit extrac- tion and representation of the probing signal.
source, and posing questions based on the source. The Figure illustrates the prompt we employ, and we will now explain our design approach step by step. To guide the LLM to achieve this task step by step, we constructed reading scenario to describe the requirements. With the source of each sample in the dataset acting as the reading comprehension passage, the L.LM formulates ques- tions based on the passage and generates answers to construct QA pairs.
3.2 Multi30K-VQA: LLM-Driven Data Generation
To explicitly strengthen the correlations with the help of probing signals, we need solve the chal- lenge of how to generate high-quality question- answering pairs. The key point of the issue lies in the manual selection of appropriate tokens for questioning, task that proved to be both time- consuming and difficult to implement.
Task Description This section primarily structs reading comprehension scenario and pro- vides a comprehensive description of the problem. We engaged the Language Learning Model (LLM) in the role of an experienced author of professional English test papers, specifically focusing on read- ing comprehension questions. We then designated four types of probing questions to guide the LLM in ascertaining the correct answer. In the final step. charged the LLM with the task of generating reading comprehension question and supplying answer rooted in the content the source sentence.
Due to the limitations of supervised training with small-scale data, we turned our attention to large language models (LLMs). Recently, there has been a surge in the capabilities of LLMss, such GPT3.5 (Ouyang et al., 2022). One can design specific prompting guide the generation results by providing several task-specific instructions and demonstrations. Here, we summary three impor- tant aspects for constructing high-quality QA pairs: determine the answer words, comprehending the
Generation Requirements During this phase, our principal focus was the enhancement

---

Description
Hruie senior
peofeniccal Eughsh paper, muidy respessibe reading comprebeniion quesione, will you passage of reodin reading comprehencion question according content passage suddentss mastery
comprehencion tople, the passage, and finaly you will give the auswer need you to make what type of quesiin about the tope's numbers nouss, colors, and charcctess The charactes cype of question needs answered by: 'mann, woman' people, 'men', 'sin' and boy.
Generation Requirements
Voit:
Don't
peisnity ukking quesiio the parts that most nced mike
duat
the
ckaaiee from paoag!!!
relerunt pictures
judgueen!!!
Give peiurity to sokinng quesiion alout tae puns that are mas ledy to be anbiganu and coufising in undeostnndin die tost, auch sa the part of palgacmny
Questinn cun only be esked for our and the typei only bi. Nankers, Nours, Cobor, and Character.
The consent
and
must be limited the passagn, and ask questioss sbout exterall inowledge.
neec
exprossion
original paosage,
word
the queeion, only contsins word phrne correponcdngg
content the original passage!!
The anower should be no mone than top words Do not comment on the noveon for the arewe; and try not complese semennce!
In-Contect Learning
will provide
examppee
exampies rephing
Proage tan young, whise males are cubide near many beales
Type: Cole Queetion: What coloris is woman?? Anover: White.
aevesud lurd lais opecning giant pulkyy system..
Type: Chanctee
Quation: Who upenating the giunt palloy my sysem!
After underscanding what said, please reepond me according to this sencence:
Figure 2:
The prompt utilized for generating Question-Answering (QA) pairs in accordance with the provided source sentence using Large Language Models (LLMs). The prompt consists of three parts: task description, generation requirements and the demonstration.
Source
Type
Question
Answer
A group of fourteen is assembled in a hall with dining tables and stage.
Number
How many people are in the group?
Fourteen.
Two young, white males are outside near many bushes.
Noun
What are the males near?
Bushes.
A little girl climbing into wooden playhouse. A man in bluc shirt is standing on a ladder cleaning a window.
Character
Who climbing into the wooden playhouse?
Girl.
Color
What color is the man's shirt?
Blue.
Table 3: Four types of cases in the Multi3K -QQ dataset.
prompts through human feedback. We imple- mented set of generative guidelines designed to augment data quality. Notably, we hand-crafted several strategies (as illustrated the blue segment of Figure 2) for identifying and addressing transla- tion pitfalls, thereby enhancing the overall quality of the generated content.
struction of model for MMT, while VQA serves as a secondary, supporting task. Upon alignment of data from both tasks, we work with three in- puts: Source, Question and Image. While, the corresponding target labels for MMT and VQA are Target and Answer
We adopted the selective attention of Li et al. (2022a)'s work as its strong performance. In this way, our model leverages a Transformer encoder in combination with an advanced vision model as the encoder for text and visual modalities respectively. By sharing the parameters of the Text Encoder, we enhance the ability of MMT to facilitate an interplay information between the two encoders, catering to the needs of VQA for modeling the question.
In-Context Learning In the final phase, we se- lected two carefully designed examples as the demonstration to foster in-context learning for the Large Language Model (text-davinci-e -0nn) This process could be regarded as a 2-shot prompt- ing strategy. Consequently, we obtained 29, 000 data entries a Type-Questinn format, where several samples are shown in the Table 3. Note that we set the temperature to O, and utlimately obtained the Multi30K-VQA dataset, which could be found in our supplementary.
Given the differences in the fusion methods of the two tasks' information and the variances in the output languages of the tasks, potential significant effects on the performance of model due to discrepancies between the two lan- guage representation spaces. As consequence, we independently initialize the parameters in Selective Attention layer and Decoder.
acknowledged
3.3 MMT-VQA Multi-Task Learning Model
After obtaining the Multi30K-VOL -QQA dataset, our central design is enable the MMT model "ask questions to image" via a multi-task learning frame- work. This leads us to the main task: the con-

---

Souree
Question
Target Decoder
Answer Decoder
Vision Eneoder
Text Eneoder
Target
Image
Source Question
Answer
Figure The overall architecture of MMT-VQA multi-task learning framework. SA indicates the Selective Attention proposed by Li et al. (2022a).
Model Architecture The proposed overall model architecture is depicted in Figure 3. The training process is initiated by feeding two distinct textual inputs, namely Source (X**)) and Question (X4*)), into the Text Encoder. This operation results in two corresponding text representations, H and H Simultaneously, we apply pre-trained vi- sion model to the associated image, thereby obtain- ing Himg
Training Objective In order facilitate the learning process MMT with the assistance of VQA, we regard MMT as the primary task and VQA as an auxiliary task. The training objective for MMT is the standard Maximum Likelihood Es- timation (MLE) loss, while simultancously optimiz- ing the training objective for VQA in conjunction with the MMT loss:
CTaal LMMT ALvQA
(7)
H = TexiEncoderrXXm
(1)
where i hyperaarameter controlling the balance (2) between the MMT loss and the VQA loss. During
H = TextEncoder( (exEEnoode((X99)) (X99)),
Hing = W. PT.PPM(XX))
(3) training, the training objective to minimize the total loss LToal
where W a projection matrix to convert the shape of PTM(X ( rmng into that of X and X PTM(-)
Experiments
denotes MAE or other advanced vision models. After obtaining the three representations, the text is associated with the image patch using single- head attention network, where the queries derive from Il and I respectively, and the key and value are both Himg Formally, these two processes can be expressed as:
4.1 Data
Multi30K We evaluate methods two stan- dard benchmarks: Multi30K English-German (En- De) and English-French (En-Fr) (Elliott al., 2016). Multi30K is a widely used MMT dataset, containing 31, 014 images with one English de- scription and the manual translation in German and French. The training and validation sets con- sist of 29, 000 and 1, 014 instances, respectively. We reported the results the Test2016, Test2017, Test2018 and MSCOCO test sets (Elliott al., 2017), which includes 1, 000, 1, 000, 1071 and 461 instances, respectively.
QKT
H img
Softmax
V
(4)
ddl
where d is the same the dimension of s or gsn because single head is used.
Then calculate two gate and the fused output as follow:
A Si + V (5)
As detailed in Section 3.2, we introduce the Mult30K-VQA dataset, featuring 29, 000 rigorously curated QA pairs (the same size as the training set) for text-image tasks. To en- sure quality and minimize hallucination, generated QA pairs underwent iterative reviews and refinements on 500 randomly selected samples. Despite these efforts, there are even 1, 012 pairs fell short our expectations. address this issue,
Multi30K-VQA
Out (1 ()- Htext + Himg
where lext is H H4th U and V are trainable variables. À controls the mixing ratio of visual information. Then, the fusion vectors Hout and H computed via Eq. 6 are sent to the Target Decoder and the Answer Decoder, to predict the target sentence and answer, respectively.
LLM-

---

Rnglich->German
English-SFreechh
System
Feature
Tes20116
Test2017
MSCOCO
Tes12016
Tea12017
MSCOCO
Text only
Transformer Tiny (Vaewani al., 2017)
41,02
68.22
33,36
62.05
29.88
56.64
61.80
81002
53.46
75.62
44.52
69.43
Existing MMT Systems
Deubly-ATT (Calixto ol., 2017)
ResNet
41,45
68.04
33,95
61.83
29.63
59.21
61.99
81.12
53.72
75.71
45.16
70.25
Imagination (Elliot and Kūdiz, 2017)
ResNet
32.89
UVR-NMT (Zhang et al., 2020)
ResNet
40.79
32,16
29.02
61:00
53.20
43.71
5
Goted Fasion (Wa et al., 2021)
ReaNet
41.96
67.84
33.59
61.94
29.04
56.15
61.69
80.97
54.85
76.34
44.86
70.51
6
Selective Attention (Li et al., 2022n)
ViT-hase
41.93
68,55
33.60
61,42
31.14
56,77
62.48
81.71
54.44
76.46
44.72
71.20
7
IKD IMD (Peng al., 2022)
ResNet
41.28
58.93
33,83
53.21
30.17
48,93
62.53
77.20
54.84
71.87
PLUVR (Fung and Feng, 2022)
ResNei
40.30
33:45
30,28
61.31
53.15
43.65
Uli 2022)
ResNet
VALHAELLA al., 20222)
VQGAN VAR
42.60
35.10
62.80
30.70
57.60
63.10
81.80
56.00
77.10
46.40
71.30
11
Noise robust (Ye et al., 2022)
Resnet
42,56
59.98
35.09
54.51
31.09
50.46
63.24
77.54
55.48
72.62
46.34
67.40
12
Multimodal-minee mixup (ee and Gan, 2022)
Resnet
41.77
58.93
33:07
51.85
29.90
49.09
62:23
76.85
55.18
73.37
44.42
66.41
Our MMT
Systems
Selective Attention
MAB-bisc
41.65
68,43
34,27
62,08
29.75
56.68
61,87
80.76
54.07
76,
44.65
70.43
MMT-VQA
MAF-hase
42,55
69.00
34,58
61.99
30.96
57.23
62.24
81.77
54.89
76.53
45.75
71.21
Table 4: BLEU (B) and METEOR (M) scores of Multi30K En- and En-Fr tasks. Some of the results are from Li et al. (2022a)'s work.
English->>
Frenchh
Feature
Method
Test2016
Test2017
MSCOCO
System
Feature
M
MMT
40.68
32.49
28.61
Transformer tiny
30.52
57.37
38.05
64.01
ViT-hase
MMT-VQA
41.1410.46
33.3510.86
28,53-9.08
Selective Aliention
ViT
30.41
57.29
38.10
64.17
MMT
41.65
34.27
29.75
Selective Atiention
MAE
31.09
38.18
64.55
MAE-base
MMT-VQA
MAE
31,74
58.49
38.72
64.87
MMT-VQA
42,5510.90
34.5g10.31
30.96"
MMT
41.01
33.73
29.41
BLIP-base
Table 5: BLEU (B) and METEOR (M) scores of Multi30K En-De and En-Fr tasks on the Test2018 test set.
MMT-VQA
41.9110.90
34.0310.30
29.81 "0.40
MMT
41.28
33.83
29.66
CLIP-hase
MMT-VQA
41.4310.15
33
33.9019.07
30.32 7.06
Type
Noun
Character
Color
Number
Count
5133
18423
5303
141
Table 7: BLEU scores of Multi30K En-De task of MMT- VQA model compared with MMT (Selective Attention) model with Different Pre-trained Vision Models.
Table 6: Answer type statistics of Multi30K- VQA.
we employed hand-crafted rules to refine 605 of these pairs and annotated the remaining 407 with sentence-level annotations via 3 annotators. Our fi- nal prompt, fine-tuned through 5 iterations, yielded the best output among various tests. Answer-type distributions are shown in Table 6.
brary (Ott et al., 2019), supplemented by the in- tegration of Vision Pre-trained Models from the Huggingface'. The optimization process is facili- tated by the Adam optimizer, with parameters &1 and f3 set at 0.9 and 0.98 respectively, and estab- lished at 10 Learning rate is set to 0.005 and a warmup phase consisting of 2000 update steps. The model incorporates a dropout rate of 0.3 and smbot-nno of 0.1. The training process is conducted on 2 GeForce RTX 3090 GPUs, with each training batch comprising 1096 tokens per GPU. To ensure fair comparison with the pre- vious work, we adopt an early stopping strategy, triggered if there is no performance improvement on the validation set over 10 epochs. For infer- ence, we average the last 10 checkpoints for robust performance. Note that our À is set to 0.3, which achieves the best in our preliminary experiments.
4.2 Model Settings
Considering the small size of the Multi30K data set, we follow the prior work using Transformer- Tiny as the base configuration (Wu et al., 2021). In general, the MMT-VQA Multi-task Learning Model consists of Pre- Tre-Trained Image Encoder, a 4-layer Text Encoder, two Selective Attention layers, and two 4-layer Decoders. For each encoder and decoder layer, the hidden size is set to 128, the intermediate size of Ferd-Forward Networks is set to 256, and the number of heads set 4.
Our implementation was based on Fairseq li-
'htnss//huggingffcc..o/

---

VQA-Loss
MMT-Loss
43.0
3.5
18.0
ALBU
31.0
41.0
3.0
15.0
40.0
33.0
2,5
12.0
39.0
32.0
Test2017
2.0
9.U
1.5
6.0
MMT:
31.0
1.0
3.0
30.0
MMT-VQA:
29.0
5
10
15
20
25
HATT-VQA-Wroo---Ae
28.0
Lpoch
MAMT-VQA-Wromg-QOaee
27.0
MSCOCO
Figure 4: The change curve of two loss in the training process of multi-task learning model.
Figure 5: BLEU scores of different data processes to Question and Answer.
4.3 Results
losses, LMMT (in blue) and LVQA (in red) during the training. Given the minimal changes observed in the latter stages of training, we have chosen to present only the trends from the initial 25 epochs. It is evident that both models were able to converge stably at certain level, validating that the auxil- iary loss is not a regularization. We also observed rapid convergence during the training of the VQA tasks. This can attributed to our strategic choice of using shorter vocabulary as answer words dur- ing data generation, which significantly reduced the learning difficulty for the model.
Table 4 presents the results of various MMT sys- tems evaluated on the Multi30K dataset. All mod- els were evaluated in terms of BLEU and METEOR scores on six separate test for En-De and En- Fr tasks. Upon comparison with the most recent and strong prior works, MMT-VQA can outper- form most of competing MMT systems terms of the BLEU and METEOR metrics, demonstrating the effectiveness of strengthening cross modality interactions. For more convincing result, we fur- ther conducted experiments on Test2018 in Table 5, where similar phenomenon could be observed. Experimental results in Table demonstrate the general applicability of MMT-VQA multitask learning framework. We can see that MMT-VQA is often beneficial when transitioning to other ad- vanced vision features, encompassing both pre- trained vision models and vision-languag pre- trained models, such as CLIP and BLIP. The MMT- VQA method exhibits improvements across various sets, with the magnitude of these enhancements varying. Remarkably, the most substantial incre- ment observed was as high as 1.21, thus reinforcing the general applicability and robustness of our pro- posed methodology.
5.2 Incongruent Question-Answrring Pairs
We further explored the effects of incongruent question-answering pairs training substantiate our claims. Under the assumption that if VQA acts as a noise signal during training, randomizing the correct Question and Answer the training data should not significantly impact the results. To scru- tinize this, we conducted additional experiments, employing the BLIP model for visual feature ex- traction due to its notable performance improve- ment. The experimental results, depicted in Figure 5, reveal significant impact on the performance of MMT-VQA- Saern-A--rro---oeeee Wrong- and MMT-VQA- across all three test sets. Some metrics even regressed below the performance of the Selective Attention, which did not employ the MMT-VQA method. We attribute these perfor- mance decreases to the failure of model to cor- rectly learn the correspondences and probing sig- nals among the Question, Answer and Image triad. This further emphasizes the importance of correct alignment and interaction in improving the efficacy of MMT systems.
5 Analysis
Here, we aim to take more in-depth analysis on the importance of MMT-VQA. Without specific notion, all results were conducted on Multi30K En-De task.
5.1 Illustrating Training Loss
First, would like validate whether the im- provements come from the strengthened interaction between modalities or just act as an regularization signal. Figure plots the curves of two training

---

Model
Process
Test2016
Test2017
MSC0CO
MMT-VQA
MMT MMT-VQA MMT-VQA
41.01
33.73
29.41
43.00
Question-xSource
41.21
32.74
29.07
42.50
Table 8: BLEU scores of exploring whether the improve- ment comes from introduction of Questions training phase and contains other information.
42.00
41.50
41.00
0.1
0.2
0.4
5.3 Is VQA really helping for cross-modal interaction?
Confusion Rate
Figure 6: BLEU scores of the two models on the Test2016 test set with different levels of confusion.
Since the complete VQA task input consists two parts of data, Question and Image, we intend to evaluate whether VQA brings stronger cross-modal interaction capabilities to the MMT model from the perspective of processing the two data.
System
Feature
Test2016-con
Transformer Tiny
37.89
2
MMT
BLIP-base
38.53
MMT-VQA
BLIP base
39.62
MMT
MAE-bas 39.12
Question Perspective A defining feature of MMT-VQA compared to traditional MMT mod- els is the added question input, thus raising the concern: Is MMT-VQA's performance boost due this extra information? We validated this by re- placing all questions in the Multi30K- VQQA dataset with source text during training. Table (row 3) shows the model's Ei-De task performance, with the first two rows for comparison against bascline MMT and MMT-VQA. The substitution leads to marked decline in performance across all test sets. Against the baseline, minor gains appeared only one set, while losses were significant on the others. These results confirm that MMTsVQA's im- provements are not solely due to the added question data, but rather indicate the model's ability to lever- age semantic cues from questions for enhanced multimodal interactions.
VMT-VQA
MAE-base
39.71 19.71
10.55
Table BLEU scores of Multi30K En-De task on the Test2016-com test set.
5.4 Test2016-con: LLM-Driven Model Evaluation
We used LLMs to identify confusing words within samples of Test2016 and subsequently formed a subset, Test2016-con, which consists of samples with these words. This subset prioritizes image disambiguation, necessitating the interplay of two modalities. Table 9 illustrates that our MMT-VE VQA method notably exceeds the performance of the baseline system with MAE and BLIP vision fea- tures, suggesting a stronger integration with image modality for disambiguation. It is noteworthy that we will also open-source this testset in our code- base.
Image Perspective We evaluated the visual con- text sensitivity using adversarial methods, follow- ing the inconsistent decoding setup in prior studies (Caglayan et al., 2019, 2021). By replacing ran- dom proportions of image samples with irrelevant images, we performed ablation experiments and compared BLEU performance changes between the MMT baseline (Selective Attention) and MMT- VQA methods. As seen in Figure 6, without our strengthened modality interactions, the baseline shows higher sensitivity and more overall fluctu- ation without clear downward trend, suggest- ing inadequate use of image information. On the other hand, MMT-VQA, despite similar fluctuation levels, maintained a downward trend, indicating stronger dependence on visual information accu- racy and improved visual modality interaction.
5.5
Case Study
We also make quantitative study shown in the Table 10. In the first case, we observed that the MMT system, despite employing selective attention mechanism and Vision Transformer (ViT) features, faltered the translation of the word "up" in the German context. Specifically, omitted this word entirely. In contrast, our MMT-VQA system produced the accurate German translation "hochgelegten" which effectively captures the no- tion of "elevated" or "up"" Another noteworthy ob- servation that conventional MMT systems strug- gle with accurate color translation when a sentence contains multiple color descriptors. This limitation was effectively mitigated in our MMT-VQA sys-

---

SRC
: A man talks on the phone with his feet up.
REF
Ein mann telefoniert mit hochgelegten füßen.
MMT
Ein mann telefoniert mit dea füßßn.
MMT-VQA:
Ein mann telefoniert mit hochgelegten füßen.
SRC
boat with red, white and blue sails docking at pier.
REF Ein boot mit roten, weißen und blauen segeln dockt an einem pier an
MMT (SA= : Ein boot mit rot blauen segeln an einem pier
MMT-VQA: Ein boot mit roten, weißen und blauen segeln an einem pier.
Table 10: Qualitative examples to demonstrate enhanced cross-modal capability. Strikethrough and bold words present the incorrect and good lexical choices.
tem by the incorporation of probing signals. These signals. particularly those covering color question- answering pairs, significantly enhanced the sys- tem's ability to resolve such translation ambiguities. Our analysis suggests that the failure the MMT system with selective attention in translating seem- ingly simple sentences stems primarily from its inability to disambiguate certain words effectively.
dealing with complete textual contexts, concern that also forms the crux of the present study, Dis- tinctively, we focus on fortifying text-visual inter- actions through the deployment of probing signals during training. Our approach shares thematic sim- ilarity with Ji et al. (2022), who explored this issue from mutual information standpoint.
7 Conclusions
6 Related Work
The task of MMT forms an intersection between NLP and CV, with the goal of enhancing translation outcomes through the integration of image-base based context. The earlier approaches focused primarily on the effective umalgamation of these dual-modal representations (Calixto and Liu, 2017: Elliott and Kádár, 2017; Delbrouck and Dupont, 2017). Sub- sequent research, however, began to question the significance of visual modality, finding that irrele- vant images did not substantially influence transla- tion quality. Indeed, the absence of images did not lead to a substantial drop in BLEU scores (Elliott, 2018). Caglayan et al. (2019) provided further in- sight into this phenomenon by demonstrating that visual modality could offer utility in contexts with limited linguistic data, but its influence was less pronounced when full sentences were available. In recent studies, Wu al. (2021) attributed im- provements to the effects of regularization during training. They further showed that even a tiny con- figuration could outperform earlier models in terms of BLEU score. Concurrently, additional efforts include the use of hallucination representations aug- mentation (Li et al., 2022c), Noise-Input to mask the additional noise thereby guiding the model to focus on meaningful interactions (Ye al., 2022), and Dual-level interactive mixup for feature align- ment (Ye and Guo, 2022) have emerged. However, existing MMT systems still face difficulties when
In this work, we offer a novel perspective on mul- timodal machine translation (MMT), attributing reduced system sensitivity to visual information in complete text inputs insufficient cross-modal in- teraction rather than image information redundancy. We propose a unique approach that generates Vi- sual Duestion-Annserrinn (VQA) style pairs from source text via LLMs to model the probing sig- nal in MMT explicitly. As a bonus, we propose a Multi30K-VO dataset and design MMT-VQA multitask learning framework. Our methodology is validated by experimental results two represen- tative benchmarks, evidencing its effectiveness and driving forward the MMT research.
A Acknowledgmentts
This work was supported in part by the National Science Foundation China (No.62276056), the National Key R&D Program of China, the China HTRD Center Project N0..2020AA0011 Natural Science Foundation of Liaoning Province of China (2022-KF-16-01) the Yunnan Provin- cial Major Science and Technology Special Plan Projects (N0.202103A the Fundamental Research Funds for the Central Universities (Nos. N2216016, N2216001, and N2216002), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No.B 6009).
the

---

Limitations
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 66x 16 words: Transformers for image recognition scale.
The performance of our proposed model will de- pend on the quality of the generated dataset, and the quality of the data set mainly depends on the hints we prepare for the large model. Therefore, although we make good use of the ability of large model, how to design a better prompt and reduce the influence of human factors on prompt quality is still worth exploring in the future. In addition, we haven't explored a better model architecture in depth, and it is very likely that the model we proposed is not the most suitable for this task.
Desmond Elliott. 2018. Adversarial evaluation of mul- timodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2974-2978, Brussels, Belgium. Association for Computational Linguistics.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine transla- tion and multilingual image description. In Proceed- ings of the Second Conference on Machine Transla- tion, pages 215-233, Copenhagen, Denmark. Associ- ation for Computational Linguistics.
References
Ozan Caglayan, Loic Barrault, and Fethi Bougares. 2016. Multimodal attention for neural machine trans- lation. CoRR, 1e09.0300..
Desmond Elliott, Stella Frank. Khalil Sima' an, and Lu- cia Specia. 2016. Multi30K: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop Vision and Language, pages 70- 74, Berlin, Germany, Association for Computational Linguistics.
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, and Lucia Specia. 2021. Cross-lingual visual pre- training for multimodal machine translation. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1317-1324, Online. Association for Computational Linguistics.
Desmond Elliott and Ákos Kádár. 2017. Imagination improves multimodal translation. In Proceedings of the Eighth International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 130-141, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loic Barrault. 2019. Probing the need for visual context in multimodal machine translation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume Min-
Qingkai Fang and Yang Feng. 2022. Neural machine translation with phrase-level universal visual repre- sentations. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers). pages 5687-5699 Dublin, Ireland. Association for Computational Linguistics.
I (Long and Short Pupers). pages neapolis, Minnesota. Association for Computational Linguistics.
Iacer Calixto and Qun Liu. 2017. Incorporating global visual features into attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing, pages 992-1003, Copenhagen, Denmark. Asso- ciation for Computational Linguistics.
Stig-Ame Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, and Raúl Vázquez. 2018. The MeMAD submission to the WMT18 multimodal translation task. In Proceedings of the Third Con- ference on Machine Translation: Shared Task Papers, pages 603-611, 1, Belgium. Brussels. Association for Computational Linguistics.
Iacer Calixto, Qun Liu, and Nick Campbell. 2017. Doubly-attentive decoder for multi-modal neural ma- chine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 1913-1924, Vancouver, Canada. Association for Computational Linguisties.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Dollár, and Ross Girshick. 2022. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009.
Bannit Delbrouck and Stéphane Dupont. 2017. An empirical study the effectiveness of images in multimodal neural machine translation. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 910-919, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Baijun Ji. Tong Zhang, Yicheng Zou, Bojie Hu, and Si Shen. 2022. Increasing visual awareness mul- timodal neural machine translation from an infor- mation theoretic perspective. In Proceedings of the 2022 Conference on Empirical Methods in Natu- rul Language Processing, pages 6755-6764, Abu

---

Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Xin Qian, Ziyi Zhong, and Jieli Zhou. 2018, Multi- modal machine translation with reinforcement learn- ing. CoRR, abs/I 805.02356.
Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, and Lucia Specia. 2018. Sheffield submis- sions for WMT18 88 multimodal translation shared task. In Proceedings of the Third Conference Machine Translation: Shared Task Papers, pages 624-631, Belgium, Brussels. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 144888763. PMLR.
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022a. On vision features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 6327-6337, Dublin, Ireland. Association for Computational Linguistics.
Lucia Specia, Stella Frank, Khalil Sima' an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image descrip- tion. In Proceedings of the First Conference on Ma- chine Translation: Volume 2, Shared Task Papers, pages 543-553, Berlin, Germany. Association for Computational Linguistics.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b. Blip: Bootstrapping language-image pre-training for unified visionllaguuag understand- ing and generation. International Conference on Machine Learning, pages 12880- PMLR.
Yi Li. Rameswar Panda, Yoon Kim, Chun Fu Richard Chen, Rogerio S Feris, David Cox, and Nuno Vascon- celos. 2022c. Valhalla: Visual hallucination for ma- chine translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 521 5226-5.
Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021. Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6153-6166 Online. Association for Computational Linguisties.
Jindfich Libovický and Jindřich Helcl. 2017. Atten- tion strategies for multi-source sequence-to-sequene learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 196-202, Vancouver, Canada. Association for Computational Linguisties.
Junjie Ye and Junjun Guo. 2022. Dual-level inter- active multimodal- mixup encoder for multi-modal neural machine translation. Applied Intelligence. 12(12):14194-1422 1419-144203
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tionfor Computational Linguistics (Demonstrations). pages 48-53, Minneapolis. Minnesota. Association for Computational Linguistics.
Junjie Ye, Junjun Guo, Yan Xiang, Kaiwen Tan, and Zhengtao Yu. 2022. Noise-robust cross-modal in- teractive learning with text2image mask for multi- modal neural machine translation. In Proceedings of the 29th International Conference on Computational Linguistics, pages
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurlPS.
Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal vi- sual representation. In International Conference on Learning Representations.
Ru Peng, Yawen Zeng, and Jake Zhao. 2022. Distill the image to nowhere: Inversion knowledge distillation for multimodal machine translation. Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 2399, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.

---

A Sample of Multi30K-VQA
Table Il shows samples that we have picked out from the -uLI dataset.
B Case Study on Type of Error
We try figure what type of error additional VQA training helps the most. We conduct case studies from three perspective:
Perspective of Part-of-Speech (POS): We ama- lyzed translation quality using POS tagging. For each POS type in the hypothesis, occurrence in sentences and then segmented the test set accordingly. We compared BLEU scores for the MMT-VQA and MMT (Selective Attention) methods across these subsets. To isolate the effect specific POS types, we calculated the score dif- ferences and adjusted for word frequency. Our re- sults highlight that adjectives (JJ) and plural nouns (NNS) most significantly improve translation qual- ity when using the VQA method.
counted its
Perspective of Long and Difficult Words: We found that 3% vocabulary consists long and difficult words. After calculating that the 97th per- centile of all word lengths is 9, we divided the test set based on whether sentences contained long and difficult words. We found that the MMT-VQA method improved the BLEU score by 0.98 points on the test subset with long and difficult words com- pared the selective attention baseline method (Li et al., 2022a). The improvement on the subset with- out long and difficult words was 0.67 points.
Perspective of Long and Difficult Sentences: We then completed the identification of long and difficult sentences. We first determined the number of clusters for sentence length categorization to be 2 based on the elbow method, dividing sentence lengths into two categories: long sentences and short sentences. We found that during the transla- tion of long sentences, the BLEU score difference between the VQA and SA methods was 1.08, and for short sentences, it was 0.63 1.63-nearly provement.
-nearly a 70% im-

---

Source
Two young, white males are outside near many bushes.
Question:
What are the males near?
Answer
Bushes.
Type
Noun.
Source A man in a neon green und orange uniform is driving on green tructor.
Question:
What colors are on the man's 's uniform?
Answer
Neon green and orange.
Type
Color.
Source
Two young toddlers outside on the grass.
Question:
Who is outside on the gruss?
Answer Toddlers.
Type
Character.
Source A young man is presenting another woman with a cake with three candles on top-
Question: How many candles are the cake?
Answer Three.
Type
Number.
Source The small child climbs on red ropes on playground.
Question:
What child climbing on? Answer Rope.
Type None.
Source A man in blue shirt is standing ladder cleaning window
Question: What color the mun's shirt?
Answer
Blue.
Type
Color.
Source
A beautiful bride walking sidewalk with her husband. Who walking the sidewalk?
Question:
Answer
Bride and hushand.
Type
Character.
Source sports team made up of 14 waring white t-shirt and pink tags.
Question: How many women are on the sports team?
Answer
14.
Type
Number.
Source Woman blue jacket shows off her vintage car in front nathalie's dollhouse.
Question: What the woman showing off?
Answer
Vintage car.
Type
Noun.
Source A beautiful photogruph of church at sunset with birds flying overhead and orange-red sky.
Question:
What color is the sky? Answer Orange-red.
Type
Color.
Source
A tisherman reeling his rod while another relaxes in boat on water..
Question: Who is reeling the rod?
Answer Fisherman.
Type Character.
Source A group people at a party with two cakes on the table
Question: How many cakes are on the table?
Answer Two.
Type Number.
Table 11: Qualitative samples of Multi30K-VOA -VQA.


Reviews:
[{"review_id": "FTBxToTqvI", "reviewer": "Reviewer_ntqX", "summary": "", "questions": "Could you give a case study to show the superiority of your method?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments on our paper. We think all the issues would be well addressed in our improved version.\n\nHere are the detailed response for your main concerns:\n\n- We will add the comparison with the pointed missing reference. Thanks for pointing out this issue. And we have already included these results in the following table.\n\n- Since most of previous work reported the results on test2016, test2017 and MSCOCO. Towards a fair comparison with theirs,  we only report results on these three testsets. For a more convincing result, we further conducted experiments on Test2018.\n\n  | System              | Feature | Test2018 (BLEU) | Test2018 (METEOR) |\n  | ------------------- | ------- | --------------- | ----------------- |\n  | Transformer-tiny    | -       | 30.52           | 57.37             |\n  | Selective Attention | ViT     | 30.41           | 57.29             |\n  | Selective Attention | MAE     | 31.09           | 58.17             |\n  | MMT-VQA             | MAE     | 31.74           | 58.49             |\n\n  As is shown in the table, our MMT-VQA can achieve strong performance on test2018 in terms of both BLEU and METEOR\n\n- Thanks for pointing out the missing meteor scores and we show them as below:\n\n  | #    | System              | Feature   | Test2016 BLEU | Test2016 METEOR | Test2017 BLEU | Test2017 METEOR | MSCOCO BLEU | MSCOCO METEOR | Test2018 BLEU | Test2018 METEOR |\n  | ---- | ------------------- | --------- | ------------- | --------------- | ------------- | --------------- | ----------- | ------------- | ------------- | --------------- |\n  | 1    | Transformer Tiny    | -         | 41.02         | 68.22           | 33.36         | 62.05           | 29.88       | 56.64         | 30.52         | 57.37           |\n  | 2    | Doubly-ATT          | ResNet    | 41.45         | 68.04           | 33.95         | 61.83           | 29.63       | 59.21         | -             | -               |\n  | 3    | Imagination         | ResNet    | 41.31         | 68.06           | 32.89         | 61.29           | 29.90       | 56.57         | -             | -               |\n  | 4    | UVR-NMT             | ResNet    | 40.79         | -               | 32.16         | -               | 29.02       | -             | -             | -               |\n  | 5    | Gated Fusion        | ResNet    | 41.96         | 67.84           | 33.59         | 61.94           | 29.04       | 56.15         | -             | -               |\n  | 6    | Selective Attention | ViT-base  | 41.93         | 68.55           | 33.60         | 61.42           | 31.14       | 56.77         | -             | -               |\n  | 7    | IKD-MMT             | ResNet    | 41.28         | 58.93           | 33.83         | 53.21           | 30.17       | 48.93         | -             | -               |\n  | 8    | PLUVR               | ResNet    | 40.30         | -               | 33.45         | -               | 30.28       | -             | -             | -               |\n  | 9    | IVA                 | ResNet    | 41.77         | 68.60           | 34.58         | 62.40           | 30.61       | 56.70         | -             | -               |\n  | 10   | VALHALLA            | VQGAN VAE | 42.60         | 69.30           | 35.10         | 62.80           | 30.70       | 57.60         | -             | -               |\n  | 11   | Noise-robust        | Resnet    | 42.56         | 59.98           | 35.09         | 54.51           | 31.09       | 50.46         | -             | -               |\n  | 12   | Multimodal-mixup    | Resnet    | 41.77         | 58.93           | 33.07         | 51.85           | 29.90       | 49.09         | -             | -               |\n  | 13   | Selective Attention | ViT-base  | 40.68         | 67.80           | 33.35         | 61.00           | 28.61       | 55.30         | 30.41         | 57.29           |\n  | 14   | Selective Attention | MAE-base  | 41.65         | 68.43           | 34.27         | 62.08           | 29.75       | 56.68         | 31.09         | 58.17           |\n  | 15   | MMT-VQA             | MAE-base  | 42.55         | 69.00           | 34.58         | 61.99           | 29.94       | 57.23         | 31.74         | 58.49           |\n\n- Our experiments reveal that the model is notably sensitive to the quality of question-answer pairs within the Multi30k-VQA dataset. Specifically, when a correct question is erroneously paired with an incorrect answer, the model exhibits a marked decline in performance after being trained on such a flawed dataset. This observation necessitates a cautious approach in creating high-quality datasets for training. We will restructure this section to provide a more comprehensive understanding of this critical dependency, thereby aiding in the interpretation and generalization of our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They introduce LLM to MMT, which is interesting.\n2. Experimental results on Multi30K En-De and En-Fr demonstrate the effectiveness both in terms of BLEU and specific test sets.\n3. The figures are clear and beneficial for readers to understand the paper.", "reasons_to_reject": "1. The paper don't compare their models with more recent SOTAs [1-3], so it can not get higher soundness.\n2. You should provide the results on more datasets, such as Test2018.\n3. You should provide the METEOR results, which is also reported in recent works [1-5].\n4. The Figure 5 is not clear, you should give more explanation about it.\n\n[1] Yi Li, Rameswar Panda, Yoon Kim, Chun-Fu Richard Chen, Rogerio S Feris, David Cox, and Nuno Vasconcelos. 2022. VALHALLA: Visual Hallucination for Machine Translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5216–5226.\n\n[2] Junjie Ye, Junjun Guo, Yan Xiang, Kaiwen Tan, and Zhengtao Yu. 2022. Noise\u0002robust Cross-modal Interactive Learning with Text2Image Mask for Multi-modal Neural Machine Translation. In Proceedings of the 29th International Conference on Computational Linguistics. 5098–5108.\n\n[3] Junjie Ye and Junjun Guo. 2022. Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation. Applied Intelligence 52, 12 (2022), 14194–14203.\n\n[4]  Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 6153–6166.\n\n[5] Li B, Lv C, Zhou Z, et al. On Vision Features in Multimodal Machine Translation[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 6327-6337.", "questions_to_authors": "Could you give a case study to show the superiority of your method?", "paper_topic_and_main_contributions": "The paper release a Multi30k-VQA dataset via using LLMs API and propose a MMT-VQA Multi-Task Learning Framework to explicitly model the probing signal in MMT.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Yi Li, Rameswar Panda, Yoon Kim, Chun-Fu Richard Chen, Rogerio S Feris, David Cox, and Nuno Vasconcelos. 2022. VALHALLA: Visual Hallucination for Machine Translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5216–5226.\n\n[2] Junjie Ye, Junjun Guo, Yan Xiang, Kaiwen Tan, and Zhengtao Yu. 2022. Noise\u0002robust Cross-modal Interactive Learning with Text2Image Mask for Multi-modal Neural Machine Translation. In Proceedings of the 29th International Conference on Computational Linguistics. 5098–5108.\n\n[3] Junjie Ye and Junjun Guo. 2022. Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation. Applied Intelligence 52, 12 (2022), 14194–14203."}}, {"review_id": "LQXFN4cecS", "reviewer": "Reviewer_EZME", "summary": "", "questions": "See [Reasons To Reject]. Additional suggestion: \n- The colors used in Figure 5 are difficult to read.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and the time you've invested in reviewing our paper. We highly value your feedback and wish to address each of your concerns in detail.\n\n1. **Novelty and Contributions of the Paper**: We appreciate the opportunity to clarify the unique aspects of our work. While we acknowledge that incorporating additional signals for training is not a new concept, the novelty of our work resides in the specific design and utility of probing signals during the training phase.\n\n   - **Effective Probing Signals**: Our probing signals are designed to assess the efficacy of Multimodal Translation (MMT) systems in leveraging visual patterns, drawing inspiration from previous work (Caglayan et al., 2019; Li et al., 2022). However, what sets our work apart is the application of these probing signals to actively enhance the interaction between visual and textual modalities—a line of inquiry that has not been previously explored. We believe that the incorporation of these signals into a multi-task learning framework, while straightforward, yields robust results.\n   - **Multi30k-VQA Dataset**: Another noteworthy contribution is the Multi30K-VQA dataset that we intend to release. This dataset comprises high-quality, task-specific question-answer pairs in a text-image setting, and aims to provide a fertile ground for subsequent research. We envision this dataset as a valuable resource for the community, potentially inspiring future works in this space.\n\n   We believe that these elements, taken together, distinguish our work from incremental advancements in the field and provide meaningful contributions to the research community.\n\n2. **Missing Statistics of QA Dataset**: You're correct that comprehensive statistics of our QA dataset would strengthen the paper. We give the detailed statistic below:\n\n   As is shown in the table. We collect the Multi30k-VQA data using the strong power of LLMs. As depicted in Figure 2, our prompt design for guiding the LLM consists of three key components:\n\n   - **Task Description**: Initially, we provide an explicit task description to make the LLM aware of the intended output, focusing on generating high-quality question-answer pairs relevant to the task at hand.\n   - **Constraints and Requirements**: Following the task description, we impose specific requirements to minimize the chance of hallucination and to direct the model towards generating the intended output format, including nouns, topic numbers, colors and characters. Note that color and characters are two more representative samples for LLM to ask questions than the other two types.\n   - **Two-Shot Demonstrations**: Lastly, we supply the LLM with two demonstration examples that serve as a template for the expected format of question-answer pairs.\n\n   The resulting corpus contains 29,000 unique question-answer pairs, each generated according to our specifications. While we have designed four types of questions for this task, we intentionally provide only two simpler examples (color and characters) in the prompt. This strategic choice is made to prevent the LLM from disregarding simpler question formats and solely generating complex ones, irrespective of their suitability for a given context. Overall, the prompt here has been carefully tuned, and the output is the optimal choice as far as we tested among various prompts. The percentage of the four types are:\n\n   | Type      | Count |\n   | --------- | ----- |\n   | Noun      | 5133  |\n   | Character | 18423 |\n   | Color     | 5303  |\n   | Number    | 141   |\n\n   \n\n3. **Prompt Design for In-Context Learning**: We understand the need for deeper exploration here.  Actually, as we emphasized above, both the prompt and the demonstration have been carefully designed and varified. We found there is no significant improvement when we add more demonstrations whatever from the number or the question types. In the revised paper, we will provide further examples and discuss how these choices affect the quality of the final synthetic data.\n\n4. **Comparison to State-of-the-Art Methods**: We appreciate this comment and will add the missing references as well as comparative evaluations with state-of-the-art methods[1] [2] to substantiate our contributions. Here, we want to do a detailed comparison between our MMT-VQA with these two related work.\n\n   - VALHALLA: This work introduced a visual hallucination framework called VALHALLA, which only requires the source sentence for reasoning and employs hallucinated visual representations for multimodal machine translation. Specifically, given a source sentence, an autoregressive hallucination transformer is used to predict a discrete visual representation from the input text, and this is then combined with the text and hallucinated representations to obtain the target translation. VALHALLA achieves significant BLEU improvements and we think our Multi30K-VQA can also bring additional benefits for theirs framework.\n   - Noise-robust Cross-modal Interactive learning: This work mainly aim to reduce the noise interference within in the image via mask strategy. The method achieves quite strong performance, but the METEOR is somehow strange compared with previous work. We also include these results in the following table. \n   - Mutual information perspective: This work mainly used mutual information to quantify the source specific-information and the target-specific information and proposed two strategies to better utilize the visual signals. The core motivation is to increase the visual awareness since MMT system concentrates more on the textual input. The main framework is orthogonal to ours and we will investigate the fusion of two methods in the revised manuscript.\n\n   [1] Yi Li et al. VALHALLA: Visual Hallucination for Machine Translation. CVPR 2022\n\n   [2] Junjie Ye et al. Noise-robust Cross-modal Interactive Learning with Text2Image Mask for Multi-modal Neural Machine Translation. COLING 2022\n\n   We will add the aforementioned comparisons in the improved version. More experiments on whether MMT-VQA can work well upon these strong MMT systems would be conducted.\n\n5. **Ablation Study**: Your point about unclear sources of performance gains is well taken. An ablation study has been already conducted which was shown in Table 4. By comparing #10 and #11, both two systems were based on the MAE-base model. The only difference is that MMT-VQA used the probing signals during training. Thus the improvements over #10 show the effectiveness of the additional VQA pairs. Moreover, Table 7 also shows that MMT systems based on both BLIP-base and MAE-base vision features can gain benefits on the special designed test set. On the other hand, we refer you to the results in Figure 5, that when the answer or the question (of the VQA pairs generated by LLM) is wrong, our model suffers from BLEU degradation. This also indicates the training signals provided by the additional VQA pairs are indeed helpful. We think these results can address your concerns. We also make a quantitative case study as follows:\n\n   ```\n   src:  a man talks on the phone with his feet up . \n   \n   ref:  ein mann telefoniert mit hochgelegten füßen . \n   \n   MMT(SA):  ein mann telefoniert mit den füßen . \n   \n   MMT-VQA:  ein mann telefoniert mit hochgelegten füßen .\n   ----------------------------------------------------------------------------\n   src:  a boat with red , white and blue sails docking at a pier . \n   \n   ref:  ein boot mit roten , weißen und blauen segeln dockt an einem pier an . \n   \n   MMT(SA):  ein boot mit rot-blauen segeln an einem pier . \n   \n   MMT-VQA:  ein boot mit roten , weißen und blauen segeln an einem pier .\n   ```\n\n   In the first case, we observed that the MMT system, despite employing a selective attention mechanism and Vision Transformer (ViT) features, faltered in the translation of the word \"up\" in the German context. Specifically, it omitted this word entirely. In contrast, our MMT-VQA system produced the accurate German translation \"hochgelegten,\" which effectively captures the notion of \"elevated\" or \"up.\"\n\n   Another noteworthy observation is that conventional MMT systems struggle with accurate color translation when a sentence contains multiple color descriptors. This limitation was effectively mitigated in our MMT-VQA system by the incorporation of probing signals. These signals, particularly those covering color question-answer pairs, significantly enhanced the system's ability to resolve such translation ambiguities.\n\n   Our analysis suggests that the failure of the MMT system with selective attention in translating seemingly simple sentences stems primarily from its inability to disambiguate certain words effectively.\n\n6. **Comparison with Close Work**: We agree that a comparison with work [b] is crucial. Similar question with the Weakness 4. \n\n7. **Comprehensive Assessment and METEOR Metric**: We understand that incorporating the METEOR metric can provide a more comprehensive assessment. We will include this in the main tables.\n\n   ## Multi30k En-De:\n\n   | #    | System              | Feature   | Test2016 BLEU | Test2016 METEOR | Test2017 BLEU | Test2017 METEOR | MSCOCO BLEU | MSCOCO METEOR | Test2018 BLEU | Test2018 METEOR |\n   | ---- | ------------------- | --------- | ------------- | --------------- | ------------- | --------------- | ----------- | ------------- | ------------- | --------------- |\n   | 1    | Transformer Tiny    | -         | 41.02         | 68.22           | 33.36         | 62.05           | 29.88       | 56.64         | 30.52         | 57.37           |\n   | 2    | Doubly-ATT          | ResNet    | 41.45         | 68.04           | 33.95         | 61.83           | 29.63       | 59.21         | -             | -               |\n   | 3    | Imagination         | ResNet    | 41.31         | 68.06           | 32.89         | 61.29           | 29.90       | 56.57         | -             | -               |\n   | 4    | UVR-NMT             | ResNet    | 40.79         | -               | 32.16         | -               | 29.02       | -             | -             | -               |\n   | 5    | Gated Fusion        | ResNet    | 41.96         | 67.84           | 33.59         | 61.94           | 29.04       | 56.15         | -             | -               |\n   | 6    | Selective Attention | ViT-base  | 41.93         | 68.55           | 33.60         | 61.42           | 31.14       | 56.77         | -             | -               |\n   | 7    | IKD-MMT             | ResNet    | 41.28         | 58.93           | 33.83         | 53.21           | 30.17       | 48.93         | -             | -               |\n   | 8    | PLUVR               | ResNet    | 40.30         | -               | 33.45         | -               | 30.28       | -             | -             | -               |\n   | 9    | IVA                 | ResNet    | 41.77         | 68.60           | 34.58         | 62.40           | 30.61       | 56.70         | -             | -               |\n   | 10   | VALHALLA            | VQGAN VAE | 42.60         | 69.30           | 35.10         | 62.80           | 30.70       | 57.60         | -             | -               |\n   | 11   | Noise-robust        | Resnet    | 42.56         | 59.98           | 35.09         | 54.51           | 31.09       | 50.46         | -             | -               |\n   | 12   | Multimodal-mixup    | Resnet    | 41.77         | 58.93           | 33.07         | 51.85           | 29.90       | 49.09         | -             | -               |\n   | 13   | Selective Attention | ViT-base  | 40.68         | 67.80           | 33.35         | 61.00           | 28.61       | 55.30         | 30.41         | 57.29           |\n   | 14   | Selective Attention | MAE-base  | 41.65         | 68.43           | 34.27         | 62.08           | 29.75       | 56.68         | 31.09         | 58.17           |\n   | 15   | MMT-VQA             | MAE-base  | 42.55         | 69.00           | 34.58         | 61.99           | 29.94       | 57.23         | 31.74         | 58.49           |\n\n   ## Multi30k En-Fr:\n\n   | #    | System              | Feature   | Test2016 BLEU | Test2016 METEOR | Test2017 BLEU | Test2017 METEOR | MSCOCO BLEU | MSCOCO METEOR | Test2018 BLEU | Test2018 METEOR |\n   | ---- | ------------------- | --------- | ------------- | --------------- | ------------- | --------------- | ----------- | ------------- | ------------- | --------------- |\n   | 1    | Transformer Tiny    | -         | 61.80         | 81.02           | 53.46         | 75.62           | 44.52       | 69.43         | 38.05         | 64.01           |\n   | 2    | Doubly-ATT          | ResNet    | 61.99         | 81.12           | 53.72         | 75.71           | 45.16       | 70.25         | -             | -               |\n   | 3    | Imagination         | ResNet    | 61.90         | 81.20           | 54.07         | 76.03           | 44.81       | 70.35         | -             | -               |\n   | 4    | UVR-NMT             | ResNet    | 61.00         | -               | 53.20         | -               | 43.71       | -             | -             | -               |\n   | 5    | Gated Fusion        | ResNet    | 61.69         | 80.97           | 54.85         | 76.34           | 44.86       | 70.51         | -             | -               |\n   | 6    | Selective Attention | ViT-base  | 62.48         | 81.71           | 54.44         | 76.46           | 44.72       | 71.20         | -             | -               |\n   | 7    | IKD-MMT             | ResNet    | 62.53         | 77.20           | 54.84         | 71.87           | -           | -             | -             | -               |\n   | 8    | PLUVR               | ResNet    | 61.31         | -               | 53.15         | -               | 43.65       | -             | -             | -               |\n   | 9    | IVA                 | ResNet    | -             | -               | -             | -               | -           | -             | -             | -               |\n   | 10   | VALHALLA            | VQGAN VAE | 63.10         | 81.80           | 56.00         | 77.10           | 46.40       | 71.30         | -             | -               |\n   | 11   | Noise-robust        | Resnet    | 63.24         | 77.54           | 55.48         | 72.62           | 46.34       | 67.40         | -             | -               |\n   | 12   | Multimodal-mixup    | Resnet    | 62.23         | 76.85           | 55.18         | 73.37           | 44.42       | 66.41         | -             | -               |\n   | 13   | Selective Attention | ViT-base  | 61.65         | 80.96           | 54.13         | 76.06           | 44.40       | 70.48         | 38.10         | 64.17           |\n   | 14   | Selective Attention | MAE-base  | 61.87         | 80.76           | 54.07         | 76.10           | 44.65       | 70.43         | 38.18         | 64.55           |\n   | 15   | MMT-VQA             | MAE-base  | 62.24         | 81.77           | 54.89         | 76.53           | 45.75       | 71.21         | 38.72         | 64.87           |\n\n8. **Statistical Significance**: The results were averaged over three trials and include tolerance scopes to ensure the statistical robustness of our results. We both run the models in seed 1, 42 and 2023, respectively. Thanks for pointing out this important issue. We will carefully revise the experimental setup to avoid the unnecessary misunderstandings!\n\n9. **Intuitive Evidence for Cross-Modal Interactions**: We think this is a similar concern with 5. The provided two cases may address your concern well. Moreover, we do other analyses from three aspects:\n\n   - **From the Perspective of Part-of-Speech (POS):** We analyzed translation quality using POS tagging. For each POS type in the hypothesis, we counted its occurrence in sentences and then segmented the test set accordingly. We compared BLEU scores for the MMT-VQA and SA methods across these subsets. To isolate the effect of specific POS types, we calculated the score differences and adjusted for word frequency. Our results highlight that adjectives (JJ) and plural nouns (NNS) most significantly improve translation quality when using the VQA method.\n\n   - **From the Perspective of Long and Difficult Words:** We found that 3% of vocabulary consists of long and difficult words. After calculating that the 97th percentile of all word lengths is 9, we divided the test set based on whether sentences contained long and difficult words. We found that the MMT-VQA method improved the BLEU score by 0.98 points on the test subset with long and difficult words compared to the selective attention baseline method (Li et al., 2022 ). The improvement on the subset without long and difficult words was 0.67 points.\n\n   - **From the Perspective of Long and Difficult Sentences:** We then completed the identification of long and difficult sentences. We first determined the number of clusters for sentence length categorization to be 2 based on the elbow method, dividing sentence lengths into two categories: long sentences and short sentences. We found that during the translation of long sentences, the BLEU score difference between the VQA and SA methods was 1.08, and for short sentences, it was 0.63—nearly a 70% improvement.\n\n10. **Clarity of Contribution Claims**: We apologize for any confusion generated by the inconsistencies in Lines 85-87. Our contributions are primarily two-fold:\n\n    - **Advanced Visual Features**: Our work delves into the effectiveness of advanced visual features, particularly in the context of MMT systems built on Vision Transformers (ViT). In doing so, we expand upon the research initiated by Li et al., 2022. Although one might intuitively expect that CLIP and BLIP would outperform MAE and ViT after optimization on extensive text-image paired data, our findings indicate otherwise. This observation serves as an invaluable guide for future research in the field, including the study of multimodal Language Models (LLMs).\n    - **Probing Signals**: While probing tasks and other visual-grounded test sets have garnered significant attention, the unanswered question remains: can these evaluative measures be used to actively strengthen MMT systems? In our work, we focus on leveraging these probing signals to enhance the interaction between visual and textual modalities during training. Although we concede that multi-task learning is a well-established method, it serves as an effective conduit to achieve our research goals.\n\nWe hope the aforementioned responses can well addressed your concerns, and we really appreciate for your valuable feedbacks for further improving the quality of our paper. Wish you can have another turn to make a new evaluation towards our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is easy to follow and well-written.\n- The introduction of QA pairs and the design of the model architecture are reasonable and intuitive.\n- The analysis of the experiments conducted is thorough.", "reasons_to_reject": "- The novelty of the paper seems limited in my opinion. As the authors claimed, the main contribution lies in the probing task proposed in [a] for training, it should be noted that the practice of incorporating additional signals for training is not new. The whole paper can be seen as its follow-up incremental work.\n- The paper would benefit from including missing statistics of the constructed QA dataset. \n- In the prompt design phase, there is a lack of further exploration of two examples for in-context learning, which seems to be essential for the quality of the final synthetic data.\n- The paper lacks some comparisons to state-of-the-art methods, please see [Missing References].\n- It is difficult to ascertain whether the improvement in performance primarily stems from better visual features or the additionally constructed QA pairs. It would be beneficial to include an ablation study to analyze the contribution of each component.\n- It is recommended to provide more comparisons with close work [b] that address the same problem, to clarify the merits of the approach. \n- The assessment provided in the paper is not comprehensive. Please include the widely adopted METEOR metric in the main tables as well.\n- The results presented in the paper lack statistical significance. It is unclear whether they are reported for a single trial or averaged over multiple trials. Additionally, the tolerance scope is not mentioned. These details are not reflected in the paper (maybe I missed them, please clarify). \n- In section 5.3, there is a lack of intuitive evidence for cross-modal interactions, as only quantitative analysis is currently provided. It is recommended to include additional qualitative evidence or visualizations to support and enhance the understanding of the cross-modal interactions.\n- Contribution claims are unclear and inconsistent. The authors mention better visual features are beneficial for MMT in Line85-87, which seems to be separated from the previous introduction; in addition, the author shows better results of MAE and CLIP in Table 1, while Line85-87 mentions MAE, BLIP, this is confusing.\n\n[a] Bei Li et al. On Vision Features in Multimodal Machine Translation. ACL 2022\n\n[b] Baijun Ji et al. Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective. EMNLP 2022", "questions_to_authors": "See [Reasons To Reject]. Additional suggestion: \n- The colors used in Figure 5 are difficult to read.", "paper_topic_and_main_contributions": "The main contribution of this paper is the generation of QA pairs according to the source sentences through LLM API, followed by the incorporation of probing signals into MMT, aiming to cross-modal interaction. The claim of the paper is supported by a comprehensive set of detailed experiments and analyses.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please proofread the manuscript."}, "extra_text": {"Missing_References": "[c] Yi Li et al. VALHALLA: Visual Hallucination for Machine Translation. CVPR 2022\n\n[d] Junjie Ye et al. Noise-robust Cross-modal Interactive Learning with Text2Image Mask for Multi-modal Neural Machine Translation. COLING 2022"}}, {"review_id": "Ux1Z9hm3nJ", "reviewer": "Reviewer_acWW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. If you have any other concerns toward this paper, we are glad to make a further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "• This work released a Multi30k-VQA dataset consisting of question-answer pairs.\n• This work proposed an MMT-VQA Multi-Task Learning Framework to explicitly model the probing signal in MMT to strengthen the interactions between text and visual modalities.\n- Experiments and analysis prove the effectiveness of the proposed framework.", "reasons_to_reject": "-", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposed to generate Visual Question-Answering style pairs (Multi30k-VQA dataset) from source text via LLMs to model the probing signal in MMT explicitly. Experimental results on two representative benchmarks show the proposed MMT-VQA framework's effectiveness.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ePj54OJIT", "reviewer": "Reviewer_gdQP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for you comments. We hope the following clarification can address your concerns towards our contribution and the novelty.\n\n1. **Novelty and Contributions of the Paper**: We appreciate the opportunity to clarify the unique aspects of our work. While we acknowledge that incorporating additional signals for training is not a new concept, the novelty of our work resides in the specific design and utility of probing signals during the training phase.\n   - **Effective Probing Signals**: Our probing signals are designed to assess the efficacy of MMT systems in leveraging visual patterns, drawing inspiration from previous work (Caglayan et al., 2019; Li et al., 2022). However, what sets our work apart is the application of these probing signals to actively enhance the interaction between visual and textual modalities—a line of inquiry that has not been previously explored. We believe that the incorporation of these signals into a multi-task learning framework, while straightforward, yields robust results.  \n   - **Multi30k-VQA Dataset**: Another noteworthy contribution is the Multi30K-VQA dataset that we intend to release. This dataset comprises high-quality, task-specific question-answer pairs in a text-image setting, and aims to provide a fertile ground for subsequent research. We envision this dataset as a valuable resource for the community, potentially inspiring future works in this space. While, it is non-trivial to create a high-quality QA dataset or the MMT model may learn from noise signals. Here we also want to highlight some details when using LLM to generate the Multi30K-VQA dataset. Our prompt underwent over five iterations to mitigate the hallucination issue which the question-answer pairs does not obey what we pre-defined. Despite these meticulous adjustments, 1,012 of the generated QA pairs fell short of our expectations. To address this, we employed hand-crafted rules to refine 605 of these pairs. For the remaining cases (407), our team conducted sentence-by-sentence annotations to ensure quality (three annotators). Consequently, our final Multi30k dataset comprises 29000 rigorously vetted QA pairs (the same size as the training set).\n\n2. **Marginal gains:** While BLEU is a widely-used metric for evaluating machine translation performance, it's worth noting that it is not the sole criterion, particularly for multimodal machine translation. Previous works such as Wu et al. (2021) and Li et al. (2022) have argued for the inclusion of visually grounded tests to provide a more comprehensive evaluation. In Table 4, our MMT-VQA method demonstrates its efficacy by outperforming the strong baseline of selective attention + MAE-base. Specifically, it achieves an average BLEU score increase of 0.5 points for En-De and 0.72 points for En-Fr. Further, on the Test2016-con dataset—specially designed to assess translation of sentences laden with ambiguous terms, MMT-VQA improves BLEU scores by 1.09 and 0.59 points for BLIP-base and MAE-base MMT models, respectively. Additional discussion will be included in the revised manuscript for a more nuanced understanding.\n\nWe believe that these elements, taken together, distinguish our work from incremental advancements in the field and provide meaningful contributions to the research community. Should you have any additional questions or require further clarification, we would be pleased to engage in further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposition of utilizing multi-task learning to strengthen the interaction capability between two modalities is logical.", "reasons_to_reject": "The paper offers limited novelty, and the observed performance enhancement is marginal.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study delves into the intricacies of multi-modal machine translation (MMT). The authors argue that current MMT models fall short in addressing cross-modal interactions. To rectify this, they introduce a multi-task learning methodology, aiming to heighten sensitivity to visual elements. The experimental results reveal that the proposed approach surpasses well-established benchmarks across several datasets.\n\nThe research presents two pivotal contributions:\n1. The introduction of the Multi30K-VQA dataset, which is helpful to the research community.\n2. Achievement of state-of-the-art (SOTA) performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CXhfPHnchm", "reviewer": "Reviewer_6B61", "summary": "", "questions": "* See weaknesses.\n* There are other ways to improve visual information utilization (e.g through regularizations), how are your results compare to these regularization based approaches?\n* The proposed method is simple yet effective, however, as the vision encoder gets stronger (from ViT to CLIP), the gain of using VQA as extra signals diminishes. It would be very interesting to know for what type of error additional VQA training helps the most.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your useful feedback! We hope the following response can address your concerns!\n\n**W1: As LLMs often don't follow instructions and hallucinate a lot based on recent studies, there aren't any comments on the quality of the generated Multi30k-VQA corpus.**\n\n- We concur with your observation that a rigorous evaluation of the generated Multi30K-VQA corpus is indispensable. Our overarching aim is to leverage probing signals to facilitate a more effective interaction between visual patterns and textual inputs within the MMT framework. Importantly, our experimental findings indicate performance gains across various test sets. These gains provide empirical evidence that MMT benefits from the enhanced interaction between the modalities, which serves as an indirect but persuasive indicator of the high quality of our Multi30K-VQA corpus. We believe that this substantiates the corpus's efficacy and intend to include these evaluation aspects in the revised version of the paper.\n- To mitigate hallucination issues, we implemented a rigorous QA pair generation process. We initially selected 500 random samples for which the LLM was tasked with generating QA pairs. These generated pairs underwent multiple rounds of review, as described in Table 3 of our manuscript. Based on these reviews, we iteratively refined the prompt and added additional constraints to guide the LLM's outputs. Our observations suggest that inadequate constraints can lead to the generation of unrelated questions. However, when constraints were properly calibrated, we noted a marked improvement in the quality of the generated QA pairs. Creating a high-quality QA dataset using LLM is far from straightforward. Our prompt underwent over five iterations to mitigate the aforementioned issues. Despite these meticulous adjustments, 1,012 of the generated QA pairs fell short of our expectations. To address this, we employed hand-crafted rules to refine 605 of these pairs. For the remaining cases (407), our team conducted sentence-by-sentence annotations to ensure quality. Consequently, our final Multi30k dataset comprises 29000 rigorously vetted QA pairs.\n\n**W2: The idea of using multi-task FT is less novel**\n\nThank you for your observations regarding the novelty of multi-task fine-tuning (FT). While we recognize that this approach is not new in and of itself, it serves as one of several means to effectively incorporate probing signals into the training phase. Alternative approaches like two-stage training could similarly be employed. The key contribution of our work is not necessarily the multi-task FT per se, but rather the successful incorporation of probing signals to enhance multi-modal training (MMT). This serves to validate our main thesis—that probing signals can effectively strengthen inter-moda interactions. We plan to expand upon this in future discussions, exploring various techniques to leverage these probing signals for more effective multi-modal interactions.\n\n**Q1: There are other ways to improve visual information utilization (e.g through regularizations), how are your results compare to these regularization based approaches?**\n\nThanks for your advice, we agree that more comparison with additional strong baselines are beneficial. So far, the selected baseline is quite strong which follows the previous tiny configuration setup raised in Wu et al., 2021. And they draw the conclusion that visual features may act as regularization when the capacity is large but the training data is hungry. We have already included their work in Table4. More comparisons with other studies for strengthening the visual information utilization will be added in our revised version.\n\n\n\n**Q2: The proposed method is simple yet effective, however, as the vision encoder gets stronger (from ViT to CLIP), the gain of using VQA as extra signals diminishes. It would be very interesting to know for what type of error additional VQA training helps the most.**\n\nThanks for pointing out this issue, we are conducting experiments on CLIP-large for further investigation. In this work, we only used the CLIP visual encoder rather, More discussion would be included in the next version for a clearer clarification. Here, we try to figure out what type of error additional VQA training helps the most. We conduct studies from three perspective:\n\n- **From the Perspective of Part-of-Speech (POS):** We analyzed translation quality using POS tagging. For each POS type in the hypothesis, we counted its occurrence in sentences and then segmented the test set accordingly. We compared BLEU scores for the MMT-VQA and SA methods across these subsets. To isolate the effect of specific POS types, we calculated the score differences and adjusted for word frequency. Our results highlight that adjectives (JJ) and plural nouns (NNS) most significantly improve translation quality when using the VQA method.\n\n- **From the Perspective of Long and Difficult Words:** We found that 3% of vocabulary consists of long and difficult words. After calculating that the 97th percentile of all word lengths is 9, we divided the test set based on whether sentences contained long and difficult words. We found that the MMT-VQA method improved the BLEU score by 0.98 points on the test subset with long and difficult words compared to the selective attention baseline method (Li et al., 2022 ). The improvement on the subset without long and difficult words was 0.67 points.\n\n- **From the Perspective of Long and Difficult Sentences:** We then completed the identification of long and difficult sentences. We first determined the number of clusters for sentence length categorization to be 2 based on the elbow method, dividing sentence lengths into two categories: long sentences and short sentences. We found that during the translation of long sentences, the BLEU score difference between the VQA and SA methods was 1.08, and for short sentences, it was 0.63—nearly a 70% improvement.\n\nWe trust that these results provide a clearer understanding of the landscape in question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea of using VQA to make MMT more grounded to the images is a nice contribution.\n* Method for question generation and Multi30k-VQA dataset could be useful for VQA or mVQA studies.\n", "reasons_to_reject": "* As LLMs often don't follow instructions and hallucinate a lot based on recent studies, there aren't any comments on the quality of the generated Multi30k-VQA corpus. \n* The idea of using multi-task FT is less novel.", "questions_to_authors": "* See weaknesses.\n* There are other ways to improve visual information utilization (e.g through regularizations), how are your results compare to these regularization based approaches?\n* The proposed method is simple yet effective, however, as the vision encoder gets stronger (from ViT to CLIP), the gain of using VQA as extra signals diminishes. It would be very interesting to know for what type of error additional VQA training helps the most.", "paper_topic_and_main_contributions": "This paper focuses on multimodal machine translation (MMT). Currently, the MMT system often underutilizes visual information. Hence, the paper proposes to generate parallel VQA pairs from the source text and proposes a multi-task learning framework to incorporate explicit signals from the generated VQA dataset into MMT. The paper shows the effectiveness of the proposed method on multiple datasets. The generated corpora, named Multi30k-VQA dataset, is an additional contribution to the paper. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper works on the MMT problems. It focuses on enhancing the multimodal interactions to achieve a better results. The proposed method is to connect MMT and VQA. Overall, the paper achieves both technical novelty (might be limited due to the reviewer's feedback) and empirical improvements.

A concern from the reviewing process is that the paper does not include the results from some 2022 papers. It's hard to fully stand with author's point that these results are just "missing references", since some of the paper achieves higher results. Higher previous results does not directly change the paper's evaluation, but it might change the claims made with this paper. 

It would be great if the author can have an update over the paper claims as well (e.g., the claim of "state-of-the-art" in abstract; I want to mention once more that not claiming SotA would not hurt the paper performance, but a claim with uncertainty could). Thus it can resolve confusions for potential readers.
Decision: Accept-Findings