Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion
Authors: Yexing Du
Youcheng Pan
Zekun Wang
Zheng Chu
Yichong Huang
Kaiyuan Liu
Bo Yang
Yang Xiang
Ming Liu
Bing Qin
Venue: ICLR 2026 Poster (2026)
Keywords: Speech
Multimodal Machine Translation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A Speech-guided framework that leverages synthetic speech and a self-evolution mechanism to improve translation

Abstract: Multimodal Large Language Models (MLLMs) have achieved notable success in enhancing translation performance by integrating multimodal information.
However, existing research primarily focuses on image-guided methods, whose applicability is constrained by the scarcity of multilingual image-text pairs.
The speech modality overcomes this limitation due to its natural alignment with text and the abundance of existing speech datasets, which enable scalable language coverage.
In this paper, we propose a Speech-guided Machine Translation (SMT) framework that integrates speech and text as fused inputs into an MLLM to improve translation quality.
To mitigate reliance on low-resource data, we introduce a Self-Evolution Mechanism.
The core components of this framework include a text-to-speech model, responsible for generating synthetic speech, and an MLLM capable of classifying synthetic speech samples and iteratively optimizing itself using positive samples.
Experimental results demonstrate that our framework surpasses all existing methods on the Multi30K multimodal machine translation benchmark, achieving new state-of-the-art results.
Furthermore, on general machine translation datasets, particularly the FLORES-200, it achieves average state-of-the-art performance in 108 translation directions. Ablation studies on CoVoST-2 confirms that differences between synthetic and authentic speech have negligible impact on translation quality. The code and models are released at https://github.com/yxduir/LLM-SRT.

Published as a conference paper at ICLR 2026
SCALABLE MULTILINGUAL MULTIMODAL MACHINE TRANSLATION WITH -PEECH-TEXT FUSION
12, Yexing Du Youcheng ? Zekun Wang', Zheng Chu', Yichong Huang'
Kaiyuan Liu Bo Yang", Yang Xiang"; Ming Liu Bing Qin
Harbin Institute of Technology
PPengcheng Laboratory
ABSTRACT
Multimodal Large Language Models (MLLMs) have achieved notable success enhancing translation performance by integrating multimodal information. How- ever, existing research primarily focuses on image-guided methods, whose ap- plicability is constrained by the scarcity of multilingual image-text pairs. The speech modality overcomes this limitation due to its natural alignment with text and the abundance of existing speech datasets, which enable scalable language coverage. this paper, we propose a Speech-guided Machine Translation (SMT) framework that integrates speech and text fused inputs into MLLM to improve translation quality. To mitigate reliance on low-resource data, we in- troduce Self-Evolution Mechanism. The core components of this framework include a text-to-speech model, responsible for generating synthetic speech, and MLLM capable of classifying synthetic speech samples and iteratively opti- mizing itself using positive samples. Experimental results demonstrate that our framework surpasses all existing methods on the Multi30K multimodal machine translation benchmark, achieving new state-of-the-an art results. Furthermore, on general machine translation datasets, particularly the FLORES- 200, it achieves average state-of-the-art performance in 108 translation directions. Ablation stud- ies on Co confirms that differences between synthetic and authentic speech have negligible impact translation quality. The code and models released at https: / / thub com/ com/yxduii r LLM- SRT.
INTRODUCTION
Multimodal Machine Translation (MMT) leverages complementary information from multiple modalities, such as images, to enhance machine translation (MT) quality. These modalities pro- vide supplementary contextual information for source texts, thereby mitigating ambiguities caused by polysemy or omissions (Shen et al., 2024).
Traditionally, image-based MMT models (Cheng et al., 2024) process image-text pairs to generate translations, leveraging visual context for semantic disambiguation. However, these models require an associated image for each input text, which limits their applicability. Recent image-free ap- proaches (Guo et al., 2023) have employed diffusion models (Rombach et al., 2022) to generate synthetic images to enhance translation. While these studies address the issue of image dependency, those methods still face two limitations: (1) Generalizability: While MMT models perform well on ambiguous datasets (Elliott et al., 2016), they struggle generalize to general translation datasets and even introduce noise in some scenarios (see Figure 1). (2) Multilinguality: Existing image MMT datasets (Guo al., 2022) support only few languages, with limited of languages coverage (see Table 1). Advances in diffusion Texe-to-Speech (TTS) models (Du et al., 2024) have achieved high-quality, zero-shot multilingual speech synthesis. This raises question: Can we leverage speech modalities to enhance translation quality?
Recent studies have revealed that, alongside lexical information, speech signals also convey prosodic cues, which offer valuable supplementary information (Chi et al., 2025). Inspired by fusion of text and prosody features, we propose the framework of Speech-guided Machine Translation (SMT),
"Corresponding Authors: {panych, xi angy) @pcc cn. iue1 hdt..du..cn

---

Published as a conference paper at ICLR 2026
Diffusion Mode
Image Dutzee
Language
Speech Dutasel
Language
WAAT Madel
Phing chre
LAPR TC-12 (Graabigger. 2000)
eug, den
MuSIC (D) Gangi al. 2019)
cog-re 15
MaRiBOK (Eilion
ong, den, fia
CaNaST-2
21-bong
MATT
MLT (1 ala Specis, 2018)
eng. des
FunoparkSTT (lamro-Stincher at 2030)
9++9
2019)
eng, spa
Fleuns (Connean al., 2022)
Amaiggaap (L) et al, 2021)
eng,
Granary (Kolaguni al., 20125
25 -eeg
Fit (Song
2021)
oug. cmn
CCFQA (Du al., 20125))
84+8
FMMT (7hu et al, 2023)
RlaesAmmend (Sinkar 2025)
enge4
1T5 Modes
+
MLLM
(3) SMT
poein parce
BLATID (Chen
OCRMTSOK M0 (Lam et al,, 2023)
Datnset (Ma et al. 2022)
2023)
eng. deu, cmn
eng. cmn
[Liang 2022)
eng.
BIG-C (Sikasote et al., 20123
cug. ben eng.
HaVQA (Pardd 2025)
M (Goo 2022)
Figure 1: Image-Guided vs. Speech-Guide Guided Machine Translation.
Table I: Dataset Statistics. For the languages supported by the im- age datasets, please refer to Table 7. Our MLLM supports 28 lan- guages, shown in Table 8.
which maps speech-text fusion inputs (speech, text} } to (translation) outputs. Specifically, our SMT framework integrates TTS model with an MLLM through a self-evolution mechanism (Tao et al., 2024) that leverages synthetic speech to enhance translation performance.
The framework consists of two core components: (1) MLLM Pre-training: We employ a multi- stage curriculum learning strategy with progressively complex objectives, beginning with speech recognition (ASR) for speech-text mapping, then speech-to-text translation (S2TT) for cross-lingual and cross-modality bridging, and culminating in SMT training for joint speech-text processing. (2) Self-Evolution Mechanism: This component synthesizes training data via the TTS model, where the MLLM classifies speech samples based on translation scores. The MLLM undergoes continuous training using positive samples, while translation performance metrics serve as evolution objectives, enabling continuous framework improvement through iterative refinement cycles.
The experimental results demonstrate that our framework achieves new state-of- state-of-the-ard (SOTA) re- sults the Multi30K benchmark (Elliott al., 2016), surpassing all existing MMT approaches. Our framework further achieves SOTA average machine translation (MT) performance across 108 lan- guages directions the FLORES-20 benchmark (Team et al., 2022), outperforming much larger language models. Ablation studies on the CoVoST-2 dataset (Wang et al., 2020) also reveal that the discrepancy between synthetic and authentic speech has negligible effect on translation perfor- mance. In summary, our key contributions are as follows:
We propose a novel speech-guided machine translation framework, which consists a TTS model and an MLLM. Our framework leverages prosodic cues speech enhance translation performance and supports 28 languages.
We propose self-evolution framework that autonomously generates training data for itera- tive self-enhancement. The framework employs continual training for the MLLM, utilizing synthetic data to improve the model's low-resource translation quality.
Our framework achieves state-of-the-art results on MMT and MT tasks across multiple benchmarks (Multi30K, FLORES- 200). Ablation studies on the CoVoST-2 benchmark show that the difference between authentic and synthetic speech has negligible impact on translation performance.
METHODOL.GGY
2.1
MODALITY-AGNOSTII HYPOTHESIS
Assumption 1.
Any auxiliary modality can enhance machine translation performance when:
The modality provides semantically relevant information the source text.
The modality representation can be aligned and jointly optimized with textual features in a shared latent space, given sufficient training data to learn discriminative embeddings.
2

---

Published as a conference paper at ICLR 2026
MULM pre-troin
I. Experience Acquisiton
II. Experience Refinement
Transkation
Score
Score
LLM
TTS Model
MILM
MLLM
Speech Adoptend
Encoderai
+ I|I
MULM
Ref Text
Text
Text
-Pol]| liji
Text
TTS
MT
SMIT
T
Self- Evolution
Evaluation
Model Updataing
Self-Evolution Objective: COMET Score
Translation
COMET Scone
LLM
#1
-ÍÍ]|| Authentic Speech
MLLM
Speech Adepter
-*d]|| Synthetic Speech
[t]lti Positive Speech
Spench Encoderth
MILLM
*||||||||| Negative Speech
M (1x
Text
Evaluction
Pos/Neg Sampling
Training
Figure 2: Overview Our SMT Framework. The proposed system architecture comprises two core components: (1) MILM pretraining and (2) Self-Evolution. This framework takes text input, synthesizes speech of the text via TTS model, and leverages the MLLM to process both text and speech features for higher-quality translation output. Self-evolution mechanism can autonomously generate training data to iteratively optimize the framework.
2.2 OVERALL DESIGN
Figure illustrates the SMT framework, comprising MILM and TTS model. The processing pipeline operates as follows: First, the system accepts textual input and synthesizes speech via the TTS model. Then, the MLLM processes both the text and synthetic speech to generate translations. The following subsections detail two key components: MLLM pretraining (Section 2.3) and self- evolution mechanism (Section 2.4).
2.3
MLLM PRE-TRAINING
The MLLM is built upon large language model (LLM) (Cui et al., 2025), adopts Whisper's coder (Radford et al., 2023) as the speech encoder, followed by a Q-Former (Li et al., 2023a) and MLP layer for speech adapter. We design three-stage training pipeline and perform instruction tuning, The sequential fine-tuning stages comprise: (1) automatic speech recognition, (2) speech- to-text translation, and (3) speech-guided machine translation.
ASR. The MILM learns speech-text alignment through ASR pre-training while keeping only the speech adapter trainable.
Modules
Param Stage Details
Speech Encoder
6655M
Whisper's 's encoder
Speech Adapter
80.55
All
Q-Former and MI.P
S2TT. Given speech input and instruc- tions, the MLLM simultaneously gener- ates transcriptions and translations.
LIM
~9.2
GemmuX2-28-998
LLM adapter
~..9
III
LoRA (r=16, alpha=32)
Total
~0B
SMT. The MLLM processes joint speech-text inputs to generate transla- tion outputs by leveraging complemen- tary multimodal information.
Table 2: MILM Pre-training. The blue color indicates the number of trainable parameters.
3

---

Published as a conference paper at ICLR 2026
2.4 SELF-EVOLUTION MECHANISM
Self-evolution mechanism allows models to autonomously learn through four phases: experience acquisition, experience refinement, updating, and evaluation. Our SMT framework is based on (1) MLLM, (2) TTS model, and (3) a S2TT dataset with authentic speech, text, and translation.
2.4.1 STAGE I: EXPERIENCE ACQUISITION
The purpose of this stage is to generate synthetic speech. During this stage, the prompt text and the predicted speech duration strictly aligned with authentic speech and text pairs.
TTS Inference. We employ a TTS model to synthesize speech signals from the text in the S2TT dataset. Given a reference text, the TTS model generates a new speech utterance while cloning randomly selected voice from the same dataset. This process ensures diverse set of synthetic speech data with varied prosody, which is crucial for our framework's 's training.
2.4.2 STAGE II: EXPERIENCE REFINEMENT
This stage implements quality-aware labeling strategy for speech samples. We find that not all speech is beneficial for translation, so we need to classify the samples. This process is achieved by comparing the scores of MT and SMT.
MT and SMT Inference. The MLLM operates in two distinct modes. In MT mode, the model processes textual inputs text to generate translations faans producing score Sj. SMT mode, the model accepts either authentic speech šzef synthetic speech 8gen paired with its corresponding text input to generate translations, producing score S2.
2.4.3 STAGE III: MODEL UPDATING
This stage is dedicated to optimizing the MILM by leveraging the synthetic data generated in the previous stage. The primary goal is to enhance the MILM'ss ability to effectively utilize prosodic cues from speech input for improved translation quality.
Positive/Negative Sampling. We first perform comparative analysis to categorize each synthe- sized speech-text pair into either a positive (Spos) or negative (Sheg) sample. Let S₁ be the translation quality score with text input only, and S2 be the score when the MILLM receives both text and speech input.
sample is categorized as positive sample (Spos) if the additional speech input improves trans- lation performance (S2 > Si). Conversely, a sample is labeled as negative sample (Sneg) if the speech input provides no benefit (S2, S S1). The scores are computed as:
S COMET MLLLM (frext, ttrans
S2 = COMET MLLM (Srof or 8gens lrext), Lerans
MILM Continuous Training. The MLLM is then continually fine-tuned using only the identified positive samples (spor). This targeted training strategy guides the model to prioritize and learn from the most beneficial speech-text interactions, thereby enhancing its ability to leverage prosody for superior translation performance.
2.4.4 STAGE IV: MODEL EVALUATION
In this final stage, we evaluate the framework's translation performance to determine whether to continue the self-evolution loop. We synthesize speech for the evaluation text using fixed refer- ence voice and measure the SMT framework's 's performance with the COMET score. This process iterates until the COMET score on the evaluation set converges and no longer shows significant improvement.
4

---

Published as a conference paper at ICLR 2026
3 EXPERIMENTS
3.1
DATASETS
We conduct comprchensive evaluations on several benchmarks. For multimodal machine translation, we use Multi30K (Elliott et al., 2016). For machine translation, we use FLORES-200 (Team et al., 2022) and WMT24++ (Deutsch et al., 2025). Additionally, we perform ablation studies on the CoVTST-2' dataset (Wang et al., 2020). Detailed information for datasets is provided in Table 10.
3.2 EXPERIMENT SETUP
Model Architecture Our MLLM consists of frozen speech encoder, specifically the encoder from Whisper-large-v3 (Radford al., 2023), and trainable adapter layer. This adapter comprises a Q-Former (Li et al., 2023b) and a multilayer perceptron (MLP). The LLM backbone is GemmaX2- 28-9B (Cui al., 2025). Following the configuration in (Yu al., 2024), Q-Former uses 80 queries, each with dimension of 768. The datasets used for MLLM training are detailed in Table For the TTS model, we adopt the CosyVoice2
Voice2 (Du al., 2024) model.
Training Details. Experiments are conducted four A100 GPUs (80GB). Following the exper- imental setup (Ma et al., 2026), we used the AdamW optimizer (Loshchilov, 2017) with peak leamning rate of x 10 The learning rate linearly warmed up over 1K steps and then linearly decayed for the remainder of the training. The models can be trained in under a week.
Evaluation Metries. For evaluation, we employ BLEU (Post, 2018), spBLEU (Team et al., 2022), and COMET (Rei et al., 2020). We compute spBLEU using the tokenizer "flores200". For fair comparison, our LLM inference uses vLLM (Kwon al., 2023), with all beam search settings and temperature uniformly set to I and 0, respectively.
3.3 COMPARING MODELS
MT Models. We evaluate the translation performance of four models: Deepseek- V3.1 API (Guo et al., 2025), Gemma3-27B-it (Team et al., 2025), Qwen3-Nent -eentn-eex--000 80B- 90B-A3B- -3B-tnrrttttt (Team, 2024), and NLLB-54B (Team et al., 2022).
MMT Models. We compare our framework against two categories of existing multimodal ma- chine translation models. We compare against four traditional MMT models that use text and authen- tic image: Soul-Mix (Cheng et al., 2024), RG-MMT-EDC (Tayir Li, 2024), WRA-guided (Zhao et al.,, 2022), and ConsQA-MMT (Gao et al., 2025b), Additionally, we compare against four image- free MMT models that rely on text and synthetic image: VALHALLA (Li et al., 2022), Bridge (Guo et al., 2023), DreamLLM (Dong et al., 2024), and IMAGE (Chen et al., 2024a).
3.4 OVERALL RESULTS
Our comprehensive experiments demonstrate the significant effectiveness of our proposed speech- guided machine translation approach. Our framework achieves new state-of-the-an results on the Multi3KK benchmark, surpassing traditional text-only and image-based MMT models. SMT-9B also consistently outperforms much larger text-only language models. Furthermore, our framework shows strong generalization, achieving state-of-the--e results in 108 translation directions on the FLORES-200 benchmark. Finally, ablation studies confirm that the performance difference between authentic and synthetic speech is negligible.

---

Published as a conference paper at ICLR 2026
Models
deu Tes22017
eng Dra
Tes22016
MSCOCO
Tes12016
Tesz2117
MSC000
Tea22016
Test2018
Models
Text
V3e (Guo al., 2025)
44.2/87.3
41.1/ 86.8
36.4/83.2
55.3 88.2
54.//777
53.5 85.8
37.9/ 90.2
35.9/ 89.7
Gemma3-27B-it (Team et al,, 2025)
43.7/87.1
40.3/ 86.3
36.1/ 7832
15./8779
54.//879
49.6
36.4/ 89.9
35.9 89.1
NLLB moe 54B (Team et al., 2022)
41,4/8622
39.7/85.8
34.7 /2.1
55.1
$4.8/87.7
53.3/85.3
35.7 / 88.9
35.8 88.3
QwenI-Neut-0B- (Team, 2025)
31.9/82.5
53.2/
51.9/87.6
50.1/85.1
29.2/
Models based Text Authentic Image
WRA-guided (Zhno et 2022)
39.3/
32.31
28.5/
61.8/
54.1
43.4/
RG-MMT-EDCT (Tayir et al., 2024)
42.0/
33.47
30.07
62.97
55.8
45.1
Soul-Mix (Cheng al., 2024)
442/
57.1/
36.51
BesSAA-MMI (Gao et al., 2025g)
44.2/
37.6/
34.3/
64.8/
58.31
48.5
34.31
30.3/
Models hased on Text & Synthetic Image
VALHALLA (Li al., 2022)
42.7/
35.11-
30.71
63.17
56.0/-
46.3/
Rridge (Gio al,, 2023)
42.5/
360/
32.01
56.2
35.2/
31.2/
D. (Dong al., 2024)
27.2/748
19.5/73.5
69.3
/6.9/111
34.7 80.6
36.6/79.2
IMAGE| | (Chen et al., 2025)
45,3/83.1
38.6/199
37.5/
67.5 88.3
61.5 96.6
19,3
Models based on Text Synthetic Speech
Baseline (Text only)
42.9/87.0
3888 86.4
34.3/8277
57.4/87.7
52.0/8799
86.6786.1
34.1 / 89.9
34.8/ 89.0
Baseline (Test only) only)
39.4
35.3/83.0
55.5 88.1
$4.07
$3.4/8599
37.2
SMT-9B
87.8
41.8 87.3
38.5 84.0
/9.0
62.1 89.6
55.3 86.7
414 90.8
39.9 89.8
Underlined denotes previous state-of-the-art models, while highlighted surpasses previous models.
Table 3: Translation Performance on Multi30K (BLEU / COMET) MMT Benchmark. The average character length of the input English text is 59.3. t indicates that the scores were directly cited from other research papers.
3.4.1 MAIN RESULTS FOR MULTIMODAL MACHIINE TRANSLATION
Comprehensive Performance Improvement from Speech-Text Fusion Input. Table 3 show- cases the remarkable performance of our SMT-9B model, which expertly integrates both synthetic speech and text inputs. The results clearly demonstrate substantial performance gain across all evaluated test sets. Specifically, for the eng->deu task, our model attains impressive BLEU scores of 47.0, 41.8, and 40.3 on the Test2016, Test2017, and MSCOCO datasets, respectively. Similarly, for the eng-frra task, it achieves high BLEU scores of 67.0, 62. and 55.3. These scores consis- tently and significantly outperform all text-only baselines. The clear advantage our approach holds provides compelling evidence that synthetic speech, as an auxiliary modality, can furnish crucial prosodic and contextual information that is not available in text alone, thereby effectively enhancing machine translation performance.
Competitive Advantage of Synthetic Speech in Multimodal Translation. The table clearly demonstrates the significant performance advantage of our proposed method, which leverages syn- thetic speech, over existing multimodal machine translation models that primarily rely on visual inputs. Our SMT-9B model establishes new benchmark by achieving state-of-the-ar average BLEU score of 52.0. This score not only surpasses the performance of all previous methods but does so by substantial margin, regardless of whether those models used authentic or synthetic im- ages. For direct comparison, our model outperforms the best rest-performing image-basee model by an impressive 2.1 points (which only achieved an average BLEU of 49.9). This result suggests that the speech modality is a rich and unique source of contextual information that is both distinct from and complementary to the visual modality.
Comparative Analysis with Large-Scale Language Models. Although not shown in the table, our SMT-9B model, despite having a parameter count that is only 1/67th of the DeepSeek-V3. V/-677 IB model, achieves superior translation performance. This result highlights the significant potential of multimodal lcarning: even a smaller model can achieve or surpass the performance of much larger text-only model by effectively leveraging cross information. This demonstrates that modality fusion can compensate for lack of scale, offering viable path for developing high-performance translation systems in resource-constrained environments.

---

Published as a conference paper at ICLR 2026
FLORES-200
WMT24+
Models
eng 27
Jpn 27
kor - 27
cma
eng 22
eng 22 (c2000)
Models based on Text
DeepSock-V3, V3.1 (Gioo al., 2025)
/9.3/88.9
26.1 85.7
27.7 / 85,9
27.5/ 86.2
/8.1/
31.8
Gemma3-222 3-2--ii (Team et al, 2025)
37.4788..
23.8/8110
25.07 81.2
21.5/81.5
34.3/8299
31.8 82.6
NLLB moe 54B (Team et al., 2022)
35.7 /6.3
21.8/811
23.6/ 83.7
22.8/ 82.1
25.4/76.9 76.9
77.7
Owend-Next-80B--1 (Team, 2025)
34.5/ 86.6
/3.9/83.8
23.9/839
24,3
30.5
29.6/ 81.6
Models bused on Text & Synthetic Speech
Baseline (Text oTey)
/ 88.3
26.6/85.4
27.4/85.6
27.5/85.7
33.9/82.7
32.1 /2.99
SMT-9R
40.4 / 89.5
27.3 / 86.9
28.3 18.1
28.3 / 1874
33.4/83.0
32.2 83.4
Underlined denotes previous state-of-thh-art models, while highlighted surpasses previons models.
Table 4: Translation Performance FLORES-200 and WMT24++ (spBLEU / COMET) MT Benchmarks. The average character length of the input English text is 130.4 for FLORES-200 and 191.3 for WMT24++. The notation < 200 indicates that the input English text length is within 200 characters. Detailed results are summarized in Tables 11, and 12 in the Appendix.
3.4.2 EXPERIMENTAL RESULTS FOR MACHINE TRANSLATION
Language Support. Our model exhibits strong language support, surpassing existing MMT mod- els. Specifically, Table 4 details results for 108 translation directions the FLORES-200 bench- mark, encompassing major source Lunguages-Englihh (eng), Japanese (jnn). Korean (kor), and Chi- nese (emn)- to 27 target languages. Furthermore, we evaluate on the WMT24++ benchmark for en +22 directions. The complete list of supported languages is provided in Table in the Appendix.
Scalable Multilingualism. The consistent performance gain underscores our method's advan- tages: scalability and multilingual capability. As shown in the Table our model not only performs exceptionally well on the eng-xx task, but also delivers impressive gains on jpn-xxx, kor -xx,+xx, and emn->xx directions. The average spBLEU scores for these language groups 27.3, 28.3, and 28.3 respectively, all of which are the highest in their respective categories.
SMT in Low-Scoring Directions. As shown in Figure 3, the SMT-9B model outperforms both the Baseline and DeepSeek models, particularly in low-resource translation directions like Khmer (khm), Lao (lao), and Burmese (mya), indicating its greater robustness in data-scarce language pairs. Beyond this, we note an underperforming high-resource language, Hindi (hin), whose translation metries are lower than many low-resource counterparts.
Translation Text Length. As shown in Table the WMT24++ dataset contains numerous tremely long texts, leading to noise (e.g., word omissions or duration exceeding 30s) in the syn- thesized speech. Although the model's performance on the overall dataset moderate, it exhibits good performance within the < 200 range. More importantly, the model's performance does not significantly degrade compared the baseline, even when receiving noisy speech input, which fully demonstrates the model's 's robustness.
COMET
79
Low
Medium
High
74
0
12
40
108
Direction
SMT-98
Baseline
DeepSeek-V3.1
Figure 3: COMET Results by Resource Level, Categorized as Low, Medium, and High. Our model shows improvement in translation scores, particularly for low-scoring translation directions.
7

---

Published as a conference paper at ICLR 2026
Input
eng
spBLEU COMET
Text
AS
SS
ara
deu
ind
jpn
tur
Avg.
37.7 / 86.3
45.2 88.0
32.1 86.9
47.9/911
31.5 90.7
36.7/ 88.8
38.5/
88.7
/
32.6 / 82.2
36.6/ 82.2
27.9/826 82.6
36.8 85.99
26.9/86.5
29.3/83.6
31.7 83.8
34.1/83.5
34.0/84.0
/
36.9/87.4
27.1/8744
30.3 85.0
3 85.4
1
40.1 / 86.8
46.5 88.33
/7.4
48.4/916
33.6/908
37.9 89.1
40.0 89.0
40.1 86.88
46.5 / 88.2
37.6
/8.6
33.5
$7.8/89.1
40.0 89.0
Table Ablation Study on the CoVoST-2 Benchmark. A comparison configurations with differ- ent modality inputs. (AS denotes authentic speech; SS denotes synthetic speech)
Models
eng
spBLEU COMET
jpn
cmm
tha
khm
lao
mya
Avg.
Bascline
33.3 /91.3 35.2 /92.7
41.6 89.2
42.5 / 88.7
24.1 / 84,2
31.5/ 84.7
20.1 / 88.1
32.2 / 87.7
SMT-9B
42.6/91.
44.1/90.3
25.6 83.6
34.2 86.3
24.3 / 88.5
34.3 / 88.8
w/o SE
34.8
42.3 / 89.3
/
23.0/81.7
31.7/84.5
23.4/ 86.8
33.0/ 87.3
Table 6: Ablation Study on Self-Evolution (SE) Mechanism on the FLORES-200 benchmark.
3.4.3 ABLATION STUDY
Authentic Speech Synthetic Speech. As shown in Table 5, experimental results reveal that the difference between authentic and synthetic speech has minimal impact on multimodal machine translation performance. Surprisingly, synthetic speech achieves better S2TT performance, likely due to the absence of background noise. Experimental results demonstrate strong semantic consis- tency between authentic and synthetic speech.
The Impact of the Self-Evolution Mechanism. As shown in Table 6, we found that after MLLM pre-training, model's performance on high-resource languages improved. However, due to the imbalance of multilingual data, the performance on low-resource languages like Khmer (khm), Lao (lao), and Burmese (mya) actually decreased on the COMET metric. Therefore, we introduced the self-evolution mechanism to enhance the model's ''' performance these low-resource directions.
Self-Evolution Rounds on Low- Low-Resource Languages. Figure 4 shows the improvements from self-evolution for low-resource languages, with round achieving best average gains of +1.9, +2.0, and +1.7 COMET on khm, lao, and mya, respectively. We observe that the first round yields the most significant improvement, later rounds give fewer benefits. The average improvement peaks round and then remains stable.
Human Evaluation for MT and SMT. Manual review evaluation samples revealed that the performance gain from adding the speech modality is likely due to a reduction in under-translation, which decreased from 5.2% to 3.5%, as shown in Figure 5. The introduction of the speech modality provides prosodic cues as additional signals that effectively help correct the attention weighting, thereby mitigating this problem.
sPBLEU
COMET
35
90
86
27
82
23
80
2
Round
my
lo
km
lo
km
Figure 4: Self-Evolution Rounds of spBLEU COMET (eng->x0 on FLORES -20 benchmark.
8

---

Published as a conference paper at ICLR 2026
Care
Tranelation Breen Faglish ts Chinees, Japanesn, and Spusich
Iopus
buy atnuet anything sestring.
drmand Trem
1011 AH. RAABE 1PP 32
P PI
Buacann
SMT--6
Ieper
The potient lad been Nightka wiee siree caes tfe thik vina hase secured. COBBE. 5.09
Ground Trus
Boodlan
R11, 2.69 www.o..
5M7--6
18 8 2 127377775 B8 Toriett P-C TNd 9 ( b Chrg.
Warke's MAAK eliee daii sugetkee appeoval for any decaiass they mall, and expected to obes thatr superian intructers walkor quesian.
Groand Tres
Con Droca modia los trrt debes castar con apriiaction do tea rppertats para sorr de decacnesy apeer obetierss intrucdioo de ww..crteemmmntt.
Boodino
Lea simmeppoonn dabin Jtrnner sprebccdie do ras raparoees para nalguter dedidt gue toevny of aeprra que cbedersn nerrrccees de experiom cannterartas
SMT-36
Con Encea cocii, los trabggiorm deber oieese la agrebadko de so aper.ore pane coalgaer decisón que sower eggen qae do In intrucddam de no agenters sin cuestoonaia
Figure 5: Case Study for text words with speech. The SMT model, which receives this speech-text fusion input, is prevented from ignoring the input text, thereby mitigating omission errors.
Having undergone speech pre-training, MLLMs align
RELATED WORK
Multimodal Machine Translation. MMT research has primarily followed two distinct paths: image-based and image-free approaches. image-based methods, exemplified by foundational work on the Multi30K dataset (Elliott et al., 2016), utilize paired visual and textual data to improve trans- lation quality. In contrast, image-free approaches emerged to tackle the challenges of data scarcity. These methods employ various techniques, such as target-end retrieval (Hitschler et al., 2016), multi- task learning (Elliott & Kádár, 2017), and even visual generation using advanced models like GANs and diffusion models (Rombach et al., 2022), to generate or retrieve supplementary information without relying on a pre-existing image dataset.
Multimodal Large Language Model. MLLMS (Chen al., 2024b; Du et al., 2025c;b) typically feature three core components: an LLM backbone, modality encoder, and a modality adapter. Our framework specifically leverages this architecture to handle both speech and text. The speech encoder, inspired by models like Whisper (Radford et al., 2023), is responsible for extracting rich speech features from the audio input. Following this, the speech adapter (Li et al., 2023b) projects these features into the same hidden dimension as the LLM, enabling seamless integration. The processed speech features are then concatenated with the original text embeddings. This unified representation is fed into the LLM backbone, which processes both modalities jointly to generate the final translated text.
Self-Evolution. The concept of self-evolution (Liu et al., 2021) empowers models to au- tonomously acquire, refine, and learn from self-generated experiences. As outlined in recent sur- veys (Tao et al., 2024), this process typically involves a four-phase iterative cycle: (1) experience acquisition, (2) experience refinement, (3) updating, and (4) evaluation. Each iteration is designed to achieve a specific evolutionary objective. In our implementation, the process begins with the expe- rience acquisition phase, where generate synthetic speech data. This is followed by refinement phase that involves the annotation of positive and negative samples. This newly labeled data is then used update the model, which is subsequently evaluated for machine translation performance.
5 CONCLUSION
In this paper, we present the Speech-guide Machine Translation (SMT) framework, novel ap- proach that overcomes the limitations of traditional image-based multimodal translation. Our frame- work integrates a TTS model with MLLM, leveraging speech as complementary modality to text. A key feature the Self-Evolution Mechanism, which autonomously generates and re- fines training data. This significantly reduces the need for human-annotated data in low-resource languages, making the system more scalable and practical. Our experiments show that SMT-9B achieves SOTA performance benchmarks such Multi30K and FLORES--00..

---

Published as a conference paper at ICLR 2026
6 LIMITATION
Unlike image-based methods, our speech-guided machine translation approach can cover a broader range of languages. However, we are still limited by the languages supported by the TTS models, as we need to synthesize speech from text. Although recent advancements in TTS technology have enabled the synthesis of dozens of languages, open-source TTS models still have limited language coverage.
7 THE USE OF LARGE LANGUAGE MODELS
In this paper, LLMs are not used for ideation but are utilized for checking grammatical rules.
REPRODUCIBILITY STATEMENT
All models and datasets tested this research are open-source.
ACKNOWLEDGEMENTS
The research in this article is supported by the National Science and Technology Major Program (Grant No. 20242D0INL00101, the National Science Foundation of China (U22B2059, 62276083, 62506182), National Key Research and Development Program of China (2055FE02000000, the Key Research and Development Program of Heilongjiagg Province (2022XX1A22) and the 5G Application Innovation Joint Research Institute's Project (A003), and the Major Key Project of PCL.
REFERENCES
Andong Chen, Yuchen Song, Kehai Chen, Muyun Yang. Tiejun Zhao, and Min Zhang. Make imag- ination clearer! stable sesttusiioonbbssee visual imagination for multimodal machine translation. arXiv preprint arXiv:2412.12627, 2024a.
Andong Chen, Yuchen Song, Kehai Chen, Xuefeng Bai, Muyun Yang. Liqiang Nie, Jie Liu, Tiejun Zhao, and Min Zhang. Make imagination clearer! stable diffusion- based visual imagination multimodal machine translation. Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 26567-26583, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 1779-8-91176-22 2025 a025.acl-long. 1289. URL ptattts://aclaattten www.:ogy.oor//..... 2025 acl acl- low...n.....9..
doi: 10.18653/v1/
Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, et al. Slam-ommn conro--cnntrollabee voice interaction system with single-stage training. arXiv preprint arXiv:2412. 2024b.
Zhuo Chen, Fei Yin, Qing Yang, and Cheng-Lin Liu. Cross-lingual text image recognition via multi-hierarchy cross-modal mimic. IEEE Trans. Multim,, 25:4830 2023. 2
Xuxin Cheng, Ziyu Yao, Yifei Xin, Hao An, Hongxiang Li, Yaowei Li, and Yucxian Zou. Soul- mix: Enhancing multimodal machine translation with manifold mixup. In Lun-We Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 11283-11111 1124,, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: URL hwtps:ttp//.mmm wwwaacannnooooyoo.. www.loo..orgg//2/0 2024 acl long. 608. 5,
Jie Chi, Maureen de Seyssel, and Natalie Schluter. The role of prosody in spoken question answer- ing. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 8468-8479 2025.
10

---

Published as a conference paper at ICLR 2026
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal repre- sentations of speech. arXiv preprint arXiv:2205. 12446, 2022. URL https: pttlpp://ariiv.. /ttp//aaxvvvv////////..a org/
abs/2205. 12446.
Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, and Bin Wang. Multilingual machine trans- lation with open large language models at practical scale: An empirical study, arXiv preprint arXiv:2502.02881 2025. 3.5
Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rehecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, et al. Wm124++ Expanding the language coverage of wmt24 to 55 languages dialects. arXiv preprint arXiv:2502. 12404, 2025.
Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. Must- C: a multilingual speech translation corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2012-2017. Association for Computational Linguistics, 2019. 2
Rumpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- net/ forum? 1 KGVd9 5,
Review.net, 2024. URL
Yexing Du, Kaiyuan Liu, Youcheng Pan, Zheng Chu, Bo Yang, Xiaocheng Feng, Yang Xiang, and Ming Liu. Ccfqa: A benchmark for cross-lingual and cross-modal speech and text factuality evaluation. arXiv preprint arXiv:2508.0722 2025a.
Yexing Du Kaiyuan Liu, Youcheng Pan, Bo Yang, Keqi Deng, Xie Chen, Yang Xiang, Ming Liu, Bin Qin, and Wei Wang. Mcat: Scaling many-to-many speech-to-text translation with milms to 70 languages, 2025b. URL https: /awxivivvvvv///////// org/ hasgg//bb///// /25//25122 2512. 51512.
Yexing Du Youcheng Pan, Ziyang Ma, Bo Yang. Yifan Yang. Keqi Deng, Xic Chen, Yang Xiang, Ming Liu, and Bing Qin. Making LLMs better many-to-many speech-to-text translators with curriculum learning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12466-112220000 12478, Vienna, Austria, July 2025c. Association for Computational Linguistics. ISBN 979-8-89176-25144 doi: 10. 10.18853/v1l /0253/1111225.clloooog URL https::///// /tttpp::/aclaaee http///aaaaaatthhoeeeeeee org/ 2025
acl long. 610/.9
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Sigi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based supervised semantic tokens. arXiv preprint arXiv:2407.0544 2024. 1.5
Desmond Elliott and Ákos Kádár. Imagination improves multimodal translation. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 December 1, 2017 - Volume 1: Long Papers, pp. 130-141. Asian Federation of Natural Language Processing, 2017. URL https: / //cc//lll/////// ho ogy. wtgg/1111100 4/.
Desmond Elliott, Stella Frank, Khalil Sima" an, and Lucia Specia. Multi30k: Multilingual english- german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, hosted by the 54th Annual Meeting of the Association for Computational Linguistics, VL ACL 2016, August 12, Berlin, Germany. The Association for Computer Linguistics, 2016. doi: 10.18653/v1/ w16-3210. URL https::/dool org/10/g0000 1851186633111 w/ 6 3210. 2,
Yue Gao, Jing Zhao, Shiliang Sun, Xiaosong Qiao, Tengfei Song, and Hao Yang. Multimodal machine translation with text-image in-depth questioning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Findings of the Association for Computational Linguistics: ACL 2025, pp. 9287-9287, Vienna, Austria. July 2025a. Association for Computational Linguistics. ISBN 979-8-891 76-91661666555 doi: 10. 10.18653/v 12202////222..///2 ..w.02....inng--a.... acl. http//claanntoooooo 483. URL hwwpttttt///tc //acle/acaaee antho org 2225 findings- 483/.0

---

Published as a conference paper at ICLR 2026
Yue Gao, Jing Zhao, Shiliang Sun, Xiaosong Qiao, Tengfei Song, and Hao Yang. Multimodal machine translation with text-image in-depth questioning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), 2025b. 5
Spandana Gella, Desmond Elliott, and Frank Keller. Cross-lingual visual verb sense disambiguation.
In Jill Burstein, Christy Doran, and Thamar Solorio (. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume (Long and Association for Computational Linguisties, 2019. doi: 10.18653/
Short Papers), pp.
v1/n19- 1200.
URL https: /oi. org/10 . 18611866333 h0///3///v//// 1200.
M. Grubinger. The iapr benchmark: A new evaluation resource for visual information systems. Language Resources and Evaluation, Jan 2006.
Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang. Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning Ilms through reinforce- ment learning. Nature, 645(808 (1:388))):3363333 2025. 5, 6.7
Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, Dongdong Zhang, and Zheng Cui. LVP-M3: language-aware visual prompt for multilingual multimodal machine trans- lation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2862-2872. Association for Computational org/2022 emn p-main. 184. 1,2
Linguistics, 2022. URL https::tp//.cc
Wenyu Guo, Qingkai Fang, Dong Yu, and Yang Feng. Bridging the gap between synthetic and authentic images for multimodal machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2863-2874, 2023. 1,5,
Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. Multimodal pivots for image caption translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties, ACL 2016, August 7-12, 2016. Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/116- 1227. URL https: 1227.
//doi.org//1/
Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerda. Javier Jorge, Nahuel Roselló, Adria Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8229 8233. IEEE, 2020.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282. Association for Computa- tional Linguistics, 2020. 16
Nithin Rao Koluguri, Monica Sekoyan, George Zelenfroynd, Sasha Meister, Shuoyang Ding, Sofia Kostandian, He Huang, Nikolay Karpov, Jagadeesh Balam, Vitaly Lavrukhin, et al. Gra- nary: Speech recognition and translation dataset in 25 european languages. arXiv preprint arXiv:2505. 13404, 2025. 2
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 5
Chiraag Lala and Lucia Specia. Multimodal lexical translation. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Köiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.). Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA), 2018 URL http:// wwwww//www/.///// erec- conf. org/ proceedi ngs rec201 8/summa r es/ 62.9 html. 2
12

---

Published as a conference paper at ICLR 2026
Zhibin Lan, Jiawei Yu, Xiang Li, Wen Zhang, Jian Luan, Bin Wang. Degen Huang, and Jin- song Su. Exploring better text image translation with multimodal codebook. In Anna Rogers, Jordan L. Boyd-Gred Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational I Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 3479 3491. Association for Computational Linguisties, 2023. doi: 10. 2023. acl loogoooggo..o 192.
A0T.///2222.22LOOOCN 192. URL /ttp::://do...o org/10. /tttppss//ooo.coor/0001815111/1
Jiaoda Li, Duygu Ataman, and Rico Sennrich. Vision matters when it should: Sanity checking multimodal machine translation models. In 2021 Conference Empirical Methods in Natural Language Processing, pp. 8556 8562. Association for Computational Linguistics, 2021. 2
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a. 3
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. Andreas Krause, Emma Brunskill. Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023b. URL https: http:s///oooceeeeeeeeeee
mlr www.prressvvvee. 1i23q.12... html. 5,
Yi Li, Rameswar Panda, Yoon Kim, C Chen, Rogerio Feris, David Cox, and Nuno Vasconcelos. Valhalla: Visual hallucination for machine translation. in 2022 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5206-5216, 2022. 5,
Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. MSCTD: A multimodal sentiment chat translation dataset. In Smaranda Muresan, Preslav Nakov, and Aline Villavi- cencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 2601- 2613. Association for Computational Linguistics, 2022. doi: 10. 10.18655/1122222.acl URL hnttps://dee http//dii.or11..1.1.....
186.
http:///2222..1111mmeeaa wawaaccolo....e011e 186. 2
Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang. Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IEEE transactions on neural networks and learning systems, 34(2):550-570, 2021.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017. 5
Cong Ma, Yaping Zhang, Mei Tu, Xu Han, Linghui Wu, Yang Zhao, and Yu Zhou. Improving end-to-end text image translation from the auxiliary text translation task. In 26th International Conference on Pattern Recognition, ICPR 2022, Montreal, QC, Canada, August 21-25, 2022, pp. 1666416700 IEEE, 2022. doi: 10.1109/C636666 2.1....22..999665.. URL https:: //d.. org/ 10. 1009/ ICPRS 6361 2022. 0...222.9955555 9956695. 2
Ziyang Ma, Guanrou Yang, Wenxi Chen, Zhifu Gao, Yexing Du, Xiquan Li, Zhisheng Zheng, Haina Zhu, Jianheng Zhuo, Zheshu Song, et al. Slam-llm: A modular, open-source multimodal large language model framework and best practice for speech, language, audio and music processing. IEEE Journal of Selected Topics in Signal Processing, 2026.
Shantipriya Parida, Idris Abdulmumin, Shamsuddeen Hassan Muhammad, Aneesh Bose, Guneet Singh Kohli, Ibrahim Said Ahmad, Ketan Kotwal, Sayan Deb Sarkar, Ondrej Bo- jar, and Habeebah A. Kakudi. Havqa: A dataset for visual question answering and mul- timodal research in hausa language. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 10162-10188, Association for Computational Linguistics, 2023. doi: 10. 100.18653//11 1 ACL.646. URL https:/ //oi. org/ 10. wtt../............... 18653/v1/ 2023. fi findings-acll 646. 2
13

---

Published as a conference paper at ICLR 2026
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https: wwwwwawlllan wwwp:/...gggagootooooooooo W1 - 9.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International Conference on Machine Learning, pp. 28492-28518. PMLR, 2023. 5,5,9
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, 2020. 5
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022. 1,9
Ashwin Sankar, Sparsh Jain, Nikhil Narasimhan, Devilal Choudhary, Dhairya Suman, Mohammed Safi Ur Rahman Khan, Anoop Kunchukuttan, Mitesh Khapra, and Raj Dabre. Towards build- ing large scale datasets and statc-of-the-arr automatic speech translation systems for 14 Indian languages. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11944533996,, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 979-8-89176-2514 doi: 10. 112/8663/112225..oo acl- long. 1582. URL https::/aclanthense
aon.oggg.... 1582/. 2
Huangjun Shen, Liangying Shao, Wenbo Li, Zhibin Lan, Zhanyu Liu, and Jinsong Su. A survey on multi-modal machine translation: Tasks, methods and challenges. arXiv preprint arXiv:2405.12669 2024.
Claytone Sikasote, Eunice Mukonde, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. BIG- C: multimodal multi-purpose dataset for bemba. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 2062-2078. Association Computational Linguistics, 2023. doi: 10.18653/V1/2033. ACL-LONG.11S URL https: / http:///oo.ooooooo/oo/oo///oooot 1861386633311 /tt.//1122222.2 acl- 115.
Yuqing Song, Shizhe Chen, Qin Jin, Wei Luo, Jun Xie, and Fei Huang. Product-oriented machine translation with cross-modal cross-lingual pre- traininnnn.. In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo César, Florian Metze, and Balakrishnan Prabhakaran (eds.), MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 - URL https:: org 1145/
24, 2021, pp. 2843-
2852. ACM, 2021. doi: 10. 10.1145/347408553455333.
3474085. 3475303. 2
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou A survey on self-evolution of large language models. arXiv preprint arXiv:2404.1433 2024. 2,
Turghun Tayir and Lin Li. Unsupervised multimodal machine translation for low-resource distant language pairs. ACM Trans. Asian Low Resour. Lang. Inf. Process., 23(4):55, 2024. doi: 10. 1145/3652161. URL https: /ww.///////////////// wow./rg//... /1114//52/6665555
Turghun Tayir, Lin Li, Bei Li, Jianquan Liu, and Kong Aik Lee. decoder calibration for multimodal machine translation. IEEE Transactions on Artificial Intelligence, 588)3396533973 2024. 6
Gemma Team, Aishwarya Kamath, Johan Ferret. Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.1978 2025. 5, 6,
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler
14

---

Published as a conference paper at ICLR 2026
Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shan- non Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang- No language left behind: Scaling human-centered machine translation. 2022. 2, 5, 6,
Qwen Team. Qwen3: Think deeper, act faster, 2024. URL attps://qwents github..hh.......... io/ wh/bloog/wwwn//
Qwen Team. Qwen3-next: More power and less cost, 2025. URL https::/gwen.alt aii/ ott.:/a3311110.eeeeeaaa.61611111 latest Latest--ddvancementts-Lt... 6,
Changhan Wang. Anne Wu, and Juan Pino. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310, 2020. 2,
Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for asr. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp-
12637-12641. IEEE, 2024. 5
Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara, and Chenhui Chu. Word-region alignment- guided multimodal neural machine translation. IEEE ACM Trans. Audio Speech Lang. Process., 30:244-259, 2022. doi: 10. 1A1100//SSLLPP22211138811 URL https: http:s::/doi..org//// 1109/TASLP
6,6
Yaoming Zhu, Zewei Sun, Shanbo Cheng, Luyang Huang, Liwei Wu, and Mingxuan Wang. Beyond triplet: Leveraging the most data for multimodal machine translation. In Anna Rogers, Jor- dan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 14, 2023, pp. 2679-2697 Association for Com- putational Linguistics, 2023. doi: 10. 10..665////2023FINNNNGGS- //doi.org/10. 18653
URL https:
findings-ndiine 11e....68.. 168.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
ISO-3
Language
Script
Family
Subgrouping
Resource
ben
Bemba
Latin
Atlantie-Congo
Benue-Congo
Low
ces
Czech
Latin
Ando-European
Balto-Slavic
High
cmn
Chinese
Han
Sino-Tibetan
Sinitic
High
deu
German
Latin
audo-Euroeean
Germanic
High
eng
English
Latin
Indo-Europent
Germanic
High
fra French
Latin
Indo-Furoo European
Italic
High
hau Hausa
Latin
Afro-Asioe Asiatic
Chadic
Low
hin Hindi
Devanagari
Indo- mndo-European
Indo-Aryan
High
lav
Latvian Latin
Indo-European
Balto-Slavic
High
spa Spanish
Latin
Indo- Europennn
Italic
High
tur Turkish
Latin
Turkic
Common Turkic
High
Table 7: 11 Languages Supported by Image-Guided MMT datasets. The resource of each language determined according to the taxonomy classes by (Joshi et al., 2020).
ISO-3 ara
Language
Script
Family
Subgrouping
Resource
Arabic Bengali
Arabic
Afro-Asiatic
Semitic
High
ben
Bengali
Indo-Eu ando-European
Indo-Aryan
Med
Czech
Latin
Indo-Furopean
Balto-Slavic
High
cmn
Chinese
Sino-Tibetan
Sinitic
High
deu
German
Latin
Indo-Furopean
Germanic
High
eng
English
Latin
Germanic
High
fas
Persian
Arabic
Mudo-European
Iranian
Hlgg
French
European
Italic
High
heb
Hebrew
Hebrew
Afro-Asiati
Semitic
Med
hin
Hindi
Devanagari
ando-Buropean
Indo-Aryan
High
ind
Indonesian Latin
Austronesian
Polyy--Polnnnsian
Med
ita
Italian
Latin
Indo-
Muro-European
Italic
High
jpn
Japanese
Japanese
Japonic
Japanesic
High
khm Khmer
Khmer
Austroasiatic
Khmeric
Low
kor
Korean
Hangul
Koreanic
Korean
High
lao
Lao
Lao
Fai-Kadai
Kam-Tai
Low
msa
Malay
Latin
Austronesian
Malayo-Poly Palayo--oo
Med
mya
Burmese
Myanmar
Aino-Tbbetan
Burmo- Miano-Qiiiicc
Low
nld
Dutch
Latin
Indo-European
Germanic
High
pol
Polish
Latin
Indo-Europer
Balto-Slavic
High
por
Portuguese
Latin
Indo-European
Italic
High
rus
Russian
Cyrillic
nuro-en
Balto-Slavic
High
spa
Spanish
Latin
mudo-European
Italic
High
igl
Tagalog
Latin
Austronesian
Med
tha Thai Thai
Tai Kadai
Kam-Tai Tai
Med
Turkish Latin
Turkic
Common Turkic
High
urd Urdu
Arabic
Indo-Europe aad--European
Indo-Aryan
Med
vie
Vietnamese
Latin
Austroasiatic
Vietic
High
Table 8: 28 Languages Supported by Our Model. The resource of each language determined according to the taxonomy classes by (Joshi et al., 2020).
16

---

Published as a conference paper at ICLR 2026
Model
Task
Description
Dataset
Split
Data Size
Metric
ASR
Automatic Speech Recognition
FLEURS!
train
160h
WER
Common Voice
3300h
MLLM
Speech Guided
FLEURS!
160h
pPLEU COMET
SMT
Multimodal Machine Translation
Mulii0K
train
m.40h
BLEU COMET
Table Summary of Training Datasets for SMT Models. Data size refers to the actual amount used for training, as we removed some overly long samples. i indicates that we performed data cleaning on the dataset. Since there an overlap between FLEURS and FLORES datasets, we removed the overlapping portions from the FLEURS training set.
Task
Description
Dataset
Split
Metric
FLORES-200
devtest
MT
Machine Translation
spBLEU/ COMET
WMT24++
test
MMT
Multimodal Machine Translation
Multi30K
test
BLEU / COMET 1
S2TT
Speech-to- Translation
CoVoST-2
test
spBLEU/ COMET 1
Table 10: Summary of Evaluation Benchmarks.
Direction
DeepSeek -V3.1
Gemmad -27B
NLLH-moe -54R
Qwent-Next -80R-A1R
Biseline
SMT
-9B
20.0 78.5
20.0/ 78..
18.5/74.6
19.4/776
19.4/77.3
19.1/
bem
25. 93.3
25. 82.7
23,5/79.7
16.5/785
28.8/82.1
23.6/ 80.7
eng
eng cmn
34.3 784,9
36.17
18.0169
37.4 81.9
36./83388
36.5 /5.3
dea
37.9 $7.9/92.6
37.9 $1.9
28.5/76.3
36.2 81.7
37.7 97.7/822.5
37.3 82.3
29.9/83.
32.9
25.8/78.0
23.7
32.1/83.1
31.9 83.8
eng heb
ong
57.47821
36.6 82.5
33.9/19.4
26.8/ 76.3
18.8/83.5
38.31 84.5
hin 19.6
19.5 73.4
16.0165.
12.6 68.5
19.3
19.6 70.2
38.8
37.6.
50.6
36.6/86.0
/
37.2/ 86.0
ang
ong jn
187.6
24.0
11.7 79.1
22.6 86.9
22.5
22.5
kor
27.1 /37.3
26.91 86.4
20.7 181.9
23.9 86.1
26.07 85.6
25.0 85.4
194.4
29.3/83.7
177.8
ang por
ong
40.7 /85.4
40,07 82.9
17.2
38.6 82.7
39.5 183.0
39:31 K5.4
29.6/983.4
31.4/ 82.7
23.2 176.6
23.8 $1.9
29.2/81.9
29,9 K3.5
48.1
83.6
36.0/77.7
46.2
18.7
83.8
ang
29.6
ong U2
36.0/85.5
36.6 84.3
79.0
30.6 83.0
36.6 14..5
36.3/ 84.2
und
50.5 790.8
30.3/79.0
29.0/73.7
23.5/ 75.8
33.3 79.8
32.4 B0L5
36.6/84.8
26.5777.7
35.9/855
833.7
Avg.
34.1 183.6
34.3/82.9
25.4/76.9
30.5/815
33.9/82.7
33.4/83.0
Table 11: spBLEU / COMET Scores on the WMT24++ Benchmark.
17

---

Published as a conference paper at ICLR 2026
Direction
DeepSeok -V3.1
Gemmañ -27B
NLLB-aa -44B
Qwen' Next
SSB A3B
Bascline
SMTT
-96
41.6/ 59,
41.7 87.8
41.87 96.8
38.3 87
42.9 87
42.6 89.5
eng
hen
35.61
83.8
3001
66.6
34.51 S6.4
28.77 85.6
M6.6
34.31 7866
6ng
ces
192.9
35.7
10.3
99.
13.4
89
426/91.2
eng
deu
48.5
59.0
46.91
45. 87
46.1 88
188.5
478 80.3
mg
58.
8
38
90.3
eng
47.6
454/89
45.01 SS
33 1814
46. 46.3/50
468 91
eng Jin
37.
80
5681817
38./997 90
119/76
413/81
4107 81 R
mg
1nd
02.6
48.1
4877
88.00
526/
911
15.01 92
526/918
18. 83
290/911
333/91
353 92
eng
khm
23. 81.7
17 1771813
22.01 79
150/ 76
184
83
mg
90.8
26 00.0
1901
45. 90
S7 86
46.4
39 88
90.9
474 91
ens IIVA
39.3
85
16.1
14.1
21 88.5
eng
9006
39.9
19
$5.3
$1./ 58.3
90.1
512 5000
911
eng RUS
90.9
1
M1
eng
86.
389785.9
82.5
362/513
10088
tha
11.5 85.8
428189.
405 88.8
M1 90.3
eng II
42.
402/50.5
89.3
413 90.5
eng
90.4
43.6 50.0
428 89.6
46.71 1900
46 91.7
Jp0 Jra
35. 6.778 7
19
2091 794
197 1804
24 1
0 84.3
J00 CES
26.7 4904
25
2021 96
251
1898
21 96
J00 cmn
279188
J0 OKU
32.7 88
334/885
128188
S7
22.7 85
1531676
185/ 709
241819
J0 rr
0/ 96.0
208/800
1231834
J hn
34.2
7488
310/714
1981731
26
484188
2531670
27 888
304130
m T1a
26 269187
27
26
79.9
kor
25.6 58.7
25.5 88
19% S48
216 88.5
235 89.6
36 Ja0
30.4
200 79.6
21.1 20
1207 70.6
84.2
16 mEA
36.0
27.8
78.9
193
26.7 86.3
21.1 1663
202 82
233 85.8
36 poi
39.5
89.3
18.1
36 JXE
26.1 59.0
27. 177
15.6 3600
261
269/58.
STU
26.5 56.0
2151
209 83.5
2291855
30 (g)
82.8
80.0
30 tha
15.9
55.9/500 57.0
25.67 26.
219
1865
26
JM bnd
20.6 791
18.7 78.1
19.3
156/ 76.1
20
26.7
86.9
kor ben
88.9
08/30
20 8006
2812
6
285 8786
33 89
kot
deu
30.7
85.6
29 86.3
85.8
kor
25.5
919 K3.
353 M49
25D M6.
kot heb
35.9
29.1
21 NO.T
28.
hi
30
R9/1 89
S 01
kor RO
86.0
24 10
3 84
5 86
26 16
26
kot
260
kor ISo
22.7 81
210 80
S1 9
12.5 1173
25 7
24
kot MAB
27
5
22 84.
26. 96.0
22.7 86.8
20
36.5
25.7 755
12.9 S4.6
23. 85.5
263 87
2or pol
25.5 89.3
25.5 38.9
21.5 96.9
22.8878
21.4 13.9
24. 90.0
kot por
51.6
85.5
85.4
31.1
$1.7
24.1 85
83.6
2307 85.0
266/351
25.
bor 121
83.6
319183
21./
182 90.8
25. 84.2
kot tha
61.6
3557875
33.3
33.1 86.8
88
kor und
5
19.6 78.6
20.3/ 77.
163 76.2
210/79)
214/82
kor Vic
32.47
58.7
29.1
298 7860
32.5 28.4
33.1 89.6
25.1
28./818
/5.5
260/
276/815
280/87.
cmm hen
24.7 83.6
102162 62.2
21 79.9
2067 8
24
1828
24 815
cmm
277
86.4
ang
35.0
87.0
87.6
35. 86.6
35 17
1376
39 88.6
cmn
fat
24.9
36.5
20.5 SL.7
848
cmn
heb
8
1555
1 82
210 80.9
360
29
cmm
hin
75.0
75
91 7
2147739
26 78
cmm
ind
27
595186
26 99
25
14.5/87
25.1 90
26 90.6
91
cmm
khm
19.7 0
1391 77.3
16 7
124
220
203 82
cmm
1979
25.4 84.5
con msa
26.7 87.2
24.9 36.0
26 846
250/ 96.5
28.6 36.
29 88
cmn mya
18.7 36.4
02 22.1
13:31 79
1207 79.5
16.5 34.6
19.5 87
cmm nid
89.8
25.7 89.
88.2
25 38.8
25.9
cmn par
22.2 57.5
3191372
115
329 1774
$4.0 87
cmm TES
787188
1ps
246/833
20 78
18. 76
24 243/33.0
26 84
cmm thn
37.5
5777
23
27.1 82.6
34.1 /47
3557966
56.3 88
cmm Ior
2607 96/9663
90
740
vic
32.7 80.2
30.0 96.5
116/8888
04.0 88.8
34.7 90.4
Avg.
30.1/86.7
27.7/33.0
26.0/835
264/84.7
30.3/86.2
31.1 387.7
Table 12: spBLEU/ COMET Scores on the FLORES-201 Benchmark.
18


Reviews:
[{"review_id": "M6P2aDwG2M", "reviewer": "Reviewer_s3NW", "summary": "This paper addresses Speech-guided Multimodal Machine Translation (SMMT), i.e., improving text translation by providing semantically equivalent speech at the input. During evaluation, this \"missing\" speech is generated by TTS.\nThe model uses a Whisper speech encoder which is frozen, a speech adapter based on Q-Former and an MLP, and a strong multilingual LLM based MT system (GemmaX2-28-9B) which is LoRA adapted.\n\nThe approach uses a multistage curriculum pretraining with progressively complex objectives: ASR, S2TT and SMMT. The authors also propose a \"self-evolution algorithm\", i.e. iteratively complementing the training data with synthesized speech while making sure that the additional speech improves performance.\n\nThe model is compared with other multimodal MT system which use the image modality (authentic and synthetic images) on the Multi30k benchmark. It performs best for 5 out of 6 tasks. The model also outperforms SOTA on Flores200, but the model is only test on 28 languages. Also, the base MT system, i.e., to which speech input is added, already outperforms the other systems for 5 out of 6 languages.", "questions": "- it's unclear on what data the model was trained on. Table 9 mentions Fleurs train, Common Voice train and Multi30k, but no sizes (per language) are given.\n\n- why you support 28 languages only? Is this limited by the TTS system? What type of training data do you need to scale to more languages?\n\n- the gains form the baseline text-only MT system and the proposed SMMT-10B are much higher for FLICK and MSCOCO than Flores. How can you explain this? What are the characteristics of the sentences in FLICK/MSCOC, e.g. very short, caption like?\n\n- Table 9 caption: \"we removed overlapping portions from the FLORES devtest set\". You report results on Flores devtest (cf. Table 10). You should never modify a test set! This makes it impossible to compare results. Which version of Flores was used in Tab 4, your modified one? Did you recalculate the scores for the other systems?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It's interesting to see that adding synthetic speech to the input of a text translation system improves performances. This seems to be more effective than adding images to the input (cf. Table 3).\n\nThe incremental training strategy is interesting and I wonder whether similar techniques could be used in other LLM settings than MT.", "weaknesses": "The experimental results show that the method works, but the authors fail to justify, or to try to explain, why it works! The only argument is that the prosody in speech helps translation. I could agree with this if human speech were used. However, this work addresses the issue how to improve text translation by providing in addition *synthetic speech*. I am not well aware of the current SOTA in TTS, but I would be surprised that the prosody is very rich.\n\nIt is not clear which data was used to train the speech and LLM adapter. The data mentioned in Table 9 gives no data sizes. I wonder whether the observed improvements are mainly the result of the additional training data. An important ablation would be to LoRa adapt the MT systems on the data, but without speech input.\n\nIn Table 4 and 12, good improvements are reported for several languages when translating from 4 languages into 27 foreign languages. Given that the algorithmic improvements are on the input only, I am puzzled how we can achieve a large variance on the improvements when translating from the same language into many different ones. The paper would benefit from a deeper analysis, instead of numerical metrics only.\n\nI would appreciate some examples, i.e., some sentences from Flores that are better translated with speech augmented input.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qb9Et8c1HF", "reviewer": "Reviewer_w9b6", "summary": "In this work, a Speech-guided Multimodal Machine Translation (SMMT) framework is proposed. The proposed system accepts textual input and synthesizes speech via the TTS model. Then, the MLLM processes both the text and synthetic speech to generate translations. The model is trained with two stages: pre-training and self-evolution training. The pre-training includes ASR to align speech and text modalities, speech translation and MMT to enable model translation capacity. The self-evolution training is a loop based training, i.e., TTS generates speech with different voices, and are assigned negative/positive tags by comparing the MT score and MMT score. Only those positive samples are used to update the MMT model. The model is evaluated with COMET score. The whole training loop is repeated if the model keeps improving COMET score. \nThe experimental results show the proposed system can achieve very competitive results on the MLLM dataset Multi30K, and MT dataset FLORES-200. \nThere are some details are missing in the paper and more detailed analysis is required. For example, what're the key factors that synthetic voice could be a positive sample or beneficial for MMT? Why speech data could help to reduce under-translation issues?", "questions": "- ``We synthesize speech for the evaluation text using a fixed reference voice''. The model is updated with different voices (sec 2.4.1). How do you select the reference voice?\n- what're the sizes of models in table 3?\n- why speech data helps to reduce under translation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Propose a Speech-guided Multimodal Machine Translation (SMMT) framework\n- The system achieves very competitive results on multiple datasets", "weaknesses": "- It is understandable that information from multiple sources might help the machine learn better representation. But those features are real data instead of synthetic data. In this work, synthetic speech is used augmented MMT. Deep analysis is required to reveal the main factors that contribute the good performance. For example, we don't know the synthetic speech samples are negative or positive during inference time. What's the impact if the provided speech sample is negative one? How to select the reference voice? \n- The self-evolution training could be very expensive. It is helpful to provide the training time spend on this stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNT5r194Gw", "reviewer": "Reviewer_wxJV", "summary": "The authors propose a Speech-guided Multimodal Machine Translation (SMMT) framework that integrates both speech and text as fused inputs into a multimodal large language model (MLLM) to enhance translation quality. The system comprises a multimodal LLM and a text-to-speech (TTS) module, augmented by a self-evolution mechanism that enables iterative model refinement. Experimental results demonstrate competitive performance on traditional machine translation benchmarks and promising improvements on the Multi30K multimodal dataset.", "questions": "- Could the authors clarify the conceptual and methodological differences between the proposed self-evolution mechanism and reinforcement learning approaches typically used for LLM fine-tuning?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Innovative self-evolution mechanism: The framework introduces a novel self-improvement process that leverages translation performance metrics as optimization objectives, facilitating continuous enhancement through iterative evolution cycles.\n- Comprehensive multimodal and multilingual support: The approach effectively unifies speech and text modalities and extends to multiple languages, showing potential for broader generalization.", "weaknesses": "- Limited evaluation scope: Experiments are primarily conducted on a small set of benchmarks (e.g., Multi30K, FLORES-200), leaving open questions about generalization to large-scale and domain-diverse datasets.\n- Dependence on external components: The framework relies heavily on open-source models, such as the TTS model, which may constrain performance consistency and scalability.\n- Lack of detailed ablations: It remains unclear how much each component (speech input, multimodal fusion, self-evolution) contributes to the observed performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)