Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning
Authors: Yexing Du, Youcheng Pan, Ziyang Ma, Bo Yang, Yifan Yang, Keqi Deng, Xie Chen, Yang Xiang, Ming Liu, Bing Qin
Accepted at: acl25

Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning

Yexing Du1,2  Youcheng Pan2  Ziyang Ma3  Bo Yang2  Yifan Yang3
Keqi Deng4  Xie Chen3  Yang Xiang2*  Ming Liu1,2*  Bing Qin1,2
1Harbin Institute of Technology  2Pengcheng Laboratory
3Shanghai Jiao Tong University  4University of Cambridge
{yxdu, mliu}@ir.hit.edu.cn, {panych,xiangy}@pcl.ac.cn

Abstract
Multimodal Large Language Models (MLLMs) have achieved significant success in Speech-to-Text Translation (S2TT) tasks. While most existing research has focused on English-centric translation directions, the exploration of many-to-many translation is still limited by the scarcity of parallel data. To address this, we propose a three-stage curriculum learning strategy that leverages the machine translation capabilities of large language models and adapts them to S2TT tasks, enabling effective learning in low-resource settings. We trained MLLMs with varying parameter sizes (3B, 7B, and 32B) and evaluated the proposed strategy using the FLEURS and CoVoST-2 datasets. Experimental results show that the proposed strategy achieves state-of-the-art average performance in \(15 \times 14\) language pairs, requiring fewer than 10 hours of speech data per language to achieve competitive results. ¹

1 Introduction
Speech-to-Text Translation (S2TT) involves converting speech from a source language into text in a target language. Traditionally, S2TT tasks have relied on a cascaded system, as shown in Figure 1(a), where an Automatic Speech Recognition (ASR) module transcribes speech into text (Baevski et al., 2020; Gulati et al., 2020), followed by a Machine Translation (MT) module that translates the transcribed text into the target language (Cheng et al., 2019; Beck et al., 2019). However, this cascade system often suffers from error propagation (Sperber and Paulik, 2020). Recently, Multimodal Large Language Models (MLLMs), illustrated in Figure 1(b), have demonstrated advantages in simplifying model architecture and mitigating error propagation in both ASR (Zhang et al., 2023; Ma et al., 2024) and S2TT tasks (Chu et al., 2024).

Figure 1: Comparison of S2TT Methods. (a) adopts a cascaded system; (b) directly generates translated text; (c) generates both transcription and translation text in an end-to-end process, with <[eng]>|[zh]o> indicating transcribing English and translating it into Chinese.

Current MLLMs process \(\{speech, instruction\}\) inputs to directly generate \(\{translation\}\), but this approach heavily relies on large-scale S2TT datasets. Existing datasets (Wang et al., 2020c; Di Gangi et al., 2019) predominantly focus on English, while datasets supporting many-to-many S2TT, such as FLEURS (Conneau et al., 2022), remain limited. Meanwhile, Large Language Models (LLMs) have demonstrated strong many-to-many multilingual MT capabilities. This raises the question: can the MT capabilities of LLMs be effectively transferred to the S2TT task with limited data?

Inspired by advances in transfer learning (Pham et al., 2024; Mueller et al., 2024), we transform the S2TT task into a **Speech Recognition and Translation (SRT)** task, which involves training \(\{speech, instruction\}\) to generate \(\{transcription, translation\}\), as shown in Figure 1(c). LLMs possess robust MT capabilities, which can be adapted to S2TT tasks with minimal data. This approach allows MLLMs to harness MT capabilities for many-to-many S2TT, effectively combining the advantages of both cascade and end-to-end models.
To connect MT and S2TT tasks, we propose a three-stage curriculum learning strategy: (1) ASR, which trains the MLLM for multimodal alignment, enabling the model to understand speech and generate transcriptions; (2) Speech-Aided Machine Translation (SMT), where both speech and transcription are provided, and the MLLM generates translations to improve cross-lingual capabilities; (3) SRT, where only speech is provided, and the MLLM generates both transcription and translation. The training proceeds sequentially through these three stages, with each stage resuming from the checkpoint of the previous one. The resulting MLLM achieves many-to-many S2TT through the SRT task. Additionally, we designed specialized instructions for many-to-many S2TT and implemented an optimized lightweight speech adapter for efficient speech feature compression to accelerate inference.

To evaluate our strategy, we trained three MLLM variants (3B, 7B, and 32B). In low-resource scenarios (with fewer than 10 hours of data per language), our MLLM, trained on the FLEURS dataset, demonstrated strong many-to-many S2TT capabilities, outperforming existing state-of-the-art end-to-end models. We also assessed performance on the EN-X direction of the CoVoST-2 dataset, where sufficient training data is available, and found that our strategy remains effective, surpassing state-of-the-art models.

The key contributions of this work are:

• This paper adopts a strategy that transforms the S2TT task into an SRT task, leveraging the machine translation capabilities of LLMs to enhance the many-to-many S2TT performance of MLLMs, particularly in low-resource settings.

• We propose a three-stage curriculum learning strategy and systematically evaluate our strategy across datasets of varying scales and model sizes (3B, 7B, 32B). To the best of our knowledge, our model is the first MLLM to support many-to-many S2TT at the 32B scale.

• Our model achieves state-of-the-art average performance across 15×14 translation directions in the S2TT task under low-resource settings on the FLEURS dataset, while also demonstrating strong robustness on the CoVoST-2 dataset in high-resource scenarios.

2 Methodology

In this section, we present the methodology of our approach. Section 2.1 defines the tasks involved in our method. Section 2.2 presents the architecture of the LLM-SRT model. Section 2.3 explains our curriculum learning strategy, which sequentially fine-tunes the model for ASR, SMT, and SRT tasks.

2.1 Problem Formulation

In this section, we define the following tasks:

ASR: Given the audio input \( \mathbf{X} \) and the instruction text \( \mathbf{T} \), the goal is to produce the transcribed text \( \mathbf{Y} \).

SMT: Given the audio input \( \mathbf{X} \), its corresponding transcription \( \mathbf{Y} \), and the instruction text \( \mathbf{T} \), the goal is to produce the translated text \( \mathbf{Z} \).

SRT: Given the audio input \( \mathbf{X} \) and the instruction text \( \mathbf{T} \), the goal is to produce both the transcription \( \mathbf{Y} \) and the translated text \( \mathbf{Z} \).

2.2 LLM-SRT

The LLM-SRT architecture is shown in Figure 2. The speech encoder extracts features from the speech input, and the speech adapter layer connects these features to the LLM, aligning their dimensions and incorporating speech feature compression. Finally, the LLM generates textual output by processing the concatenated embeddings derived from both text and speech features.

Speech Encoder. The speech encoder processes the audio input \( \mathbf{X} \) into a high-dimensional representation using the frozen Whisper encoder (Radford et al., 2023), which has been pretrained on large-scale supervised datasets for speech recognition and translation.

\[
\mathbf{H} = \text{Encoder}(\mathbf{X}),
\]

where \( \mathbf{H} \in \mathbb{R}^{T \times D} \) is the encoder’s output, with \( T \) representing the time dimension and \( D \) the hidden dimension of the encoder.

Speech Adapter. The speech adapter compresses the time dimension \( T \) and adjusts the hidden dimension \( D \) to match the LLM’s hidden dimension \( d_{\text{LLM}} \).

We use a Q-Former to convert input sequences into fixed-length query representations.

\[
\mathbf{Q}' = \text{Q-Former}(\mathbf{Q}, \mathbf{H}),
\]

where \( \mathbf{Q}' \in \mathbb{R}^{n_q \times D_q} \) is the output of the Q-Former, \( \mathbf{Q} \in \mathbb{R}^{n_q \times D_q} \) is the trainable query matrix, \( n_q \) is the
Figure 2: The Architecture of LLM-SRT. LLM-SRT consists of a speech encoder, speech adapter, and LLM. A three-stage curriculum learning strategy sequentially trains the ASR, SMT, and SRT tasks, as shown in Table 1. In stages 1 and 2, the speech adapter is continuously trained to enable efficient fine-tuning. In stage 3, the LLM is additionally unfrozen, while the speech adapter continues to be trained.

<table>
  <tr>
    <th>Task</th>
    <th>Audio</th>
    <th>Instruction</th>
    <th>Prediction</th>
  </tr>
  <tr>
    <td>ASR</td>
    <td>✓</td>
    <td>&lt;|eng&gt;</td>
    <td>Will it rain tomorrow?</td>
  </tr>
  <tr>
    <td>ASR</td>
    <td>✓</td>
    <td>&lt;|zho&gt;</td>
    <td>明天会下雨吗？</td>
  </tr>
  <tr>
    <td>SMT</td>
    <td>✓</td>
    <td>Will it rain tomorrow?&lt;|eng|&gt;&lt;|deu|&gt;</td>
    <td>Regnet es morgen?</td>
  </tr>
  <tr>
    <td>SMT</td>
    <td>✓</td>
    <td>明天会下雨吗? &lt;|zho|&gt;&lt;|jpn|&gt;</td>
    <td>明日は、雨かな？</td>
  </tr>
  <tr>
    <td>SRT</td>
    <td>✓</td>
    <td>&lt;|eng|&gt;&lt;|deu|&gt;</td>
    <td>Will it rain tomorrow?&lt;|eng|&gt;&lt;|deu|&gt;Regnet es morgen?</td>
  </tr>
  <tr>
    <td>SRT</td>
    <td>✓</td>
    <td>&lt;|zho|&gt;&lt;|jpn|&gt;</td>
    <td>明天会下雨吗? &lt;|zho|&gt;&lt;|jpn|&gt;明日は、雨かな？</td>
  </tr>
</table>

Table 1: Instruction Design. The instruction design is intended for fine-tuning instructions for three tasks: ASR, SMT, and SRT, using simple yet effective instructions to distinguish between them.

number of queries, and \( D_q \) is the hidden dimension of the Q-Former.

After the Q-Former layer, a multilayer perceptron (MLP) projects the feature dimensions from \( D_q \) to \( d_{\text{LLM}} \):

\[
\mathbf{E}^{\mathbf{X}} = \text{ReLU}(\mathbf{Q}'\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2,
\]

where \( \mathbf{E}^{\mathbf{X}} \in \mathbb{R}^{n_q \times d_{\text{LLM}}} \) is the output of the MLP layer, ready for LLM processing.

Tokenizer and LLM. The tokenizer and embedding layer process the instruction \( \mathbf{T} \) and produce \( \mathbf{E}^{\mathbf{T}} \in \mathbb{R}^{n_t \times d_{\text{LLM}}} \), where \( n_t \) is the length of the text tokens.

The speech features \( \mathbf{E}^{\mathbf{X}} \) and text features \( \mathbf{E}^{\mathbf{T}} \) are concatenated and fed into the frozen LLM:

\[
\mathbf{E}^Z = \mathbf{E}^{\mathbf{X}} \oplus \mathbf{E}^{\mathbf{T}},
\]

where \( \mathbf{E}^Z \in \mathbb{R}^{(n_t + n_q) \times d_{\text{LLM}}} \) is processed by the LLM to generate the output text.

The output text varies depending on the task. During training, the parameters of the adapter layer are updated based on the loss of the LLM’s output.

2.3 Curriculum Learning

LLM-SRT adopts a curriculum learning approach that incorporates three training tasks: ASR, SMT, and SRT.

Instruction Design. We designed minimalist instructions to help the model distinguish between tasks while reducing the instruction token length, as shown in Table 1. This design ensures that instructions like &lt;|eng|&gt;&lt;|deu|&gt; appear in the generated answers, effectively segmenting transcription and translation content in the SRT task.
ASR. In this stage, the model is pre-trained to develop ASR capabilities with a focus on multimodal alignment, while expanding language support by training on all intended languages. The speech adapter is trained with as much data as possible to ensure efficient fine-tuning.

SMT. This stage enhances the model’s cross-lingual abilities. Starting from the ASR checkpoint, the model takes both transcribed text and audio as input to generate translations based on the instruction. The purpose of this step is to activate the LLM’s inherent machine translation capabilities and establish the connection between the MT and S2TT tasks.

SRT. This stage activates the SRT capabilities of the MLLM, finalizing the model. Training continues from the SMT checkpoint, with the model receiving only audio input and a task-specific instruction, outputting both the transcription and translation of the speech. This extends the MT capabilities of LLMs to the S2TT task.

3 Experiments

3.1 Datasets

FLEURS. FLEURS² (Conneau et al., 2022) serves as the speech counterpart to the FloRes³ (Team et al., 2022) machine translation benchmarks. It includes 102 languages, with each training set containing approximately 10 hours of supervised speech data per language.

CoVoST-2. CoVoST-2⁴ (Wang et al., 2020c) is a large-scale multilingual S2TT corpus derived from the Common Voice dataset (Ardila et al., 2020). It contains translations from 21 languages to English and from English to 15 other languages.

3.2 Experiment Settings

Model Architecture. The baseline model consists of an LLM (Qwen 3B, 7B, 32B), a frozen speech encoder (Whisper-large-v3), and a trainable adapter layer comprising a Q-Former and an MLP. Following the configuration in Yu et al. (2024), we use 80 queries, each with a dimension of 768. Training can be minimized by freezing the LLM, or LoRA (Hu et al., 2021) can be applied for training.

Training Details. We used bf16 precision with Distributed Data Parallel (DDP), a learning rate of \(1 \times 10^{-4}\), 1000 warmup steps, and the AdamW optimizer. The models were trained on four A100 GPUs. We provide detailed settings in Table 11, 12, 13, and 14 of the Appendix.

For the ASR task, we used the Common Voice⁵ dataset, and for the SMT and SRT tasks, we used the FLEURS and CoVoST-2 datasets. In a 4-card A100 environment, the 3B and 7B models train in 3 days, while the 32B model trains in 7 days.

3.3 Compared Methods

We compare both cascade and end-to-end S2TT models, all of which support many-to-many S2TT.

• Cascaded systems are pipeline-based approaches, where an ASR model first transcribes speech into text, which is then translated by a machine translation model.

• SeamlessM4T⁶ (Barrault et al., 2023) is a foundational multilingual and multitask model capable of seamlessly translating and transcribing both speech and text. It supports speech-to-text translation for nearly 100 input and output languages.

• Qwen-Audio⁷ (Chu et al., 2023) is the multimodal extension of Qwen-LLM, designed to process diverse audio modalities, including speech, natural sounds, music, and songs, alongside text, generating text-based outputs.

Evaluation Metric. We use WER⁸ (Morris et al., 2004)(for the ASR task) and BLEU⁹ (Post, 2018) (for the S2TT task) as evaluation metrics.

<table>
  <tr>
    <th>Metric</th>
    <th>Details</th>
  </tr>
  <tr>
    <td>WER</td>
    <td>Text normalization follows Whisper<br>SacreBLEU signature:<br>needs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.4.3</td>
  </tr>
  <tr>
    <td>BLEU</td>
    <td>Except for jpn, kor, tha, yue, zho with char:<br>needs:1|case:mixed|eff:no|tok:char|smooth:exp|version:2.4.3</td>
  </tr>
</table>

Table 2: Metric Details. We followed the settings of SeamlessM4T-V2 (Barrault et al., 2023).

²https://huggingface.co/datasets/google/fleurs
³https://huggingface.co/datasets/facebook/flores
⁴https://github.com/facebookresearch/covost
⁵https://commonvoice.mozilla.org/en/datasets
⁶https://github.com/facebookresearch/seamless_communication
⁷https://github.com/QwenLM/Qwen-Audio
⁸https://huggingface.co/spaces/evaluate-metric/wer
⁹https://github.com/mjpost/sacrebleu
<table>
  <tr>
    <th rowspan="2">X→12 Languages</th>
    <th rowspan="2">S2TT Data (hour)</th>
    <th colspan="6">FLEURS</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng</th>
    <th>Deu</th>
    <th>Fra</th>
    <th>Jpn</th>
    <th>Rus</th>
    <th>Zho</th>
  </tr>
  <tr>
    <th colspan="8">Cascaded ASR+MT Methods</th>
  </tr>
  <tr>
    <td>Whisper + Qwen2.5-3B</td>
    <td>-</td>
    <td>23.4</td>
    <td>21.0</td>
    <td>20.9</td>
    <td>13.6</td>
    <td>19.6</td>
    <td>14.5</td>
    <td>18.8</td>
  </tr>
  <tr>
    <td>Whisper + Qwen2.5-7B</td>
    <td>-</td>
    <td>26.7</td>
    <td>23.3</td>
    <td>22.5</td>
    <td>15.1</td>
    <td>21.5</td>
    <td>16.4</td>
    <td>20.9</td>
  </tr>
  <tr>
    <td>Whisper + Qwen2.5-32B</td>
    <td>-</td>
    <td>29.9</td>
    <td>26.0</td>
    <td>24.6</td>
    <td>17.3</td>
    <td>23.8</td>
    <td>18.5</td>
    <td>23.3</td>
  </tr>
  <tr>
    <th colspan="8">End-to-End Models</th>
  </tr>
  <tr>
    <td>SeamlessM4T-V2 (2.3B)</td>
    <td>351,000 in-house</td>
    <td>33.1</td>
    <td>20.5</td>
    <td>19.6</td>
    <td>13.2</td>
    <td>19.6</td>
    <td>15.2</td>
    <td>20.2</td>
  </tr>
  <tr>
    <td>Qwen2-Audio (7B)</td>
    <td>in-house</td>
    <td>22.6</td>
    <td>20.1</td>
    <td>20.6</td>
    <td>4.0</td>
    <td>15.0</td>
    <td>13.7</td>
    <td>16.0</td>
  </tr>
  <tr>
    <td>Baseline-3B</td>
    <td>52</td>
    <td>11.8</td>
    <td>9.0</td>
    <td>9.5</td>
    <td>5.2</td>
    <td>9.7</td>
    <td>6.2</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B</td>
    <td>52</td>
    <td>27.2</td>
    <td>22.6</td>
    <td>22.0</td>
    <td>14.3</td>
    <td>21.3</td>
    <td>16.5</td>
    <td>20.6</td>
  </tr>
  <tr>
    <td>LLM-SRT-7B</td>
    <td>52</td>
    <td>27.4</td>
    <td>23.7</td>
    <td>22.8</td>
    <td>15.5</td>
    <td>21.8</td>
    <td>16.9</td>
    <td>21.4</td>
  </tr>
  <tr>
    <td>LLM-SRT-32B</td>
    <td>52</td>
    <td>32.5</td>
    <td>26.8</td>
    <td>26.1</td>
    <td>17.5</td>
    <td>25.6</td>
    <td>19.2</td>
    <td>24.6</td>
  </tr>
</table>

Table 3: BLEU Scores on 6x12 Directions in FLEURS. Underlined denotes previous state-of-the-art end-to-end models, while bold indicates models that outperform them. "-" indicates no S2TT data was used due to the cascade system. The baseline uses the same instruction-tuning strategy as Qwen2-Audio.

3.4 Overall Results

As shown in Tables 3 and 4, we evaluate the S2TT performance in low-resource settings on the FLEURS dataset. The results indicate that our model achieves state-of-the-art performance in the many-to-many S2TT task. Similarly, Table 5 presents the results under high-resource conditions on the CoVoST-2 dataset. Table 7 provides a comparison of inference speed, and Table 8 presents an ablation study of the three-stage curriculum learning.

Language Support. As an LLM-based model designed for many-to-many S2TT, conducting comprehensive baseline comparisons is both essential and challenging. To ensure a thorough evaluation, we compare baselines across 6x12 language directions and benchmark our model against state-of-the-art approaches in the 15x14 setting. The supported languages are listed in Table 10, while the complete experimental results are provided in Table 18.

Baseline-3B vs. LLM-SRT-3B. For the baseline model, we first conduct ASR pretraining as in Qwen2-Audio, followed by S2TT instruction-tuning with the same setup. Due to limited data, the baseline performs poorly, highlighting the limitations of traditional fine-tuning in low-resource settings.

In contrast, our curriculum learning training strategy achieves state-of-the-art performance (8.6→20.6) in low-resource scenarios, as we transform S2TT into an SRT task, effectively leveraging the machine translation capability of the LLM to achieve many-to-many S2TT.

SeamlessM4T-V2 vs. LLM-SRT-3B. The LLM-SRT-3B demonstrates superior performance over SeamlessM4T-V2 on non-English languages (e.g., for French 18.8→22.0), while it lags behind SeamlessM4T-V2 in English-to-X translation. This discrepancy can largely be attributed to the larger amount of S2TT data available for SeamlessM4T-V2, which includes 351,000 hours of training data compared to just 52 hours for LLM-SRT-3B. In situations where data resources are limited, our LLM-based approach has a greater advantage.

Cascaded Systems vs. LLM-SRT. As shown in Table 3, when using the same LLM as in the cascaded system, our MLLM demonstrates a clear performance advantage (e.g., for 3B, 18.8→20.6), highlighting the benefits of the end-to-end approach. The MLLM’s superior performance stems from its integrated framework, which eliminates intermediate steps and enables more efficient knowledge transfer, resulting in improved translation accuracy and robustness. This makes the model more effective in handling complex language tasks.

Scaling Law of LLM-SRT. As shown in Table 3, experiments on models with 3B, 7B, and 32B parameters demonstrate that our method follows the scaling law of LLMs (e.g., 20.6 for 3B, 21.4 for 7B, and 24.6 for 32B). Notably, the 32B model achieved state-of-the-art performance across all directions, confirming the generalizability of our approach. Our model’s performance is strongly correlated with the machine translation capability of the LLM, making the choice of an appropriate LLM foundation crucial for optimal performance.
<table>
  <tr>
    <th rowspan="2">X→14 Languages</th>
    <th colspan="2">S2TT Data (hour)</th>
    <th colspan="10">FLEURS</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng</th>
    <th>Deu</th>
    <th>Fra</th>
    <th>Ind</th>
    <th>Ita</th>
    <th>Jpn</th>
    <th>Kor</th>
    <th>Nld</th>
    <th>Por</th>
    <th>Rus</th>
    <th>Spa</th>
    <th>Tha</th>
    <th>Vie</th>
    <th>Yue</th>
    <th>Zho</th>
  </tr>
  <tr>
    <td colspan="17">Machine Translation</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td>-</td>
    <td>29.5</td>
    <td>25.4</td>
    <td>25.2</td>
    <td>24.6</td>
    <td>22.7</td>
    <td>18.4</td>
    <td>18.8</td>
    <td>22.1</td>
    <td>27.1</td>
    <td>23.4</td>
    <td>22.1</td>
    <td>18.3</td>
    <td>22.6</td>
    <td>19.0</td>
    <td>20.5</td>
    <td>22.6</td>
  </tr>
  <tr>
    <td colspan="17">Speech to Text Translation</td>
  </tr>
  <tr>
    <td>SeamlessM4T-V2 (2.3B)</td>
    <td>351,000</td>
    <td>33.3</td>
    <td>21.6</td>
    <td>21.1</td>
    <td>18.4</td>
    <td>19.1</td>
    <td>14.5</td>
    <td>17.2</td>
    <td>18.4</td>
    <td>18.8</td>
    <td>20.6</td>
    <td>17.6</td>
    <td>12.8</td>
    <td>17.5</td>
    <td>15.2</td>
    <td>16.7</td>
    <td>18.8</td>
  </tr>
  <tr>
    <td>SeamlessM4T-V2 +Lora</td>
    <td>351,129</td>
    <td>32.8</td>
    <td>22.0</td>
    <td>21.3</td>
    <td>19.1</td>
    <td>19.4</td>
    <td>14.4</td>
    <td>17.0</td>
    <td>18.6</td>
    <td>18.6</td>
    <td>21.3</td>
    <td>18.1</td>
    <td>13.2</td>
    <td>17.5</td>
    <td>14.9</td>
    <td>17.0</td>
    <td>19.0</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B-V2</td>
    <td>129</td>
    <td>27.8</td>
    <td>23.4</td>
    <td>24.0</td>
    <td>22.4</td>
    <td>22.0</td>
    <td>15.3</td>
    <td>17.9</td>
    <td>20.4</td>
    <td>25.7</td>
    <td>22.8</td>
    <td>21.9</td>
    <td>12.9</td>
    <td>18.2</td>
    <td>15.5</td>
    <td>19.2</td>
    <td>20.6</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B-V2 +Lora</td>
    <td>129</td>
    <td>29.1</td>
    <td>24.5</td>
    <td>24.3</td>
    <td>22.9</td>
    <td>22.8</td>
    <td>16.5</td>
    <td>18.0</td>
    <td>20.9</td>
    <td>26.2</td>
    <td>23.3</td>
    <td>22.6</td>
    <td>14.7</td>
    <td>19.0</td>
    <td>16.3</td>
    <td>19.6</td>
    <td>21.4</td>
  </tr>
</table>

Table 4: BLEU Scores on 15×14 Directions. The complete results are in Table 15 and 18 in the Appendix.

Many-to-Many S2TT on FLEURS. As shown in Tables 4 and 18, we compared performance across 15 languages and 210 translation directions. Table 4 reports the average performance across the 15 languages, where our model achieves state-of-the-art BLEU performance (18.8→21.4). Table 11 presents detailed results for all 210 directions, showing that our model outperforms SeamlessM4T-V2 in 154 directions.

Train Adapter Only vs. Fine-tune LLM. As shown in Table 4, LLM-SRT-3B-V2 achieves high translation performance (20.6) by freezing the speech encoder and LLM, while training only the speech adapter. Further performance improvement (20.6→21.4) can be achieved by unfreezing the LLM, such as through LoRA training.

MT vs. S2TT. As shown in Figure 3, we compared the MT performance of Qwen2.5-3B with the S2TT performance of LLM-SRT-3B-V2 across 210 translation directions. The results show a strong correlation between our MLLM’s S2TT and MT performance, confirming that the strong S2TT capability of LLM-SRT-3B stems from the LLM’s machine translation ability.

![BLEU Scores for 15x14 Directions: Comparison between MT and S2TT. The results show a strong correlation, suggesting that our S2TT capability is derived from the MT model. Table 16 includes an error analysis showing that S2TT outperforms MT.](page_101_1342_627_377.png)

Figure 3: BLEU Scores for 15×14 Directions: Comparison between MT and S2TT. The results show a strong correlation, suggesting that our S2TT capability is derived from the MT model. Table 16 includes an error analysis showing that S2TT outperforms MT.

<table>
  <tr>
    <th>Eng→X</th>
    <th>S2TT Data (hour)</th>
    <th colspan="3">CoVoST-2</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Deu</th>
    <th>Jpn</th>
    <th>Zho</th>
    <th></th>
  </tr>
  <tr>
    <td colspan="6">Cascaded ASR+MT Methods</td>
  </tr>
  <tr>
    <td>Whisper+NLLB-3.3B</td>
    <td>-</td>
    <td>33.4</td>
    <td>31.0</td>
    <td>32.0</td>
    <td>32.1</td>
  </tr>
  <tr>
    <td>Whisper+Qwen2.5-3B</td>
    <td>-</td>
    <td>21.1</td>
    <td>28.1</td>
    <td>35.0</td>
    <td>28.0</td>
  </tr>
  <tr>
    <td>Whisper+Qwen2.5-7B</td>
    <td>-</td>
    <td>24.1</td>
    <td>29.8</td>
    <td>37.6</td>
    <td>30.5</td>
  </tr>
  <tr>
    <td>Whisper+Qwen2.5-32B</td>
    <td>-</td>
    <td>28.3</td>
    <td>35.0</td>
    <td>41.1</td>
    <td>34.8</td>
  </tr>
  <tr>
    <td colspan="6">End-to-End Models</td>
  </tr>
  <tr>
    <td>SeamlessM4T-V2</td>
    <td>351,000</td>
    <td>37.0</td>
    <td>39.7</td>
    <td>35.9</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>Qwen-Audio (7B)</td>
    <td>3,700</td>
    <td>25.1</td>
    <td>--</td>
    <td>40.3</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2-Audio (7B)</td>
    <td>in-house</td>
    <td>29.9</td>
    <td>39.7</td>
    <td>44.0</td>
    <td>37.9</td>
  </tr>
  <tr>
    <td colspan="6">with FLEURS Data</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B</td>
    <td>52</td>
    <td>24.9</td>
    <td>37.3</td>
    <td>40.7</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>LLM-SRT-7B</td>
    <td>52</td>
    <td>26.6</td>
    <td>40.1</td>
    <td>41.4</td>
    <td>36.0</td>
  </tr>
  <tr>
    <td>LLM-SRT-32B</td>
    <td>52</td>
    <td>30.2</td>
    <td>43.1</td>
    <td>43.5</td>
    <td>38.9</td>
  </tr>
  <tr>
    <td colspan="6">with CoVoST-2 Data</td>
  </tr>
  <tr>
    <td>Baseline-3B*</td>
    <td>430</td>
    <td>23.6</td>
    <td>34.7</td>
    <td>38.6</td>
    <td>32.3</td>
  </tr>
  <tr>
    <td>Baseline-7B*</td>
    <td>430</td>
    <td>25.3</td>
    <td>36.5</td>
    <td>40.2</td>
    <td>34.1</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B*</td>
    <td>430</td>
    <td>26.5</td>
    <td>39.4</td>
    <td>44.0</td>
    <td>36.6</td>
  </tr>
  <tr>
    <td>LLM-SRT-7B*</td>
    <td>430</td>
    <td>28.7</td>
    <td>41.6</td>
    <td>47.1</td>
    <td>39.1</td>
  </tr>
</table>

Table 5: BLEU Scores on CoVoST-2. * indicates trained on CoVoST-2 dataset.

Eng→X S2TT. As shown in Table 4, the relatively weaker performance of our model in Eng→X translations is primarily due to the limited amount of English data, stemming from the balanced FLEURS dataset, which contains fewer than 10 hours of data per language. Consequently, as shown in Table 5, our 3B model underperforms compared to SeamlessM4T-V2, but our method is more efficient in data-constrained scenarios.

Scaling Law of S2TT Data. As shown in Table 5, we compare the performance of Eng→X translations in both low-resource and high-resource scenarios, with data scales ranging from 52 hours to 430 hours. The results indicate that both our 3B (34.3→36.6) and 7B (35.0→39.1) models show consistent performance improvements as the data scale increases, demonstrating that our approach is effective in low-resource settings and scales well with more data.
MLLM Architecture. Our Baseline-7B model follows the Qwen2-Audio setup but keeps the speech encoder frozen, leading to lower performance (34.1 vs. 37.9).

LLM-SRT-7B employs a curriculum learning strategy that incrementally fine-tunes only the adapter layer, allowing the model to adapt more effectively while maintaining efficiency. This approach leads to notable improvements in performance by enabling stable and gradual optimization. Although unfreezing both the speech encoder and the LLM could yield further performance gains, it would also result in significantly higher computational costs.

Multi-task Model LLM-SRT. LLM-SRT is a multi-task model that supports ASR, SMT, and SRT tasks. As shown in Table 6, we evaluated the performance of all tasks.

We found that performing the SRT task did not degrade, but rather slightly improved, ASR performance. Moreover, with the correct transcription, the model achieved a high BLEU score in the SMT task.

<table>
  <tr>
    <th>Model</th>
    <th>Task</th>
    <th>WER<sub>↓</sub></th>
    <th>Deu</th>
    <th>Jpn</th>
    <th>Zho</th>
  </tr>
  <tr>
    <td rowspan="3">LLM-SRT-7B*</td>
    <td>ASR</td>
    <td>11.1</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SMT</td>
    <td>0</td>
    <td>32.8</td>
    <td>43.6</td>
    <td>55.6</td>
  </tr>
  <tr>
    <td>SRT</td>
    <td>10.9</td>
    <td>28.7</td>
    <td>41.6</td>
    <td>47.1</td>
  </tr>
  <tr>
    <td>Eng→X</td>
    <td>SRT</td>
    <td>10.9</td>
    <td>28.7</td>
    <td>41.6</td>
    <td>47.1</td>
  </tr>
</table>

Table 6: Performance of Different Tasks. We evaluated the model on the CoVoST-2 dataset and found that the SMT task, which uses both speech and ground-truth transcription as inputs, achieved a notably high BLEU score.

Inference Speed. As shown in Table 7, our method achieves nearly a 3x improvement in inference speed compared to the Qwen2-Audio model, which has a similar parameter size. Even with beam search enabled (5 beams) in our model, the speed remains faster than the greedy search of Qwen2-Audio.

The speed difference is mainly due to our optimized speech adapter design, which compresses the audio features to a fixed size of 80, significantly reducing the token length input to the LLM and accelerating inference.

<table>
  <tr>
    <th>Model</th>
    <th>Strategy</th>
    <th>Batch</th>
    <th>Time(s)<sub>↓</sub></th>
  </tr>
  <tr>
    <td rowspan="2">Qwen2-Audio</td>
    <td>Greedy Search</td>
    <td>4</td>
    <td>59</td>
  </tr>
  <tr>
    <td>Beam Search 5</td>
    <td>8</td>
    <td>/</td>
  </tr>
  <tr>
    <td rowspan="4">LLM-SRT-7B*</td>
    <td>Greedy Search</td>
    <td>4</td>
    <td>74</td>
  </tr>
  <tr>
    <td>Beam Search 5</td>
    <td>8</td>
    <td>39</td>
  </tr>
  <tr>
    <td>Greedy Search</td>
    <td>16</td>
    <td>28</td>
  </tr>
  <tr>
    <td>Beam Search 5</td>
    <td>32</td>
    <td>22</td>
  </tr>
  <tr>
    <td>Beam Search 5</td>
    <td>64</td>
    <td>19</td>
  </tr>
  <tr>
    <td>Beam Search 5</td>
    <td>4</td>
    <td>93</td>
  </tr>
  <tr>
    <td>Beam Search 5</td>
    <td>8</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Beam Search 5</td>
    <td>12</td>
    <td>56</td>
  </tr>
</table>

Table 7: Inference Speed Comparison. We compared the inference time for processing 1,000 speech samples between Qwen2-Audio and LLM-SRT-7B*, both with similar parameter sizes, using a 4-card 4090 DDP FP16 inference setup. LLM-SRT-7B* demonstrated a 3x speed improvement. / Indicates out-of-memory issues.

decrease in BLEU score, emphasizing its importance in S2TT. This section explores the effect of skipping the SMT task and proceeding directly to SRT. As shown in Table 8, removing SMT and SRT resulted in performance drops of 1.1 and 4.9 points, respectively. While direct SRT maintains an MT-S2TT link with only minor degradation, omitting SRT and relying solely on instruction fine-tuning leads to a substantial performance drop.

<table>
  <tr>
    <th>Model</th>
    <th>Deu</th>
    <th>Jpn</th>
    <th>Zho</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>LLM-SRT-7B*</td>
    <td>28.7</td>
    <td>41.6</td>
    <td>47.1</td>
    <td>39.1</td>
  </tr>
  <tr>
    <td>w/o ASR</td>
    <td>26.4(-2.3)</td>
    <td>38.6(-3.0)</td>
    <td>45.5(-1.6)</td>
    <td>36.8(-2.3)</td>
  </tr>
  <tr>
    <td>w/o SMT</td>
    <td>27.6(-1.1)</td>
    <td>39.7(-1.9)</td>
    <td>46.5(-0.6)</td>
    <td>38.0(-1.1)</td>
  </tr>
  <tr>
    <td>w/o SRT</td>
    <td>25.6(-3.1)</td>
    <td>36.7(-4.9)</td>
    <td>40.4(-6.7)</td>
    <td>34.2(-4.9)</td>
  </tr>
</table>

Table 8: Ablation Study. We evaluated the model on the CoVoST-2 Eng→X dataset. Removing the SRT task leads to a substantial reduction in BLEU score, underscoring its critical role in overall performance.

Case Study. As shown in Tables 9 and 17, SeamlessM4T-V2 demonstrates poor performance in Japanese, achieving the lowest BLEU score. Qwen-Audio outperforms it, while our method outperforms significantly. Our approach follows a two-step process: first, it generates a transcription of the input speech, which is then used to produce the translation. This strategy ensures that the translation benefits from the transcribed text, leveraging its structure and context to improve accuracy. By incorporating transcription into the translation process, our method minimizes ambiguities and enhances translation quality, especially in complex or context-dependent scenarios.
<table>
  <tr>
    <th>Model</th>
    <th>BLEU↑</th>
  </tr>
  <tr>
    <td>Audio</td>
    <td>三国是中国古代历史上最血腥的时代之一。成千上万的人为了争夺西安豪华宫殿的最高权力而死去。</td>
    <td></td>
  </tr>
  <tr>
    <td>Ground-truth</td>
    <td>三国時代は、古代中国の歴史の中で最も血なまぐさい時代の一つでした。西安の大宮殿の最高権力を争う戦いの中で何千人もが戦いました。</td>
    <td></td>
  </tr>
  <tr>
    <td>SeamlessM4T-V2</td>
    <td>三国は中国の歴史で最も血なまぐさい時代の一つに千人以上の人が安東宮殿の最高権力を争うために死ん huan</td>
    <td>14.7 | 5.6</td>
  </tr>
  <tr>
    <td>Qwen2-Audio</td>
    <td>三国は中国の歴史で最も血なまぐさい時代の一つです。何千人も人が、西安の豪華宮殿の権力を得るために死んでいます。</td>
    <td>27.9 | 12.5</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B</td>
    <td>三国是中国古代历史上最血腥的时期之一，成千上万人为了争夺西安豪华宫殿的最高权力而死去。<[zh]><[jp]></td>
    <td>34.9 | 16.5</td>
  </tr>
  <tr>
    <td>LLM-SRT-32B</td>
    <td>三国是中国古代历史上最血腥的时期之一，成千上万人为了争夺西安豪华宫殿的最高权力而死去。<[zh]><[jp]></td>
    <td>49.3 | 40.4</td>
  </tr>
</table>

Table 9: Case Study. We compare the BLEU scores of our method with those of other approaches, using the 'char' tokenizer (denoted in regular font) and the 'ja-mecab' tokenizer (presented in italics). With a transcription Character Error Rate (CER) score of 11.4 for the 3B model and 4.6 for the 32B model.

4 Related Work

Cascaded S2TT. This method follows a two-step process: first, ASR transcribes the spoken language into text, and then MT translates the transcribed text into the target language. This approach leverages the strengths of specialized ASR (Radford et al., 2023; Baevski et al., 2020) and MT (Fan et al., 2021) models, utilizing extensive training data and advanced techniques. ASR models accurately convert speech to text, while sophisticated MT models, benefiting from large multilingual datasets, translate with high accuracy and fluency. However, the cascaded approach is prone to error propagation.

End-to-End S2TT. In this paradigm, a single model is trained to directly map speech from the source language to text in the target language, skipping the intermediate transcription step (Wang et al., 2020a,b; Inaguma et al., 2020). Early work on joint speech recognition and translation primarily used streaming models, aiming to provide real-time multilingual synchronization (Sperber et al., 2020; Dong et al., 2021; Papi et al., 2024). These pioneering efforts focused more on reducing latency and enhancing efficiency than offline speech translation systems. End-to-end ST offers several advantages, including reduced latency, simplified system architecture, and the elimination of error propagation between the ASR and MT stages. Despite these benefits, end-to-end ST models face challenges, such as the need for extensive parallel speech-to-text data, which is resource-intensive and difficult to obtain.

Audio MLLMs. Recent advancements in audio MLLMs (Li et al., 2025) have significantly improved speech recognition and translation. SpeechGPT (Zhang et al., 2023) uses prompting to enhance speech recognition in large language models. BLSP-KD (Wang et al., 2024) refines speech-text alignment through knowledge distillation. SALMONN (Tang et al., 2023) aims to improve auditory comprehension of language and music in models. Qwen-Audio (Chu et al., 2023) advances audio recognition and translation by re-training speech encoders within a multi-task framework. Qwen2.5-Omni (Xu et al., 2025), the end-to-end multimodal model, is designed for comprehensive multimodal perception, seamlessly processing heterogeneous input modalities.

5 Conclusion

In this paper, we propose a novel strategy that reformulates the speech-to-text translation task as a combination of speech recognition and translation tasks, leveraging the machine translation capabilities of LLMs to enhance the performance of MLLMs in S2TT. To validate our approach, we train three MLLMs with sizes 3B, 7B, and 32B, and implement a three-stage curriculum learning strategy, which proves highly effective in low-resource scenarios while further improving performance when sufficient training data is available. Our model achieves state-of-the-art results across 15×14 translation directions, excelling in low-resource learning on the FLEURS dataset and supervised training on the CoVoST-2 dataset. These results highlight the robustness and effectiveness of our approach across diverse linguistic and data availability settings.

For future work, we aim to further optimize the LLM-SRT model to push its performance boundaries and extend its application to a broader range of languages.
Limitations

This paper presents a method for training an MLLM for languages with less than 10 hours of speech translation data.

However, the performance of S2TT and the range of supported languages are constrained by the capabilities of the LLM. MLLMs trained using this method may not perform well on languages that are not supported by the LLM or on those with poor machine translation performance.

Acknowledgement

The research in this article is supported by the National Natural Science Foundation of China (Grants No. U22B2059 and 62276083) and the National Science and Technology Innovation 2030 Major Program (Grant No. 2024ZD01NL00101). We also appreciate the support from China Mobile Group Heilongjiang Co., Ltd. @ on our research, the research is a joint collaboration between the involved parties.

References

R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4211–4215.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.

Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. arXiv preprint arXiv:1809.01431.

Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. Seamlessmt4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596.

Daniel Beck, Trevor Cohn, and Gholamreza Haffari. 2019. Neural speech translation using lattice transformations and graph networks. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 26–31.

Qiao Cheng, Meiyuan Fang, Yaqian Han, Jin Huang, and Yitao Duan. 2019. Breaking the data barrier: Towards robust speech translation via adversarial stability training. arXiv preprint arXiv:1909.11430.

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.

Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446.

Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. Must-c: a multilingual speech translation corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012–2017. Association for Computational Linguistics.

Qianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021. Consecutive decoding for speech-to-text translation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12738–12748.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48.

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Enrique Yalta Soplin, Tomoki Hayashi, and Shinji Watanabe. 2020. Espnet-st: All-in-one speech translation toolkit. arXiv preprint arXiv:2004.10234.

Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. Pre-training for speech translation: Ctc meets optimal
transport. In International Conference on Machine Learning, pages 18667–18685. PMLR.

Yunxin Li, Zhenyu Liu, Zitao Li, Xuyan Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. 2025. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv e-prints, pages arXiv–2505.

Ziyang Ma, Guannou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. 2024. An embarrassingly simple approach for llm with strong asr capacity. arXiv preprint arXiv:2402.08846.

Andrew Cameron Morris, Viktoria Maier, and Phil Green. 2004. From wer and ril to mer and wil: improved evaluation measures for connected speech recognition. In Eighth International Conference on Spoken Language Processing.

David Mueller, Mark Dredze, and Nicholas Andrews. 2024. Multi-task transfer matters during instruction-tuning. In Findings of the Association for Computational Linguistics ACL 2024, pages 14880–14891.

Sara Papi, Peidong Wang, Junkun Chen, Jian Xue, Naoyuki Kanda, Jinyu Li, and Yashesh Gaur. 2024. Leveraging timestamp information for serialized joint streaming recognition and translation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10381–10385. IEEE.

Trinh Pham, Khoi M Le, and Luu Anh Tuan. 2024. Unibridge: A unified approach to cross-lingual transfer learning for low-resource languages. arXiv preprint arXiv:2406.09717.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels. Association for Computational Linguistics.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR.

Matthias Sperber and Matthias Paulik. 2020. Speech translation and the end-to-end promise: Taking stock of where we are. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7409–7421.

Matthias Sperber, Hendra Setiawan, Christian Gollan, Udhyakumar Nallasamy, and Matthias Paulik. 2020. Consistent transcription and translation of speech. Transactions of the Association for Computational Linguistics, 8:695–709.

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonnn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, et al. 2022. No language left behind: Scaling human-centered machine translation. Authors abbreviated for brevity. Full list at https://arxiv.org/abs/2207.04672.

Changhan Wang, Juan Pino, and Jiatao Gu. 2020a. Improving cross-lingual transfer learning for end-to-end speech recognition with speech translation. arXiv preprint arXiv:2006.05474.

Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, and Juan Pino. 2020b. Fairseq s2t: Fast speech-to-text modeling with fairseq. arXiv preprint arXiv:2010.05171.

Changhan Wang, Anne Wu, and Juan Pino. 2020c. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310.

Chen Wang, Minpeng Liao, Zhongqiang Huang, and Ji-ajun Zhang. 2024. Blsp-kd: Bootstrapping language-speech pre-training via knowledge distillation. arXiv preprint arXiv:2405.19041.

Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215.

Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. Connecting speech encoder and large language model for asr. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12637–12641. IEEE.

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.

A Appendix

In this chapter, we report the language support in Table 10, the case study in Table 17, and the BLEU scores for the 15x14 directions of the FLEURS dataset in Table 18.
<table>
  <tr>
    <th>Code</th>
    <th>Language</th>
    <th>Family</th>
    <th>Subgrouping</th>
    <th>Script</th>
  </tr>
  <tr>
    <td>deu</td>
    <td>German</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>English</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>fra</td>
    <td>French</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>ind</td>
    <td>Indonesian</td>
    <td>Austronesian</td>
    <td>Malayo-Polynesian</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>ita</td>
    <td>Italian</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>jpn</td>
    <td>Japanese</td>
    <td>Japonic</td>
    <td>Japanesic</td>
    <td>Jpan</td>
  </tr>
  <tr>
    <td>kor</td>
    <td>Korean</td>
    <td>Koreanic</td>
    <td>Korean</td>
    <td>Kore</td>
  </tr>
  <tr>
    <td>nld</td>
    <td>Dutch</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>por</td>
    <td>Portuguese</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>rus</td>
    <td>Russian</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>Cyril</td>
  </tr>
  <tr>
    <td>spa</td>
    <td>Spanish</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>tha</td>
    <td>Thai</td>
    <td>Tai-Kadai</td>
    <td>Kam-Tai</td>
    <td>Thai</td>
  </tr>
  <tr>
    <td>vie</td>
    <td>Vietnamese</td>
    <td>Austroasiatic</td>
    <td>Vietic</td>
    <td>Latn</td>
  </tr>
  <tr>
    <td>yue</td>
    <td>Cantonese</td>
    <td>Sino-Tibetan</td>
    <td>Sinitic</td>
    <td>Hant</td>
  </tr>
  <tr>
    <td>zho</td>
    <td>Chinese</td>
    <td>Sino-Tibetan</td>
    <td>Sinitic</td>
    <td>Hans</td>
  </tr>
</table>

Table 10: Language Support. The table lists language codes, names, families, subgroups, and scripts. Language support is extensible based on ISO 639-3. 6×12 translation directions cover source languages (eng, deu, fra, zho, rus, jpn) and target languages (eng, deu, fra, spa, por, ita, nld, rus, jpn, kor, vie, ind, zho).

<table>
  <tr>
    <th>Stage</th>
    <th>Dataset</th>
    <th>Hour</th>
    <th>Batch</th>
    <th>Step</th>
    <th>Learning Rate</th>
    <th>Warmup Step</th>
    <th>Optimizer</th>
  </tr>
  <tr>
    <td>ASR</td>
    <td>Common Voice 19</td>
    <td>4498</td>
    <td></td>
    <td>472000</td>
    <td>1e-4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ASR</td>
    <td>FLEURS</td>
    <td>129</td>
    <td></td>
    <td>472000</td>
    <td>1e-4</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>SMT</td>
    <td>FLEURS</td>
    <td>129</td>
    <td>16</td>
    <td>44000</td>
    <td>1e-4</td>
    <td>1000</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>SRT</td>
    <td>FLEURS</td>
    <td>129</td>
    <td></td>
    <td>83000</td>
    <td>1e-5</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 11: Training Details for LLM-SRT-3B-V2. The step count refers to the number of steps on a single GPU.

<table>
  <tr>
    <th>Model</th>
    <th>Encoder</th>
    <th>Adapter</th>
    <th>LLM</th>
    <th>Language Support</th>
  </tr>
  <tr>
    <td>LLM-SRT-3B</td>
    <td></td>
    <td></td>
    <td>Qwen2.5-3B</td>
    <td>6 × 12</td>
  </tr>
  <tr>
    <td>LLM-SRT-7B</td>
    <td>Whisper large-v3’s</td>
    <td></td>
    <td>Qwen2.5-7B</td>
    <td>6 × 12</td>
  </tr>
  <tr>
    <td>LLM-SRT-32B</td>
    <td>encoder</td>
    <td>Q-Former + MLP</td>
    <td>Qwen2.5-32B</td>
    <td>6 × 12</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B-V2</td>
    <td></td>
    <td></td>
    <td>Qwen2.5-3B</td>
    <td>15 × 14</td>
  </tr>
</table>

Table 12: Model Settings. The V2 model employs the same architecture but supports more languages.

<table>
  <tr>
    <th>Model</th>
    <th>Rank (r)</th>
    <th>Alpha</th>
    <th>Dropout</th>
    <th>Target Keys</th>
    <th>Bias</th>
  </tr>
  <tr>
    <td>SeamlessM4T-V2</td>
    <td>32</td>
    <td>64</td>
    <td>0.2</td>
    <td>.*-proj</td>
    <td>none</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B-V2</td>
    <td>8</td>
    <td>32</td>
    <td>0.05</td>
    <td>q_proj, v_proj</td>
    <td>none</td>
  </tr>
</table>

Table 13: LoRA Configuration Settings. We only fine-tuned the LLM with LoRA for LLM-SRT-3B-V2.

<table>
  <tr>
    <th>Modules</th>
    <th>Param</th>
    <th>Training stage</th>
    <th>Details</th>
  </tr>
  <tr>
    <td>Speech Encoder</td>
    <td>~635M</td>
    <td>–</td>
    <td>Whisper’s encoder</td>
  </tr>
  <tr>
    <td>Speech Adapter</td>
    <td>~73.7M</td>
    <td>I&II&III</td>
    <td>Q-Former and MLP</td>
  </tr>
  <tr>
    <td>LLM</td>
    <td>~3.1B</td>
    <td>–</td>
    <td>Qwen2.5-3B</td>
  </tr>
  <tr>
    <td>LLM adapter</td>
    <td>~1.8M</td>
    <td>III</td>
    <td>LoRA</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>~3.8B</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 14: LLM-SRT-3B-V2 Training Parameters. The red indicates trainable parameters.
<table>
  <tr>
    <th>X→14</th>
    <th>SeamlessM4T-V2</th>
    <th>SeamlessM4T-V2+Lora</th>
    <th>LLM-SRT-3B</th>
    <th>LLM-SRT-3B+Lora</th>
  </tr>
  <tr>
    <td>deu</td>
    <td>21.6 / 76.2</td>
    <td>22.0 / 76.8</td>
    <td>23.4 / 83.4</td>
    <td>24.5 / 83.5</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>33.3 / 84.8</td>
    <td>32.8 / 84.3</td>
    <td>27.8 / 85.5</td>
    <td>29.1 / 85.4</td>
  </tr>
  <tr>
    <td>fra</td>
    <td>21.1 / 77.6</td>
    <td>21.3 / 78.0</td>
    <td>24.0 / 83.5</td>
    <td>24.3 / 83.6</td>
  </tr>
  <tr>
    <td>ind</td>
    <td>18.4 / 73.8</td>
    <td>19.1 / 75.0</td>
    <td>22.4 / 82.3</td>
    <td>22.9 / 82.2</td>
  </tr>
  <tr>
    <td>ita</td>
    <td>19.1 / 76.9</td>
    <td>19.4 / 77.4</td>
    <td>22.0 / 83.5</td>
    <td>22.8 / 83.5</td>
  </tr>
  <tr>
    <td>jpn</td>
    <td>14.5 / 72.4</td>
    <td>14.4 / 72.5</td>
    <td>15.3 / 80.1</td>
    <td>16.5 / 80.3</td>
  </tr>
  <tr>
    <td>kor</td>
    <td>17.2 / 76.6</td>
    <td>17.0 / 76.5</td>
    <td>17.9 / 81.8</td>
    <td>18.0 / 81.9</td>
  </tr>
  <tr>
    <td>nld</td>
    <td>18.4 / 74.8</td>
    <td>18.6 / 75.3</td>
    <td>20.4 / 81.3</td>
    <td>20.9 / 81.4</td>
  </tr>
  <tr>
    <td>por</td>
    <td>18.8 / 73.3</td>
    <td>18.6 / 73.5</td>
    <td>25.7 / 84.1</td>
    <td>26.2 / 84.2</td>
  </tr>
  <tr>
    <td>rus</td>
    <td>20.6 / 75.9</td>
    <td>21.3 / 76.7</td>
    <td>22.8 / 82.5</td>
    <td>23.3 / 82.5</td>
  </tr>
  <tr>
    <td>spa</td>
    <td>17.6 / 76.1</td>
    <td>18.1 / 76.9</td>
    <td>21.9 / 83.9</td>
    <td>22.6 / 83.9</td>
  </tr>
  <tr>
    <td>tha</td>
    <td>12.8 / 71.1</td>
    <td>13.2 / 71.7</td>
    <td>12.5 / 78.6</td>
    <td>14.7 / 78.8</td>
  </tr>
  <tr>
    <td>vie</td>
    <td>17.5 / 74.1</td>
    <td>17.5 / 74.1</td>
    <td>18.2 / 79.7</td>
    <td>19.0 / 79.8</td>
  </tr>
  <tr>
    <td>yue</td>
    <td>15.2 / 70.9</td>
    <td>14.9 / 70.5</td>
    <td>15.5 / 78.9</td>
    <td>16.3 / 79.0</td>
  </tr>
  <tr>
    <td>zho</td>
    <td>16.8 / 77.5</td>
    <td>17.0 / 77.7</td>
    <td>19.2 / 82.5</td>
    <td>19.6 / 82.6</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>18.8 / 75.5</td>
    <td>19.0 / 75.8</td>
    <td>20.6 / 82.1</td>
    <td>21.4 / 82.2</td>
  </tr>
</table>

Table 15: BLEU/COMET Scores for 15 Languages Across 15×14 Translation Directions on FLEUR. Detailed results are summarized in Tables 18.

<table>
  <tr>
    <th>Case</th>
    <th>BLEU↑</th>
  </tr>
  <tr>
    <td>Audio<br>There may be more maria on the near side because the crust is thinner. It was easier for lava to rise up to the surface.<br>地が薄いため、近い方には海が多くなることがあります。溶岩が浮上しやすくなっていました。</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-3B (MT)<br>近には可能にマリアがあるかもしれませんbecause the crustははより薄いからマagmaが表面に上りやすかったlavaです。<br>LLM-SRT-3B (S2TT)<br>近にはもっとマリアがある可能性があります、地が薄いためです。溶岩が表面に上りやすかったからです。</td>
    <td>2.9<br>30.6</td>
  </tr>
</table>

Table 16: Error Analysis. We observe that the Qwen2.5-3B model exhibits language mixing (e.g., Japanese and Cantonese) in certain translation directions. Consequently, S2TT outperforms MT in these scenarios.

<table>
  <tr>
    <th>Case</th>
    <th>BLEU↑</th>
  </tr>
  <tr>
    <td>Audio # 1<br>它让玩家可以通过在空中移动设备来控制电子游戏中的运动和操作。<br>This will allow players to control actions and movements in video games by moving the device through the air.</td>
    <td></td>
  </tr>
  <tr>
    <td>SeamlessM4T-V2<br>It allows players to control the movement and operation of electronic games through mobile devices in the air.<br>它让玩家可以通过在空中移动设备来控制电子游戏中的动作和操作。<i>|zh|</i>|<i>|eng|</i></td>
    <td>14.0</td>
  </tr>
  <tr>
    <td>LLM-SRT-3B<br>It allows players to control the actions and operations of an electronic game by moving the device in the air.<br>它让玩家可以通过在空中移动设备来控制电子游戏中的动作和操作。<i>|zh|</i>|<i>|eng|</i></td>
    <td>23.2</td>
  </tr>
  <tr>
    <td>LLM-SRT-32B<br>It allows players to control actions and operations in electronic games by moving the device in the air.<br>它让玩家可以通过在空中移动设备来控制电子游戏中的动作和操作。<i>|zh|</i>|<i>|eng|</i></td>
    <td>43.0</td>
  </tr>
  <tr>
    <td>Audio # 2<br>In fact, it is not easy to find at all even if one knew it existed. Once inside the cave, it is a total isolation.<br>事实上，即便知道它的存在，也不容易找到。一旦进入洞穴，就完全与世隔绝了。</td>
    <td></td>
  </tr>
  <tr>
    <td>SeamlessM4T-V2<br>事实上,即使有人知道它存在,也很难找到它。一旦进入洞穴,<br>LLM-SRT-3B<br>In fact, it is not easy to find at all, even if one knew it existed. Once inside the cave, it is a total isolation.<i>|zh|</i>|<i>|eng|</i><br>事实上,即便知道它存在,要找到它也是很困难的。一旦进入洞穴,就是完全的隔离。</td>
    <td>22.7<br>49.3</td>
  </tr>
  <tr>
    <td>LLM-SRT-32B<br>In fact, it is not easy to find at all, even if one knew it existed. Once inside the cave, it is a total isolation.<i>|zh|</i>|<i>|eng|</i><br>事实上,即便知道它的存在,也很难找到。一旦进入洞穴,就完全与世隔绝了。</td>
    <td>86.8</td>
  </tr>
</table>

Table 17: Case Study. We compare the BLEU scores of our models with SeamlessM4T-V2.
<table>
  <tr>
    <th>src_lgt</th>
    <th colspan="3">SeamlessM4T-V2</th>
    <th colspan="3">LLM-SRT-3B-V2</th>
  </tr>
  <tr>
    <th></th>
    <th>BLEU Avg.</th>
    <th>+Lora Avg.</th>
    <th>Avg.</th>
    <th>BLEU Avg.</th>
    <th>+Lora Avg.</th>
    <th>Avg.</th>
  </tr>
  <tr><td>deu_ens</td><td>37.1</td><td>36.9</td><td>34.3</td><td>25.3</td><td>25.1</td><td>24.9</td></tr>
  <tr><td>dts_deu</td><td>17.9</td><td>21.2</td><td>23.8</td><td>25.0</td><td>25.0</td><td>25.1</td></tr>
  <tr><td>fra_jpt</td><td>19.1</td><td>19.4</td><td>17.2</td><td>18.8</td><td>17.9</td><td>18.5</td></tr>
  <tr><td>fra_ta</td><td>12.5</td><td>15.5</td><td>14.3</td><td>14.8</td><td>15.1</td><td>15.2</td></tr>
  <tr><td>fra_zho</td><td>26.6</td><td>30.2</td><td>31.9</td><td>25.6</td><td>27.5</td><td>27.0</td></tr>
  <tr><td>kin_kor</td><td>29.1</td><td>30.9</td><td>23.1</td><td>24.6</td><td>24.9</td><td>25.2</td></tr>
  <tr><td>orjDeu</td><td>13.2</td><td>21.8</td><td>13.6</td><td>12.0</td><td>24.4</td><td>15.0</td></tr>
  <tr><td>tng_ara</td><td>29.7</td><td>21.4</td><td>35.1</td><td>26.0</td><td>25.3</td><td>28.0</td></tr>
  <tr><td>tng_ben</td><td>14.7</td><td>14.5</td><td>15.9</td><td>16.4</td><td>18.6</td><td>17.0</td></tr>
  <tr><td>tng_faa</td><td>14.9</td><td>41.9</td><td>43.6</td><td>45.2</td><td>41.9</td><td>43.6</td></tr>
  <tr><td>tng_nda</td><td>32.4</td><td>21.0</td><td>27.2</td><td>12.8</td><td>24.8</td><td>25.5</td></tr>
  <tr><td>tng_zhu</td><td>21.4</td><td>30.8</td><td>28.5</td><td>12.4</td><td>30.8</td><td>28.5</td></tr>
  <tr><td>eng_esp</td><td>25.4</td><td>34.7</td><td>33.7</td><td>18.9</td><td>30.4</td><td>30.7</td></tr>
  <tr><td>eng_fra</td><td>34.6</td><td>41.8</td><td>38.7</td><td>34.5</td><td>43.5</td><td>39.9</td></tr>
  <tr><td>eng_hin</td><td>16.1</td><td>22.2</td><td>18.8</td><td>27.3</td><td>30.0</td><td>28.6</td></tr>
  <tr><td>eng_jaf</td><td>24.5</td><td>24.3</td><td>24.4</td><td>21.3</td><td>21.2</td><td>21.7</td></tr>
  <tr><td>eng_jpn</td><td>34.0</td><td>34.2</td><td>34.7</td><td>33.1</td><td>39.1</td><td>36.6</td></tr>
  <tr><td>eng_per</td><td>16.9</td><td>23.2</td><td>24.9</td><td>26.0</td><td>26.4</td><td>26.3</td></tr>
  <tr><td>eng_zhi</td><td>24.4</td><td>33.3</td><td>23.1</td><td>32.8</td><td>27.8</td><td>31.9</td></tr>
  <tr><td>eng_zhj</td><td>25.8</td><td>19.8</td><td>24.4</td><td>26.9</td><td>24.4</td><td>26.9</td></tr>
  <tr><td>spa_dac</td><td>23.7</td><td>24.0</td><td>27.7</td><td>20.2</td><td>31.2</td><td>27.2</td></tr>
  <tr><td>spa_deu</td><td>31.2</td><td>30.6</td><td>28.7</td><td>33.3</td><td>36.0</td><td>36.4</td></tr>
  <tr><td>spa_fra</td><td>23.7</td><td>25.7</td><td>27.5</td><td>26.0</td><td>25.3</td><td>25.6</td></tr>
  <tr><td>spa_jaf</td><td>36.1</td><td>35.9</td><td>37.6</td><td>29.7</td><td>29.7</td><td>32.9</td></tr>
  <tr><td>spa_yue</td><td>16.6</td><td>17.2</td><td>29.9</td><td>30.3</td><td>26.8</td><td>27.9</td></tr>
  <tr><td>spa_zho</td><td>20.9</td><td>24.3</td><td>27.1</td><td>15.4</td><td>16.4</td><td>18.2</td></tr>
  <tr><td>fra_deu</td><td>10.6</td><td>12.0</td><td>16.0</td><td>17.0</td><td>18.5</td><td>17.6</td></tr>
  <tr><td>fra_jpt</td><td>33.5</td><td>33.7</td><td>32.3</td><td>24.7</td><td>21.1</td><td>27.7</td></tr>
  <tr><td>fra_tng</td><td>19.0</td><td>21.1</td><td>18.9</td><td>19.8</td><td>21.0</td><td>20.9</td></tr>
  <tr><td>fra_zho</td><td>19.4</td><td>15.5</td><td>16.5</td><td>17.1</td><td>14.1</td><td>15.7</td></tr>
  <tr><td>jat_jvt</td><td>27.9</td><td>26.6</td><td>30.6</td><td>31.1</td><td>26.6</td><td>30.4</td></tr>
  <tr><td>jat_tglr</td><td>39.6</td><td>29.4</td><td>41.9</td><td>24.9</td><td>42.8</td><td>22.7</td></tr>
  <tr><td>kor_orj</td><td>29.1</td><td>28.9</td><td>23.1</td><td>25.4</td><td>23.8</td><td>24.1</td></tr>
  <tr><td>ukr_zho</td><td>49.0</td><td>20.0</td><td>21.3</td><td>21.6</td><td>24.0</td><td>22.6</td></tr>
  <tr><td>ukr_zjv</td><td>21.1</td><td>14.4</td><td>20.1</td><td>18.4</td><td>18.7</td><td>19.0</td></tr>
  <tr><td>yue_jpt</td><td>17.2</td><td>16.5</td><td>28.4</td><td>24.5</td><td>28.6</td><td>26.2</td></tr>
  <tr><td>swe_zhj</td><td>17.9</td><td>45.6</td><td>48.1</td><td>45.7</td><td>48.1</td><td>45.1</td></tr>
  <tr><td>viet_zjo</td><td>23.7</td><td>20.5</td><td>25.3</td><td>23.2</td><td>23.8</td><td>23.9</td></tr>
  <tr><td>viet_ztj</td><td>34.2</td><td>28.7</td><td>37.5</td><td>26.5</td><td>28.3</td><td>27.1</td></tr>
  <tr><td>zhu_ens</td><td>13.0</td><td>17.5</td><td>14.7</td><td>11.4</td><td>13.6</td><td>13.6</td></tr>
  <tr><td>zhu_jaf</td><td>18.3</td><td>15.4</td><td>20.8</td><td>19.0</td><td>22.0</td><td>20.5</td></tr>
  <tr><td>zhu_zjo</td><td>10.3</td><td>26.9</td><td>19.5</td><td>12.2</td><td>24.2</td><td>12.2</td></tr>
  <tr><td>yue_zjo</td><td>9.7</td><td>19.8</td><td>16.4</td><td>15.3</td><td>24.3</td><td>16.8</td></tr>
  <tr><td>ind_ta</td><td>26.5</td><td>26.0</td><td>22.1</td><td>28.5</td><td>27.2</td><td>26.1</td></tr>
  <tr><td>ind_jaf</td><td>24.7</td><td>26.5</td><td>30.0</td><td>19.7</td><td>22.0</td><td>23.6</td></tr>
  <tr><td>ind_tng</td><td>23.7</td><td>28.7</td><td>28.6</td><td>23.3</td><td>28.2</td><td>26.6</td></tr>
  <tr><td>pah_tng</td><td>25.6</td><td>19.1</td><td>7.4</td><td>10.2</td><td>30.6</td><td>22.8</td></tr>
  <tr><td>pah_zhu</td><td>19.5</td><td>15.9</td><td>14.7</td><td>12.4</td><td>21.5</td><td>18.1</td></tr>
  <tr><td>hin_tng</td><td>14.2</td><td>14.5</td><td>10.7</td><td>8.3</td><td>15.0</td><td>12.0</td></tr>
  <tr><td>hin_zjv</td><td>46.7</td><td>14.2</td><td>17.7</td><td>17.9</td><td>27.3</td><td>19.8</td></tr>
  <tr><td>tng_fip</td><td>41.5</td><td>14.7</td><td>14.5</td><td>14.7</td><td>7.8</td><td>15.3</td></tr>
  <tr><td>per_tng</td><td>8.1</td><td>6.5</td><td>9.4</td><td>7.6</td><td>8.6</td><td>8.0</td></tr>
  <tr><td>per_tng</td><td>16.4</td><td>15.4</td><td>11.6</td><td>16.4</td><td>11.9</td><td>15.4</td></tr>
  <tr><td>",ya5_tng</td><td>30.4</td><td>38.4</td><td>40.2</td><td>37.2</td><td>39.0</td><td>40.9</td></tr>
  <tr><td>jat_zjo</td><td>15.8</td><td>15.6</td><td>15.3</td><td>14.7</td><td>17.2</td><td>16.4</td></tr>
  <tr><td>jat_zhu</td><td>12.8</td><td>5.6</td><td>19.1</td><td>19.2</td><td>19.1</td><td>19.7</td></tr>
  <tr><td>yue_zhu</td><td>16.5</td><td>15.6</td><td>29.3</td><td>26.7</td><td>30.1</td><td>28.6</td></tr>
  <tr><td>yue_zny</td><td>16.8</td><td>6.3</td><td>20.1</td><td>14.0</td><td>18.3</td><td>16.5</td></tr>
  <tr><td>yng_jpt</td><td>9.4</td><td>15.2</td><td>17.3</td><td>15.0</td><td>18.5</td><td>16.8</td></tr>
  <tr><td>yng_zhu</td><td>10.4</td><td>25.5</td><td>21.3</td><td>21.9</td><td>25.4</td><td>23.9</td></tr>
  <tr><td>jae_zjo</td><td>37.5</td><td>37.7</td><td>33.3</td><td>17.7</td><td>28.8</td><td>21.9</td></tr>
  <tr><td>jae_zhu</td><td>7.5</td><td>29.6</td><td>19.3</td><td>17.7</td><td>28.5</td><td>20.8</td></tr>
  <tr><td>ekd_zjo</td><td>16.0</td><td>28.4</td><td>23.9</td><td>19.8</td><td>28.7</td><td>23.8</td></tr>
  <tr><td>ekd_zhu</td><td>18.3</td><td>20.5</td><td>25.2</td><td>19.3</td><td>25.7</td><td>22.3</td></tr>
  <tr><td>eko_zhu</td><td>15.9</td><td>8.5</td><td>17.5</td><td>11.9</td><td>22.6</td><td>13.4</td></tr>
  <tr><td>upo_zhu</td><td>10.9</td><td>8.9</td><td>11.8</td><td>8.1</td><td>11.6</td><td>9.5</td></tr>
  <tr><td>upo_zjo</td><td>14.8</td><td>14.8</td><td>18.7</td><td>14.3</td><td>24.4</td><td>16.5</td></tr>
  <tr><td>nur_jpr</td><td>28.0</td><td>18.1</td><td>13.2</td><td>14.5</td><td>12.5</td><td>14.5</td></tr>
  <tr><td>nur_jaf</td><td>11.6</td><td>13.0</td><td>8.9</td><td>10.3</td><td>13.2</td><td>11.1</td></tr>
  <tr><td>nur_jlf</td><td>8.7</td><td>12.2</td><td>9.8</td><td>11.1</td><td>12.6</td><td>11.0</td></tr>
  <tr><td>nur_jpt</td><td>19.3</td><td>15.2</td><td>19.4</td><td>16.2</td><td>20.5</td><td>19.8</td></tr>
  <tr><td>nur_kor</td><td>13.8</td><td>9.4</td><td>13.2</td><td>12.9</td><td>11.2</td><td>12.2</td></tr>
  <tr><td>nur_tng</td><td>16.4</td><td>10.9</td><td>8.7</td><td>8.0</td><td>8.9</td><td>8.0</td></tr>
  <tr><td>nur_tng</td><td>18.0</td><td>10.0</td><td>10.2</td><td>10.5</td><td>8.9</td><td>9.3</td></tr>
  <tr><td>oko_zjo</td><td>23.4</td><td>18.7</td><td>16.5</td><td>23.2</td><td>22.3</td><td>23.3</td></tr>
  <tr><td>oko_zhu</td><td>21.0</td><td>16.0</td><td>17.1</td><td>17.0</td><td>21.0</td><td>17.3</td></tr>
  <tr><td>oko_zno</td><td>13.2</td><td>17.5</td><td>18.2</td><td>17.0</td><td>23.4</td><td>25.0</td></tr>
  <tr><td>uko_zjo</td><td>23.4</td><td>23.3</td><td>23.7</td><td>23.1</td><td>20.3</td><td>22.1</td></tr>
  <tr><td>uko_zhu</td><td>11.3</td><td>10.0</td><td>9.6</td><td>10.0</td><td>11.0</td><td>10.9</td></tr>
  <tr><td>uko_zno</td><td>23.4</td><td>30.0</td><td>27.6</td><td>27.1</td><td>38.0</td><td>32.5</td></tr>
</table>

<table>
  <tr>
    <th>src_lgt</th>
    <th>BLEU</th>
    <th colspan="3">SeamlessM4T-V2</th>
    <th>BLEU</th>
    <th colspan="3">LLM-SRT-3B-V2</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Av según</th>
    <th>+Lora Av. según</th>
    <th>Avg. según</th>
    <th></th>
    <th>Av según</th>
    <th>+Lora Av. según</th>
    <th>Avg. según</th>
  </tr>
  <tr><td>deu_enu</td><td>6.8</td><td>7.5</td><td>18.2</td><td>18.5</td></tr>
  <tr><td>deu_zhe</td><td>38.6</td><td>37.7</td><td>40.3</td><td>38.8</td></tr>
  <tr><td>inf_frontend</td><td>18.3</td><td>28.8</td><td>29.6</td><td>29.6</td></tr>
  <tr><td>ind_rah</td><td>13.9</td><td>15.2</td><td>20.5</td><td>21.6</td></tr>
  <tr><td>ind_tuh</td><td>16.3</td><td>8.4</td><td>14.7</td><td>14.9</td></tr>
  <tr><td>ind_zfn</td><td>10.4</td><td>14.7</td><td>17.5</td><td>17.5</td></tr>
  <tr><td>ind_zju</td><td>18.4</td><td>23.4</td><td>23.8</td><td>23.8</td></tr>
  <tr><td>lap_zha</td><td>17.8</td><td>18.8</td><td>25.0</td><td>24.8</td></tr>
  <tr><td>pfi_cha</td><td>14.2</td><td>34.9</td><td>27.8</td><td>23.5</td></tr>
  <tr><td<textarea>phn_sum</td><td>12.1</td><td>20.9</td><td>26.6</td><td>27.0</td></tr>
  <tr><td>phe_zne</td><td>10.0</td><td>9.3</td><td>17.1</td><td>18.0</td></tr>
  <tr><td>phe_znh</td><td>11.3</td><td>45.2</td><td>46.6</td><td>47.1</td></tr>
  <tr><td>plo_pal</td><td>19.0</td><td>22.6</td><td>23.4</td><td>22.4</td></tr>
  <tr><td>poh_prn</td><td>19.1</td><td>21.8</td><td>25.9</td><td>25.2</td></tr>
  <tr><td>AddNet-Fine</td><td>17.6</td><td>18.5</td><td>18.9</td><td>18.9</td></tr>
  <tr><td>bld_LAVE6</td><td>18.2</td><td>16.2</td><td>17.7</td><td>17.8</td></tr>
  <tr><td>nf_PAN_TRAFIC_V4</td><td>18.4</td><td>19.2</td><td>25.9</td><td>26.2</td></tr>
  <tr><td>avg cả</td><td>18.8</td><td>19.0</td><td>20.6</td><td>21.4</td></tr>
</table>

Table 18: BLEU Scores for 15 Languages Across 15×14 Translation Directions on FLEURS.

Reviewer paper summaries: This paper proposes a 3-stage finetuning for increasing adapting LLMs to the Speech-to-Text Translation (S2TT). The authors connect an audio frontend to an LLM in order to provide speech input capabilities. Specifically, the use a (frozen) Whisper encoder and Q-former + MLP audio adapter. The output of the adapter is concatenated to the text embedding which can contain information like the instructions. For the LLM they use Qwen-2.5 (3B / 7B / 32B).

The finetuning stage is as follows:
(1) ASR: The model is adapted in Transcription
(2) SMT: The model is adapted in S2TT, while the transcription is also provided as input
(3) SRT: The model is trained to generated both the transcription and the translation

The authors conduct experiments in low-resource (FLEURS, ~10h) and high-resource (CoVoST2, ~500h) S2TT. For ASR training the used CommonVoice. They use 15 languages in their experiments.

In their results they show that the proposed 3-stage method easily outperforms a simpler 2-stage finetuning approach (using the same LLM), where they finetune on ASR and then finetune on ST. When scaling the base LLM size, the approach can also outperform strong cascades and Seamless (with also LoRA finetuning), and Qwen2-Audio. One exception is the En->X case, where previous models (Seamless) are better since they have used a significant amount of audio in English, compared to the 10h used in the FLEURS experiments in this work. The authors show that using CoVoST2 data En->(De/Ja/Zh)  their 7B model can outperform SeamlessM4T.
Summary of strengths: - The paper is very well written and easy to follow.
- The proposed finetuning method is well-motivated and is proven to be superior compared to a simpler method of (ASR followed by ST).
- The proposed method seems to be very data-efficient in terms of S2TT data, which is very crucial since these data are very limited, especially for lower-resource languages.
- The proposed method obtains very good results outperforming other strong models (cascades, SeamlessM4T, Qwen2-Audio)
Summary of weaknesses: I have only some minor concerns about this work.

- Translation quality is evaluated only with BLEU, which is known to have several issues. The authors most of the times evaluate into many directions and then average which makes the scores more reliable. Still, it would have been great if evaluations with a different metric (like COMET which I think natively supports all the 15 languages studied here [maybe not yue?]) were included in the Appendix for some important comparisons, like those of Table 3 or 4.

- Although most of the focus of this work is on low-resource data scenarios (which is great), aside maybe from yue, all other languages are actually considered high-resource. This means that the base LLM and audio front-end (Whisper) are likely to be good at them, which makes it easier for the method to be successful. I think it would be very beneficial for this work and the community to have some additional experiments on languages from FLEURS that are really low-resource. For example Occitan is a good low-resource candidate, while Mongolian can also be used as a very hard test case, since it is also from a unique language family.
Comments, suggestions and typos: ### Citations

- L031-L032: Needs citation: However, this cascade system often suffers from error propagation. Consider citing [1]
- L046-L048: Needs citation: Large Language Models (LLMs) have demonstrated strong many-to-many multilingual MT capabilities.
- L161: I think it's not necessary to provide [Rosenblatt, 1958] as a citation for an MLP. Especially considering that not even the Transformer [Vaswani et. al, 2017] is cited in this work.
- L250: Consider also citing [2] since you are using the v2 version of SeamlessM4T which was introduced there

### Typos

- Figure 1: Englash -> English
- Table 3: the results of the bottom two lines should also be mostly bolded

### Questions / Suggestions

- You established in the experiments that the choice of base LLM is very important for the performance of the proposed method (Figure 3). I would very curious to see results with an LLM that is already tailored for MT (for example Tower[3]), or if you included a step-0 in your curriculum where you finetune the LLM with LoRA on MT. I would expect that you can obtain better results do any of those.

- For completeness it would have been nice to show some cascade results in Table 3 with traditional MT models. For example either Whisper + NLLB3.3 or Whisper + SeamlessM4T-v2 (in MT mode).

- Consider renaming the "Baseline-3B" to "LLM-ST-3B"

- Table 4: Why is the model now named v2? I did not find any reason for this change in the text. Is it not the same as presented in Table 3?

- Table 5: Is there a reason why LLM-SRT-32B is not trained on CoVoST2? Is this due to computing resources?

- Studied Languages: Is there a motivation behind choosing these specific 15 languages?


[1] Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

[2] Seamless: Multilingual Expressive and Streaming Speech Translation

[3] Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
Overall assessment scores: 4
Metareview score: 4
Metareview: This work introduces a 3-stage LLM fine-tuning framework for low-resource Speech-to-Text Translation (S2TT), leveraging LLMs' pre-trained knowledge to reduce dependence on large S2TT-specific datasets. The method demonstrates strong empirical performance across 15 languages (expanded to 27 in the rebuttal stage, including two genuinely low-resource languages). While the technical approach is straightforward, its practical success in low-resource scenarios offers significant value to the community. Key concerns about insufficient implementation details and task analysis raised by reviewers must be addressed to solidify the paper's impact.
Summary of reasons to publish: 1. Achieves effective S2TT performance for low-resource languages, addressing a critical bottleneck in multilingual speech translation.
2. Demonstrates extensibility to 27 languages, including extreme low-resource cases, with a systematic 3-stage approach.
3. Proves LLMs can be adapted for S2TT with limited task-specific data, reducing annotation costs.
4. The method’s simplicity can lower barriers for community adoption.