LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models (2024)

Abstract: We introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems.We address the limitations of end-to-end speech translation (E2E ST) models by exploring model architecture design and optimization techniques tailored for LLMs. Our approach includes LLM-based speech translation architecture design, ASR-augmented training, multilingual data augmentation, and dual-LoRA optimization. Our approach demonstrates superior performance on the CoVoST-2 benchmark and showcases exceptional scaling capabilities powered by LLMs. We believe this effective method will serve as a strong baseline for speech translation and provide insights for future improvements of the LLM-based speech translation framework1.

heading: 1 Introduction
text: The speech-to-text translation (ST) task, which transcribes spoken language into written text in a different language, is pivotal for bridging communication barriers. This capability has a wide array of applications, including facilitating global communication, enabling automatic subtitling, and aiding in language learning.
Conventional ST systems are typically composed of two distinct components: an automatic speech recognition (ASR) module that transcribes spoken speech into written text in the source language, and a machine translation (MT) module that subsequently translates this text into the target language. These modules can be trained using paired ASR and text-to-text translation data, significantly enhancing the overall performance of ST systems. Despite their modular design, cascade systems are prone to error accumulation, where inaccuracies from the ASR stage are compounded in the MT
1We release the data, code and models in https://github.com/openaudiolab/LLaST
phase, often leading to sub-optimal translations. Recently, the focus has shifted towards the development of end-to-end speech translation (E2E ST) models that bypass the need for separate automatic speech recognition (ASR) and machine translation (MT) modules by directly converting spoken input into text in the target language. Nonetheless, these approaches often necessitate extensive training datasets and are contingent upon sophisticated model architectures to achieve strong performance.
Speech translation is intrinsically linked to natural language processing (NLP), as it involves the conversion of spoken language into written text in a target language, necessitating a deep understanding of both the source and target languages’ linguistic structures and semantics. The unprecedented capabilities that large language models (LLMs) have demonstrated across a variety of NLP tasks (Touvron et al., 2023a,b; Achiam et al., 2023) have opened up new possibilities to construct potent speech translation systems by leveraging these LLMs as a foundation. Recent research has seen some preliminary attempts exploring this direction (Chu et al., 2023; Wu et al., 2023; Huang et al., 2023). Despite these advancements, the question remains on how to most effectively harness the vast potential of LLMs to develop a high-performance ST system in an efficient manner, without compromising on quality or scalability.
In this study, we focus on the exploration of best practices for constructing an effective speech translation system powered by Large Language Models (LLMs), which we term LLaST. The paper delves into the core aspects of the development process, specifically the model architecture design and optimization techniques. Our exploration begins with the creation of a minimalist model architecture, examining the selection of key modules such as the speech encoder and LLMs. Subsequently, we investigate training strategies, including ASR-augmented training and dual-LoRA optimization. Moreover, to
6976
deepen our understanding of scaling laws in LLMbased ST, we also scrutinize the impact of model size variations. Through these concerted efforts, we aim to uncover insights that can significantly enhance the performance and training efficiency of LLaST.
Our contributions are listed as follows. • We explore the LLMs-based speech translation method, including model architecture design, training strategies, and data recipe. • Extensive evaluations demonstrate the superiority of our approach, surpassing the previous SOTA methods (Barrault et al., 2023) and achieving 45.1 BLEU on the fr→en test set of CoVoST-2. • We are dedicated to making all data recipes, training methodologies, and model weights associated with LLaST openly accessible to the community. By doing so, we foster transparency, collaboration, and advancement in the field of LLM-based speech translation technology.

heading: 2 Related Work

heading: 2.1 Cascaded Speech Translation
text: Historically, the construction of speech translation systems has been approached in a cascading fashion, incorporating both an ASR and an MT subsystem (Stentiford and Steer, 1988; Ney, 1999; Nakamura et al., 2006). The procedure involves initially converting the input speech into text in the source language, which is subsequently translated into the target language. The primary objective of this line of research has been to mitigate error accumulation, including the use of multiple recognition outputs and the development of robust MT models (Casacuberta et al., 2008; Kumar et al., 2014; Sperber et al., 2017). Sperber et al. (2019b) introduces a self-attention mechanism to handle the lattice inputs, and Zhang et al. (2019) proposes a lattice transformer, equipped with a controllable lattice attention mechanism, to derive latent representations. Lam et al. (2021) establishes a feedback cycle in which the downstream performance of the MT system serves as a signal to enhance the ASR system via self-training.

heading: 2.2 End-to-End Speech Translation
text: The development of end-to-end speech translation (E2E ST) models, which bypass the requirement for intermediary stages such as ASR outputs and lattices, has been a significant stride in mitigating error propagation. Research indicates that these
E2E ST models demonstrate encouraging results and offer performance on par with cascaded models (Sperber et al., 2019a; Ansari et al., 2020; Bentivogli et al., 2021; Ye et al., 2021). Moreover, these models present additional benefits such as lower latency and the potential to be applied to languages that lack a written form (Bérard et al., 2016).
Data scarcity and the modeling burden are recognized as two significant obstacles impeding the performance of E2E ST (Xu et al., 2023). Firstly, the intrinsic complexity of speech translation, which integrates transcription and translation, presents a challenge in optimizing a single model to accomplish these cross-modal and cross-lingual tasks in one step. Secondly, ASR datasets are typically less extensive than MT datasets, and the extension to ST datasets further exacerbates this size discrepancy. To address this issue of data scarcity, researchers have employed strategies such as data augmentation (Tsiamas et al., 2023; Lam et al., 2022), pretraining (Wang et al., 2020c; Ao et al., 2022), and knowledge distillation (Liu et al., 2019), which leverage external datasets.
To alleviate the modeling burden, a variety of multi-task learning strategies have been investigated (Zhang and Yang, 2018). Originating from the multi-task encoder-decoder architecture (Weiss et al., 2017), some researchers have chosen to split the decoder into two separate components (Liu et al., 2020a; Anastasopoulos and Chiang, 2018): one dedicated to transcription and the other to translation. Parallel research efforts (Liu et al., 2020b; Cheng et al., 2023) have similarly decoupled the encoder, with further work showing that a shared encoder can be independently partitioned (Tang et al., 2021; Ye et al., 2022) to make better use of ASR data. In addition, non-autoregressive (NAR) modeling has been explored as a means to decrease latency (Inaguma et al., 2021; Chuang et al., 2021).
Significantly, recent advancements have also delved into multi-tasking within the context of large-scale training, leading to impressive results on ST benchmarks. For instance, Whisper (Radford et al., 2023) and SeamlessM4T (Barrault et al., 2023) have incorporated 680k and 470k hours of multilingual speech data in their training.

heading: 2.3 LLM-based Speech Translation
text: Inspired by the robust linguistic capabilities of LLMs (Brown et al., 2020; Touvron et al., 2023b), recent initiatives have sought to harness the
power of LLMs to address various speech tasks, aided mainly by instruction tuning. The prevailing method involves integrating an LLM (backend) with a speech encoder (frontend). Models like LauraGPT (Chen et al., 2023) and Qwenaudio (Chu et al., 2023) support a range of multimodal speech tasks, demonstrating performance comparable to task-specific E2E ST models. VioLA (Wang et al., 2023) employs a neural codec model (Défossez et al., 2022) to discretize the speech input while tuning the LLM. Similarly, AudioPaLM (Rubenstein et al., 2023) discretizes the speech input and achieves commendable results on CoVoST-2 (Wang et al., 2020b).
Salmonn (Tang et al., 2023) employs two encoders as the frontend and uses LoRA (Hu et al., 2021) for efficient fine-tuning. However, the extent of its performance improvement on ST remains largely unexplored. Some recent studies (Wu et al., 2023; Zhang et al., 2023a) specifically target the ST task and delve into efficient tuning strategies, but their performance enhancements have been somewhat limited. In an industrial study focusing on translation between Chinese and English, Huang et al. (2023) additionally incorporates the Chainof-Thought (CoT) technique (Wei et al., 2022), enabling a step-by-step approach using LLMs.

heading: 3 Method
text: This section presents our method in detail. We begin by introducing the problem setting of the speech-to-text translation task in Sec. 3.1. Then, we explain the structure of the proposed model in Sec. 3.2, followed by the description of the training
and inference processes in Sec. 3.3.

heading: 3.1 Problem Setting
text: We now present the problem setting of speech translation. Given a speech translation dataset D = {(S,Ysrc,Ytgt)}, the source language speech S’s acoustic features (e.g., mel-spectrogram) are denoted as Xs, and we have:
Xs = Fa(S), Xs = {x1, x2, . . . , xT }
where Fa is the acoustic feature extraction operation, and T is the timesteps of the input features. Ysrc and Ytgt are the transcripts of S in the source and target languages, respectively. The goal of speech translation is to generate the prediction text of target language Ŷtgt from the source speech S.
We can formulate the whole process as:
Ŷtgt = F(S)
and F represents the entire ST system. Performance of ST system is typically assessed by comparing the predicted output Ŷtgt with the ground truth Ytgt using metrics like BLEU (Papineni et al., 2002).

heading: 3.2 Model Architecture
text: Our objective is to develop the LLaST model with a simple architecture, as depicted in Figure 1. The design of LLaST comprises three key components: a speech encoder to process the input speech, an adaptor that projects these speech features into the compatible feature space for Large Language Models (LLMs), and finally, a decoder-only LLM for multi-modality decoding.
Speech Encoder Acoustic features Xs encapsulate a wealth of information, including speaker traits, emotions, prosody, background noise, and more. The role of the speech encoder is to disentangle these variabilities and generate robust linguistic representations, denoted as Zs. We define this process mathematically:
Zs = Fse(Xs)
where Fse represents the speech encoder function. Our work investigates various options for the speech encoder, with a focus on mHubert (Hsu et al., 2021; Lee et al., 2021) and Whisper (Radford et al., 2023). For an in-depth analysis and discussion on the speech encoder selection, please refer to Sec. 5.1.
Adaptor The adaptor acts as a bridge between the speech encoder and the Large Language Model (LLM), consisting of a lightweight set of trainable parameters. Fine-tuning these parameters aligns speech features more effectively with the LLM’s representation space. Its function is to project the extracted linguistic representations, Zs, into the embedding realm of the LLM, thus yielding Hs:
Hs = Fada(Zs)
This transformation process facilitates a smooth integration of speech data into the LLM’s text-based context. We adopt a 3-layer multilayer perceptrons(MLPs) for adaptor.
Large Language Model Equipped with the projected speech feature Hs, our objective is to utilize the Large Language Model (LLM) for generating the translated text of the original speech. To facilitate this, we construct a speech-text prompt input for the LLM. The text component of this prompt, denoted as Xq, conveys the specific translation
task instruction, such as "Translate the French sentence into English". Post-tokenization and embedding, Xq is transformed into the LLM’s input representation, Hq. Subsequently, the LLM generates translation predictions based on the concatenated speech-text features (for simplicity, we omit bos and eos tokens in the equation below):
Ŷtgt = Fllm([Hs,Hq])
This process allows the model to fuse speech and textual information effectively to produce translations.
In summary, the entire process can be expressed as:
Ŷtgt = F(S) = Fllm([Fada(Fse(Fa(S))),Hq])

heading: 3.3 Training and Inference
text: This section delves into the optimization techniques employed in LLaST and elucidates its inference methodology.
Optimization with Dual-LoRA Fintuning To enhance training efficiency, we employ the LoRA (Hu et al., 2021) tuning method for model optimization. This technique significantly reduces trainable parameters by introducing trainable rank decomposition matrices to each Transformer layer, while keeping the pre-trained weights frozen.
In LLaST, we introduce the dual-LoRA finetuning, applying LoRA separately to both the speech encoder (S-LoRA) and the Large Language Model (L-LoRA). This approach ensures effective adaptation to speech translation tasks with minimal parameter updates. Specifically, we perform instruction-tuning on prediction tokens using the original auto-regressive training objective of LLM. For a target translation result Ytgt of length N , its
probability is calculated as:
P (Ytgt|Xs,Xq) = N∏
i=0
Pθ(yi|Xs,Xq,Ytgt,<i)
This strategy allows us to efficiently tune LLaST without extensive retraining, maintaining both computational efficiency and task-specific effectiveness.
Training with ASR-augmentation To enhance the performance of LLaST, we adopt the strategy from prior work (Barrault et al., 2023; Radford et al., 2023) to incorporate Automatic Speech Recognition (ASR) tasks for data augmentation during training. Given the structural similarity between ASR and ST tasks—both involve converting speech to text, we can simply modify the ASR prompt to match ST objectives, such as "Transcribe the French sentence into English". The examples of prompts are listed in Fig. 2. This ASR-augmentation significantly boosts the effectiveness of LLaST across various language pairs, as detailed in Sec. 5.2.
Inference Methodology During inference, we construct prompts in the same format as depicted in Fig. 1. To generate translation text sequences Ŷtgt, we employ a beam search algorithm with a beam size of 5.

heading: 4 Experiments
text: In this section, we conduct a series of experiments to validate the effectiveness of our method. We start by detailing experimental configurations in Sec. 4.1, followed by an overview of quantitative results in Sec. 4.2.

heading: 4.1 Configurations
text: Datasets Our speech translation models are trained and evaluated on CoVoST-2 (Wang et al., 2020b), a large-scale multilingual dataset that supports translations between English and 15 other
languages, as well as from 21 languages into English. For monolingual experiments, we utilize six subsets with source languages translating to English, focusing on French-English for training and testing. In the multilingual setup, we employ Fr→En, Es→En, De→En, It→En, Zh→En, and Ja→En subsets and three English-to-X subsets: En→Zh, En→Ja, and En→De . Audio samples are downsampled from 48kHz to 16kHz in all experiments.
Model Architecture Tab. 1 presents the three LLaST model configurations. Each model utilizes a Whisper-large-v2 speech encoder, contributing approximately 1B parameters. The adaptor is a compact multilayer perceptron with three layers, ingesting 1280-dimensional inputs and adjusting its output dimensions to match those of the subsequent LLMs. Consequently, the overall parameter count is predominantly influenced by the LLM component. Hence, we denote our models as LLaST-2B, LLaST-8B, and LLaST-14B.
Hyperparameters All models are optimized with AdamW, setting β1 = 0.9 and β2 = 0.98. A warmup-then-linear decay learning rate schedule is adopted, peaking at 0.0002. Training spans one epoch for each model. By default, the rank of S-LoRA (Whisper LoRA) is set to 128, while L-LoRA (LLM LoRA) rank is 512 unless specified otherwise. The LLaST-8B and LLaST-14B models are trained using 32 NVIDIA A100 GPUs, each with a batch size of 32, while the smaller LLaST2B model is trained on a setup consisting of 8 A100 GPUs, maintaining the same batch size per GPU.

heading: 4.2 Main Results
text: Comparisons with Other Models Tab. 2 presents a comparison between our proposed LLaST models and previous methods, with SacreBLEU scores evaluated across six language pairs: Fr→En, Ja→En, De→En, Zh→En, Es→En, and It→En. Notably, LLaST-2B outperforms SeamlessM4T(medium) and demonstrates competi-
tive performance against SeamlessM4T(large-v2). LLaST-8B significantly excels by improving upon the Qwen-audio model of similar scale with an impressive 5.6 BLEU point gain on the Fr→En task. Furthermore, LLaST-14B achieves state-ofthe-art (SOTA) results, attaining a BLEU score of 45.1 on CoVoST-2’s Fr→En subset, surpassing SeamlessM4T(large-v2) by 3.0 BLEU points. These results convincingly demonstrate the superiority of LLaST and highlight the promising potential of exploring LLMs for speech translation tasks.

heading: 5 Ablation Analysis
text: In this section, we delve into a meticulous ablation study and analysis of LLaST. We begin by examining the impact of model architecture in Sec. 5.1, followed by an exploration of optimization strategies in Sec. 5.2. Finally, we investigate the relationship between model scale and performance in Sec. 5.3.

heading: 5.1 Model Architecture Design
text: Choice of Speech Encoder We experiment with various speech encoder architectures, including
mHuBERT (Hsu et al., 2021; Lee et al., 2021) and Whisper (Radford et al., 2023) model. For the mHuBERT, we adhere to the preprocessing approach from (Dong et al., 2023; Lee et al., 2021) to extract semantic units. For a fair comparison, we select the Whisper-base model, which is comparable in size to the mHuBERT model. Performances reported in Tab. 3 indicate that the Whisper model yields superior performance, demonstrating a 4.3 BLEU score improvement over mHuBERT. This improved performance can be attributed to the fact that Whisper has been trained on significantly more data, thus generating more representative linguistic features.
Choice of Large Language Models We examine the impact of different large language models within LLaST to discern how variations in language modeling performance affect its speech translation capabilities. We present X→en results in Figure 3. Notably, Qwen achieves a score of 47.3 on the en→zh test set, outperforming Llama2 (Touvron et al., 2023a) by 4.9 BLEU points. Similarly, InternLM2(Cai et al., 2024) surpasses Llama2 by 5.0 BLEU points. These findings suggest that Chinese-oriented LLMs notably enhance performance on Chinese-related ST tasks, exemplified by En→Zh and Zh→En. The LLaST model, when coupled with Llama2, demonstrates exceptional performance particularly in the Fr→En and De→En language pairs. This intriguing observation underscores the potential of LLM-based ST approaches, as they allow for effortless integration of diverse LLM strengths tailored to specific languages or tasks.

heading: 5.2 Optimization
text: Training with ASR Augmentation Automatic Speech Recognition (ASR) is a task akin to speech translation, as both involve converting speech into text. Prior research has leveraged ASR tasks as auxiliary objectives for ST training (Zhang and Yang, 2018; Ye et al., 2022; Zhang et al., 2023b), or used models pre-trained on ASR data (Wang et al., 2020a). In LLaST, we adopt this concept and incorporate ASR tasks to optimize LLaST performance. An example of the speech-text prompt structure can be found in Fig. 2, where ST and ASR samples are randomly mixed during training, with the focus remaining on the ST task at inference time. The results presented in Fig. 4 demonstrate the efficacy of ASR augmentation in optimizing LLaST. We observe across nearly all test sets that ASR augmentation improves ST performance, suggesting that leveraging ASR or multi-task training within LLM-based ST frameworks is a promising direction with significant potential for future work.
Multilingual Data Augmentation In our experiments, we explore both monolingual and multilingual settings. Specifically, for the monolingual setup, we employ the Fr→En language pair. In the multilingual scenario, we introduce additional language pairs while maintaining the Fr→En data identical to that in the monolingual experiment.
The results presented in Tab. 4 reveal that incorporating other language pairs indeed benefits the Fr→En translation task, with a 1.6 BLEU score improvement observed upon adding multilingual data augmentation. This finding aligns with similar phenomena reported in LLM research (Team, 2023; Zeng et al., 2022), where exposure to multilingual corpora has been shown to enhance the language modeling capabilities of these models.
Dual-LoRA Optimization We investigate the impact of employing dual-LoRA for both speech encoders and large language models. In the ablation experiments, we utilize Whisper-large-v2 and Llama2-7B. The results from scenarios without any LoRA, with LoRA applied only to Whisper, LoRA applied only to Llama2, and dual-LoRA are reported in Table 5. From these outcomes, it is evident that even with a lightweight adaptor, leveraging a strong speech encoder and LLM can yield commendable performance. We also discover that
applying single LoRA to either Whisper or Llama2 separately leads to substantial gains, improving scores from 40.5 to 41.3 and 43.6, respectively. More notably, when dual-LoRA is used to jointly optimize both speech encoder and large language model, an additional improvement is achieved, culminating in a 44.1 BLEU score on test set.

heading: 5.3 Impact of Model Scale
text: Different Size of Speech Encoder We maintain a constant language model, Llama2-7B, and vary the size of Whisper models acting as speech encoders to examine the effect of encoder size on performance. The range of encoder sizes spans from 40M to 800M parameters. As shown in Table 6, we observe that as the encoder size increases, the BLEU score of the model consistently improves; however, the rate of improvement diminishes with each incremental increase in size. The base encoder achieves a BLEU score of 37.0, while the large encoder attains a peak score of 44.1. This considerable leap underscores the importance of scaling up speech encoders for better speech-totext translation. However, future research should consider the trade-offs between model size, computational efficiency, and overall performance to strike the right balance for practical applications.
Different Size of LLMs We further investigate the impact of varying LLM sizes on speech translation performance. With the speech encoder consistently set as Whisper-large-v2, we assess three different scale LLMs: TinyLlama-1B, Llama2-7B, and Llama2-13B. The outcomes are presented in Tab. 2. Our findings reveal that there is a positive correlation between the size of the language model and the BLEU scores across all test sets. As the capacity of the LLM increases, so does the overall performance in terms of translation quality, indicat-
Speech Encoder Encoder Size BLEU
ing that larger models can capture more nuanced linguistic patterns and generate more accurate translations.
Comparison between the Encoder Scaling and Decoder Scaling Given the Tab. 6 and Tab. 2, we observe some interesting phenomena. Despite Whisper-small+Llama2-7B-Chat and LLaST-2B demonstrating nearly equivalent performance on the fr->en subset, the former operates with approximately 7B parameters, whereas LLaST-2B functions with only about 2B parameters. This suggests that, in terms of parameter efficiency, scaling the encoder is a more effective strategy. It also indicates that, in these experiments, the encoder may play a more significant role. Meanwhile, the performance of the LLM-based system has yet to converge with respect to scale. To draw more comprehensive conclusions, we may need to continue scaling up the Whipster model and experiment with LLMs larger than 13B. For instance, in the domain of vision-language models, LLaVA (Liu et al., 2023) and InternVL (Chen et al., 2024) demonstrate that achieving optimal performance with a larger vision encoder (6B) necessitates employing correspondingly larger LLMs, such as Yi-34B (AI et al., 2024).

heading: 6 Limitation
text: While our study has yielded significant findings, it is crucial to recognize the limitations that may impact the interpretation and broad applicability of our results. Although we delved into the architecture design and optimization strategies, our reliance on a relatively narrow data source and the use of short voice samples could potentially affect the generalizability of our outcomes. To address this, future research will expand to encompass a more diverse array of data. Moreover, due to the constraints of our current resources, we have not ventured into exploring larger language models or
a broader range of language pairs in this study. In speech translation, LLaST’s use of LLMs raises concerns in actual application as teh following: (a)Probabilistic inaccuracy, mistranslations may occur due to nuances or dialects, impairing accuracy and cultural relevance. (b)Data imbalances, insufficient representation in training data can lead to biased translations or reduced effectiveness for underrepresented groups. (c)Deployment challenges, large model sizes and complexity may cause latency, high energy usage, and device compatibility issues. (d)Harmful content generation, despite post-processing, risks persist; ongoing monitoring, filter refinement, and expert collaboration are needed.

heading: 7 Conclusion
text: We presents the development and analysis of LLaST, a novel speech translation model that harnesses LLM in this work. The study demonstrates that integrating well-tuned speech encoders like Whisper with different sizes of LLMs significantly improves speech-to-text translation performance. Through meticulous ablation studies, it is shown that applying dual LoRA optimization to both speech encoders and LLMs leads to substantial gains in BLEU scores. Additionally, experiments confirm that increasing the scale of either the speech encoder or the LLM positively impacts performance, though the rate of improvement decreases as size increases. Furthermore, incorporating ASR augmentation and multilingual training further enhances the model’s performance on specific language pairs. Overall, LLaST underscores the potential of large language models for advancing speech translation tasks and offers valuable insights into their effective integration.
Ethical Considerations
We use the public LLMs to build LLaST, the LLMs may produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Addtionally, we use ChatGPT and Grammarly to polish the writing.

heading: Future Work
text: To address this, future research will expand to encompass a more diverse array of data. Moreover, due to the constraints of our current resources, we have not ventured into exploring larger language models or
a broader range of language pairs in this study. In speech translation, LLaST’s use of LLMs raises concerns in actual application as teh following: (a)Probabilistic inaccuracy, mistranslations may occur due to nuances or dialects, impairing accuracy and cultural relevance. (b)Data imbalances, insufficient representation in training data can lead to biased translations or reduced effectiveness for underrepresented groups. (c)Deployment challenges, large model sizes and complexity may cause latency, high energy usage, and device compatibility issues. (d)Harmful content generation, despite post-processing, risks persist; ongoing monitoring, filter refinement, and expert collaboration are needed