GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators (2024)

Abstract: Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by its reduced representation errors and incorporated external knowledge. However, both translation tasks typically utilize beam search decoding and top-1 hypothesis selection for inference. These techniques struggle to fully exploit the rich information in the diverse N -best hypotheses, making them less optimal for translation tasks that require a single, high-quality output sequence. In this paper, we propose a new generative paradigm for translation tasks, namely “GenTranslate”, which builds upon LLMs to generate better results from the diverse translation versions in N -best list. Leveraging the rich linguistic knowledge and strong reasoning abilities of LLMs, our new paradigm can integrate the rich information in N -best candidates to generate a higher-quality translation result. Furthermore, to support LLM finetuning, we build and release a HypoTranslate dataset that contains over 592K hypotheses-translation pairs in 11 languages. Experiments on various speech and machine translation benchmarks (e.g., FLEURS, CoVoST-2, WMT) demonstrate that our GenTranslate significantly outperforms the state-ofthe-art model1.

heading: 1 Introduction
text: Recent advances in large language models (LLMs) have attracted a surge of research interest due to their strong abilities in logical reasoning and language generation (OpenAI, 2022, 2023; Touvron et al., 2023a,b). These models have achieved surprisingly wide-ranging success across various natural language processing (NLP) tasks (Brown et al., 2020; Wang et al., 2022; Wei et al., 2022a,b; Ouyang et al., 2022).
1This work is open sourced at: https://github.com/Y UCHEN005/GenTranslate
In the realm of NLP, the translation tasks, which encompasses speech and machine translation (ST & MT), hold significant practical importance for global communication. Similar to other NLP tasks, translation tasks also gain a notable progress thanks to the recent advancement of LLMs (Zhang et al., 2023a; Lyu et al., 2023). In the domain of speech translation, Whisper (Radford et al., 2023) demonstrates superior performance by collecting 680K-hour data for web-scale model training. AudioPaLM2 (Rubenstein et al., 2023) integrates both text- and speech-based language models into a unified architecture to process and generate text and speech, thereby augmenting speech translation performance to a great extent. On the other hand, LLMs also show remarkable ability in machine translation. NLLB (Costa-jussà et al., 2022) is the first to extend LLMs’ linguistic capability to over 200 languages. BigTranslate (Yang et al., 2023b) is finetuned on LLaMA (Touvron et al., 2023a) with multilingual instruction tuning, which achieves comparable performance to ChatGPT (OpenAI, 2022) and Google Translate. Most recent work
74
proposes SeamlessM4T (Barrault et al., 2023a), a foundational multilingual and multitask model that can translate across speech and text, which achieves the state-of-the-art on both ST and MT tasks on various public datasets.
Despite the superior performance, most existing translation models employ the typical beam search algorithm for inference and select the top-1 hypothesis as final output (see Fig. 1 (a)), following that in automatic speech recognition (ASR) (Tsunoo et al., 2021). However, this strategy discards the 2 to N -best hypotheses that could be advantageous to the generation of ground-truth translation. As illustrated in Fig. 2, the discarded 2 to N -best hypotheses contain abundant semantic information that is the key to composite the ground-truth utterance, while the 1-best hypothesis lacks this part of information. As a result, the typical top-1 hypothesis selection is sub-optimal to the translation tasks that require a single informative and high-quality output sequence (Li et al., 2022; Xiao et al., 2022).
Inspired by the recent works on LLMs-enhanced ASR (Ma et al., 2023b; Chen et al., 2023; Yang et al., 2023a; Radhakrishnan et al., 2023), we propose a new generative paradigm for translation tasks, namely GenTranslate (see Fig. 1 (b)). Leveraging the rich linguistic knowledge and strong reasoning ability of LLMs, our paradigm integrates the diverse translation versions in the N -best list from foundation model to generate a higher-quality translation result. Furthermore, in order to support LLM finetuning, we also build and release a HypoTranslate dataset that contains over 592K pairs of N -best hypotheses and ground-truth translation in 11 languages. Experimental evidence on various ST and MT benchmarks (e.g., FLEURS, CoVoST-2, WMT) demonstrate that our proposed GenTranslate significantly outperforms the state-of-the-art model with efficient LLM finetuning.
Our contributions are summarized as follows:
• We propose GenTranslate, a new generative paradigm for translation tasks that leverages LLMs to generate higher-quality translation results from the diverse N -best hypotheses decoded from foundation translation model.
• We release a HypoTranslate dataset to support LLM finetuning, which contains over 592K pairs of N -best hypotheses and ground-truth translation in 11 languages.
• Experiments on various ST and MT bench-
marks show that our GenTranslate significantly outperforms the state-of-the-art model.

heading: 2 Related Work

heading: 2.1 Large Language Models
text: There is recently a surge of research interests in Transformer-based large language models, such as ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023) and LLaMA (Touvron et al., 2023a,b). Benefiting from the giant model size and oceans of training data, LLMs can understand better the linguistic structures and semantic meanings behind raw text, which thus shows remarkable performance on a wide range of natural language processing (NLP) tasks (Brown et al., 2020; Wei et al., 2022a; Ouyang et al., 2022). Thereafter, with techniques like incontext learning (Xie et al., 2021) and efficient finetuning (Hu et al., 2021; Yang et al., 2021b), LLMs further show powerful ability on downstream generative and reasoning tasks (Lampinen et al., 2022; Yang et al., 2023a; Hu et al., 2023b; Zhang et al., 2023b). Our proposed GenTranslate is exactly inspired by the promising generative ability of LLMs.

heading: 2.2 Speech and Machine Translation
text: The advancement of LLMs has notably enhanced the capabilities of translation tasks. In the domain of speech translation (Liu et al., 2021), Whisper (Radford et al., 2023) demonstrates commendable effectiveness, leveraging extensive web-scale data. AudioPaLM2 (Rubenstein et al., 2023) integrates text- and speech-based language models,
thereby augmenting the speech translation performance. In the context of machine translation, NLLB (Costa-jussà et al., 2022), a model finetuned on LLMs, extends its linguistic range to over 200 languages. Additionally, BigTranslate (Yang et al., 2023b) utilizes instruction tuning to enhance the translation capabilities of LLMs. The most recent innovation, SeamlessM4T (Barrault et al., 2023a), represents a highly-unified model capable of fluid translation between speech and text, setting new benchmarks in both ST and MT tasks. However, it is noteworthy that the majority of these methodologies rely on beam search decoding (Yang et al., 2021a; Hu et al., 2023a) and top-1 hypothesis selection for inference. How to leverage N -best hypotheses to deliver better translation result remains to be an open question.

heading: 2.3 LLMs-Enhanced ASR
text: Recent works investigate LLMs to enhance the ASR output by error correction (Ma et al., 2023a; Chen et al., 2023), which serves as a postprocessing technique to improve the recognition result (Leng et al., 2021). In particular, they leverage LLM finetuning (Zhang et al., 2023b) and in-context learning (Wang et al., 2023) to correct the wrongly recognized tokens in hypotheses by second-pass reasoning, which achieves promising improvement. Inspired by them, in this work we leverage LLMs to integrate the diverse translation versions in N -best list to generate a informative and higher-quality translation result.

heading: 3 Methodology
text: In this section, we introduce the proposed method. First, we describe the latest foundational translation model, SeamlessM4T, which we employ for beam search decoding and hypotheses generation (§3.1).
Then, we introduce our LLMs-based GenTranslate paradigm by N -best hypotheses integration (§3.2). Finally, we present the details of our released HypoTranslate dataset for GenTranslate training (§3.3).

heading: 3.1 Foundational Translation Model: SeamlessM4T
text: Recent work (Barrault et al., 2023a,b) proposes SeamlessM4T2 (Massively Multilingual & Multimodal Machine Translation), a single Transformerbased (Vaswani et al., 2017) model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. During development process, it is firstly pre-trained on 1 million hours of speech data by self-supervised learning, and it is then finetuned on a 406K-hour multimodal corpus of automatically aligned speech translations named SeamlessAlign. Experiments show that SeamlessM4T yields superior performance on all of the five supported tasks. In particular, it has achieved the stateof-the-art on both ST and MT tasks in terms of BLEU score on various public benchmarks.
Considering its effectiveness, generality and popularity, we employ SeamlessM4T as the foundation model for both speech and machine translation in our system, as depicted in the left part of Fig. 3. Given an input speech Ssrc or text T src in source language (e.g., German), SeamlessM4T translates it into target language (e.g., English) text by beam search decoding, which generates N -best hypotheses list T tgtN = {T tgt 1 , T tgt 2 , · · · , T tgt N }.
2https://github.com/facebookresearch/seamless _communication

heading: 3.2 GenTranslate

heading: 3.2.1 Overall Framework
text: To solve the information loss in typical top-1 hypothesis selection, we leverage LLMs to generate a final translation result based on the decoded N - best hypotheses. Since each candidate in N -best list represents one unique version of translation for source language input, our GenTranslate can integrate their rich information to generate a higherquality translation result, thanks to the strong linguistic and reasoning ability of LLMs. This new generative paradigm can be formulated as:
T tgt = MGT(T tgtN , I), (1)
where I is a proper instruction for LLM prompting. The goal of GenTranslate is to learn a mapping MGT from N -best hypotheses to the true translation. Following typical sequence-to-sequence learning strategy, we employ the ground-truth translation T tgt* as supervision signal and optimize the LLM to learn MGT in an auto-regressive manner. The cross-entropy-based training loss is defined as:
LGT = L∑
l=1
− logPθ(ttgt*l |t tgt* l−1, · · · , t tgt* 1 ; T tgt N , I),
(2) where ttgt*l is the l-th token of T
tgt*, L denotes the sequence length, and θ denotes the learnable parameters in LLM (i.e., adapter).

heading: 3.2.2 Efficient LLM Finetuning
text: Considering the giant scale of LLMs, we adopt the popular efficient finetuning strategy, LLaMAAdapter (Zhang et al., 2023b), which is comparable to LoRA tuning (§4.3.4). As shown in Fig. 3 (right), it inserts a set of learnable adaptation prompts into the top-L of total H Transformer layers in a pretrained LLM to learn high-level semantics. Denote the prompt for l-th layer as Pl ∈ RU×D, where U is prompt length and D is embedding size.
Assume we gain M tokens including instruction and already generated response, i.e., Tl ∈ RM×D, now we aim to predict the (M + 1)-th token as response. The learnable adaptation prompt is concatenated with Tl as prefix, i.e., [Pl;Tl] ∈ R(U+M)×D, which provides learned instruction knowledge to guide the subsequent response generation.
Furthermore, considering the prompt Pl is randomly initialized and thus could disturb the LLM tuning at early training stage, a zero-initialized attention mechanism is devised to mitigate such
disturbance. Denote the current M -th token as T (M) l ∈ R1×D, in attention there are three projection layers to generate query, key and value:
Ql = Linearq(T (M) l ), Kl = Lineark([Pl;Tl]), Vl = Linearv([Pl;Tl]),
(3)
Then the attention score is calculated as Al = Ql · Kl/ √ D ∈ R1×(U+M), which captures the correlation between current token and the history tokens as well as prompts to predict the next token. Therefore, it can be split into two parts accordingly:
Al = [A P l ;A T l ] T , (4)
where APl ∈ RU×1 is the attention score of U adaptation prompts and ATl ∈ RM×1 is that of M history tokens. Since the adaptation prompts are randomly initialized, their attention scores may cast disturbance on next-token prediction at early training stage. To this end, a learnable gating factor gl with zero initialization is introduced to adaptively control the weight of prompt in attention:
Agl = [gl · softmax(APl ); softmax(ATl )]T , (5)
Finally, the attention output of l-th Transformer layer is obtained with a linear projection:
O (M) l = Linearo(A g l · Vl) ∈ R1×D, (6)
It is then employed to predict the next token T (M+1) l as response. The zero-initialization mechanism yields an effective trade-off between the pretrained knowledge of LLM and the learned instructional knowledge through adaptation prompt.

heading: 3.3 HypoTranslate Dataset
text: In order to support the LLM finetuning for GenTranslate, we release a HypoTranslate dataset that contains over 592K pairs of N -best hypotheses and ground-truth translation in 11 languages. In particular, we use the state-of-the-art SeamlessM4TLarge as foundation translation model to decode N -best hypotheses from input speech by beam search algorithm, where the beam size N is set to 5. Specifically, for ST task we investigate two popular pipelines in literature, i.e., end-to-end ST and cascaded ASR+MT. Thanks to the universal ability of SeamlessM4T on ST, ASR and MT tasks, we only need one model to build above two pipelines.
To build HypoTranslate dataset, we select several public ST and MT corpora in both X→En and
En→X language directions. For speech translation, we select FLEURS (Conneau et al., 2023), CoVoST2 (Wang et al., 2020), and MuST-C (Di Gangi et al., 2019). For machine translation, we select FLORES (Costa-jussà et al., 2022), WMT’16 (Bojar et al., 2016), WMT’19 (Barrault et al., 2019), and WMT’20 (Loïc et al., 2020) corpora. As a result, we obtain over 592K hypotheses-translation pairs in 11 languages. The details of dataset statistics are presented in §A.3 and Table 15, 17.
Since the hypotheses-translation data pairs in HypoTranslate dataset are monolingual, we can also use ASR dataset to benefit GenTranslate training, especially for low-resource language pairs. Relevant studies are illustrated in §4.3.2 and Table 7. Our best result was obtained by first performing translation with SeamlessM4T and then integrating the N -best candidates using LLMs.
3Our experiments are mainly conducted on SeamlessM4TLarge as they had already been done before Meta released the latest SeamlessM4T-Large-V2 on November 30th, 2023. For comprehensive evaluation, we rerun the main experiments on V2, which demonstrate similar effectiveness of our paradigm.

heading: 4 Experiments

heading: 4.1 Setup

heading: 4.1.1 Model Selection
text: LLMs. We select the popular LLaMA-2 (Touvron et al., 2023b) for our paradigm. Specifically, we employ LLaMA-2-7b4 for English-target directions (X→En) and LLaMA-2-13b for non-English-target directions (En→X), as LLaMA-2 shows superior ability on English language while less-optimal on other languages. In addition, for En→X we also try some latest multilingual LLMs like BigTranslate5 (Yang et al., 2023b) and ALMA6 (Xu et al., 2023b) that are finetuned on LLaMA-13b. Adapter. We follow the default settings of LLaMAAdapter (Zhang et al., 2023b). The number of tunable Transformer layers L is set to H−1, which means all layers except the first one are tunable
4https://huggingface.co/meta-llama/Llama-2-7 b-hf
5https://huggingface.co/James-WYang/BigTransl ate
6https://huggingface.co/haoranxu/ALMA-13B
with inserted prompts. The prompt length U is set to 10. More details are provided in §B.1.

heading: 4.1.2 Training Details
text: The batch size is set to 4, with accumulation iterations set to 8 (i.e., real batch size is 32). We train 2 epochs with AdamW optimizer (Loshchilov and Hutter, 2018), with learning rate initialized to 1e−2 and then linearly decrease to 1e−5 during training.

heading: 4.2 Comparison with the State-of-the-art

heading: 4.2.1 Speech Translation
text: X→English (En). Table 1 and 2 present the X→En speech translation performance on FLEURS and CoVoST-2 datasets. We can observe from Table 1 that all the strong baselines like Whisper, AudioPaLM2 and SeamlessM4T-Large perform well on 15 X→En directions, where SeamlessM4TLarge is the best (27.1 BLEU). With LLMs in-
troduced for N -best integration, our GenTranslate achieves consistent improvements on various source languages X, where further analysis on language family is presented in §4.4.1. As a result, our GenTranslate shows 3.0 BLEU improvement over SeamlessM4T-Large, which verifies the effectiveness of LLMs for generative translation7.
Following the speech translation literature, we also investigate cascaded ASR+MT methods for evaluation. We can observe from Table 1 that, with the same SeamlessM4T-Large backbone, cascaded system outperforms end-to-end system by 4.8 BLEU score, which is consistent with previous findings (Xu et al., 2023a). Latest SeamlessM4TLarge-V2 further improves V1 model, and our GenTranslate shows significant and consistent gains of performance over theses two backbones.
Table 2 presents the X→En ST results on more language directions of CoVoST-2 dataset, where we introduce more latest baselines for comprehensive comparison. In end-to-end methods, SeamlessM4TLarge achieves a good 34.5 BLEU score though underperforms the state-of-the-art AudioPaLM28. In comparison, our GenTranslate achieves a promis-
7Latest SeamlessM4T-Large-V2 achieves significant gains over V1, based on which the proposed GenTranslate also shows similar effectiveness in our study.
8We speculate it could be attributed to the train-test domain mismatch because SeamlessM4T-Large outperforms AudioPaLM2 by a large margin on FLEURS dataset in Table 1.

heading: End-to-end ST Methods

heading: Cascaded ASR+MT Methods

heading: ASR+GenTranslate Method
text: ing improvement over SeamlessM4T. Similar phenomenon can be observed in cascaded systems, where SeamlessM4T significantly outperforms the competitive baselines that combine state-of-theart ASR and MT models, and our GenTranslate moves one step forward with 1.8 BLEU improvement. Similar improvements can be observed on SeamlessM4T-Large-V2 backbone.
English (En)→X. For comprehensive evaluation, we also present En→X ST results on three datasets in Table 3. SeamlessM4T (both Large and LargeV2) achieves excellent performance on En→X ST tasks under both end-to-end and cascaded systems. In comparison, our proposed GenTranslate achieves significant performance improvements (∼3 BLEU score) in various language directions. Since En→X translation tasks produce non-English N -best hypotheses for LLM integration, such performance gains indicates the excellent multilingual abilities of LLMs (i.e., LLaMA-2).

heading: 4.2.2 Machine Translation
text: X→English (En). Table 4 presents the X→En MT results on FLORES dataset. The baseline methods ALMA-13b and BigTranslate show limited performance. NLLB-3.3b achieves an improved performance of 37.5 BLEU, which is comparable to
SeamlessM4T-Large. Based on that, our GenTranslate achieves the state-of-the-art with consistent gains on all language directions except Ja→En. English (En)→X. Table 5 presents the En→X MT results on WMT test sets. Similar to previous results, we observe much higher BLEU scores of NLLB-3.3b than ALMA-13b and BigTranslate. SeamlessM4T-Large surpasses NLLB-3.3b by large-scale multitask training. The proposed GenTranslate achieves the state-of-the-arts on all language directions with a gain of 2.4 BLEU score. Please note that SeamlessM4T-Large-V2 underperforms V1 on selected MT datasets, but our GenTranslate achieves consistent gains on both of them.
In summary, we observe consistent improvements of GenTranslate over various baselines (i.e., SeamlessM4T, Whisper, etc.), various tasks (i.e., ST and MT), various test data (i.e., FLEURS, WMT, etc.), and various language directions (i.e., X→En and En→X). Therefore, the effectiveness and generality of our approach are well verified.

heading: 4.3 Ablation Study

heading: 4.3.1 Effect of Different LLMs
text: According to Table 3 and 5, LLaMA-2 has shown excellent multilingual ability. To further investigate the role of this ability in GenTranslate, we select
two latest multilingual LLMs for comparison, i.e., BigTranslate and ALMA-13b. Table 6 shows that both of them perform worse than LLaMA-2-13b for ST and MT tasks. One explanation is, BigTranslate and ALMA-13b are finetuned on MT task that requires cross-lingual ability, while the En→X GenTranslate mainly requires strong monolingual ability of language X, such mismatch may explain why MT finetuning fails to enhance GenTranslate.

heading: 4.3.2 Role of LLMs in GenTranslate
text: To further investigate the role of LLMs in our GenTranslate, we build an ASR+GenTranslate system for ST task as shown in Fig. 4. Take De→En as an example, we first send the German speech input into ASR to produce N -best transcriptions, which are then fed by LLMs to generate English translation. In other words, LLMs are assigned N -best integration and translation tasks at the same time. As shown in Table 7, among the three evaluated LLMs, ALMA-7b achieves the best performance thanks to its MT finetuning during development, but it still underperforms the best cascaded method (40.6 vs. 41.6). We can conclude from such observations that 1) LLaMA-2 provides reasonable translation
ability and it can be further improved via MT task finetuning (i.e., ALMA). 2) In this study, LLM underperforms SeamlessM4T in translation task, but it shows remarkable ability in N -best integration. Therefore, future work may focus on how to better engage LLMs into the translation part.

heading: 4.3.3 Effect of N -best List Size
text: GenTranslate relies on powerful LLMs and informative N -best hypotheses to generate higherquality translation output. Therefore, the amount of information in N -best hypotheses could be a key factor of GenTranslate’s performance. We can observe from Table 10 that with the increase of N, the performance of GenTranslate first improves and then drops, where the best choice ranges from 5 to 10. We believe that small N results in insufficient information for generation of ground-truth translation, while too large N leads to information redundancy and thus increases the miscorrection and hallucination. In this work, we set N to 5 for the best trade-off between efficiency and quality.

heading: 4.3.4 LLaMA-Adapter vs. LLaMA-LoRA
text: Apart from LLaMA-Adapter, low-rank adaptation (LoRA) (Hu et al., 2021; Yu et al., 2023) is another popular efficient LLM finetuning strategy. Table 8 compares the performance between LLaMAAdapter and LLaMA-LoRA for proposed GenTranslate, in terms of the BLEU results of ST task on FLEURS X→En test sets. We can observe similar BLEU performance of these two strategies on GenTranslate (34.0 vs. 34.1), indicating that the efficient LLM finetuning strategy is not a key factor in GenTranslate paradigm.

heading: 4.4 Analysis

heading: 4.4.1 Effect of Language Family
text: Table 9 analyzes the effect of language family using the X→En ST results. The source language X is grouped into two categories depending on whether it belongs to Indo-European family (English is also Indo-European language). First, we observe better results of SeamlessM4T when X belongs to Indo-European family, indicating that translation within same family is easier than across different families. Then, we also observe larger BLEU improvement of GenTranslate over baseline when X is Indo-European language (2.6 vs. 1.3). The reason could be, within-family translation produces N -best hypotheses with higher quality and richer information, which is beneficial to GenTranslate.

heading: 4.4.2 Case Study
text: Table 11 shows a case study where GenTranslate outperforms the 1-best hypothesis by a large margin. We may speculate two key points about its working mechanism, where it first extract the word “Television” from 3rd/4th hypotheses to replace “TV” and then reason out the word “coming” that does not exist in N -best list. Therefore, our paradigm may not only integrate the N -best sentences for better result, but also improve the translation quality by itself. Another non-English case study is in Appendix C.1.

heading: 4.4.3 Visualizations of GenTranslate Output
text: Fig. 5 visualizes the n-gram tokens in GenTranslate output, which contains sufficient semantic information to match the ground-truth translation. In comparison, the 1-best hypothesis lacks such information to produce high-quality translation output, which highlights the contribution of N -best hypotheses in GenTranslate paradigm (see Fig. 2).

heading: 5 Conclusion
text: In this paper, we propose a generative paradigm for translation tasks, namely GenTranslate, which leverages LLMs to integrate the diverse candidates in the decoded N -best list and generate a higherquality translation result. Furthermore, we release a HypoTranslate dataset to support LLM finetuning, which contains over 592K hypotheses-translation pairs in 11 languages. Experimental evidence on various speech and machine translation benchmarks shows that our GenTranslate significantly outperforms the state-of-the-art model.

heading: Limitations
text: There are two limitations existed in this work. First, the contribution of LLMs in our GenTranslate paradigm focuses on N -best hypotheses integration, while the translation part is actually done by SeamlessM4T model. Experiment results in Table 7 also indicate that LLMs are good at N - best hypotheses integration and SeamlessM4T is good at translation. Therefore, our future work could focus on how to better engage LLMs into the translation part to further improve the translation quality. Another limitation is about the latest
second version of SeamlessM4T released by Meta, which indicates a stronger baseline for GenTranslate. In fact, our experiments had already been done on SeamlessM4T-Large before Meta released the latest SeamlessM4T-Large-V2 on November 30th, 2023. For comprehensive evaluation, we also rerun our main experiments on this latest V2 backbone, and our GenTranslate has shown similar effectiveness on it (highlighted in gray in Table 1 to 5). For brevity, we prefer to leave the ablation study and analyses on SeamlessM4T-Large backbone only, as our GenTranslate paradigm has shown similar effectiveness and patterns on V1 and V2 backbones.

heading: Ethics Statement
text: This work does not pose any ethical issues. All the data used in this paper are publicly available and are used under following licenses: Creative Commons BY 4.0 License, Creative Commons CC0 License, Creative Commons BY-NC-ND 4.0 License, and Creative Commons BY-SA 4.0 License.

heading: Acknowledgements
text: This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-GC-2022-005). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg).

heading: A HypoTranslate Dataset Details
text: In this section, we introduce the details of our proposed HypoTranslate dataset. We first introduce the speech and machine translation corpora that we utilize to build HypoTranslate in §A.1 and §A.2. Then, we present the dataset statistics in §A.3.

heading: A.1 Speech Translation Corpus Selection
text: For speech translation task, we select three popular and public datasets that cover multiple languages:
FLEURS9 (Conneau et al., 2023): Few-shot Learning Evaluation of Universal Representations of Speech (FLEURS) benchmark provides an n-way parallel speech dataset in 102 languages built on top of the machine translation FLORES-101 benchmark (Goyal et al., 2022), with approximately 12 hours of speech supervision per language. In this work, we select 15 X→En and 6 En→X language directions of speech translation data for evaluation.
CoVoST-210 (Wang et al., 2020): CoVoST-2 is a popular multilingual speech translation corpus based on Common Voice (Ardila et al., 2019) that consists of 2,880 hours speech data recorded from 78K speakers. In this work, we select 15 X→En and 3 En→X language directions for evaluation. Specifically, for En→X language directions, we randomly select 1,000 testing samples from the original test split for higher evaluation efficiency.
MuST-C11 (Di Gangi et al., 2019): MuST-C is a multilingual speech translation corpus whose size and quality facilitate the training of end-to-end systems for spoken language translation from English into 15 languages. In this work, we select 3 En→X language directions for evaluation.

heading: A.2 Machine Translation Corpus Selection
text: For machine translation task, we select two popular and public datasets that cover multiple languages:
FLORES12 (Costa-jussà et al., 2022): FLORES consists of 3001 sentences sampled from Englishlanguage Wikimedia projects for 204 total languages. Approximately one third of sentences are collected from each of these sources: Wikinews, Wikijunior, and Wikivoyage. The content is professionally translated into 200+ languages to create
9https://huggingface.co/datasets/google/fleurs 10https://github.com/facebookresearch/covost 11https://mt.fbk.eu/must-c-releases/ 12https://huggingface.co/datasets/facebook/flo
res
FLORES dataset. In this work, we select 10 X→En language directions for evaluation.
WMT: The Conference on Machine Translation (WMT) is a popular evaluation benchmark for MT task. In this work, we select the newstest data of Ro→En language direction from WMT’1613 (Bojar et al., 2016), Cs→En and It→En directions from WMT’1914 (Barrault et al., 2019), Ja→En and Zh→En directions from WMT’2015 (Loïc et al., 2020) for evaluation, and corresponding newdev data is used for validation. The training data is obtained from ParaCrawl-V916 (Bañón et al., 2020) and JParaCrawl17 (Morishita et al., 2020) datasets.

heading: A.3 Statistics
text: After performing beam search decoding on the selected speech and machine translation corpora introduced above, we collect over 592K pairs of N -best hypotheses and ground-truth translation to build the HypoTranslate dataset. The statistics are illustrated in Table 15 and 17, which present the number of hypotheses-translation pairs and the average utterance length. We plan to release the HypoTranslate dataset to public upon publication and open the development venue for more data.

heading: B Experimental Setup Details

heading: B.1 Model Setups
text: We select two latest foundation LLMs for evaluation, including LLaMA-2-7b (Touvron et al., 2023b) and LLaMA-2-13b (Touvron et al., 2023b). In addition, in order to evaluate the multilingual ability of LLMs for GenTranslate with non-Englishtarget directions, we also select two latest finetuned LLMs on MT task, including BigTranslate (Yang et al., 2023b) and ALMA-13b (Xu et al., 2023b). Table 12 compares their main configurations. For efficient LLM finetuning, we follow the default settings of LLaMA-Adapter18 (Zhang et al., 2023b).
13https://www.statmt.org/wmt16/translation-tas k.html
14https://www.statmt.org/wmt19/translation-tas k.html
15https://www.statmt.org/wmt20/translation-tas k.html
16https://paracrawl.eu/ 17https://www.kecl.ntt.co.jp/icl/lirg/jparacra wl/ 18https://github.com/Lightning-AI/lit-gpt/blob/ main/lit_gpt/adapter.py
B.2 Inference Setups In the response generation during inference stage, we set a temperature of 0.2 and top-1 sampling, i.e., greedy search. We have observed over-confidence phenomenon in our experiments (i.e., output probability distribution for decision is close to one-hot), which results in similar performance with different k for top-k sampling. Therefore, we select top-1 sampling for higher decoding efficiency.

heading: B.3 Translation Baselines
text: To comprehensively evaluate our GenTranslate model, we selected some of the latest and most advanced baselines in speech and machine translation for comparison. We will introduce these in the following subsections.

heading: B.3.1 Speech Translation
text: XLS-R19 (Babu et al., 2021): XLS-R is a largescale model for cross-lingual speech representation learning based on Wav2vec 2.0 (Baevski et al., 2020). They train models with up to 2B parameters on 500K hours of publicly available speech audio in 128 languages, which achieves superior performance on a wide range of multilingual speech processing tasks, including speech translation, speech recognition and language identification. Whisper20 (Radford et al., 2023): Whisper is a large-scale automatic speech recognition (ASR) system trained on 680K hours of multilingual and multitask supervised data collected from the web, which shows excellent robustness to accents, background noise and technical language. Moreover, it enables transcription in multiple languages, as well as translation from those languages into English. AudioPaLM2 (Rubenstein et al., 2023): AudioPaLM2 fuses text-based and speech-based language models, PaLM-2 (Anil et al., 2023) and AudioLM (Borsos et al., 2023), into a unified multimodal architecture that can process and generate
19https://huggingface.co/models?other=xls_r 20https://github.com/openai/whisper
text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM2 inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. The resulting model significantly outperforms existing systems for speech translation and has the ability to perform zero-shot speech-totext translation for many unseen languages. ComSL21 (Le et al., 2023): ComSL is a speechlanguage model built atop a composite architecture of public pre-trained speech-only and languageonly models and optimized data-efficiently for spoken language tasks. Particularly, they propose to incorporate cross-modality learning into transfer learning and conduct them simultaneously for downstream tasks in a multi-task learning manner, which has demonstrated effectiveness in end-to-end speech-to-text translation tasks.

heading: B.3.2 Machine Translation
text: NLLB22 (Costa-jussà et al., 2022): No Language Left Behind (NLLB) is a first-of-its-kind, AI breakthrough project that open-sources models capable of delivering evaluated, high-quality translations directly between 200 languages. BigTranslate5 (Yang et al., 2023b): BigTranslate adapts LLaMA-13b (Touvron et al., 2023a) that covers only 20 languages and enhances it with multilingual translation capability on up to 102 languages by instruction-following finetuning, which achieves comparable MT performance to ChatGPT (OpenAI, 2022) and Google Translate. ALMA6 (Xu et al., 2023b): ALMA proposes a novel finetuning approach for LLMs that is specifically designed for MT task, eliminating the need for the abundant parallel data that traditional translation models usually depend on, which includes two stages: initial finetuning on monolingual data
21https://github.com/nethermanpro/ComSL 22https://huggingface.co/facebook/nllb-200-3.3
B
followed by subsequent finetuning on a small set of high-quality parallel data. Built based on LLaMA2, it has achieved significant improvement over prior works across multiple translation directions.

heading: C Supplementary Experiment Results

heading: C.1 Supplementary Case Study
text: Table 13 supplies a case study from FLEURS En→Zh ST test set. We can observe that the N -best candidate are semantically similar to each other and only varies in sentence structure. In our GenTranslate paradigm, LLMs integrates the different patterns of N -best hypotheses to generate a new translation result with 3.7 BLEU improvement over 1-best hypothesis. Such observation verifies the effectiveness of LLMs in our GenTranslate paradigm to generate better translation output.

heading: C.2 BLEU vs. chrF++
text: We report translation performance in terms of the BLEU score (Papineni et al., 2002) in most experiments of this work. For more comprehensive evaluation, Table 16 presents both BLEU and chrF++ scores (Popović, 2017; Barrault et al., 2023a) on FLEURS X→En test sets, where we can observe consistent improvements of BLEU and chrF++ scores (2.1 ∆ BLEU and 0.9 ∆ chrF++) in GenTranslate. It indicates that both metrics are applicable for the evaluation of translation tasks.

heading: BLEU score

heading: Future Work
text: Therefore, our future work could focus on how to better engage LLMs into the translation part to further improve the translation quality. Another limitation is about the latest
second version of SeamlessM4T released by Meta, which indicates a stronger baseline for GenTranslate. In fact, our experiments had already been done on SeamlessM4T-Large before Meta released the latest SeamlessM4T-Large-V2 on November 30th, 2023. For comprehensive evaluation, we also rerun our main experiments on this latest V2 backbone, and our GenTranslate has shown similar effectiveness on it (highlighted in gray in Table 1 to 5). For brevity, we prefer to leave the ablation study and analyses on SeamlessM4T-Large backbone only, as our GenTranslate paradigm has shown similar effectiveness and patterns on V1 and V2 backbones