Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding (2024)

Abstract: Contemporary translation engines based on the encoder-decoder framework have made significant strides in development. However, the emergence of Large Language Models (LLMs) has disrupted their position by presenting the potential for achieving superior translation quality. To uncover the circumstances in which LLMs excel and explore how their strengths can be harnessed to enhance translation quality, we first conduct a comprehensive analysis to assess the strengths and limitations of various commercial NMT systems and MT-oriented LLMs. Our findings indicate that neither NMT nor MToriented LLMs alone can effectively address all the translation issues, but MT-oriented LLMs show promise as a complementary solution to NMT systems. Building upon these insights, we propose Cooperative Decoding (CoDec), which treats NMT systems as a pretranslation model and MT-oriented LLMs as a supplemental solution to handle complex scenarios beyond the capability of NMT alone. Experimental results on the WMT22 test sets and a newly collected test set WebCrawl demonstrate the effectiveness and efficiency of CoDec, highlighting its potential as a robust solution for combining NMT systems with MT-oriented LLMs in the field of machine translation.

heading: 1 Introduction
text: Over the years, the encoder-decoder framework has established Neural Machine Translation (NMT) models as the prevailing standard, achieving impressive translation quality through extensive training on large-scale and high-quality parallel data (Vaswani et al., 2017; Freitag and Firat, 2020; Fan et al., 2021). Commercial machine translation engines, e.g., Google Translate, are proficient in addressing the majority of translation requirements. More recently, with the emergence of generative large language models (LLMs), the position of traditional NMT models has been challenged (Brown
et al., 2020; OpenAI, 2023). While commercial LLMs like OpenAI’s GPT-4 currently perform well in translation (Hendy et al., 2023; Zhu et al., 2023; Lin et al., 2022; Agrawal et al., 2022), they are constrained by their interface nature, thereby limiting further customization and improvement due to privacy concerns in industrial applications. A more promising approach involves fine-tuning relatively smaller LLMs (i.e., fewer than 13B parameters) to create LLMs specifically tailored for MT (Zeng et al., 2023; Zhang et al., 2023; Jiao et al., 2023).
In this context, this study aims to investigate the following research questions: In which scenarios do MT-oriented LLMs demonstrate superior performance to conventional NMT models, and how can we leverage the strengths of the two paradigms to enhance translation quality?
To begin, we conduct a comprehensive analysis into the characteristics of translations generated by commercial NMT systems and MT-oriented LLMs. Our findings reveal that commercial NMT systems excel at producing adequate translations in specific domains or languages. Conversely, MToriented LLMs demonstrate proficiency in gener-
13275
ating authentic-sounding translations and handling infrequent words that are not effectively processed by NMT systems. In summary, MT-oriented LLMs can serve as valuable fallback systems in cases where the output of commercial NMT systems is unsatisfactory.
To complement NMT with MT-oriented LLMs, Hendy et al. (2023) introduced the Hybrid Threshold approach (Figure 1(a)), which employs the NMT system as the primary translation system. When the translation fails to meet the quality threshold determined by the quality estimation (QE) module, an alternative translation is generated using a GPT-like model. However, this approach faces two primary challenges. First, existing reference-free metrics struggle to align with human judgment, resulting in inaccuracies being propagated (Freitag et al., 2021, 2020; Ma et al., 2019; Rei et al., 2022a). Second, the integration of neural quality estimation modules and the sequential execution by LLMs leads to increased decoding time (Tay et al., 2023; Xu et al., 2021), which poses concerns for efficient translation in practical applications.
To address the above issues, we propose an efficient implementation approach for system ensembles called Cooperative Decoding (CoDec). As illustrated in Figure 1(b), the NMT system functions as the front-end module, generating an initial translation draft for a given input sentence. Subsequently, the MT-oriented LLM serves as both an evaluator and a refiner, which firstly evaluates the draft from a language modeling perspective, and then the LLM refines the partial translation starting from a specific position where the token in the draft is not among the top-k token candidates suggested by the LLM. Since the evaluation process takes advantage of parallel computation and the frontend module can handle most situations effectively, CoDec is more efficient compared to using LLMs for complete decoding.
The contributions of this paper are three-fold:
• We conduct in-depth analyses on the WMT22 test sets and a newly collected test set, WebCrawl, to identify the strengths and weaknesses of traditional NMT systems and MToriented LLMs, finding that MT-oriented LLMs can complement NMT systems.
• We present CoDec, a novel hybrid framework that synergizes the strengths of NMT systems and MT-oriented LLMs. By harnessing the complementary capabilities of MT-oriented
LLMs, CoDec effectively overcomes the limitations of traditional NMT systems1.
• We evaluate the performance of CoDec on various test sets. Our CoDec, without the need for an additional quality estimation module, achieves competitive or even better performance than Hybrid Threshold. Furthermore, CoDec offers a significant acceleration advantage, achieving an acceleration ratio of approximately 2x compared to directly using LLM for generation.

heading: 2 Related Work

heading: 2.1 Large Language Models on Machine Translation
text: Research on Large Language Models (LLMs) for machine translation can be broadly divided into two categories: utilizing LLMs as an interface and optimizing them for specific translation tasks. For the former, Hendy et al. (2023) evaluate ChatGPT, GPT3.5, and text-adavinci-002 in eighteen translation directions, while Zhu et al. (2023) assess XGLM, BLOOMZ, OPT, and ChatGPT across 202 directions and 102 languages. Other researchers explore strategies for selecting translation exemplars (Lin et al., 2022; Agrawal et al., 2022) and incorporating external knowledge (Lu et al., 2023) to enhance GPT translation. Fine-tuning smaller models (e.g., 7B) specifically for translation tasks has attracted increasing attention (Zeng et al., 2023; Zhang et al., 2023; Jiao et al., 2023). Diverging from existing approaches, our research focuses on examining the capabilities and limitations of commercial NMT systems and MT-oriented LLMs and developing efficient hybrid frameworks that leverage their respective strengths.

heading: 2.2 Accelerate Generation for Large Language Models
text: Efforts to improve the inference efficiency of LLMs have been ongoing for several years (Tay et al., 2023; Xu et al., 2021), leveraging techniques such as knowledge distillation (Hinton et al., 2015; Jiao et al., 2020; Wang et al., 2020), quantization (Shen et al., 2020; Sun et al., 2020), pruning(Fan et al., 2020), and others (Kim and Cho, 2021; Lei, 2021). The most related work is to leverage speculative execution (Burton, 1985; Hennessy and Patterson,
1We release code and the translations of different systems at https://github.com/lemon0830/CoDec
2012) for the speedup of autoregressive models. Stern et al. (2018) propose to decode several tokens in parallel to accelerate greedy decoding. For LLMs, speculative decoding (Chen et al., 2023a; Leviathan et al., 2023) uses an additional draft model and generates sequences with sampling. Yang et al. (2023) copy some tokens from retrieved reference text to the decoder, which are validated with output probabilities. Santilli et al. (2023) reframe MT’s standard greedy autoregressive decoding procedure with a parallel formulation. We are pioneers in using speculative execution as a fusion approach for commercial NMT systems and MT-oriented LLMs, without requiring an auxiliary quality estimation module or modifications to the target LLMs’ parameters.

heading: 3 Preliminary Experiments
text: In this section, we conduct a series of analyses to quantitatively investigate the characteristics of translations from different systems.

heading: 3.1 Setup
text: Commercial NMT Systems & MT-oriented LLMs. Our focus is the use of MT-oriented LLMs in industrial settings, and the chosen commercial NMT systems consist of Google Translate (GoogleMT for brevity)2 and Microsoft Translate (MicroMT for brevity)3 due to their strong performance and high reproducibility.
Regarding MT-oriented LLMs, we utilize BayLing-7B (Zhang et al., 2023). We directly use the translations released on GitHub4. Additionally,
2https://translate.google.com/ 3https://www.bing.com/translator 4https://github.com/ictnlp/BayLing/tree/main/exp/translation
_benchmark/bayling-7b
we develop an in-house MT-oriented LLM, trained on human-written validation data from previous WMT competitions5, such as the newstest20172021 of German⇔English, Chinese⇔English, Russian⇔English, and Jappanese⇔English. In addition, we have incorporated high-quality bilingual sentence pairs in Chinese⇔English, German⇔English, and Russian⇔English, resulting in a total of two million sentences in our training data. According to the data license of WMT22, the data released for the General MT task can be freely used for research purposes. We fine-tune the tigerbot-13b-base6 with TIM (Zeng et al., 2023) as the final MT-oriented LLM7.
Automated MT Metrics. We follow previous studies (Hendy et al., 2023; Zhu et al., 2023; Zeng et al., 2023; Zhang et al., 2023) to utilize COMET22 (wmt22-COMET-da) (Rei et al., 2021), and COMETkiwi (wmt22-COMET-kiwi-da) (Rei et al., 2022b) for reference-free quality estimation. We also report ChrF (Popovic, 2015) and SacreBLEU (Papineni et al., 2002) in Table 7 in Appendix.

heading: 3.2 Analyses on WMT22 test sets
text: To prevent data leakage (Garcia et al., 2023), we analyze the WMT22 test sets. Detailed statistics are reported in Appendix A.
Main Results. The experimental results are illustrated in Table 1. We have made the following observations: 1) GoogleMT and MicroMT showcase excellent performance. They consistently outperform the WMT winner in most of the language pairs, highlighting the robust capabilities of these
5https://www.statmt.org/wmt22/translation-task.html 6https://huggingface.co/TigerResearch/tigerbot-13b-base 7https://huggingface.co/Lemoooon/TIM_13B_forCoDec
well-established translation engines. 2) Despite the existing performance gap, MT-oriented LLMs still have untapped potential for further improvement. Notably, TIM outperforms BayLing by a significant margin across all language pairs. Moreover, TIM exhibits slightly inferior performance compared to MicroMT on most test sets. This suggests that employing more effective fine-tuning methods with large amounts of high-quality parallel data can enhance the translation capabilities of MT-oriented LLMs, making them close to commercial NMT systems.
Off-target Rates. Off-target indicates translations generated by machines involve segments of wrong languages or code-mixing, presenting a significant challenge in multilingual neural machine translation (Chen et al., 2023b; Zhang et al., 2020). Here, we use langdetect8 to identify the language of each translation. The off-target rate of a translation is the subtraction of the probability of the target language prediction from 1. For a test set, we compute the average off-target rate across all the sentences.
As depicted in Figure 29, the MT-oriented
8https://github.com/Mimino666/langdetect 9Due to limited space, we only present the results for
LLMs tend to produce translations with higher offtarget rates compared to NMT systems. Specifically, BayLing exhibits off-target rates of 21% and 53.83% for EN⇒RU and EN⇒JA translations, respectively, which falls outside the language scope covered by the training data. This highlights a more pronounced off-target issue in LLMs, especially in zero-shot scenarios. In contrast, TIM achieves notably lower off-target rates in EN⇒RU and EN⇒JA compared to BayLing. We speculate that this can be attributed to TIM’s incorporation of corresponding training data, which enhances its ability to handle language switching and produce more accurate translations.
Unaligned Source/Target Words. To assess the literalness of the translation, we follow Raunak et al. (2023); Hendy et al. (2023) to calculate the number of source and target words that do not align on a word-to-word basis. More details can be found in Appendix B. The left portion of Figure 3 illustrates that the MT-oriented LLMs incur a notably larger number of unaligned source words across the test sets than the NMT counterpart.
We examine the top six part-of-speech (POS)
English-to-Many translations here. The results for Many-toEnglish can be found in Figure 7 in Appendix.
tags by NLTK toolkit (Bird et al., 2009) of the unaligned source words (in Figure 4). The difference mainly lies in nouns (NN) and adjectives (JJ), indicating the possibility of increased paraphrasing or a higher degree of inadequacy, such as omitted or inserted content. However, back to the middle part of Figure 3, the number of unaligned target words of the MT-oriented LLMs does not significantly differ from those of NMT systems, suggesting that the adequacy of translations produced by LLMs is comparable to NMT.
Additionally, we calculate the average word count in the generated translations. As depicted in Figure 3, MT-oriented LLMs tend to produce shorter sentences, utilizing concise and precise language. Humans often use concise language, especially in conversations, which is abundant in the training corpus of LLMs. This influence may result in LLMs generating shorter translations.
Figure 5 presents several examples that highlight translation differences. For instance, the phrase “frustration shouting” should be translated as “大 喊大叫 (scream in frustration)”. While MicroMT aims for fidelity by using translation augmentation segments like “我非常沮丧 (I feel extremely
frustrated)”, TIM demonstrates a better understanding of the entire sentence and provides more accurate translations. However, in the third example, TIM overlooks the inclusion of the expression “the receptive, super-alert brain of a tiny child” from the source text, resulting in a certain degree of translation oversight. In summary, MT-oriented LLMs tend to generate shorter translations with substantial paraphrasing, where the original text is rephrased using different words and sentence structures while preserving the same meaning.

heading: 3.3 Analyses on Web Crawl test sets
text: The WMT22 test set is meticulously screened and annotated, with source sentences free of errors and from common domains. While the sentences have strong syntactic structures and grammatical correctness, real-world translation scenarios may not always have these ideal conditions. To reflect practical challenges, we collected a challenging test set from the open domain through web crawling. Here, we focus on Chinese⇔English directions. To acquire the data, we follow the process outlined in Appendix C.
Main Results. Similarly, we compute various evaluation metrics for NMT systems and TIM on the WebCrawl test sets. As shown in Table 3, it is noteworthy that TIM demonstrates significant improvements in both ZH⇒EN and EN⇒ZH directions. This surprising finding suggests that MToriented LLMs can serve as valuable fallback systems in cases where the quality of commercial NMT systems is unsatisfactory.
To further support our hypothesis, we calculate the COMETkiwi scores of the translations generated by GoogleMT and TIM against the source text, selecting a group of sentences where GoogleMT has higher scores than TIM by more than 3 points, and another group where TIM has higher scores than GoogleMT. To mitigate the impact of sentence lengths, we retain only those sentences containing
fewer than 60 tokens. Next, we use gpt2-large10 to calculate the perplexity for the two groups. The perplexity for sentences in which GoogleMT excels is 38.61, whereas for sentences in which TIM performs better, it is 45.51. The MT-oriented LLM showcases superior proficiency in handling complex source language sentences, as reflected by higher perplexity scores.
Case Study. In Table 2, we provide several examples that are hard for GoogleMT to handle but are solved well by TIM. It shows that the NMT system struggles to understand the meaning of some professional terms and fails to produce suitable translations for ill-informed text. In contrast, MToriented LLM demonstrates its superiority in handling such issues, which can be attributed to its enhanced ability to comprehend rare, specialized words, and informal texts.

heading: 3.4 Discussion
text: The analysis in Section 3.2 demonstrates the accuracy of commercial NMT systems, likely due to their extensive training and cross-attention capability. MT-oriented LLMs, known for their paraphrastic nature, can further enhance NMT’s ability to handle figurative text translations. Moreover, MToriented LLMs excel on WebCrawl test sets (Section 3.3), particularly with specialized terminology and ill-formed sentences. This suggests that an effective hybrid framework combining NMT and LLMs can handle challenging input domains and figurative text. Based on these insights, we propose to investigate an effective hybrid framework
10https://huggingface.co/gpt2-large
to answer the second question: How can we effectively harness the capabilities of LLMs to enhance translation quality?
Hybrid Threshold (Hendy et al., 2023) employs the NMT model as the primary translation system, with a quality estimation module (e.g., COMETkiwi) to assess the translation. If the quality falls below a certain threshold, the GPT-like model is used as an alternative translation engine. However, it faces two main challenges: autoregressive decoding latency and reliable quality estimation. The practical implementation of the hybrid approaches must ensure efficient decoding and high translation quality.

heading: 4 Cooperative Decoding
text: We propose an innovative cooperative decoding approach. This approach leverages an NMT model as a pretranslation model and incorporates an MToriented LLM as a quality assessment module and fallback model if needed. The overview of cooperative decoding is shown in Figure 6 and we will give a detailed description of each step in the following.
Step1: Draft Generation. Given an input source sentence x, the NMT system generates the translation o using an autoregressive decoding strategy like beam search. The difference is that the transla-
tion o is considered as a draft and requires further confirmation or modification before being used as the final output.
Step2: Verification. We feed o into the MToriented LLM in a forward process, which fully utilizes parallel computing. The procedure is the same as training LLMs, and we can obtain a probability distribution vt at each position, which is modeled as P (ot|o<t, x). The distribution can be regarded as the confidence of the LLM given the specific prefix of the draft o, and we use it to verify the prediction of the NMT model. One straightforward approach for verification is to check whether the token with the highest probability matches the prediction of NMT. If o is fortunately exactly the same with {argmax(v1), ..., argmax(vn)}, the inference will finish with o as the final translation. However, high-quality generation does not follow a distribution of the highest probability of the next tokens, and the tokens in o that can be regarded as accurate may appear outside of the top-1 selection, like in beam search. To address this issue, we relax the matching constraint using the top-k candidates of the LLM and define the verification criterion as
ot ∈ top−k(vt). (1)
ZH⇒EN EN⇒ZH Method COMET COMETk. Token/s Speedup Ratio COMET COMETk. Token/s Speedup Ratio
Step3: Re-decoding. The verification is performed from left to right, and we end the verification once there is a situation that does not meet the verification criteria, i.e., ot′ /∈ top−k(vt′). Then, we feed the verified prefix ot′−1 into the MT-oriented LLM and use it to re-decode the subsequent sequence. Compared to totally replacing NMT models with MT-oriented LLMs, our cooperative decoding can speed up the whole inference process due to the expensive cost of autoregressive decoding. The speedup is more significant when the longer draft is accepted. Moreover, the cooperative mechanism alleviates the issue of inaccuracy of LLMs by exploiting the output of NMT models.

heading: 5 Experiments

heading: 5.1 Main Results
text: We merge the WMT22 and WebCrawl test sets to simulate the distribution of translation requests in real-world scenarios. For CoDec, we use GoogleMT as the NMT system, and TIM as the MT-oriented LLM. In particular, we set the threshold as the 50th percentile of COMETkiwi scores of GoogleMT (Hendy et al., 2023). We use the MT-oriented LLM to generate the translation only when the COMETkiwi score of the NMT translation is under the threshold. We use beam search with a beam size of 4 for TIM during inference. The decoding and speed measurement processes are performed on a single A100 GPU.
Effect of different values of k. Intuitively, as k increases, cooperative decoding can accept a wider range of tokens in NMT translations during the verification stage. As a result, less content needs to be re-decoded by LLMs, leading to a reduction in processing time. Here, we examine the performance
of CoDec under various values of k. As shown in Table 4, with the increase of k, the ratio of tokens accepted on average and the decoding speed increase consistently. With a larger k, CoDeC-128 achieves a 3.4x and 2.2x speedup over TIM in ZH⇔EN. This signifies that CoDec effectively reduces decoding latency while maintaining translation quality. Besides, our CoDec-(*) models exhibit superior performance compared to both GoogleMT and TIM. This highlights the potential of cooperative decoding in improving translation accuracy and overall system performance. Moreover, models with lower values of k, such as CoDec-8, achieve better translation quality, suggesting that the choice of k should be considered to strike a balance between performance and efficiency.
CoDec vs. Hybrid Threshold. In our comparison between CoDec and Hybrid Threshold, we utilize different Quality Estimation (QE) methods, including HT(Random), where 50% of GoogleMT’s translations are randomly replaced with TIM’s translations, HT(BLEURT-12), which uses BLEURT-20-D1211 as the QE method; HT(BLEURT-20), which employs BLEURT-2012 as the QE method; and HT(COMETk.). Additionally, CoDec is integrated into the Hybrid Threshold pipeline as a comparative system, referred to as HT(COMETk.) w/ CoDec. Furthermore, we follow Hendy et al. (2023) to use Hybrid MaxRouting to establish an upper bound by selecting the best translation from either system based on the COMETkiwi.
The performance comparison in Table 5 reveals a notable performance disparity between GoogleMT
11https://huggingface.co/lucadiliello/BLEURT-20-D12 12https://huggingface.co/lucadiliello/BLEURT-20
and Max-Routing. This result supports our assertion that MT-oriented LLMs can play a crucial role as reliable fallback systems for NMT systems. Moreover, the different QE modules employed in Hybrid Threshold yield varying performances, highlighting the dependence of Hybrid Threshold’s performance on the precision of the QE modules and the quality of LLM translations used as replacements. In contrast, CoDec-8 surpasses most of the Hybrid Threshold models and achieves competitive results with HT(COMETk.) w/ CoDec, suggesting that the QE modules may not be necessary. The findings validate that our approach achieves a better balance between efficiency and effectiveness, resulting in enhanced translation quality without compromising system efficiency.

heading: 5.2 Human Evaluation
text: In addition, we carry out a human evaluation on the WebCrawl EN⇒ZH dataset. A total of 300 sentences are randomly selected from the test set, and two individuals are asked to evaluate the translations produced by GoogleMT, HT(COMETk.), and our CoDec-8. We use the commonly used pairwise comparison method to count the number of better, similar, and worse translations from System 1 rather than System 2. The result of CoDec vs. GoogleMT is 144:115:41, while the result of CoDec vs. HT(COMETk.) is 106:130:64. It shows that our CoDec significantly outperforms the commercial NMT system and performs better than the Hybrid Threshold without an additional quality evaluation module.

heading: 5.3 Terminology Translation
text: Unlike conventional NMT models, MT-oriented LLMs enable them to exploit instructions to handle various translation scenarios. Here, we apply CoDec to assess the effectiveness of incorporating instructions in a dedicated terminology translation test set obtained from WMT2313. The result is shown in Table 6, evaluated by COMET, ChrF, and Terminology Success Rate. The data statistics and details of baselines can be found in Appendix D.
The results indicate that the use of terminology information in instructions, as demonstrated by TIM w/ term, enables MT-oriented LLMs to achieve constrained machine translation, resulting in more accurate domain-specific terminology in the translated output. When compared to Lingua Custodia and UEDINLLM (Semenov et al., 2023), CoDec-8 combines the advantages of higher translation quality offered by NMT and the constrained translation capabilities of MT-oriented LLMs. This combination leads to higher-quality translations while maintaining a higher Terminology Success Rate.

heading: 6 Conclusion
text: We explore the strengths of both NMT and LLM and propose CoDec that integrates the two to achieve superior performance compared to existing hybrid frameworks. Notably, our CoDec offers reduced decoding latency compared to relying solely on LLMs for inference, and it does not require any modifications to the target LLMs. We believe that exploring more effective utilization of LLMs while considering practicality in both industry and academia is a valuable direction.
13https://wmt-terminology-task.github.io/

heading: 7 Limitations
text: This paper primarily concentrates on enhancing translation performance for medium and highresource language pairs. Further investigation is required to analyze the translation characteristics of different systems in low-resource languages, which we defer to future research.
Additionally, the draft translations were validated by directly utilizing the top-k candidates predicted by the target MT-oriented LLM. We acknowledge that the implementation of more meticulously designed token-level validation methods has the potential to further enhance CoDec, and we consider it as an avenue for future exploration.

heading: A WMT22 test sets
text: To prevent data leakage (Garcia et al., 2023), we analyze the WMT22 test sets, consisting of recent content from diverse domains including news, social media, e-commerce, and conversation. The test sets consist of the following number of samples for each language pair: German-to-English (DE⇒DE) - 1984 samples, English-to-German (EN⇒DE) - 2037 samples, Chinese-to-English (ZH⇒EN) - 1875 samples, English-to-Chinese (EN⇒ZH) - 2037 samples, Russian-to-English (RU⇒EN) - 2016 samples, English-to-Russian (EN⇒RU) - 2037 samples, Japanese-to-English (JA⇒EN) - 2008 samples, English-to-Japanese (EN⇒JA) - 2037 samples.

heading: B Unaligned Source/Target Words.
text: For English and German, we utilize the Moses tokenizer14. We use jieba15 and MeCab16 for Chinese and Japanese, respectively. We use awesomealign17 (Dou and Neubig, 2021) to obtain the word alignments. Unaligned source words (USW) indicate the number of words in the source text that have no corresponding translation in the target sentence. Unaligned target words (UTW) assess the degree to which words are potentially added or inserted into the translation without any basis or support from the source sentence.

heading: C WebCrawl test sets
text: To acquire the data, we follow the process outlined below:
• We extract snippets from web pages and use an in-house sentence segmentation tool to split them into individual sentences.
14https://github.com/mosessmt/mosesdecoder/tree/master/scripts/tokenizer
15https://github.com/fxsjy/jieba 16https://github.com/SamuraiT/mecab-python3 17https://github.com/neulab/awesome-align
• We employ sensitive word filters, language identification tools, length ratio checks, and perplexity scores to filter out sentences of lower quality.
• We utilize Google Translator to obtain translations of the sentences, with a primary focus on the Chinese⇔English directions.
• We calculate COMETkiwi scores and retain sentences with scores below 65.
In this way, we collected a total of 889 Chinese sentences and 1195 English sentences as our final test set, named WebCrawl test sets. We hire 2 annotators who have degrees in English Linguistics to annotate translations. Before formal annotation, annotators were asked to annotate 100 samples randomly extracted from the dataset, and based on average annotation time we set a fair salary (i.e., 30 dollars per hour) for them.

heading: D Terminology Translation
text: Terminology translation is an extensively encountered application scenario, where the NMT (Neural Machine Translation) model is expected to precisely handle the provided domain-specific terminology. In this experiment, we use the prompt “{srcWord} means {tgtWord}. Translate the following sentences from {src} to {tgt}, and muse use the given word translations.{line}” for inference of TIM. The numbers of sentences on Zh⇒En and De⇒En are 2640 and 2963, respectively. The average numbers of terms per segment on Zh⇒En and De⇒En are 3.8 and 1.1, respectively. We only highlight a few systems that achieved the best performance on specific metrics in the competition findings (Semenov et al., 2023). Lingua Custodia, which utilizes a specialized Transformer architecture to ensure the inclusion of given terminology in the translation. Additionally, the UEDINLLM employs ChatGPT with prompts specifically designed for terminology translation.

heading: E Different from traditional NMT with additional language models
text: Traditional language models, such as causal language models are usually used as decoder initialization or reranking to improve fluency. We do not consider the prediction probabilities of LLMs during the decoding process of NMT. Instead, we treat
LLMs as independent translation systems and introduce speculative execution as a fusion approach for NMT systems and MT-oriented LLMs. Notably, we do not introduce additional monolingual training data and our research does not encompass backtranslation. Building upon this insight, we propose a hybrid framework that combines the strengths of both NMT systems and MT-oriented LLMs and alleviates the limitations of individual systems.

heading: F About speedup
text: The time consumption of the Hybrid Threshold is the sum of the inference time for both the NMT systems and the MT-oriented LLM, whereas the CoDec requires only the inference time of the NMT systems and a small amount of calculation of the LLM. Considering the relatively negligible time consumption of Google Translate, we did not specifically factor in its inference time in our analysis, as it does not significantly impact the overall performance comparison.

heading: Future Work
text: Further investigation is required to analyze the translation characteristics of different systems in low-resource languages, which we defer to future research. Additionally, the draft translations were validated by directly utilizing the top-k candidates predicted by the target MT-oriented LLM. We acknowledge that the implementation of more meticulously designed token-level validation methods has the potential to further enhance CoDec, and we consider it as an avenue for future exploration