SCALE: Synergized Collaboration of Asymmetric Language Translation Engines (2024)

Abstract: In this paper, we introduce SCALE, a collaborative framework that connects a compact Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. By introducing translation from STM into the triplet in-context demonstrations, SCALE unlocks refinement and pivoting ability of LLM, thus 1) mitigating language bias of LLMs and parallel data bias of STMs, 2) enhancing LLM speciality without sacrificing generality, and 3) facilitating continual learning in a LLMtuning-free way. Our comprehensive experiments show that SCALE significantly outperforms both LLMs (GPT-4, GPT-3.5) and supervised models (NLLB, M2M) in either highresource or challenging low-resource settings. Moreover SCALE demonstrates the scalability by only updating the lightweight STM and witness consistent system improvement, an averaged 4 BLEURT score across 4 languages without tuning LLM. Interestingly, SCALE could also effectively exploit the existing language bias of LLMs by using an English-centric STM as a pivot to conduct translation between any language pairs, outperforming GPT-4 by an average of 6 COMET points across eight translation directions. Furthermore we provide an in-depth analysis of SCALE’s robustness, translation characteristics, latency costs and inherent language bias, providing solid foundation for future studies exploring the potential synergy between LLMs and more specialized models1.

heading: 1 Introduction
text: Large Language Models (LLMs) have recently revolutionized the field of natural language processing (OpenAI, 2023; Touvron et al., 2023a; Anil et al., 2023; Peng et al., 2023) and brought a paradigm
*Work done when interning at Microsoft, mentored by Tao and Xun. Correspondence Tao Ge, Xun Wang, Dongyan Zhao and Rui Yan.
1Code available at github.com/hannibal046/scale
shift in machine translation (MT) by delivering exceptional performance without relying on bilingual data (Brown et al., 2020; Garcia et al., 2023). Moreover, as a unified multi-task learner, LLMs represent a substantial step towards artificial general intelligence (Bubeck et al., 2023), with the potential to transcend not only the language barriers emphasized by previous MT research but also cultural boundaries (Yao et al., 2023).
Despite their advancements, LLM-based translation systems still confront several challenges. Firstly, there exists a significant language bias towards English (e.g., 92.1% of the GPT-3 pretraining corpus is English, while French, the second largest, represents only 1.8%), which significantly constraints multilingual ability, especially for those low-resource languages (Scao et al., 2022; Hendy et al., 2023; Zhang et al., 2023, 2024). Secondly, as the go-to approach for improved system performance, fine-tuning is non-trivial for LLMs due to (1) the trade-off between speciality and generality (Lin et al., 2023; Cheng et al., 2023a), and (2) the prohibitively high cost associated with tuning large-scale models. In contrast, traditional Specialized Translation Models (STMs)—those based on encoder-decoder architecture, trained with labeled data and significantly smaller in size (Sutskever et al., 2014; Vaswani et al., 2017)—serve as specialists for specific translation tasks and could be efficiently fine-tuned. Nevertheless, due to restricted model capacity, these models exhibit limitations in general language capabilities and may be prone to parallel data bias, such as the memorization of low-quality samples (Raunak et al., 2022).
In this paper, we demonstrate for the first time the possibility to unify these two asymmetric translation engines in a single framework. Our work, SCALE, connects LLMs and STMS by utilizing the LLM’s most enigmatic capability: in-context learning. Rather than employing source-target pairs as in conventional few-shot translation (Garcia et al.,
15903
2023; Vilar et al., 2023), SCALE would first sample translations from a STM and then use triplets as incontext demonstrations consisting of a source sentence, an STM-generated set and a target sentence, which unlocks refinement and pivoting ability of LLMs. With SCALE, we could (1) mitigate both language bias of LLMs by utilizing an STM that concentrates on a specific language pair, and parallel data bias of STMs by using a general-purpose LLM as the main body of the system; (2) enhance the speciality of LLMs without compromising generality; (3) facilitate continual learning within the framework by updating only the lightweight STM, thus avoiding expensive LLM fine-tuning. By employing SCALE, we create a more efficient and effective system that combines the best of both translation engines.
Our comprehensive experiments reveal that SCALE considerably outperforms LLMs (e.g., GPT-4) and STMs (e.g., NLLB) even in the challenging low-resource setting. Moreover, in Xhosa→English direction, SCALE experiences consistent improvement by a 4 BLEURT score without tuning LLM and surpasses few-shot GPT4 by 2.5 COMET score and 3.8 BLEURT score when equipped with a compact model consisting of merely 600M parameters. Interestingly, SCALE could exploit the existing language bias of LLMs by using an English-centric STM as a pivot to conduct translation between any language pairs, outperforming GPT-4 by an average of 6 COMET points across eight translation directions. Furthermore, we conduct an in-depth analysis of the robustness, translation characteristics, latency costs and inherent language bias associated with SCALE. Our findings provide valuable insights and encourage further research in the synergy between LLMs and small specialized models.

heading: 2 The SCALE Framework
text: In this section, we present SCALE and provide an overview illustrated in Figure 1. Popularized by GPT-3 (Brown et al., 2020), In-context Learning (ICL) allows LLMs to perform a wide variety of tasks, even newly created ones (Bills et al., 2023), by providing a few demonstrations. For a translation task from a source language X to a target language Y , an LLM with parameters θ carries out ICL by conditioning on k source-target paired examples E = (x1, y1)⊕(x2, y2)⊕...(xk, yk) and the test source sentence x, generating the target y in an auto-regressive manner as yt ∼ pθ(yt|E, x, y<t). In this scenario, the LLM must analyze the provided examples to discern the input distribution, output distribution, input-output mapping, and formatting to successfully complete the task (Press et al., 2022; Wei et al., 2023). Different from conventional ICL, SCALE introduces an intermediate variable Z as reference between source x and target y, transforming each demonstration example into a triplet (x,Z, y). The variable Z is a generation set sampled from a specialized translation model MX 7→Y trained on a labeled dataset. The final input to the LLM consists of the instruction, demonstrations, and source sentence combined in a prompt template: T ((x1,Z1, y1)⊕(x2,Z2, y2)...⊕ (xk,Zk, yk)), (x,Z)). Unlike language understanding tasks that have fixed label set (Xu et al., 2023), the hypothesis space of translation model is actually infinite, so we could sample multiple generation paths from STM for one single source sentence to provide a more comprehensive generation guide for LLM. The SCALE framework, though conceptually straightforward, demonstrates several advantages over STMs and LLMs:
Refinement For X to Y translation task, when the intermediate variable Z is from MX 7→Y(x),
SCALE essentially conduct few-shot learning in a multi-task way by introducing an additional refinement task. Refinement has long been proved effective in MT (Xia et al., 2017; Cheng et al., 2022). In this refinement process, we pass sampled sentences and their confidence score (token level probability) from STM to an LLM. The LLM then digests the information carried by the sampled set and infers the generation space of the STM, which guides the LLM to generate the output that is more consistent with the local data distribution (Xu et al., 2023). And since the final translation is delivered by an LLM, SCALE could also mitigate the parallel data bias from STMs and exhibit robustness by not merely copying and pasting the draft translation from STMs as shown in §5.3.
Pivoting Considering the predominantly English-centric nature of most LLMs (Brown et al., 2020; Touvron et al., 2023a), SCALE could employ an intermediate variable Z from MX 7→English(x) where the target language Y is not necessarily English. And here Z serves as a pivot point for LLMs to enhance their understanding of the source sentence and yield improved translations. This can also be regarded as a form of knowledge transfer from high-resource languages to low-resource languages (Kim et al., 2019).
Scalability A significant limitation of the existing LLM-based translation systems is the inherent complexity of LLM continual learning. This complexity arises from several factors, including the balance between speciality and generality (Lin et al., 2023), the catastrophic forgetting problem (Yong et al., 2023), and the substantial computational demands. In contrast, SCALE offers a more efficient and streamlined approach for scalability. By exclusively and effectively updating the lightweight MX 7→· component, SCALE ensures that the LLM remains untouched, thus preserving its general language capabilities. This selective updating process not only mitigates the issue of catastrophic forgetting but also reduces the computational burden of fine-tuning associated with large-scale models.

heading: 3 Experimental Setup

heading: 3.1 Dataset
text: Our evaluation datasets encompass a diverse set of languages, spanning both low- and high-resource ones. To facilitate reproducibility, all our evaluation datasets come from publicly available devtest
split of Flores-200 (NLLB Team et al., 2022).

heading: 3.2 Translation Systems
text: We compare our approach with cutting-edge academic systems including both specialized models and LLMs, as well as one commercial system, Microsoft Translator. To our knowledge, these models are among the strongest and most representative of their respective categories. For supervised models, we consider M2M100 (Fan et al., 2021), the first multilingual encoder-decoder translation model that can translate between any pair of 100 languages without relying on English data; NLLB (NLLB Team et al., 2022), a supervised translation model suite capable of delivering high-quality translations directly between 200 languages and remains state-of-the-art performance. For LLMs, we consider: XGLM-7.5B (Lin et al., 2022), a multilingual generative language models; GPT-3.5, a GPT model specially optimized for conversational purpose; GPT-4 (OpenAI, 2023), the latest version of GPT-series.
We use both GPT-3.5 and GPT-4 from Microsoft Azure OpenAI Service. Without further notice, the number of few-shot samples in LLM and SCALE are set to 10 and the sample selection strategy follows Agrawal et al. (2022). The prompt we use could be found in the Appendix A.

heading: 3.3 Evaluation Metrics
text: Because neural metrics have shown higher correlation with human preference (Freitag et al., 2022; Rei et al., 2020) and are widely adopted by recent literatures (Hendy et al., 2023; Garcia et al., 2023), we mainly evaluate our system with (1) COMET-22, a reference-based neural metric (Rei et al., 2022a) combining direct assessments, sentence-level score, and word-level tags from multidimensional quality metrics error annotations, (2) COMETKiwi, a refrence-free quality estimation model from Rei et al. (2022b), and (3) BLEURT (Sellam et al., 2020), a learnable evaluation metric with a regression model trained on ratings data. For completeness, we also include the results of lexical metrics such as spBLEU (NLLB Team et al., 2022) and chrF++ (Popovic, 2017).

heading: 4 Experimental Results
text: In this section, we conduct various experiments to show the effectiveness of our framework. In §4.1, we verify the effectiveness of the SCALE refinement ability. In §4.2, we focus on non-English
pairs to test the pivoting ability of SCALE. In §4.3, we show the scalability of our framework with a fixed LLM and an evolving STM.

heading: 4.1 SCALE Refinement
text: Although LLMs perform comparably with supervised models on high-resource languages, they still struggle with languages with very limited resource (Garcia et al., 2023; Vilar et al., 2023). To validate the generality of our framework, we evaluate in both low- and high-resource setting. For highresource language, we include English (eng_Latn), Czech (ces_Latn) and Chinese (zho_Hans); for lowresource ones, we include Khmer (khm_Khmr) and Nepali (npi_Deva). We adopt three kinds of baselines as described in §3.2. For supervised NLLB model suite, we choose the NLLB-3.3B version, and for SCALE-refine, the LLM is GPT-4 and the STM is NLLB-3.3B for fair comparison.
The low-resource results are displayed in Table 1. As observed, few-shot LLMs, including GPT-4, significantly trail behind specialized models in all translation directions. In contrast, our framework, by combining LLMs and STMs, demonstrates superior performance over few-shot GPT-4 by an absolute 5 COMET score on average, and surpasses the strong NLLB model in 4/4 directions. The high-resource results are shown in Table 2, leading to the following observation: (1) different from low-resource ones, the few-shot GPT-4 already surpasses supervised counterparts (NLLB and M2M100) by a significant margin; (2) SCALE continues to offer improvements, albeit less sub-
stantial than those observed in low-resource settings; (3) SCALE exhibits strong robustness when paired with a less performant STM (especially in eng_Latn→zho_Hans direction).

heading: 4.2 SCALE Pivoting
text: In this section, we demonstrate the performance of SCALE-pivot, in which the variable Z is not directly pertinent to the current translation directions but functions as a pivot. Specifically, we examine the performance of few-shot GPT-4 and SCALEpivot on Lao→ Y translations, where Y represents a language set encompassing both low-resource and high-resource languages. For the low-resource languages, we include Assamese (asm_Beng), Armenian (hye_Armn), Amharic (amh_Ethi), Xhosa (xho_Latn), and we have German (deu_Latn), Czech (ces_Latn), Bulgarian (bul_Cyrl) and Greek (ell_Grek) for the high-resource setting.
The results are presented in Figure 2. Upon examining the GPT-4 performance in isolation, it is evident that the inherent language bias has a considerable impact on translation performance. In particular, the GPT-4 model generally performs well in high-resource settings; however, it tends to struggle in low-resource scenarios. Moreover, our findings highlight that employing SCALE-pivot can effectively enhance the performance of GPT-4 across both low- and high-resource settings. Interestingly, the performance gains achieved through SCALEpivot are more pronounced in high-resource settings, with an average improvement of 6.8 COMET22 score compared to 5.2 for low-resource set-
tings. This outcome suggests that incorporating SCALE-pivot can further boost the already strong performance of GPT-4 in high-resource situations, while also providing a notable improvement in lowresource contexts.

heading: 4.3 SCALE Scalability
text: In this section, we explore the scalability of our framework by keeping the LLM fixed and solely updating the STM. Specifically, we use M2M100-12B and NLLB model suite ranging from 600M to 3.3B as our evolving STM. We conduct experiments on the Xhosa → English direction and adopt the prompt format of SCALE-refine. The experimental results are displayed in Figure 3, leading to the following observations: (1) The overall framework can be consistently improved with a fixed LLM and a continuously evolving STM; (2) SCALE, when equipped with a small model containing only 600M parameters, can outperform GPT-4 with an absolute 2.5 COMET-22 score and a 3.8 BLEURT score; (3) Equipped with an STM (M2M100) of
relatively lower performance than original few-shot GPT-4, SCALE demonstrates strong robustness by not merely copying and pasting the less satisfactory reference answer provided by M2M100, which we detailedly investigated in §5.3.
Interestingly, we also observe that the growth patterns exhibited by lexical metrics and neural semantic metrics differ. For M2M100 and NLLB600M as STM, both metrics experience substantial improvement, while for NLLB-1.3B and 3.3B as STM, SCALE maintains the same lexical accuracy while continually enhancing translation performance as measured by neural semantic metrics.

heading: 5 Further Analysis

heading: 5.1 Translation Characteristics
text: To gain a deeper understanding of the translation characteristics of different systems (LLMs, STMs, and SCALE) beyond overall translation quality, we employ the following measurements, as suggested by Hendy et al. (2023):
1. Translation Fluency: Since LLMs are optimized by predicting the next token, their translations tend to display a language modeling bias that favors fluency over adequacy. To investigate this, we utilize an independently trained open-source language model (Llama2-13B (Touvron et al., 2023b)) to measure the perplexity score of the translation output.
2. Translation Non-Monotonicity: This metric evaluates the extent to which a translation adheres to the source sentence’s structure, calculating the deviation from the diagonal in the word-to-word alignment. Translations that are more paraphrastic or less literal tend to deviate from closely tracking the source word order across language pairs (Hendy et al., 2023). We apply the non-monotonicity metric proposed by Schioppa et al. (2021).
3. Unaligned Source Words: Another measure
of literalness is the count of unaligned source words (Hendy et al., 2023). When accounting for quality, less literal translations are likely to include more words that do not align with those in the source sentence.
We present the Translation Fluency results of X → English translation in Figure 4, where X remains the same as used in Section 4.1. It is evident that regardless of the translation quality delivered by the LLM, whether superior (SCALE) or inferior (GPT-4) compared to the STM (NLLB), the LLM translation generally demonstrates higher fluency than the STM. Additionally, in 6 out of the 8 languages examined, SCALE produces lower perplexity scores than the original GPT-4 output. This suggests that the STM-generated variable Z can effectively aid the GPT-4 model in further decreasing its generation uncertainty.
For Non-Monotonicity and Unaligned Source Words, we choose Xhosa→English translation

heading: 1 80.4 73.2 35.6

heading: 2 81.2 74.3 37.1
text: with different STMs, and the results are shown in Figure 5. We also include PPL score for completeness. We find that both the USW and NM scores for STM are higher than those of GPT-4. This indicates that even though STM provides higher translation quality, it results in less literal translations. However, for SCALE, it effectively reduces GPT-4’s NM score while maintaining a moderate USW score. This suggests that during the SCALE refinement process, the model primarily adheres to the original LLM output structure while taking cues from STM’s word selection. We also show several concrete cases in Appendix C.

heading: 5.2 Multipath Sampling
text: In this section, We list the results of multi-path sampling strategy in Table 3. We test with Xhosa→English with one-shot SCALE-refine. The results show that without increasing the shot number in the few-shot learning, using STM to generate more generation paths could consistently improve the overall performance, which could be useful in the extremely low-resource setting where demonstration samples are hard to acquire.

heading: 5.3 Ablation
text: In this section, we conduct an ablation study for each key design in our framework. We examine the following variants: (1) without confidence: This model follows the same setting as the SCALErefine in §4.1, except that we do not pass the confidence score of each token as input. (2) zero-shot: This variant removes all in-context-learning examples, keeping only the translation instruction and the reference answer from STM. (3) one-shot: This model utilizes only one-shot, in contrast to the tenshot results presented in §4.1. (4) zero-shot-M2M: This model also implements zero-shot, but the STM used is M2M100, a less performant model than the
original few-shot GPT-4. This is employed to assess the robustness of our framework.
The outcomes of our ablation study are showcased in Table 4. It is evident that each component in our framework perform effectively, with the incontext-learning setting providing the most performance gain. This indicates that simply offering a reference answer to the LLM without in-context samples does not adequately guide the model in utilizing those references effectively. Furthermore, the number of ICL examples is also an essential factor in the process.
Regarding the SCALE zero-shot-M2M variant, its performance is significantly inferior to that of the few-shot LLM due to the poor quality of the M2M100 output. From this observation, we can conclude that the robustness of SCALE, as illustrated in Figure 3, primarily stems from the power of in-context learning. This learning approach informs the LLM about which elements to trust and which to disregard, ultimately improving the overall translation performance and robustness.

heading: 5.4 Generation Latency
text: In this section, we conduct a detailed evaluation of the overhead introduced by SCALE in comparison to few-shot LLM. The additional latency arises from two factors: first, the time required to generate the variable Z for the current source sentence x using STM, and second, the increased latency caused by the LLM due to the extended context.
We utilize one of the largest open-source LLMs (BLOOM-176B) for this analysis. As shown in Table 5, we observe that the incurred latency can be primarily attributed to the extended context window due to the quadratic time complexity of the transformer. Exploring methods to accelerate this process based on STM-generated output using speculative decoding techniques remains future work (Xia et al., 2022; Yang et al., 2023).

heading: 5.5 Inherent Language Bias
text: In this section, we investigate whether the inherent language bias of LLM could be alleviated by combing output from a specialized model. Due to the limited transparency of the GPT-4 model, we turn to a potential indicator, the number of native speakers, to illustrate the extent of a language’s resources. We tested on four languages following the setting of SCALE-refine. As shown in Table 6, the performance of few-shot GPT-4 diminishes with the number of native speakers, while our framework, SCALE, consistently and effectively mitigates this language bias, outperforming both few-shot GPT-4 and the supervised NLLB model.

heading: 6 Related Work
text: The use of LLM for translation tasks has garnered significant interest in recent times. Brown et al. (2020) initially demonstrated the efficacy
of prompting an LLM with a few examples to achieve noteworthy results, particularly in highresource languages (Vilar et al., 2023; Lin et al., 2022). Following the release of ChatGPT, several studies have examined its overall translation performance(Jiao et al., 2023; Hendy et al., 2023), along with works focusing on the issue of hallucination (Guerreiro et al., 2023) , literalness (Raunak et al., 2023a), multilinguality (Zhu et al., 2023) and incidental bilingualism problem (Briakou et al., 2023). A comprehensive analysis conducted by Garcia et al. (2023) revealed the unreasonable effectiveness of few-shot LLMs. Furthermore, a diverse range of research has attempted to enhance LLM-based translation systems through cultural awareness (Yao et al., 2023), refinement (Chen et al., 2023), retrieval-augmentation (Cheng et al., 2023b), post-editing (Raunak et al., 2023b), and comparison (Zeng et al., 2023).
Our work also shares similarities with a series of studies aiming to build collaboration between LLMs and other systems. Luo et al. (2023) propose equipping LLMs with a knowledge-guiding module to access relevant information without tuning LLM. Hendy et al. (2023) propose to use Microsoft Translator system as the primary translation system, and then use GPT as a fallback system. Xu et al. (2023) introduce SuperICL and achieve significant improvements in various language understanding tasks.

heading: 7 Conclusion
text: In this paper, we present a novel framework SCALE, which effectively combines the strengths of Large Language Models (LLMs) and compact Specialized Translation Models (STMs) through in-context learning. By providing triplet in-context demonstrations, SCALE unlocks the refinement
and pivoting capabilities of LLMs, demonstrated by comprehensive experiments in various settings. Our results offer crucial understanding for subsequent research investigating the possible synergy between LLMs and more specialized models.

heading: 8 Limitations
text: In this paper, we acknowledge the following limitations and strive for improvement as our future work:
(1) While SCALE has demonstrated considerable advancements over both LLMs and STMs across diverse scenarios, our evaluation has predominantly concentrated on the GPT-series as a black-box model. To comprehensively investigate the underlying mechanisms of SCALE, we aim to extend our research to future developments involving powerful multilingual LLMs with fully transparent architectures, weights, and training data distribution.
(2) Although SCALE is the first work to combine LLM and STM into a unified framework, the interaction between these two elements is on the prompting level. Future work will explore more sophisticated integrations, such as applying knowledge distillation from LLMs to STMs, to enhance the synergy between these two components.
(3) The introduction of extended contexts in SCALE is an inevitability that may present significant challenges for systems where response time is critical. Developing strategies to accelerate this process, such as using the output from STMs to perform online speculative decoding, remains an area for further investigation and improvement.

heading: A Prompt Example
text: In Table 7, we list the prompt we use for few-shot LLM and in Table 8, for our SCALE framework. We use Chat Markup Language version from Azure to format our prompt2.

heading: B Data Statistics
text: We list the detailed data information for SCALErefine and SCALE-Pivot experiments in Table ??. The number of dev set is 997 and 1012 for devtest set in flores-200 (NLLB Team et al., 2022).

heading: C Translation Cases
text: In this section, we list several translation cases from different languages in Figure 6, 7, 8, 9.

heading: D More languages covered with SCALE-pivot
text: In addition to using Lao as the source language for translations into Assamese, Armenian, Amharic, Xhosa, German, Czech, Bulgarian, and Greek with SCALE-pivot, we demonstrate the versatility of our method by also testing Xhosa as the source language. The results are depicted in Table 10, which exhibit similar patterns with Lao as source languages.

heading: E More languages covered with SCALE-update
text: In addition to using Xhosa as the source language for translations into English with SCALE-update, we demonstrate the versatility of our method by also testing Lao, Assamese and Amharic as the source language. The results are depicted in Table 11, which exhibit similar patterns with Xhosa as source languages.
2https://learn.microsoft.com/en-us/azure/ ai-services/openai/how-to/chatgpt?pivots= programming-language-chat-ml
Under review as a conference paper at ICLR 2024 SOURCE የዳይነሶር ላባዎች የዳበረ ራቺስ የሚባል ዘንግ ስለሌለው፣ ነገር ግን ሌሎች የላባ ባህርያት — ባርብስ እና ባርቡልስ — ስላለው ተመራማሪዎች ራቺስ ከእነዚህ ሌሎች ባህርያት የቆየ ዝግመተ ለውጥ ውጤት እንደሆነ ይላሉ። TARGET Because the dinosaur feathers do not have a well-developed shaft, called a rachis, but do have other features of feathers — barbs and barbules — the researchers inferred the rachis was likely a later evolutionary development that these other features. MS Translator Dinosaur feathers developed because it doesn’t have a rod called rachis, but has other feather traits — barbs and barbules — that researchers say is the result of older evolution of rachis from these other traits. NLLB dinosaur feathers did not develop a shaft called the rachis, but other feather features, such as barbs and barbels, suggest that the rachis was the result of an earlier evolution of these other features. GPT-4 As there is no known population of the extinct Laysan Rail on Laysan Island, researchers suggest that the presence of rails on the other islands—Barbados and Barbuda—indicates a prolonged period of isolation and change.
SCALE Dinosaur feathers did not develop a shaft called the rachis, however, other feather features such as barbs and barbules suggest that the rachis was the result of an earlier evolution of these other features.
Table 9: Translation case from Amharic to English.
SOURCE बाइसन, एल्क, मूस, भालु र लगभग सबै ठूला जनावरहरूले जस्ता नरम देखिए पनि आक्रमण गर्न सक्छन्।
TARGET No matter how docile they may look, bison, elk, moose, bears, and nearly all large animals can attack.
MS Translator Bison, elk, moose, bears, and almost all large animals can attack even if they look soft.
NLLB The Bible says: ”The one who is walking with wise persons will
become wise, but the one who is having dealings with the stupid
ones will fare badly”.
GPT-4 Bison, elk, moose, bears, and nearly all large animals, despite appearing gentle, can be aggressive.
SCALE Bison, elk, moose, bears and nearly all large animals can attack even though they appear docile.
Table 10: Translation case from Nepali to English.
Figure 6: Translation case from Nepali→English.
Under review as a conference paper at ICLR 2024
A.2 Data Statistics
We list the detailed data information for SCALE-refine and SCALE-Pivot experiments in
Table A.2. The number of dev set is 997 and 1012 for devtest set in flores-200 (NLLB Team et al., 2022).
code language # dev length # devtest length script family resource asm_Beng Assamese 40.55 41.67 Bengali Indo-Europea low hye_Armn Armenian 43.91 45.31 Armenian Indo-European low amh_Ethi Amharic 38.87 39.64 Ge’ez Afro-Asiatic low
xho_Latn Xhosa 35.31 36.37 Latin Atlantic-Congo low
uig_Arab Uyghur 40.77 42.41 Arabic Turkic low
TARGET Stirrups are supports for the rider’s feet that hang down on either
side of the saddle.
MS Translator The legged rickshaw helps to keep the driver’s leg which hangs down on either side of the horse’s mattress.
NLLB The foot rest helps to keep the rider’s feet which are sloping downwards on both sides of the horse’s saddle. GPT-4 A heavily loaded Rickshaw helps balance the load by tilting to both sides when going over bumps.
SCALE The stirrup helps to support the rider’s feet, which are sloping downwards on both sides of the horse’s saddle.
Table 8: Translation case from Assamese to English.
Under review as a conference paper at ICLR 2024
MS Translator Bison, elk, moose, bears, and almost all large animals can attack
GPT-4
SCALE
Es gibt eine Chance, dass es genauso verschwindet, wie es aussieht, als ob es einfach verschwindet. Es ist auch nützlich, einen Ring zu tragen, nur scheint der Ring zu teuer zu sein.
SOURCE बाइसन, एल्क, मूस, भालु र लगभग सबै ठूला जनावरहरूले जस्ता नरम देखिए पनि आक्रमण गर्न सक्छन्।
TARGET Auch das Tragen eines Rings ist hilfreich (nur keinen, der zu teuer aussieht
Figure 7: Translation case from Lao to German.
Figure 7: Translation case fro ssamese→English.
15917
features.
Table 9: Translation case from Amharic to English.
SOURCE बाइसन, एल्क, मूस, भालु र लगभग सबै ठूला जनावरहरूले जस्ता नरम देखिए पनि आक्रमण गर्न सक्छन्।
TARGET No matter how docile they may look, bison, elk, moose, bears, and nearly all large animals can attack.
MS Translator Bison, elk, moose, bears, and almost all large animals can attack even if they look soft.
NLLB The Bible says: ”The one who is walking with wise persons will become wise, but the one who is having dealings with the stupid ones will fare badly”. GPT-4 Bison, elk, moose, bears, and nearly all large animals, despite appearing gentle, can be aggressive.
SCALE Bison, elk, moose, bears and nearly all large animals can attack even though they appear docile.
Table 10: Translation case from Nepali to English.
Figure 8: Translation case fro haric→English.
GPT-4
SCALE
Es gibt eine Chance, dass es genauso verschwindet, wie es aussieht, als ob es einfach verschwindet. Es ist auch nützlich, einen Ring zu tragen, nur scheint der Ring zu teuer zu sein.
SOURCE बाइसन, एल्क, मूस, भालु र लगभग सबै ठूला जनावरहरूले जस्ता नरम देखिए पनि आक्रमण गर्न सक्छन्।
TARGET Auch das Tragen eines Rings ist hilfreich (nur keinen, der zu teuer aussieht
Figure 9: Translation case r Lao→German.

heading: Future Work
text: In this paper, we acknowledge the following limitations and strive for improvement as our future work:
(1) While SCALE has demonstrated considerable advancements over both LLMs and STMs across diverse scenarios, our evaluation has predominantly concentrated on the GPT-series as a black-box model. To comprehensively investigate the underlying mechanisms of SCALE, we aim to extend our research to future developments involving powerful multilingual LLMs with fully transparent architectures, weights, and training data distribution. (2) Although SCALE is the first work to combine LLM and STM into a unified framework, the interaction between these two elements is on the prompting level. Future work will explore more sophisticated integrations, such as applying knowledge distillation from LLMs to STMs, to enhance the synergy between these two components. (3) The introduction of extended contexts in SCALE is an inevitability that may present significant challenges for systems where response time is critical. Developing strategies to accelerate this process, such as using the output from STMs to perform online speculative decoding, remains an area for further investigation and improvement