LANGBRIDGE: Multilingual Reasoning Without Multilingual Supervision (2024)

Abstract: We introduce LANGBRIDGE, a zero-shot approach to adapt language models for multilingual reasoning tasks without multilingual supervision. LANGBRIDGE operates by “bridging” two models, each specialized in different aspects: (1) one specialized in understanding multiple languages (e.g., mT5 encoder) and (2) one specialized in reasoning (e.g., Orca 2). LANGBRIDGE connects the two models by introducing minimal trainable parameters between them. Despite utilizing only English data for training, LANGBRIDGE considerably enhances the performance of language models on low-resource languages across mathematical reasoning, code completion, logical reasoning, and commonsense reasoning. Our analysis suggests that the efficacy of LANGBRIDGE stems from the language-agnostic characteristics of multilingual representations. We publicly release our code and models.1

heading: 1 Introduction
text: Language models (LMs) are known to exhibit inferior performance in solving reasoning tasks such as math or coding in low-resource languages (Shi et al., 2023; Qin et al., 2023). This tendency primarily stems from the fact that LMs are predominantly trained on corpora comprised of a few highresource languages (Touvron et al., 2023a,b). This results in low-resource languages being represented as long-tail knowledge (Lazaridou et al., 2021; Kandpal et al., 2023).
Prior works have mainly approached this problem by adapting English-centric LMs to other languages through continual training on the target language (Marchisio et al., 2023; Oba et al., 2023; Zhu et al., 2023; Kew et al., 2023). However, scaling this approach to a large number of languages is challenging, as it requires targeted training corpora for each language. This issue is particularly
1github.com/kaistAI/LangBridge
pronounced for LMs such as MetaMath (Yu et al., 2023) and Orca 2 (Mitra et al., 2023), which have undergone continuous domain-specific adaptation from Llama 2 (Touvron et al., 2023b). These specialized, domain-specific datasets are typically in English, complicating multilingual support for the underlying LM.
In this paper, we introduce LANGBRIDGE, a novel approach that adapts LMs to solve multilingual reasoning tasks without explicitly training on multilingual data. Inspired from the multimodal literature that integrates two independently pretrained modalities (Alayrac et al., 2022; Li et al., 2023a; Merullo et al., 2023; Liu et al., 2023b; Bavishi et al., 2023), we leverage the encoder from mT5 (Xue et al., 2021) and introduce a small number of trainable parameters between the encoder and the target LM. Most importantly, our approach does not require multilingual supervision and solely relies on English data while generalizing to multiple languages during test time, resembling zero-shot cross-lingual transfer (Pires et al., 2019; Conneau
7502
et al., 2020; Xue et al., 2021). We demonstrate the effectiveness of LANGBRIDGE by applying our method to LMs specialized in diverse reasoning tasks of mathematical reasoning, code completion, logical reasoning. Our empirical results show LANGBRIDGE substantially enhances the multilingual reasoning performance of LMs. For example, LANGBRIDGE applied to MetaMath-13B leveraging mT5-XL encoder (2.2B) boosts the average accuracy on MGSM (Shi et al., 2023) from 40.5% to 53.5%, matching the performance of PaLM-540B (Chowdhery et al., 2023), which stands at 51.3%. We observe LANGBRIDGE also significantly boosts LM performance on reasoning datasets that require intrinsic linguistic understanding such as specific subtasks of Big-Bench Hard (Suzgun et al., 2023) and XCOPA (Ponti et al., 2020).
We hypothesize that the effectiveness of LANGBRIDGE is anchored in the language-agnostic characteristics of multilingual representations (Pires et al., 2019; Libovický et al., 2020). By mapping these representations to the target LM’s input space, we conjecture that the LM is able to grasp the semantics of these representations. Since these representations are language-neutral, understanding them allows the LM to become less dependent on the specific language of the input, thereby enabling it to tackle tasks in languages it rarely encountered during pretraining. Our empirical analysis of LANGBRIDGE, using principal component analysis (PCA) and qualitative methods, supports this hypothesis.

heading: 2 Related Work

heading: 2.1 English-centric Language Models
text: Previous works have enhanced the reasoning capabilities of LMs in mathematical reasoning, code generation, and logical reasoning (Mukherjee et al., 2023; Azerbayev et al., 2023; Yu et al., 2023; Rozière et al., 2023; Mitra et al., 2023). However, the majority of these LMs are derived from Englishcentric LMs (Touvron et al., 2023a,b) and adapted with English domain-specific datasets. Consequently, these LMs inherit limited proficiency in low-resource languages, which results in a significant performance discrepancy between highresource and low-resource languages. The motivation of our proposed method, LANGBRIDGE, is to narrow this gap.

heading: 2.2 Zero-shot Cross-lingual Transfer
text: Multilingual models demonstrate remarkable zeroshot cross-lingual transfer capability (Conneau et al., 2020; Xue et al., 2021; FitzGerald et al., 2023). This ability significantly aids the lowresource language community, allowing multilingual models to handle tasks across a wide range of languages after being finetuned on high-resource languages. Our method, which leverages multilingual models, exhibits a similar ability, as it can enhance the reasoning capabilities of LMs across multiple languages while solely relying on English data during adaptation.

heading: 2.3 Aligning Pretrained Representations
text: The concept of combining independently pretrained representations has been widely explored in the realm of cross-modal alignment (Alayrac et al., 2022; Li et al., 2023a; Merullo et al., 2023; Liu et al., 2023b; Bavishi et al., 2023). These works focus on aligning visual encoder models with LMs to facilitate the visual comprehension abilities of LMs. In a concurrent work, Bansal et al. (2024) align two large language models to augment each other’s capabilities. One of their experiments showcases that aligning a multilingual LM with another LM can lead to improved performance in multilingual tasks. However, in contrast to our method, their approach employs multilingual supervision for aligning.

heading: 3 LANGBRIDGE

heading: 3.1 Hypothesis
text: Previous works show that representations of multilingual encoder models are moderately languageagnostic (or language-neutral) out-of-the-box, facilitating zero-shot cross-lingual transfer (Pires et al., 2019; Libovický et al., 2020). Based on this premise, we hypothesize that by aligning a multilingual encoder model to the space of an LM, the LM will be able to understand the semantics of the languages supported by the multilingual encoder without training on a large set of languages.
To test this hypothesis, we align multilingual encoder models with LMs using only English corpora (Left of Figure 2). Then, we evaluate the aligned models using non-English inputs to determine if the LMs exhibit enhanced capabilities in multilingual tasks (Right of Figure 2).

heading: 3.2 Model Architecture
text: Building on the findings of previous works, where effective cross-modal understanding has been achieved by mapping representations from nonlinguistic modalities to the soft prompts (Lester et al., 2021) of LMs (Merullo et al., 2023; Liu et al., 2023b; Bavishi et al., 2023), LANGBRIDGE maps the final hidden states of multilingual encoders to the soft prompts of LMs to attain multilingual understanding of the target LM. Following the studies above, we adopt the setting of using a single linear layer as the mapping function and append one trainable token to the end of the soft prompt as an [EOS] (end of sequence) token. Therefore, given the input tokens (padded if necessary) of the encoder Xenc, the derived soft prompt Henc is equivalent in sequence length to Xenc + 1, and has the same dimensionality as the hidden state of the language model. Any tokens in Henc that originate from padding tokens of Xenc are masked for the target LM. We ablate the effect of using more complex architectures in Appendix D.
Language modeling objective of LANGBRIDGE resembles that of the “prefix LM” explored by Raffel et al. (2020), as the encoder input tokens Xenc can be interpreted as prefix tokens upon which the target tokens Xlm is conditioned. Formally, given the encoder input tokens Xenc, the language modeling likelihood of the target tokens Xlm is denoted as:
p(Xlm|Xenc) = L∏
i
p(xi|Xenc, x<i) (1)
where L is the sequence length of Xlm, and xi is the ith token of Xlm.

heading: 4 Main Experiments

heading: 4.1 Overview
text: We select four task categories for our experiments: (1) mathematical reasoning, (2) code completion, (3) logical reasoning, and (4) commonsense reasoning.2 For each task category, we apply LANGBRIDGE to LMs specialized in reasoning derived from Llama 2 (Touvron et al., 2023b), such as MetaMath or Orca 2.3 We evaluate the models on existing multilingual benchmarks (e.g., MGSM) or translated English benchmarks (e.g., translations of HumanEval (Chen et al., 2021)). As the evaluation tasks necessitate both multilingual understanding abilities and advanced reasoning capabilities, this complexity poses a significant challenge for general multilingual LMs and English-centric LMs specialized in reasoning. On the contrary, models aligned with LANGBRIDGE could take advantage of both.
Since Touvron et al. (2023b) disclose the language distribution of the pretraining data of Llama 2, this enables us to identify which languages are underrepresented in LMs initialized from Llama 2 weights. Throughout the paper, we classify a language as underrepresented if it comprises less than 0.1% of Llama 2’s pretraining data.
In all of our experiments, we use the encoders of mT5 (Xue et al., 2021) as the multilingual encoders due to their availability across a wide range of parameters,4 and their support for longer input sequences compared to other multilingual encoder
2Experiments on commonsense reasoning can be found in Appendix A.1.
3We test LANGBRIDGE on general-domain LMs (Llama 2 and Mistral 7B (Jiang et al., 2023)) in Appendix C.
4270M (Small), 470M (Base), 820M (Large), 2.2B (XL) and 6.7B (XXL).
models.5 Specifically, we use the “LM adaptated” checkpoints from Vu et al. (2022). We align target LMs of 7B parameters with mT5-XL encoder to adapt 9B-sized models and 13B-sized LMs with mT5-XL encoder and mT5-XXL encoder to obtain 15B and 20B models, respectively. As LANGBRIDGE adds a considerable amount of additional parameters to the target LM, we conduct an inference throughput analysis in Appendix B. We also ablate the effect of the encoder parameter size and encoder model type in Appendix D.
We use the original continual training data of the LM for LANGBRIDGE when accessible (e.g., MetaMathQA for MetaMath). If unavailable, we opt for the closest publicly accessible dataset (e.g., OpenOrca (Lian et al., 2023) for Orca 2). In all our experiments, we fix the size of the training dataset to 200,000 instances. However, our ablation study on the effect of the training dataset size in Appendix D suggests LANGBRIDGE in practice may require much less data. We maintain the language model frozen throughout the alignment process for efficiency. We also freeze the encoder (except the embedding layer) for aligning finetuned LMs, whereas for pretrained LMs, we keep the encoder trainable.6 In Appendix D, we provide further explanations for these choices and ablate the effects of freezing. We align the models by training on the prefix LM objective described in Section 3.2. In our preliminary experiments, we find that training on various lengths of Xenc is necessary to ensure robustness on inference time, as the language model is exposed to diverse lengths of Henc.
We use a maximum input length (Xenc) of 1,024 and a maximum target length (Xlm) of 128 for training. For unlabeled data, we randomly vary the input length within the 1,024 window to introduce the LM to various lengths of Henc. For labeled data, the data naturally comes in diverse input lengths. On a machine equipped with four A100 80GB GPUs, training a 9B model takes less than four hours when the encoder layers are frozen, and under five hours when the entire encoder is trainable. In our main experiments, the setting where the encoder is fully trainable in the 20B models results in the maximum training time, which is ap-
5mT5 was trained on input size of 1024 tokens, but can take longer sequences due to its use of relative position embeddings (Shaw et al., 2018).
6We define finetuned LMs as LMs trained on labeled corpora and pretrained LMs as LMs trained on unlabeled corpora.
proximately ten hours. Further training details are available in Appendix F.

heading: 4.2 Mathematical Reasoning

heading: 4.2.1 Experimental Setup
text: Evaluation Datasets MGSM (Shi et al., 2023) comprises grade school math word problems in 11 typologically diverse languages, human translated from a sample of GSM8K (Cobbe et al., 2021). For evaluating pretrained LMs, we adopt the crosslingual transfer chain-of-thought (CoT) reasoning (Wei et al., 2022) setting (NATIVE-EXEMPLARS + EN-COT) from Shi et al. (2023), where the few-shot exemplars are given in the target language, but the CoT rationales to solve the exemplars are provided in English. For finetuned LMs, we evaluate in zeroshot7 setting. Additional evaluation on MSVAMP (Chen et al., 2023) is available in Appendix A.
Language Models Llemma (Azerbayev et al., 2023) is a set of LMs for mathematics, continually pretrained from Code Llama (Rozière et al., 2023) on Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code. MetaMath (Yu et al., 2023) was finetuned from Llama 2 (Touvron et al., 2023b) on MetaMathQA, a mathematical dataset based on GSM8K and MATH (Hendrycks et al., 2021b). As both Proof-Pile-2 and MetaMathQA are publicly available, we apply LANGBRIDGE using samples of their respective training datasets.
Baselines Llama 2 (Touvron et al., 2023b) is an English-centric LM in which 89.7% of the pretraining data consists of English but has shown considerable performance on non-English languages (Lai et al., 2023). mT58 (Xue et al., 2021), XGLM (Lin et al., 2022), and BLOOM (Scao et al., 2022) are massively multilingual LMs. MathOctopus (Chen et al., 2023) is an LM for multilingual mathematical reasoning. It was initialized from Llama 2 and finetuned on translations of the GSM8K dataset across ten languages. The ten languages seen by MathOctopus overlap with the 11 languages included in MGSM, except Telugu. We use their best-performing checkpoints, xRFT-MathOctopusP , which were further enhanced by data augmentation through rejection sampling (Yuan et al., 2023). We also re-
7Here, the term zero-shot refers to the lack of few-shot examples.
8We use the language model checkpoint from Vu et al. (2022).
port the performance of BLOOM models further trained on the training sets of LANGBRIDGE models: BLOOM-Proof-Pile-2 (PP2) and BLOOMMetaMath (MM). This is done to confirm that the capabilities of LANGBRIDGE models are derived from the LMs’ inherent strength rather than solely from the training set utilized by LANGBRIDGE. Detailed hyperparameters for training the BLOOM models are available in Appendix F. We additionally report the performance of PaLM (Chowdhery et al., 2023) measured by Shi et al. (2023). Similar to Llama 2, PaLM was pretrained on English-heavy corpora.

heading: 4.2.2 Results
text: Table 1 shows the evaluation results of baselines and LANGBRIDGE models on MGSM. We highlight six main observations. (1) Llama 2, Llemma, and MetaMath exhibit critical performance degradation across languages that are underrepresented in the training data of Llama 2. (2) Despite this, massively multilingual LMs underperform Llama 2,
even in the context of underrepresented languages.9 This disparity underscores the robust mathematical reasoning capabilities inherent in Llama 2 and absent in BLOOM, XGLM, and mT5. (3) LANGBRIDGE enhances the multilingual performance of Llemma and MetaMath, especially in underrepresented languages. Most notably, LANGBRIDGE is able to bring Llemma and MetaMath performance on Telugu (TE) from zero or near zero to a range comparable to other languages. (4) LANGBRIDGE may degrade performance on highresource languages, with Llemma-LB-9B’s English (EN) performance drop being particularly pronounced. We provide our speculations on the cause of this phenomenon in Section 6.1. (5) Mathematical reasoning capabilities of LANGBRIDGE models come from their original LMs, not their training data. This is evident from BLOOM-7.1B-PP2 and BLOOM-7.1B-MM underperforming LlemmaLB-9B and MetaMath-LB-9B, respectively, by a large margin. (6) Surprisingly, despite only be-
9Note BLOOM models were not trained in German, Russian, Japanese, and Thai.
ing trained on English math data, our MetaMathLB models are competitive against MathOctopus models, which were finetuned on translations of GSM8K on ten out of 11 languages supported by MGSM. Most importantly, performance of MathOctopus models drop to near zero on Telugu (TE), an unseen language by MathOctopus. On the other hand, LANGBRIDGE models show robust performance on all 11 languages, suggesting that even without multilingual supervision LANGBRIDGE generalizes to the large scale of languages included in the multilingual pretraining of the encoders.
Overall, LANGBRIDGE models demonstrate outstanding performance against baselines. LANGBRIDGE models vastly outperform similar-sized multilingual models, establishing LANGBRIDGE as a viable approach for developing mathematical reasoning models for low-resource languages. We provide an example of a CoT rationale generated by MetaMath-LB in Appendix G.

heading: 4.3 Code Completion

heading: 4.3.1 Experimental Setup
text: Evaluation Datasets Leveraging the strong translation capabilities of GPT-4 (OpenAI, 2023), as demonstrated in the study by Jiao et al. (2023), we extend HumanEval (Chen et al., 2021), a set of hand-written programming problems, into five underrepresented languages: Swahili, Bengali, Punjabi, Telugu, and Urdu. We name the resulting dataset HumanEval-MT. We select the five languages among those with reported MMLU (Hendrycks et al., 2021a) performance in the technical report of GPT-4 while also being included in the pretraining corpora of BLOOM. This choice is made to acquire high-quality translations from GPT-4, and provide BLOOM-based baselines a level playing field. To guide GPT-4 to only translate the embedded natural language instruction of the docstring while not modifying the code segments, we prompt GPT-4 with two human-annotated examples.10 Subsequently, the generated translations are executed in a Python interpreter environment to assert the absence of syntax errors. We provide translation quality estimation of HumanEval-MT in Appendix E.
The examples within the HumanEval dataset often feature self-explanatory function names. This suggests that language models could potentially
10We provide entire functions as inputs to GPT-4 to give it as much context as possible for accurate translation.
complete the associated code segments successfully, even without accurately comprehending the natural language comments embedded within them. Therefore, we evaluate the models on anonymized versions of each language set, wherein the target function names of the code segments are uniformly altered to “func”. An example of the anonymization and evaluation result on the original version is available in Appendix A.
Language Models Code Llama (Rozière et al., 2023) is a family of models initialized from Llama 2 model weights and pretrained on a codeheavy dataset. In our experiments, we use Code Llama-Python models, which were further pretrained on a Python-heavy dataset. Since the datasets used to pretrain the Code Llama models are not publicly available, we sample from the Python subset of StarCoder data (Li et al., 2023b) as the training dataset for LANGBRIDGE.
Baselines We use Llama 2 and BLOOM models as the baselines. Both models contain code data within their pretraining corpora. Additionally, we report the performance of BLOOM-StarCoder, a BLOOM model continually pretrained on the sample of StarCoder data used to train LANGBRIDGE models.

heading: 4.3.2 Results
text: Table 2 presents the Pass@1 scores on HumanEval and HumanEval-MT. Code Llama-LB models show consistent improvements over Code Llama across all underrepresented languages. Moreover, LANGBRIDGE models can match their larger baselines in terms of average scores: the 9B model slightly underperforms Code Llama-13B by 0.2%, while both the 15B and 20B models surpass Code Llama-34B. BLOOM trained on StarCoder data did not demonstrate noticeable improvements, reemphasizing that the strengths of LANGBRIDGE models predominantly stem from the capabilities of original LMs.

heading: 4.4 Logical Reasoning

heading: 4.4.1 Experimental Setup
text: Evaluation Datasets We assess logical reasoning capabilities with Big-Bench Hard (BBH) (Suzgun et al., 2023) and Big-Bench Hard Bengali (BBH-BN) (Shafayat et al., 2024). BBH is a collection of challenging subtasks where the application of chain-of-thought (CoT) reasoning has the potential to enhance performance substantially.
BBH-BN translates 14 of the 23 subtasks of BBH into Bengali. To facilitate meaningful comparison, we evaluate only on the 14 subtasks supported by BBH-BN for BBH.11
Language Models Orca 2 (Mitra et al., 2023) was finetuned on Llama 2 with a collection of datasets augmented with reasoning traces of GPT-4 as well as fully synthetic datasets created with GPT4. Orca 2 effectively improved the reasoning abilities of smaller LMs on complex tasks demanding advanced reasoning in zero-shot settings. As the training dataset of Orca 2 is not publicly available, we sample the training data for LANGBRIDGE from the OpenOrca dataset (Lian et al., 2023). OpenOrca follows the data distribution of the first iteration of Orca (Mukherjee et al., 2023). We employ CLD312 to filter any non-English data that mainly derives from translation datasets to ensure the zero-shot setting of our experiments. Examples were included if their input text had a 99% or greater probability of being English, while their target text also had a 95% or greater chance of being English. A slightly lower threshold was adopted for the target text to not falsely filter single-word responses, which CLD3 exhibits lower confidence.
Baselines In our evaluation of BBH, we assess whether Orca 2-LB models could acquire multilingual comprehension while retaining the zero-shot CoT capabilities of Orca 2. However, from our limited testing, we found that no existing open multilingual LMs could generate CoT reliably in a zero-shot setting. Consequently, they were not included as baselines. We do report the performance of BLOOM-OpenOrca, a BLOOM model further
11List of the 14 subtasks is available in Appendix A. 12github.com/google/cld3
trained on the same training set as Orca 2-LB.

heading: 4.4.2 Results
text: Table 3 shows the average accuracy across the subtasks for BBH and BBH-BN. Orca 2-LB-9B model shows considerable improvement in BBH-BN, surpassing the larger Orca 2-13B model. We report the performances on each subtask in Table 12 and Table 13. Remarkably, LANGBRIDGE significantly increases the performance on BBH-BN subtasks that require intrinsic linguistic understanding such as CAUSAL JUDGEMENT and SNARKS. CAUSAL JUDGEMENT requires the model to comprehend a short story and determine how a typical human would answer a given question. For SNARKS, the model is given two nearly-identical sentences and is required to indentify which one contains sarcasm. This observation, together with the results from the evaluation on XCOPA (Ponti et al., 2020), a commonsense reasoning dataset, suggests that LANGBRIDGE models are robust in grasping and interpreting nuanced linguistic details. Evaluation
results on XCOPA are provided in Appendix A. Figure 8 shows an example of a LANGBRIDGE model correctly solving CAUSAL JUDGEMENT.

heading: 5 Analysis of LANGBRIDGE
text: Based on the empirical evidence presented in the previous sections, we assert that LANGBRIDGE effectively enhances LMs’ capability to address multilingual tasks without multilingual training, especially for low-resource languages. This section presents two fundamental observations that further reinforce the hypothesis outlined in Section 3.

heading: 5.1 PCA
text: The hypothesis primarily attributes the feasibility of LANGBRIDGE to the sufficiently languageagnostic representations of multilingual encoders. If the conjecture holds, given a LANGBRIDGE model, the LM’s output representation of the soft prompt Henc should also exhibit language-agnostic characteristics. It stands to reason that the LM would not arbitrarily introduce additional languagespecific features to a language-neutral input derived from the multilingual encoder.
Therefore, to verify if the LANGBRIDGE models truly have language-agnostic output representations, we apply PCA to the mean pooled output representations from a LANGBRIDGE model and compare them with those from the original Englishcentric LM. Figure 3 shows the first two principal
components of pooled output representations obtained with FLORES (Goyal et al., 2022; Costajussà et al., 2022), a parallel corpus. For Orca 2, high-resource languages, English (EN), Spanish (ES), and German (DE), are mapped closely together. Underrepresented languages, Telugu (TE) and Swahili (SW), exhibit a more distant mapping in the representation space, forming three clusters.13 Conversely, for Orca 2-LANGBRIDGE, all languages are mapped into a single cluster, indicating that the representations of Henc maintain a relatively language-neutral status.

heading: 5.2 Accidental Translations
text: Figure 4 illustrates an example of “accidental translation” (Xue et al., 2021) by the Orca 2- LANGBRIDGE-15B model. Despite being given
13Note that Flores Swahili is in Latin script, the same as the three high-resource languages.
the options in Bengali, the LANGBRIDGE model perceives the options as Chinese and recites them in Chinese. With Bengali as input, accidental translation in a third language other than English or Bengali suggests that multiple languages may have similar representations in Henc (Li and Murray, 2023). Nonetheless, outputs in languages other than English are uncommon for Orca 2-LANGBRIDGE models. We conduct a qualitative analysis on the CoTs generated by Orca 2-LANGBRIDGE-15B for BBH-BN SNARKS subtask, which we find is the combination with the most frequent accidental translations. Out of 178 CoT rationales generated, only eight examples contained accidental translations in Chinese, Danish, Hindi, Japanese, Marathi, and Turkish, with one or two examples per language. Additionally, seven examples had Bengali in their CoT rationales. The relatively high frequency of Bengali indicates that Henc does not exhibit a perfectly language-agnostic behavior, and for some examples, the LM could discern the input language as Bengali. This is not ideal, as it suggests that the LM had to comprehend the input in Bengali, a language in which the LM lacks proficiency. We believe that LANGBRIDGE performance can be further enhanced by relieving the zero-shot constraint and adapting the mT5 encoder to have enhanced language-neutrality (Reimers and Gurevych, 2020; Feng et al., 2022) prior to alignment with the LM. However, we leave this exploration for future study.

heading: 6 Discussion

heading: 6.1 HRL Performance Degradation
text: In some cases, LANGBRIDGE models exhibit performance degradation in high-resource languages compared to their target LMs. While we leave further investigation for future studies, we outline some conjectures regarding potential causes for these performance declines.
• Given that the target LM is already proficient in a certain language, the text representations derived from the soft prompts may be more challenging for the LM to interpret compared to those from the LM’s native embedding layer.
• Due to limited computing budget, LANGBRIDGE models were trained with maximum input sequence length of 1024 (Xenc) and maximum output sequence length of 128 (Xlm). This is considerably shorter than what
the original language models were trained on, especially for language models trained solely on unlabeled data such as Llemma.
• We fixed the training hyperparameters for all of the LANGBRIDGE models regardless of the target LM for the sake of consistency and to save computing budget. This could have been suboptimal for some LANGBRIDGE models.
• For Code Llama and Orca 2, their original training corpora are inaccessible. Therefore, we opt for the closest alternative as stated in Section 4.1. Nonetheless, a minor distribution shift during LM training and LANGBRIDGE training is inevitable.

heading: 6.2 Multilingual CoT Capabilities?
text: As LANGBRIDGE solely utilizes English data for training, it is anticipated that the intermediate reasoning steps, or CoT, of LANGBRIDGE models would predominantly be in English. Given that LANGBRIDGE targets English-centric LMs specialized in reasoning, we conjecture that English CoT is one of the key components that contribute to the competitive performance of LANGBRIDGE models. This claim is supported by the findings of Shi et al. (2023), where they observe English CoT consistently leads to competitive results, and suggests that English CoT serves as a "useful baseline for future multilingual reasoning work".
While we believe incorporating non-English data into the LANGBRIDGE training data could induce multilingual CoT capabilities in LANGBRIDGE models, this approach does not align well with the original motivation of this work, which is to avoid the need for collecting multilingual reasoning data.

heading: 7 Conclusion
text: In this paper, we propose LANGBRIDGE, a simple yet effective method of extending the capabilities of LMs to solve multilingual reasoning tasks without using multilingual supervision. We show that LANGBRIDGE is surprisingly effective for enhancing multilingual reasoning capabilities for low-resource languages. Additionally, we provide analysis that indicates the effectiveness of LANGBRIDGE is due to the language-agnostic nature of the multilingual representations. We hope our findings benefit the low-resource language users and spur further research advancing the development of LMs inclusive of the entire global community.
Limitations
As LANGBRIDGE solely utilizes English data for training, LANGBRIDGE models may not be proficient in generating text in languages other than English. Although LANGBRIDGE successfully narrows the performance gap between high-resource and low-resource languages, a noticeable performance gap remains. Also, while multilingual representations are known to have language-agnostic characteristics to some degree, our analysis and previous works suggest that there is room for improvements (Libovický et al., 2020; Feng et al., 2022). While LANGBRIDGE has the potential to generalize to all languages supported by the multilingual encoder, the extent to which LANGBRIDGE enhances the reasoning capability of a specific language depends on two key factors: (1) The original proficiency of the LM in that particular language. (2) The proficiency of the encoder model in that particular language.
Ethical Considerations
While we share LANGBRIDGE models for open access, their terms for use or license adhere to those of the original LMs. The training datasets utilized in our research is primarily sourced from academic materials. As a result, we assess that the datasets likely contain a relatively fewer examples featuring offensive or personal information. Nevertheless, it is important to acknowledge that such content may be still present within the datasets.

heading: Acknowledgements
text: This work was partly supported by KAIST-NAVER Hypercreative AI Center (60%) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00113, Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework, 20%; No.2021-0-02068, Artificial Intelligence Innovation Hub, 20%).

heading: A Additional Evaluation Results
text: A.1 XCOPA
Table 10 shows the evaluation results of Orca 2 and Orca 2-LANGBRIDGE models on COPA (Roemmele et al., 2011) and XCOPA (Ponti et al., 2020), commonsense reasoning datasets. COPA is available in English, while XCOPA extends COPA to 11 languages. We do not include other LMs as baselines, as COPA was included in the training set of Orca 2, making it challenging to draw meaningful comparisons.
Despite reaching near-perfect accuracy for COPA, Orca 2 models’ performance drops close to random chance (50%) on some of the underrepresented languages of XCOPA. LANGBRIDGE successfully decreases this performance degradation, except for Quechua (QU). The discrepancy is likely due to Quechua not being included in the 101 languages covered by mT5. This observation reemphasizes that the large-scale linguistic proficiency of LANGBRIDGE models is primarily derived from mT5.
A.2 MSVAMP
MSVAMP (Chen et al., 2023) is a multilingual grade school level math word problem dataset translated from SVAMP (Patel et al., 2021) to 10 languages. We only evaluate MSVAMP in a zero-shot setting, as no CoT rationale examples are provided with the dataset.
Table 11 presents the evaluation results on MSVAMP. MetaMath and MathOctopus were not trained on SVAMP or MSVAMP, so MSVAMP can be seen as an out-of-domain test set to evaluate domain generalization (Chen et al., 2023). Performance of MetaMath-LB models indicates our models can generalize to out-of-domain test sets successfully.
A.3 BBH
Table 12 and 13 showcase the complete results for the 14 subtasks of BBH and BBH-BN. The subtasks are: CAUSAL JUDGEMENT, DATE UNDERSTANDING, DISAMBIGUATION QA, FORMAL FALLACIES, LOGICAL DEDUCTION (3, 5 AND 7), NAVIGATE, PENGUINS IN A TABLE, REASONING ABOUT COLORED OBJECTS, SNARKS, SPORTS UNDERSTANDING, TEMPORAL SEQUENCES, and WEB OF LIES.
Notably, Orca 2-LB models show noticeable performance degradation in DATE UNDERSTANDING.
From our qualitative analysis of the CoT, we observe that Orca 2-LB models frequently falsely assume an arbitrary date as the current date at the beginning of CoT (Figure 9), whereas the original Orca 2 models do not exhibit this behavior. Our exploration of the OpenOrca dataset reveals that examples often require the model to assume a specific current date. For example, an input text is given as “The current senate majority leader in the US is Chuck Schumer. Options: - yes - no”, and the target text contains “...Today’s date is October 12, 2021...”. As we do not have access to Orca 2’s original training dataset, a thorough ablation on the effect of such examples is challenging. Nevertheless, we speculate this problematic emergent behavior in Orca 2-LB models is partially due to the distribution shift of the training data from the original Orca 2 dataset to the OpenOrca dataset.
A.4 HumanEval Table 4 compares the original and anonymized prompts of HumanEval-MT. Top can be solved without comprehending Swahili whereas Bottom is not. Table 14 contains the evaluation results on the original (non-anonymized) version of HumanEval and HumanEval-MT. Compared to LANGBRIDGE models, Code Llama models show a sharper decline in performance when evaluated on the anonymized version, suggesting that they are less capable of comprehending natural text in another language.
B Inference Throughput of LANGBRIDGE Model
We report the inference throughput of models in Table 5. We measure the total time to infer 500 instances of XCOPA. For each model, we repeat
five times and report the average and the standard deviation. Additionally, we include the GPU memory utilization. We test using a single A6000 GPU on an idle server.
Results show Orca 2-LB-9B model requires considerably less compute and memory compared to Orca 2-13B, despite it outperforming Orca 2-13B in BigBenchHard Bengali (BBH-BN) and XCOPA.

heading: C General-domain Language Models
text: Table 15 probes the effect of LANGBRIDGE on general-domain English-centric LMs, Llama 2 and Mistral 7B (Jiang et al., 2023), using MGSM. Consistent with the findings on specialized LMs, LANGBRIDGE enhances the performance of lowresource languages. As training corpora for Llama 2 and Mistral 7B are unavailable, we use a sample of the SlimPajama dataset (Soboleva et al., 2023) as the training set.

heading: D Ablation Studies
text: D.1 Freezing/Unfreezing
We strictly keep the embedding layers of mT5 trainable throughout our experiments as we extend the vocabulary and the embedding layer to incorporate whitespace characters. mT5 tokenizers do not have whitespace characters in their vocabularies, and their default behavior is to truncate any consecutive whitespaces to a single space. However, this could negatively affect understanding code or following instructions considering the frequent use of whitespaces as delimiters (\n, \t, and “four spaces”). Therefore, even when we freeze the encoder, we leave the embedding layer trainable for the added whitespace vocabulary.
Table 6 presents the ablation study on the impact of parameter freezing during the alignment process. We apply LANGBRIDGE with mT5-XL (2B) encoder on multiple LMs while varying the trainable modules. Notably, freezing the encoder appears beneficial when adapting finetuned LMs (MetaMath and Orca), whereas it negatively af-
Target LM Trainable
AVG Score Enc LM
MGSM
Llama 2-7B 9.6 ! 11.3
Llemma-7B 14.4 ! 20.4
MetaMath-7B 48.8 ! 43.9
HUMANEVAL + HUMANEVAL-MT
Code Llama-7B 15.3 ! 19.4
XCOPA
fects pretrained models (Llama, Llemma, and Code Llama). We speculate this divergence stems from differing entropy levels in the datasets: unlabeled corpora typically exhibit higher entropy than relatively well-formatted finetuning datasets. Consequently, we conjecture that for unlabeled data, keeping the encoder trainable enables the model to acclimate to the training data better. Nonetheless, we leave a thorough investigation for future research. Additionally, training the LM during the alignment phase does not improve performance. We hypothesize this is due to the training datasets being in-domain of the LMs. As such, the LMs may not be learning additional information from the data.
D.2 Encoder Size
Figure 5 shows the effect of encoder size on LANGBRIDGE when applied to Orca 2-7B, measured with XCOPA. We test five different sizes of mT5 encoder: 270M (Small), 470M (Base), 820M (Large), 2.2B (XL) and 6.7B (XXL). We observe that LANGBRIDGE with the two smaller-sized encoders underperforms the base Orca 2-7B. Nonetheless, performance increases rapidly as the encoder size scales from 270M to 2.2B and saturates in the 2.2B to 6.7B range. These results highlight
that scaling the encoder size past a certain point provides diminishing returns.
D.3 Training Set Size
While we fixed the training set size to 200,000 in our main experiments, Figure 6 shows that XCOPA performance peaks on 120,000 training examples, which is 60% of our experiment setting. This suggests that in practice, LANGBRIDGE can be applied more efficiently.
D.4 Different Architectures
Table 7 shows the ablations of different LANGBRIDGE architectures. Throughout the main experiment, we adopt a single linear layer to connect the encoder and the language model. We ablate the effect of using an MLP layer following the second iteration of LLaVa (Liu et al., 2023a), and a “re-
sampler” module that adopts the architecture of perceiver resampler employed by Flamingo (Alayrac et al., 2022). In contrast to the finding of Liu et al. (2023a), we find that using an MLP layer instead of a linear layer decreases the performance slightly. Using a resampler module results in random performance.
D.5 Different Encoder Models
Table 8 presents the results of using different encoder models. We test out umT5 (Chung et al., 2023), which improves upon mT5 to include a higher proportion of low-resource languages during pretraining. As umT5 does not have “LM adapted” checkpoints, unlike mT5, we use the original checkpoints. However, we find using umT5 results in random performance. Since umT5 has a nearly identical architecture to mT5, except that it has relative position bias for every Transformer layer (Vaswani et al., 2017) in contrast to the very first, we speculate that using the encoder of a non-LM Seq2Seq model resulted in failed alignment. We also test XGLM (Lin et al., 2022), a decoder LM, as an encoder, in which we also observe similar results. In both cases, freezing the encoder made no difference. We leave further investigation for future research.

heading: E Quality Estimation of HumanEval-MT
text: Table 9 presents the quality estimation for HumanEval-MT using backtranslation (Agrawal et al., 2022; Zhuo et al., 2023). We translate
HumanEval-MT back to English and measure Code Llama Pass@1 scores. As explained in Section 4.3, we anonymize all function names. In addition, we report automatic evaluation metrics, BLEU (Papineni et al., 2002) and chrF (Popović, 2015), against the original HumanEval. We use the docstrings only for calculating the metrics.
Overall, Code Llama’s performance on the backtranslations matches the original HumanEval benchmark, with the exception of Code Llama34B’s performance on Swahili backtranslation. As Code Llama-7B achieves comparable results on the Swahili backtranslation, this suggests that for Swahili, GPT-4 may struggle with accurately translating complex examples that only larger models can solve. Lower automatic evaluation scores in Swahili further support the idea.

heading: F Experimental Details
text: For training LANGBRIDGE models, we leverage AdamW (Loshchilov and Hutter, 2019). We use a learning rate of 6e−4 for the linear layer and 2e−5 for the encoder. We do not use any learning rate scheduling. We use an effective batch size of 128.
For further training BLOOM baselines, we keep the training configuration the same as LANGBRIDGE models with two exceptions: (1) Learning rate, in which we used a uniform value of 2e−5 across the entire model. (2) Sequence length, which was set to 1,152 (1, 024 + 128).
For evaluations, we leverage code adapted from LM Evaluation Harness (version 0.3.0) (Gao et al., 2023) and Bigcode Evaluation Harness (version 0.1.0) (Ben Allal et al., 2022) packages and report single run results with fixed seed. We open source the evaluation code.
For constructing HumanEval-MT, we prompt GPT-4 with human-translated examples. The two examples used for HumanEval-MT were translated to Korean by a native Korean author. GPT-4 was then prompted to translate HumanEval to the target
languages with the examples as reference. Note that the examples were provided to guide GPT-4 to keep the format of the data intact and only translate the natural language, not necessarily to enhance the translation quality itself.

heading: G CoT Examples
text: In this section, we provide three examples of CoT generated by LANGBRIDGE models. Figure 7 is from the MetaMath-LANGBRIDGE-15B model prompted with an example from MGSM Telugu. Figure 8 and 9 are from Orca 2-LANGBRIDGE15B model prompted with examples from BBHBN CAUSAL JUDGEMENT and BBH-BN DATE UNDERSTANDING. We select these two subtasks from BBH-BN as LANGBRIDGE substantially increased the performance of Orca 2-13B for the former (+13.4%), but caused a considerable decrease for the latter (−6.4%). We show a success case for the former and a failure case for the other. In Appedix A, we provide an explanation for the performance degradation in BBH-BN DATE UNDERSTANDING.

heading: Future Work
text: Nonetheless, we leave a thorough investigation for future research. Additionally, training the LM during the alignment phase does not improve performance. We hypothesize this is due to the training datasets being in-domain of the LMs. As such, the LMs may not be learning additional information from the data. D. 2 Encoder Size
Figure 5 shows the effect of encoder size on LANGBRIDGE when applied to Orca 2-7B, measured with XCOPA. We test five different sizes of mT5 encoder: 270M (Small), 470M (Base), 820M (Large), 2. 2B (XL) and 6. 7B (XXL). We observe that LANGBRIDGE with the two smaller-sized encoders underperforms the base Orca 2-7B. Nonetheless, performance increases rapidly as the encoder size scales from 270M to 2. 2B and saturates in the 2. 2B to 6. 7B range. These results highlight
that scaling the encoder size past a certain point provides diminishing returns. D. 3 Training Set Size
While we fixed the training set size to 200,000 in our main experiments, Figure 6 shows that XCOPA performance peaks on 120,000 training examples, which is 60% of our experiment setting. This suggests that in practice, LANGBRIDGE can be applied more efficiently. D. 4 Different Architectures
Table 7 shows the ablations of different LANGBRIDGE architectures. Throughout the main experiment, we adopt a single linear layer to connect the encoder and the language model. We ablate the effect of using an MLP layer following the second iteration of LLaVa (Liu et al. , 2023a), and a “re-
sampler” module that adopts the architecture of perceiver resampler employed by Flamingo (Alayrac et al. , 2022). In contrast to the finding of Liu et al. (2023a), we find that using an MLP layer instead of a linear layer decreases the performance slightly. Using a resampler module results in random performance. D. 5 Different Encoder Models
Table 8 presents the results of using different encoder models. We test out umT5 (Chung et al. , 2023), which improves upon mT5 to include a higher proportion of low-resource languages during pretraining. As umT5 does not have “LM adapted” checkpoints, unlike mT5, we use the original checkpoints. However, we find using umT5 results in random performance. Since umT5 has a nearly identical architecture to mT5, except that it has relative position bias for every Transformer layer (Vaswani et al. , 2017) in contrast to the very first, we speculate that using the encoder of a non-LM Seq2Seq model resulted in failed alignment. We also test XGLM (Lin et al. , 2022), a decoder LM, as an encoder, in which we also observe similar results. In both cases, freezing the encoder made no difference. We leave further investigation for future research