Question Translation Training for Better Multilingual Reasoning (2024)

Abstract: Large language models show compelling performance on reasoning tasks but they tend to perform much worse in languages other than English. This is unsurprising given that their training data largely consists of English text and instructions. A typical solution is to translate instruction data into all languages of interest, and then train on the resulting multilingual data, which is called translate-training. This approach not only incurs high cost, but also results in poorly translated data due to the non-standard formatting of mathematical chain-of-thought. In this paper, we explore the benefits of question alignment, where we train the model to translate reasoning questions into English by finetuning on X-English parallel question data. In this way we perform targeted, in-domain language alignment which makes best use of English instruction data to unlock the LLMs’ multilingual reasoning abilities. Experimental results on LLaMA2-13B show that question alignment leads to consistent improvements over the translate-training approach: an average improvement of 11.3% and 16.1% accuracy across ten languages on the MGSM and MSVAMP multilingual reasoning benchmarks1.

heading: 1 Introduction
text: Large language models have recently shown a strong ability to reason in English, but performance in other languages, especially more distant languages, still trails far behind (Shi et al., 2022; Huang et al., 2023). It is unsurprising, considering that their training data is predominantly composed of English text and instructions (Blevins and Zettlemoyer, 2022; Touvron et al., 2023; Wang et al., 2023). To elicit LLM’s multilingual performance, previous approach typically follows the translatetraining paradigm (Chen et al., 2023), which first
1The project will be available at: https://github.com/ NJUNLP/QAlign.
translates English instruction data into non-English with a translation engine and then uses the multilingual data for instruction-tuning.
However, the translate-training has the following drawbacks: (1) translating English training data to numerous non-English languages incurs significant translation cost, especially considering the constant addition of large and complex instruction tuning sets (Yuan et al., 2023; Yu et al., 2023). (2) Additionaly, it is hard for the translation engine to accurately translate lengthy, logical texts containing mathematical symbols in chain-of-thought (CoT) responses, which can compromise the quality of translated data (evidence are shown in Appendix A). Consequently, we explore the following research question in this paper: Can we unlock the LLM’s multilingual reasoning ability by teaching it to translate reasoning questions into English?
In this paper, we focus on the multilingual mathematical reasoning task and explore the benefits of question alignment (QAlign), where we fine-tune the pre-trained LLM to translate reasoning questions into English with X-English parallel question data. This targeted, in-domain language alignment enables the subsequent effective utilization of English instruction data to unlock LLMs’ multilingual reasoning abilities. Following question alignment, we implement response alignment by further fine-tuning the language-aligned LLM with cuttingedge English instruction data. Even when only English supervised data is available, our alignmentenhanced LLM can achieve superior performance on non-English tasks with its transferable English expertise.
To demonstrate the advantages of question alignment, we conduct experiments on challenging multilingual mathematical reasoning benchmarks, MGSM (Shi et al., 2022) and MSVAMP (Chen et al., 2023). We use two of the most advanced open-source LLMs, LLaMA2-7B and LLaMA213B (Touvron et al., 2023), as base models. Exper-
8411
iment results show that the inclusion of the question alignment stage brings an average improvement of up to 13.2% in multilingual performance. The performance improvement on low-recourse languages, e.g. Thai and Swahili, can be 30%-40%. Compared to the translate-training baseline, MathOctopus (Chen et al., 2023), which tuned with a multilingual version of GSM8K dataset , our alignment-enhanced LLMs achieves average performance improvement of 9.6% (7B) and 11.3% (13B) on MGSM. On the out-of-domain test set MSVAMP, our fine-tuned LLMs achieve 13.1% (7B) and 16.1% (13B) average accuracy improvement, also demonstrating our approach is robust to domain shift. In general, we observe that although incorporating translated instruction data does benefit multilingual performance, our question alignment strategy provides a more efficient and effective choice. In our analysis, we also present the effects of other implementations for performing language alignment and illustrate the importance of choosing the appropriate translation direction and domain during this phase of training.
The main contributions of this paper can be summarized as:
• We present a novel X-English question alignment finetuning step which performs targeted language alignment for best use of the LLMs English reasoning abilities.
• We fine-tune open-source LLMs, LLaMA27B/13B, into strong multilingual reasoners, which beat the translate-training baseline by 9.6% (7B) and 11.3% (13B) on MGSM, by 13.1% (7B) and 16.1% (13B) on MSVAMP.
• We explore language alignment with other language directions (English-X), types and domains of data, and confirm our intuition that in fact X-English questions perform best.

heading: 2 Related Work
text: Large language model With a large number of parameters pre-trained on a large-scale corpora, large language models can memorize vast amounts of knowledge (Roberts et al., 2020) and acquire emergent abilitie, such as in-context learning (Brown et al., 2020), CoT generation (Wei et al., 2022b). Then, to better align the behavior of LLMs with human expectations, Wei et al. (2022a) propose instruction-tuning, training LLM to generate desired response based on the given instruction.
Subsequently, many efforts are put into creating effective instruction data to further unlock LLM’s potential (Wang et al., 2022; Taori et al., 2023; Longpre et al., 2023; Wang et al., 2023). However, since the proposed instruction datasets consist mainly of English, the directly fine-tuned LLMs struggle on non-English languages, especially on those languages that are dissimilar to English (Huang et al., 2023; Zhu et al., 2023; Chen et al., 2023).
Multilingual mathematical reasoning Mathematical reasoning is a challenging and representative task for evaluating the intelligence of LLMs (Ahn et al., 2024), where LLMs need to understand the given math question and produce a numerical answer through step-by-step reasoning. Shi et al. (2022) expanded the scope to a multilingual context by translating English math questions from the GSM8K test set (Cobbe et al., 2021) into non-English languages, thereby creating a multilingual benchmark called MGSM.
Subsequently, many efforts are put into enhancing LLM’s multilingual reasoning capabilities, which can be categorized into two approaches: prompting close-source LLMs and instructiontuning open-source LLMs. In the first approach, Qin et al. (2023) and Huang et al. (2023) carefully craft prompts for close-source LLMs like ChatGPT (OpenAI, 2022). Their strategy involves first prompting the LLM to explicitly translate nonEnglish questions into English, then ask the model to solve the translated problem instead. However, the effectiveness of these prompting methods are not well-examined on open-source LLMs. And it remains an open challenge to equip opensourced LLMs with strong multilingual mathematical problem-solving skills.
In the second approach, Chen et al. (2023) follow the translate-training method (Artetxe et al., 2023). Initially, they translate English instruction data in GSM8K into non-English with ChatGPT, followed by employing multilingual data for instruction-tuning. Moreover, Chen et al. (2023) investigate cross-lingual training strategies such as mixing questions and CoT responses in different languages, but fail to achieve consistent improvement. Although the translate-training approach is effective, it incurs high translation cost and is errorprone2. It also becomes increasingly impractical
2We analyze the errors in the translated dataset from Chen et al. (2023) and present both quantitative and qualitative results in Appendix A.
to translate vast quantities of augmented data into numerous languages, especially considering recent findings that augmented training data, e.g., METAMATHQA (Yu et al., 2023)—which is 50 times larger than GSM8K—greatly enhances LLM’s reasoning skills. Without relying on translated CoT responses, in this paper, we present a novel question alignment technique to utilize cutting-edge Englishonly supervised data to boost open-source LLM’s performance on multilingual reasoning tasks.

heading: 3 Methodology
text: An illustration of our devised method is shown in Figure 1. The key idea of our approach is strengthening language alignment within LLM before exposing it to English instruction-response pairs. By doing so, we can utilize LLM’s expertise in English to enhance its performance on non-English tasks. Below we introduce the two training stages of our framework: question alignment (§3.1) and response alignment (§3.2).

heading: 3.1 Stage I: Question Alignment
text: It has been found that directly fine-tuning LLMs with English instruction data does not help to improve their performance on non-English tasks (Chen et al., 2023). We suggest that this issue may arise from the insufficient alignment of multiple languages within the LLM. Ideally, in a wellaligned LLM, proficiency in one language, like
English, could easily transfer to other languages. To improve the alignment of non-English languages with English, we devise a translation task QAlign: training LLM on translating questions from non-English into English. Specifically, given a group of multilingual questions, the optimization objective can be written as:
argmax θ
∑ l∈L log pθ(Ze|Zl)
where θ denotes the parameters of the base model. Zl and Ze denote non-English and English questions respectively and L is the set of considered non-English languages. With this training objective, we equip the LLM with an implicit bias to relate non-English questions with their English counterparts when performing non-English tasks.
Note that this stage only relies on multilingual questions rather than translated CoT responses. Basically, acquiring multilingual questions is more feasible than obtaining accurate multilingual CoT responses, because translation engines often struggle to precisely translate lengthy, logical texts containing mathematical symbols (quantitative evidence are shown in Appendix A).
In this translation task, the domain of translation data is also an important factor to consider. In subsequent experiments, we demonstrate that using multilingual questions as translation data is more effective than employing general domain translation corpora.

heading: 3.2 Stage II: Response Alignment
text: After question alignment, we train LLM with specialized instruction-response pairs to unlock its potential on multilingual mathematical reasoning tasks. Specifically, we consider two data scenarios: monolingual supervision setting and mixed supervision setting.
Monolingual supervision setting In this setting, we employ English-only instruction data for response alignment, because the cutting-edge instruction datasets are often available only in English. During training, we follow the standard implementation (Wei et al., 2022a) and finetune the languagealigned LLM to maximize the generetive probability of the response Y given the question X :
argmax ϕ
∑
{X ,Y}∈D log pϕ(Y|X )
Where ϕ denotes the parameters of the stage I model and D denotes the instruction dataset. Although the training only utilizes English supervision, the previously established language alignment allows the LLM’s English proficiency to be shared across multiple languages.
Mixed supervision setting While our framework is primarily designed for the scenario where only English instruction data is available, it can also leverage additional multilingual supervision, when available, to achieve even higher multilingual performance. For instance, this multilingual dataset could be a translated version of a subset of largescale English data. In this scenario, given a set of additional multilingual superivsed data M, we sequentially fine-tune the stage I model on M and then on the English instruction data D. Subsequent experiment results show that this training recipe can further improve the LLM’s multilingual reasoning capabilities.

heading: 4 Experiment Setting
text: Base LLM In our experiments, we use two of the most advanced open-source LLMs, LLaMA2-7B and LLaMA2-13B as the base model.
Training Dataset In the question alignment stage, we utilize multilingual questions from GSM8KINSTRUCT3 (Chen et al., 2023). Dur-
3GSM8KINSTRUCT is a multilingual dataset that extends the English instruction dataset GSM8K by translating English instructions and CoT responses into nine non-English languages with ChatGPT.
ing the response alignment stage, we employ the cutting-edge English-only dataset METAMATHQA as monolingual supervision, which is built upon English dataset GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) by performing data augmentation, such as rephrasing questions and enriching answers. In the mixed supervision setting, we employ both METAMATHQA and GSM8KINSTRUCT. Dataset statistics are reported in Table 1.
Training Details We use stanford_alpaca4 as our code base. We use consistent training hyperparameters across two stages of training. At each stage, we fine-tune LLM’s full parameters for 3 epoch on eight NVIDIA A100 GPUs. The learning rate is set to 2e-5, with a batch size of 128.
Baseline Systems For comparison, we consider following systems which are instruction-tuned from LLaMA2 with diverse training recipes:
• SFT (Touvron et al., 2023), which is instruction-tuned with basic GSM8K.
• RFT (Yuan et al., 2023), which is instructiontuned with an augmented GSM8K training dataset, using rejection sampling techniques.
• MAmmoTH (Yue et al., 2023), which is instruction-tuned with GSM8K and a collection of math instruction datasets.
• WizardMath (Luo et al., 2023), which is constructed using reinforcement learning on GSM8K and MATH.
• MathOctopus (Chen et al., 2023), which is instruction-tuned with a multilingual version of GSM8K dataset, representing a standard implementation of translate-training approach. We also reproduce this model in our experiments, denoted as MultiReason.
4https://github.com/tatsu-lab/stanford_alpaca
• MetaMath, which is instruction-tuned with METAMATHQA (Yu et al., 2023). It is currently the most powerful English instruction data for mathematical reasoning. We also reproduce this model in our experiments, denoted as MonoReason.
Among these baseline systems, most models are tuned with English data and only MathOctopus and MultiReason are tuned with multilingual data.
Evaluation Dataset To assess LLMs’ performance on multilingual mathematical reasoning5, we employ the benchmark dataset MGSM (Shi et al., 2022). We also evaluate the robustness of LLMs using an out-of-domain test set MSVAMP (Chen et al., 2023). In our experiments, we report LLM’s answer accuracy in a zero-shot and greedy decoding setting. Specifically, we use evaluation scripts6 provided by Chen et al. (2023) and measure answer accuracy by comparing the last numerical number that appears in the LLMgenerated response with the gold answer.
5In this paper, we evaluate LLMs’ reasoning performance on ten languages: Bengali (Bn), Thai (Th), Swahili (Sw), Japanese (Ja), Chinese (Zh), German (De), French (Fr), Russian (Ru), Spanish (Es) and English (En).
6https://github.com/microsoft/MathOctopus

heading: 5 Main Results
text: In this section, we report our experiment results and introduce our main findings.

heading: 5.1 Monolingual Supervision Setting
text: Question alignment stage enables LLM’s proficiency in English to be transferred to nonEnglish tasks. Experiment results on the MGSM dataset are presented in Table 2. We can see that LLMs trained with augmented English data (RFT, MAmmoTH, WizardMath, MetaMath and MonoReason) typically underperform on nonEnglish tasks, despite showing improved performance in English compared to SFT model. The multilingual MathOctopus outperforms existing open-source models in terms of multilingual performance. However, as we have discussed, the translated dataset can be out-dated quickly and keeping translating cutting-edge English instuction can also be prohibitive due to the high translation cost.
Unlike the translate-training approach, our framework can easily utilize the most advanced English instruction data, e.g., METAMATHQA. With the question alignment stage (QAlign), we successfully transfer model’s proficiency in English to non-English languages. On average, this leads to a 11.2% increase in accuracy for the 7B model and a 13.2% increase in accuracy for the
13B model. These substantial improvements on non-English languages significantly reduce LLM’s performance gap between non-English and English tasks, thereby demonstrating the effectiveness of our devised method.
After question alignment, our fine-tuned LLM surpasses the translate-training baseline by a large margin More importantly, we observe that after question alignment, our fine-tuned LLM surpasses the translate-training baseline (MathOctopus) by a large margin. By transferring the model’s expertise in English to non-English scenarios, our approach outperforms MathOctopus by an average margin of +9.6% for the 7B model and +11.3% for the 13B model. These results again demonstrate the superiority of our method7.
Our fine-tuned LLMs also exhibit better robustness on the out-of-domain test set Apart from evaluating on MGSM, we further assess the robustness of our LLMs on the out-ofdomain test set MSVAMP (Table 10). The findings are generally consistent with those from
7In Appendix B, we also report the results of using question translation data for stage I training and multilingual instruction data for stage II training. This provides a more direct comparison (QAlign→MultiReason vs. MultiReason), and the added question alignment stage also improves multilingual performance in this setting.
MGSM dataset. Notably, compared to the unaligned counterpart (MonoReason), our model (QAlign→MonoReason) achieves significant improvement in average multilingual performance, with gains of 11.0% for the 7B model and 10.8% for the 13B model. Our method outperforms the translate-training approach (MathOctopus) by an even larger margin here, showing increases of 13.1% for the 7B model and 16.1% for the 13B model, which shows its more generalized and robust performance.

heading: 5.2 Mixed Supervision Setting
text: Incorporating multilingual supervised data into our framework can achieve a higher ceiling for multilingual performance Although our framework does not rely on the multilingual supervised data, we can utilize such data to attain a higher level of multilingual performance if a multilingual dataset is available. In this mixed supervision setting, we first tune the stage I model (7B) with multilingual GSM8KINSTRUCT and then tune it with English data METAMATHQA. The experiment results on MGSM are depicted in Figure 2. We find that incorporating additional multilingual supervision further leads to an average performance gain of 2.1% on multilingual tasks. Compared to the data mixing baseline (MonoReason+MultiReason), our
approach demonstrate an average improvement of 3.6%, with significant advantages in high-resource languages such as Spanish, Russian, German, and French.

heading: 6 Analysis

heading: 6.1 Ablation study
text: Impact of using different translation training data During the question alignment stage, we implement the translation task by training LLMs on translating questions from non-English to English. Now we present the ablation study to show the effects of alternative implementations (Table 4). while different implementations yield similar performance in English, their impact on non-English peformance varies significantly. For instance, training LLMs on reverse translation tasks greatly degenerates non-English performance (Question:En→X, Response:En→X). Training LLM on translating CoT responses from nonEnglish to English (Response:X→En) also results in lower performance compared to our original implementation. We suggest that this is because noises in the translated CoT responses compromise
the data quality. Training the LLM with translation data from commonly-used corpora, such as FLORES8, does not work as well, indicating that the domain of the translation data is another crucial factor in establishing language alignment.
Impact of manipulating training order We also conduct the ablation study to demonstrate the significance of the training sequence within our proposed framework. As shown in Table 5, reversing the order of the two training stages results in the LLM performing poorly in both English and nonEnglish languages. We observed that an LLM finetuned in this manner tends to repeat the question in English when presented with questions in various languages.
When we merge the training datasets from both stages and perform a single-stage, multi-task training, there is a significant drop in non-English performance as well. Although capable of responding to questions in English, the fine-tuned LLM is prone to translating the given non-English questions rather than answering them. These analysis results demonstrate that our design of two-step training framework is non-trivial.
8In this ablation study, we take the translation data in the development and test set of FLORES-101 dataset (Goyal et al., 2022) for fine-tuning.

heading: 6.2 Prediction Consistency
text: Another advantage of establishing question alignment is the improvement it brings to the consistency9 of predicted answers against multilingual queries. This means a higher degree of agreement in answers to the same question posed in different languages. Figure 3 displays the quantified results. In contrast to their unaligned counterparts (MonoReason), our alignment-enhanced LLM (QAlign→MonoReason) usually demonstrate higher answer consistency. This improvement is particularly notable for distant languages, such as Bengali, Thai, Swahili, Japanese, and Chinese. This results can serve as another strong evidence of our successful transfer of LLM’s proficiency in English to non-English languages. Appendix C presents some cases to further illustrate the advantages of achieving higher multilingual consistency.

heading: 6.3 Question Alignment vs Translate-Test
text: In our training framework, we implicitly endow the LLM with a bias that associates non-English questions with their English equivalents, sharing similar philosophy with translate-test prompting approach.
9Supposing the set of correct predictions in two languages is U and V respectively, we compute the consistency score as |U∩V | |U| .
Thus we discuss the difference between these two approaches here. Experiment results are reported in Table 6. For the MonoReason model, the translatetest approach does not yield any improvement, suggesting that this approach may not be universally applicable solution for open-source LLMs. For our alignment-enhanced QAlign→MonoReason model, direct inference and translate-test prompting achieves similar performance. But considering our approach does not rely on explicitly translating the questions during inference, it will have a more efficient inference process.

heading: 6.4 Effects of tuning LLM with different English instruction data
text: To demonstrate the universal effectiveness of question alignment, we also employ English GSM8K dataset as monolingual supervison and show the results in Table 7. Under different English instruction data, the incorporation of a question alignment stage always boost LLM’s non-English performance. These results also highlight the importance of using advanced English instruction data, because achieving better performance in English usually means an improved non-English performance with the help of inner language alignment.

heading: 7 Conclusion
text: In this paper, we introduce a novel question alignment method to empower LLMs on multilingual
mathematical reasoning tasks without requiring multilingual instruction data. Experiment results on MGSM and MSVAMP benchmarks show that our proposed question alignment stage brings an average improvement of up to 13.2% in multilingual performance. Our alignment-enhanced LLM outperforms the unaligned counterpart and the translate-training baseline by a large margin and shows a more robust performance. Generally, our devised method successfully narrows the gap between LLM’s performance between English and non-English, showing a new possibility to unlock LLM’s capabilities to solve multilingual tasks.
Limitation
Below we discuss potential limitations of our work:
• Chain-of-Thought in English: When receiving non-English questions, our language-aligned LLM typically produces an English CoT before giving the final numerical answer. While the language used for the CoT is not explicitly specified as a requirement for the multilingual mathematical reasoning task, providing a CoT consistent with the query’s language could enhance the model’s utility.
• Scale of the pre-trained LLM: Our experiment is constrained by available computational resources, leading us to utilize the LLaMA2-7B and LLaMA2-13B models. Should resources allow in the future, we aim to broaden our research to include larger-scale models, such as LLaMA2-70B.

heading: Acknowledgement
text: We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by National Science Foundation of China (No. 62376116, 62176120), the Liaoning Provincial Research Foundation for Basic Research (No. 2022- KF-26-02), research project of Nanjing UniversityChina Mobile Joint Institute. This project has also received funding from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee (UTTER grant numbers 10039436). Wenhao Zhu is also supported by China Scholarship Council (No.202306190172).

heading: A Analyzing the Quality of the Translated Dataset
text: In the work of (Chen et al., 2023), the authors employ ChatGPT to translate GSM8K into several non-English languages, resulting in the creation of the multilingual dataset GSM8KINSTRUCT. Below we analyze the translation quality of this dataset and highlight the challenges associated with translating complex CoT responses. We evaluate the translation quality of both questions and responses in a reference-free condition with COMETKiwi10 (Rei et al., 2022). The evaluation results in Table 8 show that the quality of the translated responses is significantly inferior to that of the translated questions. This gap demonstrates the difficulties inherent in translating CoT content.
Table 9 provides some examples of typical translation errors. Based on this analysis, we suggest that constructing a multilingual CoT dataset through a translation engine is fraught with errors and cannot ensure the quality of the dataset. In constrast, our devised framework provides a more effective and efficient solution, which does not require translated multilingual CoT.

heading: B Experiment Results of Using
text: Multilingual Instruction Data for Response Alignment
To more comprehensively illustrate the benefit of the question alignment approach, we use question
10Specifically, we employ wmt22-cometkiwi-da as the evaluation model: https://huggingface.co/Unbabel/ wmt22-cometkiwi-da.
translation data for stage I training and multilingual instruction data GSM8KInstruct for stage II training (QAlign→MultiReason). We can see that the added question alignment stage also brings improvement on multilingual performance in this setting.

heading: C Benefits of Achieving Higher Multilingual Consistency
text: Table 11 presents some examples to illustrate the advantages of achieving higher multilingual consistency. In this case, both MonoReason model and QAlign→MonoReason model predict correctly on the English question. However, when presented with the same question expressed in Chinese, our LLM generates a similar reasoning path and still give the right answer, but the unaligned model fails.

heading: D Used Scientific Artifacts
text: Below lists scientific artifacts that are used in our work. For the sake of ethic, our use of these artifacts is consistent with their intended use.
• Stanford Alpaca (Apache-2.0 license), a project that aims to build and share an instruction-following LLaMA model.
• Transformers (Apache-2.0 license), a framework that provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.

heading: Future Work
text: Should resources allow in the future, we aim to broaden our research to include larger-scale models, such as LLaMA2-70B