mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models (2024)

Abstract: Large language models (LLMs) with Chainof-thought (CoT) have recently emerged as a powerful technique for eliciting reasoning to improve various downstream tasks. As most research mainly focuses on English, with few explorations in a multilingual context, the question of how reliable this reasoning capability is in different languages is still open. To address it directly, we study multilingual reasoning consistency across multiple languages, using popular open-source LLMs. First, we compile the first large-scale multilingual math reasoning dataset, mCoT-MATH, covering eleven diverse languages. Then, we introduce multilingual CoT instruction tuning to boost reasoning capability across languages, thereby improving model consistency. While existing LLMs show substantial variation across the languages we consider, and especially low performance for lesser resourced languages, our 7B parameter model mCoT achieves impressive consistency across languages, and superior or comparable performance to closeand open-source models even of much larger sizes.

heading: 1 Introduction
text: Recent progress on language models shows that they can achieve surprising performance on complex reasoning tasks in natural language processing (NLP), such as symbolic reasoning (Wei et al., 2022a; Kojima et al., 2022), math word problem (Cobbe et al., 2021; Wei et al., 2022a), and commonsense reasoning (Wei et al., 2022a; Kojima et al., 2022). Most of the research focuses on prompting large language models (LLMs), where the LLMs are conditioned on a few examples or instructions describing the target task (Wei et al., 2022b; Fu et al., 2023b; Zhou et al., 2023; Kojima et al., 2022; Chuanyang et al., 2023).
While most previous works focus on reasoning with LLMs in English, Shi et al. (2023) recently have extended it to a multilingual setting leveraging a few-shot prompting approach. However,
performance for lesser resourced languages still lags behind, in a similar way as generalization of factual knowledge has been shown to vary widely across languages (Fierro and Søgaard, 2022; Qi et al., 2023), mainly due to the fact that most languages are not well represented in LLMs. The work we present here has the twofold aim of (i) better understanding and evaluating the general reasoning capabilities of LLMs beyond just English, and (ii) providing lesser resourced languages with capable but manageable models which can be used for reasoning tasks.
To this end, we propose to measure reasoning consistency across multiple languages. As shown in Figure 1, LLMs are expected to produce logically similar reasoning solutions and consistent final results for inputs which are semantically equivalent but expressed in different languages. Based on our findings, we aim to enhance multilingual
12012
reasoning abilities through instruction tuning, yielding a model that can solve reasoning tasks in various languages, with similar reasoning capabilities across those languages. Specifically, we focus on math word problems and empirically investigate the reasoning consistency of current open-source state-of-the-art LLMs across multiple languages. The model’s reasoning consistency is evaluated toward the final answer (the final results should be the same across languages). On the basis of preliminary results showing substantial reasoning gaps between different languages, we propose a multilingual Chain-of-Thought reasoning (mCoT) framework using instruction tuning, which aims to boost reasoning capability across languages, thereby improving consistency. So, first we construct the multilingual instruction training dataset (mCoTMATH) by automatically translating English data into multiple languages, and then we use it for LLM finetuning.
In summary, our contributions are:1
• We propose to study reasoning consistency of LLMs across different languages, providing insights into (the evaluation of) this ability of LLMs.
• We compile and distribute mCoT-MATH, the first large-scale multilingual math CoT reasoning dataset containing around 6.3 million samples for 11 diverse languages.
• Based on mCoT-MATH, we train and make available a 7B parameter model mCoT for multilingual math reasoning, which achieves impressive consistency across languages, and superior or comparable performance to closeand open-source models.

heading: 2 Background
text: Math Reasoning with LLMs In recent years, LLMs such as GPT-3 (Brown et al., 2020) have shown impressive performance in various NLP tasks. In particular, LLMs with CoT-based method exhibit the emergent ability to perform complex math reasoning tasks (Wei et al., 2022b; Wang et al., 2023). One popular line is prompt engineering, which aims to elicit reasoning capability of LLMs by exploring various prompts, such as basic CoT prompting (Wei et al., 2022b), complex CoT (Fu et al., 2023b), auto-CoT (Zhang et al., 2023), self-consistency CoT (Wang et al., 2023),
1Data, code, and model are available at https://github. com/laihuiyuan/mcot.
multilingual CoT (Shi et al., 2023), least-to-most prompting (Zhou et al., 2023), progressive-hint prompting (Chuanyang et al., 2023), residual connection prompting (Jiang et al., 2024), and using specific phrases like “Let’s think step by step” (Kojima et al., 2022). In addition to guiding the model through prompting methods, several works propose to control the reasoning path during inference through verifier (Cobbe et al., 2021; Khalifa et al., 2023) and decoding method (O’Brien and Lewis, 2023). Another research line is a tailor-designed reasoning model that improves the mathematical reasoning ability of LLMs through instruction tuning on reasoning data, including reinforcement learning (Uesato et al., 2022; Luo et al., 2023), knowledge distillation (Fu et al., 2023a; Hsieh et al., 2023; Magister et al., 2023; Shridhar et al., 2023; Yue et al., 2024), and data augmentation (Huang et al., 2023; Zelikman et al., 2022; Ni et al., 2023; Zhu et al., 2023; Yu et al., 2024). In this work we extend the multilingual reasoning carried out by Shi et al. (2023) in GPT-3 and PaLM (Chowdhery et al., 2022) to current open-source popular LLMs, and study multilingual reasoning consistency. Additionally, following previous works (Chen et al., 2023a; Chai et al., 2024), we employ machine translation to translate existing English data into other languages for multilingual reasoning, but we do so at a very large scale, with substantial gains in performance.
Consistency in Language Models Consistency is one of the core qualities of language models, which refers to models behaving consistently on semantically equivalent inputs (Elazar et al., 2021; Fierro and Søgaard, 2022). Intra-language consistency has been studied across different tasks, such as language inference (Li et al., 2019; Mitchell et al., 2022), explanation generation (Camburu et al., 2020), fill-in-the-blank phrases (Ravichander et al., 2020), and math reasoning (Wang et al., 2023). These works mainly consider the English language, although there is some recent research on factual consistency in a multilingual scenario (Fierro and Søgaard, 2022; Qi et al., 2023). To our knowledge, the present work is the first systematic analysis of multilingual reasoning consistency for LLMs, measuring the extent to which language models reason about the same answer to the same question written in different languages. In this context, we introduce multilingual CoT instruction tuning to boost reasoning capability across
multiple languages, thereby improving model consistency.

heading: 3 Multilingual Reasoning Consistency
text: This section provides details on the task, the data, and the experimental setup we use to investigate the LLMs’ reasoning capability in different languages and their reasoning consistency.

heading: 3.1 Task, Dataset, and Setup
text: Task Definition Generally, each math problem is fed to an LLM along with a set of manually written CoT exemplars, which is expected to generate all necessary intermediate steps up to and including the final answer. Given a set of math problems M, each problem consists of a triplet (question:qx, reasoning steps:rx, final answer:a), where the problem and intermediate steps are written in a natural language x ∈ L. We define the multilingual reasoning consistency of an LLM as the extent to which it reasons to the same answer for the same question asked in different languages, including correct consistency and incorrect consistency. For a given language pair (x, y), correct consistency (CC) is the percentage of identical math problems written in that language pair for which the correct answer is predicted. Formally:
CC(x, y) = ∑|M | i=1I (âxi = â y i = ai)
|M | (1)
Where I(·) is the indicator function. ai represents the gold answer corresponding to the i-th math question, âxi and â y i are the predicted answers to the i-th question written in languages x and y, respectively. For incorrect consistency (IC), we calculate the proportion of predicted answers that are incorrect but identical in the language pair out of the total number of incorrect answers in each respective language, and take the average of the two languages as the final result:
IC(x, y) =
(∑|M | i=1I(âxi = â
y i ̸= ai)∑|M |
i=1 I(âxi ̸= ai)
+
∑|M | i=1I(âxi = â
y i ̸= ai)∑|M |
i=1 I(â y i ̸= ai)
) /2 (2)
Reasoning Dataset As defined above and depicted in Figure 1, multilingual reasoning consistency requires both questions and reasoning steps to be written in the same language, for multiple
languages. GSM8K (Cobbe et al., 2021) is an English (EN) dataset which includes high-quality grade school math word problems, each involving basic arithmetic operations (addition, subtraction, multiplication, and division) that usually require two to eight steps to solve according to the official, provided solution. It contains approximately 7,500 and 1,319 samples for training and testing, respectively. Based on GSM8K, Shi et al. (2023) create MGSM, extending math reasoning into a multilingual setting. To do so, they select the first 250 problems from GSM8K and manually translate them into ten different languages: Bengali (BN), Chinese (ZH), French (FR), German (DE), Japanese (JA), Russian (RU), Spanish (ES), Swahili (SW), Telugu (TE) and Thai (TH). SW, BN, TE, and TH are usually heavily underrepresented languages in pre-trained language models; in PaLM (Chowdhery et al., 2022) they account for less than 0.1% of the pretraining data. We conduct experiments on these ten languages plus English, and measure the reasoning consistency between any two languages.
Model Setup We select a range of open-source state-of-the-art LLMs in three different sizes:
• 7B: LLAMA2 (Touvron et al., 2023); Qwen (Bai et al., 2023); Mistral (Jiang et al., 2023).
• 13-14B: LLAMA2-13B (Touvron et al., 2023); Qwen-14B (Bai et al., 2023).
• 56-72B: Mistral-8×7B2; LLAMA2-70B (Touvron et al., 2023); Qwen-72B (Bai et al., 2023).
We perform few-shot CoT reasoning following Wei et al. (2022a) and Shi et al. (2023), who improve the reasoning task by augmenting few-shot examples with intermediate steps. We use 8-shot for all languages except TE which only uses 2-shot due to the maximum number of input tokens. All CoT prompts in different languages are sourced from the original multilingual CoT reasoning paper (Shi et al., 2023). On the other hand, assuming that we can not access existing math problems with the corresponding reasoning solutions in some languages, a natural and simple way is to use machine translation to translate existing data (e.g., English data) into the target languages. Therefore, we compare not only the consistency between languages, but also the consistency between humantranslated (HT) prompts and machine-translated
2https://mistral.ai/news/mixtral-of-experts/
(MT) prompts to understand the impact of translated data on model performance.

heading: 3.2 Results
text: Model Performance We first compare the performance of different models, as shown in Figure 2.3
Unsurprisingly, and similar to the results reported by Shi et al. (2023), the performance generally improves for all models across all languages as the models scale up in size. However, in contrast to their findings for the large-scale model PaLM540B, open-source models still yield a performance gap between underrepresented languages (which cover less than 0.1% of the training corpora in PaLM) and high-resource languages. For instance, Qwen-7B scores above 30% in most high-resource languages (e.g., ES and FR) and below 10% in underrepresented languages (e.g., SE, BN, and TE). Particularly, Mistral-8×7B and Qwen-72B achieve
3Detailed results are in Appendix A.2.
similar or higher scores than PaLM-540B in highresource languages, while PaLM-540B has better results in underrepresented languages. Regarding HT and MT prompts, we compare all languages except EN, and observe that the results using these two prompts are very close for all models and languages we considered.
Reasoning Consistency We illustrate multilingual reasoning consistency results for different models in Figure 3. When looking at reasoning consistency between language pairs as presented in Figure 3(a), consistent with the trend in model performance, we find overall consistency to be lower for smaller-scale models (7B and 13B) and underrepresented languages (SW, BN, and TE). The correct consistency for all models improves with increasing language representation in pretraining, which is not surprising as higher-resource languages have better accuracy. Incorrect consistency, however, shows a different trend, with higher
scores between underrepresented languages and between high-resource languages in most models, and this occurs even in larger-scale models such as LLAMA2-70B and Qwen-72B. This suggests that incorrect reasoning knowledge in LLMs is similar to a certain extent between these languages.
To further analyse the impact of using machinetranslated prompts, we present reasoning consistency between HT and MT prompts in Figure 3(b). The consistency trend is similar to those in Figure 3(a), with high-resource languages having high consistency on correct answers and underrepre-
sented languages having high consistency on incorrect answers. On the other hand, consistency scores of the larger models are closer to the reasoning accuracy of using human-written prompts, indicating that using machine-translated prompts has less impact on these models.

heading: 4 mCoT Instruction Tuning
text: Based on our findings in multilingual reasoning consistency, we propose a multilingual instruction tuning framework mCoT to supervise the reasoning process, aiming to generate similar reasoning and the same results on inputs expressed in different languages but semantically equivalent. Formally, given a question q = {q1, · · · , qm} in language x and its corresponding answer – always an integer in this work – the model can generate the step-bystep reasoning solution s = {s1, · · · , sn} written in language x with the final answer.

heading: 4.1 mCoT-MATH
text: Source Data We leverage two English datasets as source data: MetaMathQA (Yu et al., 2024), which augments the training data of GSM8K and MATH with a question bootstrapping method that rewrites questions using both forward and backward reasoning paths and leverages LLMs to reformulate the question text; and MathInstruct (Yue et al., 2024), which is based on seven existing math rationale datasets annotated by humans or GPT-4 (OpenAI, 2023). This dataset contains two prompt formats: Chain-of-Thought (CoT, Wei et al. 2022b) and Program-of-Thought (PoT, Chen et al.
2023b). We select MathInstruct’s CoT samples and combine them with MetaMathQA, resulting in approximately 580,000 samples.
Automatic Translation Following Shi et al. (2023), we select the ten different languages included in MGSM as the target languages for translation. The overall framework is illustrated in Figure 4. First, we machine-translate4 all EN data into the target languages. After translation, we use the instruction format to reformulate all the data to obtain mCoT-MATH, the first large-scale multilingual math CoT dataset, containing around 6.3 million samples. This is expected to facilitate an exhaustive exploration of model reasoning consistency across languages.

heading: 4.2 Implementation
text: We use Mistral-7B as base model, training our mCoT instruction tuning framework using HuggingFace Transformers (Wolf et al., 2020) and Deepspeed (Rasley et al., 2020). During training, we set the maximum length of the input sequence to 1024, thus reducing GPU memory consumption and improving training speed. We train our model using AdamW optimiser (Loshchilov and Hutter, 2019) with a maximum learning rate of 5e-6 and a 3% learning rate warmup. The batch size is set to 32 and gradients are accumulated in 4 update steps. We train our model on 4 × NVIDIA A100 40GB GPUs for around 10 days. We report the final answer accuracy for all experiments.

heading: 4.3 Evaluation Data
text: We select two popular multilingual math reasoning datasets, in which each sample contains a question and the corresponding final answer. Specifically, in addition to MGSM, we also include MSVAMP (Chen et al., 2023a), which is constructed based on English data SVAMP (Patel et al., 2021). Chen et al. (2023a) use Google Translate to transform 1,000 questions from the SVAMP test set into nine languages: SW, BN, TH, JA, ZH, RU, ES, FR, and DE. To ensure the translation quality, they back-translate the translated text into English and ask three professional annotators to check semantic consistency manually.

heading: 4.4 Baselines
text: We consider state-of-the-art models of different sizes, both closed and open source.
4https://translate.google.com/.
Close-Source Models We include six models from three different companies as foundational benchmarks: (i) OpenAI’s GPT-3 (Brown et al., 2020), GPT-3.55, and GPT-4 (OpenAI, 2023); (ii) Anthropic’s Claude-26; and (iii) Google’s PaLM 2 (Chowdhery et al., 2022) and Flan-PaLM (Anil et al., 2023).
Open-Source Models We also compare our model mCoT with several best-performing opensource models for the sake of fairness (including size comparison): (i) WizardMath (Luo et al., 2023); (ii) MathOctopus (Chen et al., 2023a); (iii) xCoT (Chai et al., 2024); and (iv) MetaMath (Yu et al., 2024).

heading: 4.5 Results
text: Evaluation on MGSM Table 1 reports results on the MGSM benchmark. The first observation is that most existing models, including close- and opensource, perform poorly on underrepresented languages such as SW, BN, and TE. For close-source models, GPT models achieve higher accuracy in high-resource languages, with GPT-4 scoring the highest; PaLM-540 achieves competitive performance in all languages, especially reaching the highest score in low-resource languages BN, TE and TH. For open-source models, we observe that
5https://openai.com/blog/chatgpt. 6https://www.anthropic.com/index/claude-2.
WizardMath achieves less than 7% accuracy on low-resource languages which is explained by the fact that this model is trained on English data, while both MathOctopus and xCoT gain strong improvement with the help of the multilingual instruction dataset. When looking at our model, we see that mCoT significantly outperforms all previous strong baselines, and even outperforms GPT-4 in underrepresented languages such as SW, BN, TE, and TH. Particularly, mCoT has higher accuracy scores than PaLM-540B across all languages.
Evaluation on MSVAMP Table 2 reports results on the MSVAMP benchmark. We can observe that GPT-4 with 2-shot achieves the best performance in all languages except BN, where our model achieves the best results. When looking only at open-source models, similar to the observations on MGSM, WizardMath performs poorly in low-resource languages, while our model mCoT shows the highest scores across the board. In particular, mCoT outperforms MathOctopus-Mistral, the one also based on Mistral-7B, confirming that leveraging our dataset mCoT-MATH can yield substantial gains in performance. Finally, we observe that mCoT scores are very close across all languages, suggesting a lesser dependency on the low- vs high-resource aspect.
Reasoning Consistency To further evaluate mCoT’s reasoning capability in different languages, in Figure 5 we present its reasoning consistency
results on the dataset MGSM. After instruction tuning on our dataset mCoT-MATH, correct consistency shows a strong improvement as our model’s reasoning accuracy improves, and in particular we observe that: (i) this is especially true for underrepresented languages; and (ii) the scores for all language pairs are very close, with most being above 50%. It is also interesting to see an increasing trend in incorrect consistency for most language pairs, including low- and high-resource languages, confirming that our model exhibits similar reasoning capabilities across languages. Overall, these observations underscore the efficacy of our method in improving the model’s reasoning consistency.

heading: 5 Analysis
text: Case Study Table 3 shows reasoning solutions generated by our model mCoT for the same math question expressed in different languages. In this case, mCoT incorrectly reasons in the second step in TE: Yogurts cost $5.00 each, so Terry spends 60 * 5.00 =300.00 on yogurts in 30 days (EN translation of red background part), which leads to the wrong reasoning and final answer. For other languages, we observe that mCoT generates the correct final answers, while the solutions might be logically different. For example, the focus of FR is to first reason about how many packs of yogurt are needed and then calculate the total cost based on the price of each pack, while for other languages such as EN, the total cost is calculated through the total amount of yogurt and the unit price of each yogurt. Overall, these evidences demonstrate
that mCoT has a good reasoning capability across different languages.

heading: 6 Conclusion
text: We studied multilingual reasoning consistency across multiple languages for popular open-source LLMs such as Mistral and LLAMA2, which provides insights into the evaluation of LLMs. Our findings show that there is a substantial variation across the languages, with lesser resourced ones
substantially underperforming. To address this issue, we constructed the first large-scale multilingual math reasoning instruction dataset mCoTMATH, with around 6.3 million samples in eleven diverse languages. We then introduced a multilingual reasoning instruction tuning framework to train our model mCoT on mCoT-MATH. Evaluation on two multilingual benchmark datasets shows that our 7B parameter model achieves impressive reasoning consistency across all languages, and comparable or superior performance to close- and open-source state-of-the-art models even of much larger sizes.

heading: 7 Limitations
text: In this work we investigated multilingual reasoning consistency across 11 languages and eight opensource models in different sizes, but there are very many more languages and LLMs that can still bring substantial challenges and insights if considered. In addition, when considering more languages in the future, it will also be interesting to consider language families as factor. On the other hand, this work focused on reasoning consistency based on the final answer, while the consistency of intermediate reasoning steps is definitely an interesting direction. Reasoning solutions are not necessarily consistent across languages (even within the same language), since they might be logically different but still result in the same and correct answer. Therefore, automatically assessing intermediate steps is challenging and requires more explorations in the future. Regarding the dataset, while mCoTMATH boosts the reasoning capability across languages, screening high-quality machine-translated data could further improve the model. Finally, the full potential of our approach could be further explored by for example extending instruction tuning with reward learning to encourage models to generate more diverse solutions.

heading: Acknowledgments
text: The anonymous reviewers of ACL Rolling Review provided us with useful comments which contributed to improving this paper and its presentation, so we’re grateful to them. We would also like to thank the SURF organisation and the Center for Information Technology of the University of Groningen for their support and for providing access to the high-performance computing clusters Snellius and Hábrók, respectively.

heading: A Appendix
text: A.1 Prompt Examples
A.2 Multilingual reasoning results on MGSM

heading: Future Work
text: In addition, when considering more languages in the future, it will also be interesting to consider language families as factor. On the other hand, this work focused on reasoning consistency based on the final answer, while the consistency of intermediate reasoning steps is definitely an interesting direction. Reasoning solutions are not necessarily consistent across languages (even within the same language), since they might be logically different but still result in the same and correct answer. Therefore, automatically assessing intermediate steps is challenging and requires more explorations in the future. Regarding the dataset, while mCoTMATH boosts the reasoning capability across languages, screening high-quality machine-translated data could further improve the model. Finally, the full potential of our approach could be further explored by for example extending instruction tuning with reward learning to encourage models to generate more diverse solutions