MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages
Authors: Zixian Huang
Wenhao Zhu
Gong Cheng
Lei Li
Fei Yuan
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Multilingual
Reasoning
Primary area: natural_language_processing
TL;DR: Merging the built-in capabilities of LLMs and external capabilaties of multilingual model to boost multilingual reasoning.

Abstract: Reasoning capabilities are crucial for Large Language Models~(LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MergeMinds, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MergeMinds consistently outperforms all baselines, especially in low-resource languages.  Without updating the parameters of LLMs, the average accuracy improved by 6.7 and 8.0 across all languages and low-resource languages on the MGSM dataset, respectively.

MindM:rger: Efficiently Boosting LLM Reasoning in non-English Languages
Zixian Huang', Wenhao Zhu', Gong Cheng's Lei Li', Fei Yuan
State Key Laboratory for Novel Software Technology, Nanjing University
2 Carnegie Mellon University
Shanghai Artificial Intelligence Laboratory
(zixianhuang, zhwwh Aamaill nju. edu scheng@njj.edde edu. leili@cs. cmu. edu, quanfei@pjlab.oo org.cm....c..
Abstract
Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this dis- parity, some works fine-tune LLMS to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding propose new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilin- gual reasoning performance. Furthermore, a two-step training scheme is introduced first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities LLMss. Experiments on three multilingual reasoning datasets and language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively
LLMs, we
Introduction
One of the primary focuses of Artificial Intelligence research currently revolves around improving
its reasoning capabilities [Ahn et al.) 2024 Minaee et al. 2024 which is derived from the need
to enable Large Language Models (LLMs) Ouyang et al. 2022 Touvron et al. 2023 Jiang et al..
2023al to think rationally and perform functions like humans [Imani et al. 2023 Jiang al. 2023b
Substantial progress has been made English reasoning Yu et al. 2023 Yuan et al. 2023a
but the
performance in non-Ense English, especially low-resource languages, still lags behind (Shi et al.
2023
due to the scarce of multilingual training data Touvron et al. 2023].
Existing work tries to use external models to compensate for the deficiencies of LLM in multilingual reasoning. Some works use the relearning- strategy, which uses translation models to generate multilingual training data for fine-tuning L.MMs to relearn reasoning in each language [Chen et al.)
2023 Chai et al. 2024]. Some other works use the replacement-basee strategy, which use translation models to translate queries from non-English to English text for replacing the non-English input of
Corresponding author:
Our code is available on
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Query
LLM Generation
Query
Generation
Fregfsht Satie slege 8 hours on Monday For the pext 2 days the slopt 2 bouss les, each, If the not of the weck dhe stept i bout moes than thrse 2 days how many hours did she sloep local Broughou the week?
2-2
English Claire makes 3 eap soudet every mumiing for brekfast. low many doons aggs will she entin weaks?
7
6
2 4:84
6+1=7
01/12=7
(5 30
B 14 -4
Chinosn 90 HB 9CO2 8X 12 4K. ROFERANE S-1P 174682
X
7x22 21 44-64
Chimase SSHEIN 187 6.102, H 2 7 2/1EE, MER F889 RITEE 12122E 2 3E 1 ONEE, MAKER T- 7136 7-9977777
2-2
6+1=7
6x2
(- St 20 8+ 22 m48
Figure 1: Examples of multilingual mathematical reasoning from the MGSM dataset. LLM can generate correct and incorrect answers when asked in different languages.
LLM (Shi et al., 2023]. Both strategies try use the translation model to help LLMs master new capabilities, but the insufficient translation quality constrains the performance of these methods.
Moreover, certain capabilities, such as reasoning and language understanding, are built-in in LLMs and should be utilized without the need to develop them from scratch. For the reasoning capabilities, regardless of the language in which the same mathematical question is posed, the correct reasoning process remains consistent. It shows that the reasoning capabilities is universal rather than language-specifie [Brannon] 2005). For language understanding capabilities, while the Chinese question in the first example of Figure may cause a reasoning error by failing in understanding the English-origin term "dozen of". the LLM successfully differentiated between "the week" " and "the weekdays" in the Chinese question, contrasting with the failure of the English question in the second example. It shows that although the proficiency may not match that of English, expressions in non-Enne non-English languages remain valuable to LLMs.
as illustrated in Figure [
Considering that the built-in reasoning and language understanding capabilities of LLMs need to be better utilized, in this paper, we propose a new method MindMerger, which preserves the minds of reasoning and language understanding capabilities in LLMs, and merges the external language understanding capabilities from pre-trained multilingual models to boost the multilingual reasoning effects. To address the challenge of insufficient generation quality of external models, MindMerger feeds the L.M the undecoded query representation from the multilingual model rather than text. Additionally, it uses augmentation strategy that combines the encoded query representation with the input of LLM to utilize both external and built-in language understanding capabilities.
Given the inconsistency in the representation space, understanding the query representation encoded by multilingual model is not trivial for LLMs. To address this, we propose a two-stage training scheme including the mapping stage and the augmentation stage. In the mapping stage, we train MindMerger to embed the language capabilities of the multilingual model into the LLM by using accessible general bilingual pairs such as translation data. In the augmentation stage, MindMerger is further trained to collaboratively utilize the built-in capabilities of LLM and the embedded external language capabilities by using query translation task data generated from translation model. Through- out the two stages, only a small number of parameters that connect two models are trained, while both the multilingual model and the LLM are frozen to prevent their built-in capabilities from forgetting.
Extensive experiments are conducted on three multilingual reasoning datasets and a language un- derstanding dataset. Taking on the MGSM dataset (Shi et al. 2023] as an example, MindMerger outperforms all baselines and achieves lead of at least 6. 7 on the average accuracy across all languages, and its performance in low-resource languages is even more significant leading by at least 8.0% (8 4.3.. Compared with the replacement- based method that translates non-English text into English as the LLM input, MindMerger can lead by at least 6.6% in average accuracy based on the same translation model ($ 5. Benefiting from the accessible general bilingual pairs used in the mapping stage, the average accuracy across low-resource languages increased by 14.9% ($ 5.2)
Our contributions can be summarized as follows:
We propose a new method MindMerger to boost the multilingual reasoning of LLMs, which pre- serves the built-in reasoning and language understanding capabilities of LLMs while augmenting them with the external language understanding capabilities from multilingual models.
We propose two-stage training scheme to help MindMerger sequentially learn to embed external capabilities and collaboratively utilize internal and external capabilities.
2

---

MindMerger achieves the best results on four dataset about multilingual reasoning or under- standing datasets, notably improving all languages, including low-resource languages, with an average accuracy increase of at least 6.7% and 8.0% on the MGSM datasets.
Related Work
Multilingual Reasoning. There have been some attempts to improve LLM's performance
multilingual reasoning. Several works design crafted prompts to assist LLMs in reasoning (Huang Qin et al. 2023]. but their effectiveness diminishes when used with open-source LLMs
e al.
2023
like
lama
Touvron et al. [023] due to limited capacity for multi-step instruction -oollwiinnnooooiingg Guang
al. 2023
Supervised fine-tuning LLMs is another effective way, where some works translation
models to translate query-responas [Chen et al.) 2023]. or query-only [Zhu et al. 20244) to build multilingual task datasets, enabling LLMs to relearn reasoning language understanding. Some other works utilize external models to generate English text [Shi et al. 2023 to replace the non- English input of LLMs, aiming utilize the reasoning capabilities of LLMs directly. However, the above methods are limited by the generation quality of the translation model, and the built-in reasoning or language understanding capabilities of LLM are neglected. In contrast, MindMerger avoid the loss introduced by autoregressive decoding and employs an augmentation-baset strategy that utilizes the built-in capabilities of LLMs enhance multilingual reasoning performance.
Model Merging. Model Merging is popular topic in recent LLM research. Generally. it aims to combine external module to strengthen the capabilities that LLM lacks, such as multi-modal
vision capabilities (Liu et al. 2023 Li et al. [023] Zhu et al. 2023]. Some works [Sun et al. 2021 Bansal et al., 2024) find that interpolating models with multilingual capabilities using cross-attention
multilingual tasks, but research on merging multilingual models and LLMs with English reasoning capabilities is still scarce. Recently, Yoon al. [2024] merge the encoder of multilingual model and the LLM to improve the multilingual reasoning performance. However, the input of LLMs is completely replaced by the output of multilingual encoder, making its built-in multilingual capabilities underutilized. In addition, only using English task data for training limits the performance of model merging. Instead of replacing the input of LLMs, we collaboratively utilize the features of the multilingual model and use the available general bilingual pair train obtain multilingual reasoning capabilities.
can improve the performance
3 Approach
Model Structure
Training Scheme
L1M Transfomerr layer
Mapping Stage GGeerrl kinngul pain)
Quary e
LLM Frebekking
Mapprig
2. 2.A ugmenlation Slage (Quey rocddator task data)
Query
Mulllingaa Encodar
Query
Query
Figure 2: Overview of the model structure and training scheme of MindMerger, which consists of an LLM (blue) and a external model (yellow) and trained by a two-stage scheme.
Given an LLM with skilled reasoning capabilities and useful language understanding capabilities, our target is to maintain the built-in capabilities and compensate for its shortcomings in non- son-English language understanding capabilities with an external multilingual model. To this end, as shown in Figure 2 we propose MindMerger that uses the output of the multilingual model as augmentation complementing the original input ($ 3.T. We further design two-stage training scheme to learn the collaborative utilizing both the external and built-in capabilities ($ 3.2

---

3.1 Model Structure
Given query q with tokens, we first utilize the multilingual model to understand and encode it into a representation X that is more general and reduces the challenge of multilingual understanding:
X Encoder
(I)
where Encoder(-) is a pre-trained multilingual model, typically using its encoder part, and X E Rixd is the hidden state output of the multilingual model with a dimension of d1.
The representation X resides in the multilingual model space, which is separate from the LLM space and cannot be used directly. Therefore, introduce a mapping layer:
X Mapping(XX)
(2)
where X E R is the projection of X on the space of LLM. Unless otherwise stated, the implementation of Mapping(-) is two-layer multi-layer perceptron (MIP).
The acquisition of X utilizes the language understanding capabilities of the external multilingual model. In order to take advantage of the built-in capabilities of the LLM,, we prompt it directly;
T Embedding(4).
(3)
where T E R is the representation of query g in the space of LLM and Embedding(.) is the embedding layer of LLM.
Then, we concatenate the query representation from the multilingual model and the LLM for the collaborative utilizing of L.I.Mss capabilities:
(X,T) = [(bos); X: (sep): T)
(4)
where (bos) E RRd2 is the representation of the start token of LLM and (sep) Rd is trainable
variable that denotes the boundary of X. Finally, T) is fed to LLM to generate the response.
3.2 Two-Stage Training
The training scheme of MindMerger is divided into two stages: mapping stage and augmentation stage. The former helps LLM learn to the capabilities of multilingual model, and the latter helps LLM collaboratieely utilize its own and the capabilities from multilingual model. Examples of training data for each stage are shown in Appendix D
Mapping Stage. Given the representation spaces between multilingual model and LLM are distant different, it is not trivial for LLM understand and utilize the external capabilities provided by multilingual model. In order to better learn to utilize external capabilities, we force MindMerger to focus on input from the multilingual model using the replacement-baat based strategy during this stage. Specifically, we simplify the input of Equation as follows:
M = [bos); X; (sep)].
(5)
Since general bilingual pairs such as translation data in vast availability, we use it from various languages to English to train MindMerger in this stage. The loss function is outlined as follows:
arg min log P (.
where Y is the text of training target, @ and Φ are the parameters of the multilingual model and LLM respectively, which are frozen during the training to prevent forgetting, and a contains the trainable parameters of the mapping layer in Equation (2) and the boundary token (sep).
Augmentation Stage. Although MindMerger have learned to utilize external capabilities in the mapping stage, the sataacementt-bssee strategy may cause LLMM to neglect the use of its own built-in capabilities. To help MindMerger further learn to merge capabilities from external and built-in LLM, in this stage use the augmentation based strategy as described in the Equation (4). The loss function is calculated as follows: arg min log
where ē is initiated from the checkpoint of a trained at the mapping stage, and the parameters the multilingual model and LLM represented as 0 and σ, respectively, are kept constant during training. In this stage, the query translation task data generated from public translation models used as the training data to adapt MindMerger to downstream task.

---

4 Experiments
4.1 Compared Methods
Our Methods. Two variants of MindMerger were implemented. (1) The implementation described in $ $3.1]s is denoted as MindMerger-Soff.. (2) MindMerger-Haree augments the prompts of L.LM with the translated query given by the translation model (Appendix Doo the details of prompts).
Baselines. We compared our methods with three categories of baselines. (1) One basic method MonoReason Yuu et al.) 2023 Zhu et al. 2024] which is fine-tuned on the English task dataset. (2) Three relearning-basee methods that use task data with query translation, including the full- parameter fine-tuning model MultiReason-Full (Zhu et al., 2024], parameter-efficien fine-tuning model MultiReason-Loon [Hu al., ]022], and the state-of the-thherrtt method QAlign [Zhu et al.) [024] which first learns language understanding by training LLM on query translation data and then further fine-tunes L.M MonoReason. (3) Two replacement-based methods that introduce external models, including Translate-Fn [Shi et al. 2023] and LangBridge (Yoon et al., 2024). Translate- -nn is a hard seplacement-baseee which translates the query into English replace the prompt of LLM. LangBridge
is a soft replacement-based method which replaces the input of LLMs with the hidden states output
by mT5-xl [Xue al. 2021]. The prompts of each baselines are presented in Appendix D
Details. Unless otherwise stated, we used the encoder part of mT55xl [uee et al. 2021] as the multilingual model in methods, used the NLLB200-3.3B as the translation model for baselines, and used the Llama 2-7B as the LLM for all methods. The influence of different multilingual models including encoder-only, decoder-only and mecoerr-dccoed architectures will be analyzed in $ Both MindMerger and all the baselines, except QAlign, are based on the same MonoReason model. Additionally, MonoReason and QAlign are trained based the same LLM. Specifically, we used the publicly available checkpoint given by Yu et al. [2033] as MonoReason model for mathematical reasoning task. For all models, we set learning rate=2e-5, batch size=128, max length=512, and epoch=3 and used 8 NVIDIA A100 GPUs for training.
4.2 Datasets
Evaluation Datasets. We experimented MindMerger with the latest multilingual mathematical reasoning MGSM (Shi et al., 2023) and MSVAMP (Chen et al., 2023), where MSVAMP serves as out-of-domain test set. In order to evaluate the diverse multilingual reasoning capabilities
2021] in commonsense of the models, a challenging multilingual dataset X-CSQA [Lin et al.
reasoning task and language understanding dataset XNLI [Conneau et al. 2018 in natural language inference (NLI) task were used. The statistics of these datasets are presented in Appendix D
Training Datasets. Three categories of training data were used in our methods and baselines. (1) General bilingual pairs. We used the translation data from the multilingual language to English and randomly sampled 100K of data for each language (except English) from the Lego-MTT (Yuan
et al. 2023b) dataset, which is a large-scale translation dateset that contains all the languages that involved in our experiments. (2) English task data. We used MetaMathQA Yu et al. 2023 and
MultiNL WWilliams et al., 2018) datasets for mathematical reasoning and NLI task, respectively. Similar to Huang et al. [022]. we unified the training set of X-CSQA, OpenBookQA [Mihaylov et al., 2018), ARC [Bhakthavatsalam et al.] 202T] and QASC [Khot et al., 2020] to train commonsense reasoning task more fully. (3) Query translation task data. We used the translated results given by Chen et al. [2023] and the official dev set of XNLI for mathematical reasoning and the NLI task, respectively, and translated the X-CSQA training set based M2M100-1.22 [Fan et al. 2021
4.3 Experimental Results
MindMerger improves LLM performance on all datasets, especially benefiting low-resource languages. MindMerger-Sone has an average accuracy better than all other baselines at least 6.7%, 3.2% on the MGSM and MSVAMP datasets in Table I demonstrating its remarkable multilingual reasoning capabilities. For the commonsense reasoning task in Table MindMerger-Son also signifi- cantly leads all baselines by at least 4.1%. In Table 3 MindMerger-SS language understanding, which significantly outperformed all the baselines (with p-value < 0.01). MindMerger- Hard achieves the best results except MindMerger-Sont on three out of four datasets, demonstrating the advantages of augmentation- based methods over other categories of methods.
demonstrates advantages in
5

---

Table 1: Experimental results MGSM and MSVAMP datasets. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages. respectively. Referring to Shi al. 2023 regard Bn, Th, and Sw low-resourse languages, and regard the remaining languages as high- high-resource languages. We used the checkpoints given by Yu al. (0233 as the MonoReason models.
Sw
Ja
Avg.
Manoficasan
Ya 2025
2014
6.8
36.4
54.4
53.0
57.2
28.0
MatuResoon
48.3
QAlign
2la
39.6
52.4
59.5
37.5
LangBragz Yoon
M144
42.8
50.4
43.2
4000
45.2
$6.4
50.8
52.4
58.0
63.2
45.5
52.3
50.2
Tranclat-Po
48.
37.6
37.6
49.2
46.8
60.4
$6.4
47.6
59.5
65.5
41.2
55.1
50.6
MiniMerger-Hal
52.4
60.4
62,0
Mindhlegge Soft
50.4
$2.8
$7.2
$4.4
$3.6
61.2
57.6
60.3
58.4
66.8
53.5
59.0
57.3
MSVAMP
Bm
Th
Sw
Ja
7hh
Do
Fr
Ru
FA
En
1rr.
Hrl.
Avg.
Menolieaon aa 2003
Zin et 5074
12.5
15.8
17.2
$4.00
55.0
63.9
65.2
58.1
64.3
67.11
15.2
61.22
47.4
MaltReaw.n-en
39.4
43.8
39.1
.5.22
55.3
60.4
59.1
56.8
60.5
64.2
40.5
58.8
53.4
MaltiReeen-Flll
Full
34.8
38.1
39.8
43.4
42.9
45.6
45.8
45.0
46.8
57.6
45.1
47.7
54.8
58.0
61.1
63.3
LangRrang
2104
56.9
MindMerger-Tnl
43.6
46.7
50.6
61.4
60.6
63.9
65.6
62.1
65.0
67.8
47,0
64.1
58.9
MinlMerger-Sof
65.0
MindMerger- stands out notably in enhancing the reasoning capabilities low-resource lan- guages. Compared to all baselines, it delivers average accuracy improvement of at least 8.0% and 5.0% across low-resource languages on the MGSM, MSVAMP datasets.
Utilizing the built-in reasoning capabilities of the LLM rather than relearning for non-Englo non-English is more effective way boost multilingual reasoning capabilities. Is not easy for relearning- based methods to improve the performance of high-resource and low-resource languages simultaneously. For instance, MultiReason- of 6.9% to 38.4% on the MGSM dataset, but its performance in high-resource languages declines from an average of 51.8% to 45.6%. Compared with the three relearning-basse based baselines, MindMerger-Son achieves a lead on both low-resource and high-resource languages and outperforms them by at least 6.9% on the MGSM datasets for the average accuracy across all languages.
improves its performance in low-resource languages from an average
Utilizing the built-in language understanding capabilities of LLM rather than replacing them boosts multilingual reasoning capabilities more significantly. Comparing Translate-En and MindMerger- Hard which are completely consistent except for the input strategy, the augmentation based MindMerrer-HHn Hard has a higher average accuracy than the replacement-basee Translate-En of 4.4% and 11 1188% across high-resource languages in the MGSM and MS datasets respectively, showing that lonllshlish input can help LLM enhance query understanding. Although the under- standing capabilities of LLM in low-resource languages are limited, MindMerger-Haun still leads Translate-En by 2.3% on the MSGM dataset. Larger boost comes from MindMerger-Sote which leads all replacement-based methods with average accuracy of at least 6.7 % across all languages.
Table 2: Results on the X-CSQA dataset. Avg. represents the average accuracy across all languages.
X-CSQA
Sw
Ur
III
Ar
Vi
Ja
P
Zh
MI
Ru
It
De
Pt
Fr
Es
En
Arg
MonoRoveen Yogrel 2I3 Ziee 4
305
242
25.1
32.9
32.3
50.0
49.1
5066
56.5
57.5
56.0
56.0
61.3
61.7
63.5
64.0
76.5
$1.3
Mulli Rellieoon--err Cane Iorteni 2004
25.1
32.0
39.2
42.2
36.6
55.9
60.6
62.2
61.3
62.5
66.3
649
66.2
67.4
67.7
79.3
56.9
Muli Rusen-ullll Piine 2004
276
29.2
32.0
25.7
38.5
38.7
45.5
43.8
45.9
46.5
50.2
49.1
51.2
52.1
54.3
67.2
43.5
QAlian (2ueral alal Pile
35.1
32.6
37.8
36.3
5U.5
49.2
$1.3
54.8
56.3
56.5
58.3
58.5
59.8
60.3
63.1
75.7
52,5
LangBrune 1008 D. 2024
318
30.5
30.6
3066
33.3
33.9
39.8
39.8
38.4
35.1
39.1
37.4
36.3
38.2
38.4
44.4
36.1
Trandase En S01 00
363
41.3
48.4
44.6
51.3
47.1
53.3
31.5
55,0
54.4
56.3
57.3
54.7
57.2
553
71.3
52.3
MindMergr-Hall
29.9
52.9
59.7
62.5
636
53.1
MinlMeeprSSff
60.6
66.8
6R.1
5
Analysis
5.1 The Usage of Multilingual Model
MindMerger can flexibly interpolate among various multilingual models. We experimented with a wide variety multilingual models, including decoder-only models mGPT [Shliazhko et al. 2022 encoder-only models mBERT [Devlin al. 2018 and XLM- MoMM-RoBERTa-aage [Conneau et al. 2019

---

Table 3: Results the XNLI dataset. Avg. represents the average accuracy across all languages.
XNLI
Sw
Ur
Hi
Th
Ar
Tr
E1
V1
Zhi
Ru
Bg
De
Fr
Es
En
Arg.
MenoRasson No cl d 3023 Zune
2014
45.9
49.2
55.7
55.4
63.9
61.9
65.7
13.7
74.7
77.6
76.7
30.5
82.2
62.5
90.0
68.7
Multikesonn CYeumre
45.9
49.3
56.4
55.7
60.9
61.9
64.7
73.7
24.7
76.3
76.7
30.5
62.2
82.3
90.0
68.9
MultiReaun- Full Fourer 2009
56.3
57.5
61.7
60.1
61.3
65.6
67.0
73.7
79
79.7
78.7
82.3
62.9
83.9
88.6
71.9
QAlign Zha
02.2
63.3
67.0
66.3
73.7
79.2
79.4
83.1
89.1
73.5
Langbabgge
66.9
71.1
75.2
79.1
78.5
77.4
79.9
83.4
76.5
Translate
68.9
79.3
76.7
74.8
76.0
80.4
67.4
75.1
MinlMago-Haal
63.6
56.6
63.2
78.4
Table Merging with different multilingual models on the MGSM dataset. # Parm represents the number of parameters of the used external model. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resoure regh-resuure languages, and all languages, respectively. Referring to Shi et al. [2023], we regard Bn, Th, and Sw as low-resourse languages, and regard the remaining languages as high-resource languages.
MGSM
# Parm
Bn
Th
Sw
Ja
Zh
De
Fr
Ru
Es
En
Lrl.
Hrl.
Avg.
Translate-Fo
M2M100-418M
484 M
30.0
38.0
38.8
31.6
50.8
52.0
50.0
42.4
54.0
65.5
35.6
49.5
44.7
M2M100-1,2B
1,239
42.4
46.8
42.0
47.1
NLLBB-20-11338
1,371 M
46.0
32.0
40.4
47.2
45.6
55.2
51.2
46.0
55.2
65.5
39.5
52.3
47.9
NLLBB-20033.8
3,345
48.4
37.6
46.8
56.4
47.6
41.2
55.1
MiodMergoe-Hard
M2M 12M1000-18M
39.6
48.8
64.8
34.7
56.2
49.7
M2M100-1.20
1,239 M
40.0
36.0
47.6
52.4
50.8
58.0
56.4
60.8
61.2
66.8
41.2
58.1
53.0
NIL.B-200-1.3
1,371
48.0
70.8
38.9
58.4
NLLB-200-3.3B
3,345 M
46.0
36.0
48.4
52.4
54.4
60.4
56.0
60.4
62.0
71.2
43.5
89.5
54.7
MindMMerger-So Soff
mGPT
61.2
64.0
18.5
55.5
mBERT
178 M
30.8
37.6
46.8
$0.0
48.8
55.6
52.4
59.6
60.8
66.4
38.4
56.2
50.9
XLM--RBEERTD-aagge
66.4
48.9
$7.9
M2M1000-18M
282 M
49.2
$2.8
46.0
48.8
52.4
59.6
58.0
59.2
60.8
65.6
49.3
57.8
55.2
M2M100011.2B
635
58.0
51.7
NLLB 2.00-1.3
766 M
45.6
47.6
57.6
$4.4
$2.4
57.2
57.2
60.8
60.8
66.8
50.3
58.5
56.2
NLLB-200-3.38
52.4
mTsS -nrgee
564 M
40.4
47.2
53.6
47.6
$1.6
59.2
55.2
57.6
56.8
66.4
47.1
56.3
51.6
mT33xl
1,670
and the encoder part of encoder-decoder models M2M100 (Fan al. 2021 NLLB-200 [Costa-jussá et al. 2022] and mT5 [Xue et al. 2001]. The experimental results in T show that the encoder part of the encoder-decoder model and the encoder-only model are more suitable interpolate into MindMerger-SSe than the decoder-only model, which can achieve better performance than mGPT with smaller number of parameters. M2M100 the most cost-effective model, reaching or even exceeding the performance of XLM-RoBERTTa-arrg and m'55-large while using only half of the parameters of T.M-RoBBRTa--lrree and m1---arrge
Our augmentation based strategy outperforms the translate-then-replase strategy. Compar- ing Translate-En and MindMerger- Hard which only differ in input strategy, augmentation-bian MindMerger-Hart based on the same translation model, consistently exceeds Translate-Hi En by 5.0%, 5.9%, 4.7%, and 4.1% 1% in the average accuracy. MindMerger-Son further expands its lead with an increment the average accuracy by at least 6.6% based on the same translation model.
MindMerger-Soft is a better utilization of existing multilingual model. As shown in Table 4 the performance MindMerger-SSon consistently exceeds MindMerger-HHnd under the same multilingual model with increases of average accuracy of 5.5% and 3.3% on two versions of M2M100, and 3.6% and 2.5% on two versions of NLLB-200 Although only the encoder part of the multilingual model is used, MindMerger-SSof merges LLM with dense representation rather than decoded text based on sparse bag-of-words. enhancing the effectiveness of utilizing multilingual model.
A more powerful multilingual model can better enhance the multilingual capabilities. We compared the different sizes of each encoder-decodee decoder models and consistently observed that the larger version outperformed the smaller one in Table 4 With the help of the larger model size, the average accuracy improved to 1.1%. %, 1.0%, and 3.7% on M2M100, NLLB-200, and mT5, respectively. The improvement in low-resource languages is even more obvious with an average increase in accuracy of 1.1% 1.0% and 3.7% in M2M100, NI.LB-200, and mT5, respectively. This underscores the greater imperative to enhance language understanding in low-resource languages.
7

---

70
65
Augmercctionn Stage
Accura3
55
50
20
10
30
Lrl.
Hrl.
Avg.
Avg.
40
Lrl.
Hrl.
Avg.
Mapping Stage
(b) Augmentation Stage
Replacement vs. Augmentation
Figure 3: Ablation experiments of MindMerger-Soe on the MGSM dataset. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. Referring to Shi et al. [2023), we regard Bn, Th, and Sw as low-resourse languages, and regard the remaining languages as high-resource languages.
5.2 Ablation Studies
Mapping Stage. In Figure 3a ablated the mapping stage observe its necessity. A significant drop in the performance of low-resource languages can be observed when ablating the mapping stage, which shows that the accessible general bilingual pairs beneficial help Mind understand the low-resource language information that exists in the multilingual model representation space. More detailed are reported Appendix B.1
Augmentation Stage. In Figure 3b) we ablated the augmentation stage to observe its necessity. can be observed that after ablating the augmentation stage, without using any task-related data, MindMerger-Soo can still outperform MonoReason on low-resource languages, which demonstrates the generalization of MindMerger-Sool Soft that benefits from using accessible general bilingual pairs in the mapping stage. Furthermore, when the augmentation stage is added, MindMerger-S exhibits a significant improvement, demonstrating its effectiveness in learning the utilization of both external and built-in capabilities. More detailed are reported in Appendix B.2
Replacement vs. Augmentation. MindMerger uses the representation X outputted by multilingual model to augment the LLM's original representation rather than replace it to better utilize the built- in capabilities of LLMs. To verify the advantages of the augmentation-basee strategy, removed T method. As shown in Figure although with exactly the same training data and process, the performance of the replacement-batt based MindMerger-St drops significantly, indicating that is valuable to use the built-in capabilities LLM to understand the original input. More detailed are reported in Appendix B.3
to make MindMerger-Son
replacement-bes
Table 5: Results on the MGSM dataset based on MetaMath- 13B and MetaMath MetaMathh-Mistral- Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. Referring to (Shi et al. [203), we regard Bn, Th, and Sw as low-resourse languages, and regard the remaining languages as high-resource languages.
Meta Lhaathhh-Lma-199
Th
Ja
De
Ru
En
Hrl.
Avg.
MonoReason Yu et al 2023 Zhu
2024
12.0
8.8
48.0
56.0
64.0
63.6
62.0
67.2
70.8
9.1
61.7
45.9
Multieeson- Full
/10 R
2023
44.8
51.6
50.8
58.0
61.6
64.8
59.2
60.8
67.6
66.4
49.1
62.6
58.6
QAlign Zhu al.
38.4
49.6
46.0
52.4
59.2
62.0
62.4
64.4
67.2
69.2
44.7
62.4
57.1
Langänigge
33.6
55.2
58.8
65.2
Translato-En Sindlal 1023
34.8
54.0
44.4
44.4
58.0
53.6
$4.0
45.6
62.4
70.8
44.4
55.5
52.2
MindMerger
67.6
MindMergee-Soott
55.2
$9.6
56.4
60.0
6014
65.2
63.6
68.0
69.6
68.8
$7.1
65.1
62.7
MetaMatth-Mistraa-77
Bn
Th
Sw
Ja
Zh
De
Fr
Ru
Es
Em
L.tl
Hrl.
Arg.
MonoReuson et 2023
70.4
Multicaason- Losa Qi A034
46.8
51.2
39.6
$4.4
62.4
72.0
66.0
68.4
70.0
76.0
45.9
67.0
60.7
MultReason.FF Full
18.4
26.4
26.8
30.8
28.8
32.4
34.8
32.0
38.0
39.6
23.9
33.8
30.8
QAlign (7hu.
49.44
59.2
59.8
60.2
65.8
LangBriuge Tooe <
2024
50.0
60.0
47.2
58.4
65.6
68.4
68.8
68.4
65.6
65.6
52.4
65.8
61.8
Translate En
54.6
58.7
47.7
57.2
63.1
50.4
56.7
64.9
58.6
69.7
53.7
60.1
58.2
MindMerger-HHar
52.4
48.4
57.6
62.4
60.0
66.4
66.8
69.6
71.6
76.4
52.8
67.6
63.2
MindMerger-Soff
57.6
59.6
53.2
57.2
68.8
69.2
69.6
68.4
71.6
79.2
$6.8
699.1
68.4

---

5.3 Merging with Different LLMs
MindMerger can be flexibly integrated with different LLMs. verify this, we conducted experiments
on different type of LLM, MetaMath-Miisral-75 (Jaan et al. 2023a Yu al. 2023]. and larger size, HitaMath--llamas 13B [Touvron et al. 2023, Yu ct al. 2023]. The experimental results are
shown in Table 5 MindMerger-Son Soft has achieved superior performance actoss various baselines. The average accuracy of MindMerger-Sow on the MetaMath-Llame 1 13B and MctaMath-Misttra- versions is at least 4.1% 1% and 4.7% higher than all baselines, respectively, demonstrating the potential of extending MindMerger to a wider range of LLMs.
30
10
0-
-10
-20-
-00
-30
-20
-10
10
20
30
-40
-20
20
40
(a) LLM embeddings
(b) mapping layer outputs
Figure 4: T-SNE visualization in the spaces of the LLM embeddings and mapping layer outputs.
5,4 Representation Space Changes
For each language, we selected the same 100 texts from the Flores-101 [Fan al., [021] dataset, used the mean pooling operation to obtain the representation vectors of the L.LM embedding and the hidden states output by and the mapping layer, and visualized it based on T-SNE (Van der Maaten and Hinton| 2008). As shown in Figure the representation spaces of non-English on the LLM embedding are independent and away from English, especially in low-resource languages, which leads to understanding challenges for non-English and the inability to use built-in reasoning capabilities. By contrast, as shown in Figure 4b] the representations of all languages outputted by the mapping layer almost overlap with English, which reduces the difficulty for LLM to understand non-Ent Lon-Englihh/ languages and enables various languages to utilize the built-in reasoning capabilities.
5.5 Supplementary Experiments
We experimented with several supplementary experiments, including the influence of training dataset size used in augmentation stage (Appendix [).1), the selection of mapping layers structure (Ap- pendix A..)). the usage encoder-decoder model in MindMerger-S tive analysis representation space changes (Appendix A.4), and the translation performance of MindMerger-Si after mapping stage (AppendixAA.5 A.5)
(Appendix AA.3). the quantita-
6 Conclusion
This paper explores way to more fully utilize the built-in capabilities of LLMs to improve multi- lingual reasoning effects. We proposed MindMerger merge the expert multilingual capabilities in the multilingual model with the skilled reasoning and not very proficient but useful multilingual capabilities in the LLM. Through more fully utilizing the potential of LLM and more effective fusion of multilingual models, the performance of MindMerger exceeds all baselines on three reasoning datasets and language understanding dataset. In the future, we will explore the possibility of MindMerger empowering more professional skills besides reasoning, such code generation.

---

Acknowledgement
Shervin Minaee, Tomás Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. CoRR, abs/2402.06116. 2024. doi: 10.48550/ARXIV.2400001111 This work is partially supported by the National Key R&D Program of China (NO.2022ZD0161100))
References
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Neele Falk, Sara Papi, and Mike Zhang, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024: Student Research Workshop, St. Julian ss Malta, March 21-22, 2024, pages 225 237. Association for Computational Linguisties, 2024. URL ptttp::/aalannthlogg" org/ 2024. www.laaalls...... 17
URL https:////dd.... org/10. 10. 48550/ arXiv 2402. 06196
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022. URL http://:: nips cctppper_ppppppaeeeeee lnmeerffilese wiweslllappppeee wwtpe///aaaee//222///////////// http/paape22222hal//// -tap::0001171----ttattt------eo Confoettttoooooooeeeeee html
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer. Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR. abs/2307.00 09288, 2023. doi: 10.48550/ARXIV22000 09288. URL htttt::///ooi wrg/100 18555000.88/ 2ww.22337009233
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06822 2023a. doi: URL http:: org/10. 485500/ arXiv 2310. 06825
00.48550/ARXII.2310.6828
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. Sunayana Sitaram, Beata Beigman Klebanov, and Jason D. Williams, editors, Proceedings of the The 61 Annual Meeting of the Association for Computational Linguistics: Industry Track, ACL 2023, Toronto, Canada, July 9-14, 2023, pages 37-42. -2.2 Association for Computational Linguistics, 2023. doi: 10. 1rgg/0..1655..3 10...0.1113//1/22220 acl- ielliindustry... //doi.org//0.188800/
URL https
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Zhang, Zhenguo Li, and James T. Kwok. Forward-backwand reasoning in large language models for verification. CoRR, /bs/2330007758 2023b. doi: 00.48550/ARX1V.230066 U
https://ts://do 485t:e//0/0/...../////////// 2308
07758,
10

---

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. CoRR, abs/2309. 12284, 2023. doi: 10.48550/ARXIV22300. 12224, URI.
htttps://dii. ottp:///ooo.or//////// 41810.4850/// arXiv 2309 12284
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling re- lationship on learning mathematical reasoning with large language models. CoRR, abs/2308.01825. 2023a. doi: 10.4850/ARXI.2308.000 01825
htLhhttp://ddd org/ 10. 48550/ attp:///xxxv...... 2308
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual Mhainn-of-though reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. net/ ht/p/ddfff 1idddff--GCC IXp.
URL https://
Nuo Chen, Zinan Zheng. Ning Wu, Linjun Shou, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. CoRR, abs/2310.202244 2023, doi: 1048550/ARXIV23100002222 URL https (8550/arXiv 2310.20246
//doi.org//
Linzheng Chai, Jian Yang. Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xinnian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. CoRR, hass24001.070311 2024. doi: 10.48550/ARXIV. 2401.07037. URL htttp::///oooo org/10. 10/g:/1044400 485.0.485/////. 40550/85555iyy arXiv. 2002220010003
Elizabeth M Brannon. The independence of language and mathematical reasoning. Proceedings of the National Academy of Sciences, 102(9):3177-3111 2005.
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in Ilms: Improving multilingual capability by cross- lingual-thought prompting. arXiv preprint 1xXiv:335.00004 2023.
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompt- ing: Improving zero-shot of-in-of-tthughh reasoning across languages. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 2023.
Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. Question translation training for better multilingual reasoning. arXiv preprint arXiv:2401.07817, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, arXiv preprint arXiv:2310.03744 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19742. PMLR, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023.
Zewei Sun, Mingxuan Wang, and Lei Li. Multilingual translation via grafting pre-trained language models. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Puntu Canu, Dominican Republic, 16-20 November, 2021, pages 2735-2747. Association for Computational Linguistics, 2021. doi: 10. 10.18853/V 1/2221. WWINNINNNNNNNNELPPP....... URL https //doi.org// 11w.1...1151111111 findings- emnlp 233
233
Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Shikhar Vashishth, Sriram Ganapa- thy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. L.M augmented Nms: Expanding capa- bilities through composition. CoRR, abs/2401 102412, 2024. doi: 10. (00.48550/ARXXV.22444 1.241...2..... URL https: //doi. org/1o../.. 48550/arXiv 02412

---

URL Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. Langbridge: Multilingual reasoning without multilingual supervision. CORR, abs/2401.10695. 2024, doi: 10.48550/ARXIV2240...0 10695
Atl.https:///ao http://....rrg1100..468555///111 .201
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International
Conference on Learning Representations, 2022. URL.https. hattps://openrevieee net/ met/forum?? id
nZeVKeePYf9
Linting Xue, Noah Constant, Adam Roberts. Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 483 498. Association for Computational Linguistics, 2021. doi: (0.18653N1/222.NAAAACI URL battps://doo org/ 10. 18653/v1/202/ maacl- 41
Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren. Common sense beyond english: Evaluating and improving multilingual language models for commonsense reasoning. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Pupers). Virtual Event, August 1-6, 2021, pages 1274-1287. Association for Computational Linguistics, 2021. doi: 10. 10.18/53/V1/ org/10..///0.... 10. 11..00.1665////22220 acl-long..... 102
ACL-LLCL 102. URL https://doi...... http://ooooorgg/1///
Alexis Conneau, Ruty Rinott, Guillaume Lample. Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
Fei Yuan, Yinquan Lu, Wenhao Zhu, Lingpeng Kong, Lei Li, Yu Qiao, and Jingjing Xu. Lego-mt: Learning detachable models for massively multilingual machine translation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computa- tional Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 11518 11118-11550 11533. Association for Computational Linguistics. 2023b. doi:
ACL.731. URL
httts://doi.org:: org/10....111... 00653/vv/202/ f findings-nnn-ii acl. 731
Adina Williams, Nikita Nangia, and Samuel R. Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Marilyn A. Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HILT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Pupers), pages 1112-1122 Association for Computational Linguistics, 2018 8. doi: 10. (0.18653/V1/N18-111 1101. URI.https: :tttp:://do..org10 18653/v1/n18-11 -101
Zixian Huang, Ao Wu, Jiaying Zhou, Yu Gu, Yue Zhao, and Gong Cheng. Clues before answers: Generation-enhancee multiple-choice QA. Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz, editors, Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3272-3287. Associa- tion for Computational Linguistics, 2022. doi: 10.18653/V https://doi. org/10. 18653/v1/20000 nwa.l 239
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018 pages 2381-2391. Association for Computational Linguistics, 2018 doi: 10.18653/V1/D18 1260. URL https: / /ooi org/ 10. 18653/v1/418- 1260
12

---

Sumithra Bhakthavatsallm, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03315, 2021. URL htttt:t/ts:/aaattt wttpaaawwww....r//aa/aaaaaaa
org httggabbs/1002... 03315
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: dataset for question answering via sentence composition. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8082-8090. AAAI Press, 2020. doi: N0.16099AAAALV344055.00
URL https: / //doi. or.. 10. 1609/ aaai. v34105.400.0. 6319
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond english- centric multilingual machine translation. J. Mach. Learn. Res., 22:107:1-107:: 2021. URL http://jmlr www/orrgg...eeee... wwpppppppp//0-oa//ppppp 2000:/222222220022 1307. .ttll
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. mgpt: Few-shot learners go multilingual, 2022. URL httpss///////
abs/2204.07580
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http://arxiv. http:///axxvvv..aaaaba httg/abs/18110 100//0000000005555
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. 2019. URL
sttp://arxiv.org//ee htapg//bs//99// 02116
Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y. Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loie Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human- centered machine translation. CoRR, abs/2207 04672, 2022. doi: 10.4850/ARXII..207..0 04672.. 48550/ URL https:://doll 188/...40///////////...
2207 2207.04672
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
13

---

Outline
Supplementary Experiments
A.1
Training Set Size in the Augmentation Stage (Table
A.2
The Selection Mapping Layers (Table
A.3
The Usage of Encoder-Decodee Model (Table
A.4
Quantitative Analysis on Representation Space Changes (Table 9)
A.5
Translation Performance (Table TO and Figure
A.6
Comparison with Enhanced Full Fine- Tuni- using Translation Data (Table
Inference Speed (Table
Complete Results
Ablation of Mapping Stage (Table
B.2
Ablation of Augmentation Stage (Table 14)
B.3
Comparing Replacement and Augmentation (Table 15)
B
Complete Experiments on the MGSM Dataset (Table 16
Limitations
Other Tables
Table 77
Dataset Statisties
Table 18
Prompt Templates
Table 19
Examples of Training Data
A
Supplementary Experiments
Table 6: Influence of training set size (per language) used in the augmentation stage. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resouree languages, and all languages, respectively. Referring to Shi et al. [2023] regard Bn, Th, and Sw as low-resourse languages, and regard the remaining languages as high higurresuurce languages.
MGSM
Bn
Th
Sw
Ja
Zh
De
Fr
Ru
Es
En
Lrl.
Hrl.
Avg.
O
26.8
26.8
28.8
31.2
39.6
54.0
52.4
46.0
55.2
65.2
27.5
49.1
42.6
100
28.4
31.6
31.6
34.0
35.6
56.8
54.0
48.0
52.8
62.4
30.5
49.1
43.2
1,000
36.8
43.2
37.2
40.8
41.2
57.6
52.0
52.8
52.0
62.0
51.2
47.6
5,000
44.0
50.4
48.0
42.8
45.6
56.0
$7.6
58.4
58.4
66.0
47.5
55.0
52.7
10,000
48.4
53.6
52.8
54.8
58.4
54.8
54.8
59.2
58.8
66.4
51.6
58.2
56.0
20,000
45.6
54.0
54.4
52.4
53.6
60.4
54.4
58.8
59.2
68.8
51.3
58.2
55.9
30,000
50.4
52.8
57.2
54.4
61.2
57.6
60.8
58.4
66.8
59.0
57.3
40,000
51.6
49.6
54.4
50.0
54.0
57.2
56.0
58.8
66.4
51.9
57.9
56.1
A.1 Training Set Size in the Augmentation Stage
We randomly sampled different size of training set to train MindMerger-Sofff on the mathematical reasoning task and reported the results MGSM Table Despite only 100 samples per language is used, MindMerger-Soff can be improved by 3.0% in low-resource languages compared to without augmentation stage. The training data size has significant impact on the performance MindMerger within the data size range of 10,000. MindMerger-Sole achieves the best result when using 30,000 samples per language, and better results may require enhancing the model's built-in capabilities.. Considering the query training data for various tasks can be easily obtained from public translation models, and training on scale of tens of thousands is cost-effective, MindMerger can be readily adapted to wider range of tasks.

---

Table 7: The selection of mapping layers. # Parm represents parameters size of the mapping layers. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. Referring (Shi et al. 2023). regard Bn, Th, and Sw as low-resourse languages, and regard the remaining languages as high-resource languages.
MGSM
Parm
Bn
Th
Sw
Ja
Zh
De
Fr
Ru
Es
En
Lrl.
Hrl.
Avg.
Lincar
4
50.8
50.0
51.2
47.6
51.2
60.0
58.0
56.4
61.6
66.0
50.7
57.3
55.3
2 layers MLP
10 M
50.4
52.8
57.2
54.4
53.6
61.2
57.6
60.8
58.4
66.8
53,5
59.0
57.3
layers MI.P
14 M
51.6
50.0
51.2
49.6
54,4
58.0
58.4
60.0
61.2
69.2
50.9
58.7
56.4
QFormer
42 M
1.6
0.8
1.6
17.2
22.4
34.0
34.4
26.0
36.8
52.8
1.3
31.9
22.8
A.2 The Selection of Mapping Layers
experimented with the performance of MindMerger-Soff when using different mapping layers in Table The 2 layers MLP used in the main experiment achieved the best results, while the effect of Linear will lower in comparison. This may be the insufficient space transfer capability of Linear due to the small number of parameters. As for QFormer (Li et al. 2023]. which converts variable-length input into fixed-length hidden states output, is popular model for model merging in computer vision, but it does not work well MindMerger-Soff. This could be attributed the
multilingual capabilities of LLMs, which may render features diverging too much from the original expression less readily accepted.
Table 8: The usage of encoder-decoder model (M2M100-1.22) 1.20B. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, regh-resource languages, and all languages, respectively. Referring to soShh et al. [2023), we regard Bn. Th, and Sw as low-resourse languages, and regard the remaining languages high-resource languages.
MGSM
Bn
Th
Sw
Ja
De
Fr
Ru
Es
En
Lrl.
Hrl.
Avg.
Encoder
50.4
52.8
57.2
54.4
53.6
61.2
57.6
60.8
58.4
66.8
53.5
59.0
57.3
Decoder
40.4
32.0
45.6
46.8
46.0
56.8
56.0
55.2
58.0
67.6
39.3
55.2
50.4
Encoder + Decoder
44.0
36.4
44.8
49.6
49.6
55.6
53.6
56.0
58.4
66.8
41.7
55.7
51.5
A.3
The Usage of Encoder-Decoder Model
In all other experiments, for the encoder-decoder model, we only use its encoder part. In this section, we compared the performance of MindMerger-Sooe using the decoder. Specifically, we use the translation model M2M100-1.22 autoregressively to generate English text, and take the hidden states corresponding to the generated text, and input them into the mapping layer alone or after concatenating with the output of the encoder. The experimental results are shown in Table R where the performance of MindMerger-Soon Soft dropped after adding the hidden states output by the decoder.
Table 9: The changes in the representation space within MindMerger as well as its constituent models the multilingual model (mT5-xl)) and the LLM (MetaMath-Llama-77 Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. Referring Shi al. [2023). we regard Bn. Th, and Sw low-resourse languages, and regard the remaining languages as high-resource languages.
Cosine Similarity
Ba-=En
Th-+En
Sw-Ea
Ja-En
Zh-Ea
De-EE
Fr-+En
Ru-En
Ee-EEn
Lrt.
11rl.
Avg
Mutiiliguul model
Enbalding Layar
Last Layer
0.29
n91
0.27
0.89
0.92
0.43
0.31
0.92
0.51
0.93
11.00
0.95
0.51
0.95
0.37
0.92
0.45
0.92
0.33
0.91
0.41
0.34
0.38
0.93
11M
Last Layer
Embedding Layer
008
0.18
007
0.15
0008
0.34
0.11
0.28
0.28
0.13
0.39
0.04
0.42
0.12
0.31
0.06
0.41
0.09
0.22
0.02
0.35
0:09
0.31
0.07
MindMerrger
LLM Layer
Mapping Module
0.32
092
0.32
0.1
0.46
0.91
0.46
0.90
0.91
0.46
0.35
0.96
0.62
0.94
0.9
0.52
0.56
0.90
0.37
0.91
0.94
0.53
0.93
0,47
Recall& (%)
Bn-EEn
Th-+En
Su-Fn
Ja-FF 7h--Fm
De-FEn
Fr-+Rn
Ru-Fn
Re-EEn
Hrl.
Avg:
Mutilingual model
Embedding layer Last layer
99.7
3.0
97.8
16.2
69.9
4.9
28.2
100
10.7
100
29.1
588
18.2
90.9
99.6
14.4
LIM
Enbedding layer Last layer
077
0.4
1.9
1.8
0.2
59
3.8
3.2
12.6
4.7
30.2
11.5
41.7
47.3
44.9
8.7
27.0
27.3
2.8
0.8
24.9
193
13.8
16.8
MindMarggr
Mapping module 11 mochlle
53.8
99.4
98.9
76.0
90.4
64.3
91.6
99.6
90.9
96.4
95.0
100
99.2
100
98.1
90.8
99.6
88.0
64.4
99.2
90.8
95.4
99.6
85.0
15

---

A.4 Quantitative Analysis on the Representation Space Changes
We further quantitatively analyzed the changes in the representation space of the models. Two metrics were employ to quantify space relationships: (1) Cosine similarity, indicating the similarity between the same query expressed in different languages, and (2) Recall representing the percentage of top-1 retrieve results from pool of 1000 English query that correspond to the same query.
The experimental results are shown in Table 9 multilingual model demonstrates the capability to map inputs in different languages language-agnostic space. Within this space, the average cosine similarity and Recall@) from various languages to English reach as high as 0.93 and 99.6. On the contrary. LLM lacks this capability. The average cosine similarity and Recall @ in the output space of LLM are only 0.07 and 16.8, respectively. The significant disparity in the capabilities to construct language-agnostic representations suggests the potential for LLM be complemented by multilingual model.
For MindMerger-Soff consisting of multilingual model and LLM, the mapping layer generates representation with an average cosine similarity 0.93 and an average Recall 1 of 99.6. This highly language-agnostic representation input to LLM effectively helps LLM understand different languages. Compared with the huge difference between the language representations output when using only LLM, the average cosine similarity of LLM output in MindMerger-Son increased from 0.07 to 0.47, and the average Recall@) increased from 16.8 to 85.0.
A.5
Translation Performance
Table 10: Translation results on Flores- dataset. spBLEU is used as evaluation metric.
Flores- 101
Ba Ea
Th :aa
Sw
En
Ja
EEn
EEn
EE Fr
Ru EEn
Es -En
Arg.
Translation Models
SetaMatthLLlamo-- (5-shots)
2.3
2.6
M2M100-118M
25.1
18.2
26.6
20.2
21.2
36.4
39.0
28.4
25.8
M2M110-J.200 1.00
27.7
23.9
34.1
24.8
26.7
42.9
43.9
34.3
30.0
32.0
W.1.R-200-1.38
43.9
28.6
30.4
45.9
47.4
38.0
34.8
37.4
NLLB--333
38.7
33.2
46.3
30.6
32.2
47.2
48.7
39.1
36.5
39.2
MiniMerrger (w/o
MBERT
10.4
8.2
12.4
12.5
16.3
23.9
26.1
18.7
19.0
16.4
BLLM-R-BBRTa-aagee
12.6
14.2
17.5
9.2
12.6
24.6
26.7
20.2
17.2
mTS-JJuge
18.0
18.1
25.1
13.8
16.3
30.4
34.3
25.3
24.6
22.9
mTS
25.9
23.5
34.1
19.8
22.3
37.3
39.1
30.7
29.0
29.1
M2M1100418M
23.8
25.6
36.8
39.4
M2M100-1.22
26.6
20.0
32.6
24.5
27.0
39.3
41.7
33.3
31.2
NILBB20001.38 1,38
30.1
26.4
38.5
21.8
24.4
38.3
41.8
32.1
30.3
31.5
NLLB-200-3.3B
In Table 10, we used the translation dataset Flores-101 101 [aan et al. 2021 as the evaluation set and presented the spBLEU performance of various versions of MindMerger-Soue (using MetaMath- Llama-7B as backbone) trained on mapping stage. In contrast to the poor performance of the same LLM, MetaMath-L Llama, MindMerger-Sonn greatly boost it translation capabilities. Remarkably, the performance of MindMerger, based on the M2M100- RBM1000418M, 18M encoder part, even surpasses the M2M100- 418M translation model itself. Although the encoder-only models mBERT and -L.M-RBERTTa-lnn do not have translation capabilities, MindMerger can still learn translation capabilities based them.
We also compared the performance of MindMerger with the full fine-tuning method. We conducted experiments across four translation directions using two different sizes of training data, each covering translations between nine pairs of languages, with the comparison of average scores presented in Figure 5 It can be observed that MindMerger has significantly enhanced the translation performance across settings, including X-En, Xn-X, X-Zh, and Zh-X, resulting in improvements ranging from 3.7% to 7.5%. Although MindMerger primarily enhances the understanding capabilities of LLMs without directly improving their generation capabilities, its performance on translation task demonstrates that it can also benefit generation tasks. Given that understanding is the basis for generation, MindMerger represents a crucial step towards enhancing the multilingual generation capabilities of LLMs.
16

---

26.7
IPd Mintee
2
Minderleegen See
22.3
15,2
37.0
15.2
11.5
31.6
(a) 100K training pairs per languages
(b) 200K training pairs languages
Figure 5: Comparison the translation performance of MindMerger and fully supervised fine-tuning method the 101 dataset. spBLEU is used as evaluation metric.
Table 11: Comparison the effectiveness of extra translation data for full fine- tin-ttniig and MindMerger- Soft on the MGSM dataset. Three types of training data are used: (a) translation data, (b) English task data, and (c) query translation task data. represents the usage order of the training data.
MGSM
Bn
Th
Sw
Jn
Zh
De
Fr
Ru
Es
En
Lrk.
Hrt.
Avg.
(6)
33.2
36.8
34.8
36.0
37.2
39.6
39.6
38.4
37.6
40.4
34.9
18.4
37.4
36.4
36.0
37.2
31.5
34.4
42.4
40.4
40.4
42.4
41.2
36.5
39.0
38.2
57.2
(n) (h)
21.2
21.2
24.4
38.8
34.4
52.8
56.8
52.4
57.6
51.5
22.3
50.6
42.
(b) (c)
33.2
40.0
42.0
42.0
42.0
45.2
44.8
45.2
48.0
52.0
38.4
45.6
43.4
(c) (b)
40.8
46.4
45.2
44.8
50.4
52.8
52.8
52.0
56.1
59.5
64.1
52.7
50.1
60 (c) (b)
40.8
50.0
50.4
47.6
52.8
$4.8
54.0
56.0
30.0
$4.8
47.1
52.9
51.1
MindMorger Soft
50.4
52,8
57.2
54.4
53.6
61.2
$7.6
60.8
58.4
66.8
53.5
59.0
$7.3
A.6 The Influence Translation Data for Full Fine-Tuning
Recall that the training of MindMerger-Soft involves three types of data: (a) translation data, (b) English task data, and (c) query translation task data. Therefore, we compared the effects full fine-tuning LLMs with the same data to demonstrate that MindMerger-SSff can utilize data more efficiently. The experimental results presented in Table show that, regardless of how the training data utilized, MindMerger-S Soft achieves average accuracy that is at least 6.2% higher than the full fine-tuned methods. Using translation data as the initial stage of training can enhance performance in out of 4 full fine-tuning settings ((a) > (c), (a) (b), and (a) (c) (b)). The benefits of translation data for MindMerger-Son are significantly greater than full fine-tuning
methods. Specifically, with the assistance of translation data, MindMerger-Soff improved the average accuracy by 5. 5.5% (refer to B.I), while the improvements for full fine-tuning are 0.8%, 6, 3.8%, and 1.0%, respectively.
A.7 Inference Speed
Table 12: Inference speed and generation length the MGSM dataset.
Avg.
per sample (6)
QAlign
3.8
4.2
3.6
3.8
3.5
3.8
MindMenge-SSof
3.9
3.6
3.6
3.7
3.5
3.5
3.9
3.7
3.6
3.7
The iverage length of genemated text (token)
QAlign
126.3
139.9
118.3
135.9
135.9
126.5
120.2
119.8
128.2
117.6
126.9
MindMerger-See Soft
We compared the inference speed of MindMerger and the single LLM model QAlign on the MGSM dataset with a batch size of As shown in the upper part of Table the average inference speed for MindMerger-Sofft is even faster than QAlign, where MindMerger taking average of 3.7s to infer sample, compared QAlign's 3.8s. This discrepancy arises because the text generated by QAlign slightly longer than that produced by MindMerger-Soft,, indicated in the lower part of Table 12
17

---

B Complete Results
Table 13: Ablation of mapping stage.Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. Referring to Shi ct al. [023]] we regard Bn, Th, and Sw as low-resourse languages, and regard the remaining languages high-resource languages.
MGSM
Th
Ja
Zh
Fr
En
Lrl.
Hrl.
Avg.
MindMerger (M2M100)
49.6
$2.4
53.2
$2.8
54.4
60.0
56.4
60.0
58.0
66.0
$1.7
58.2
56.3
wfo mapping Stage
32.4
52.0
59.6
60.0
68.4
58.3
MindMerger (mTS)
50.4
52.8
57.2
54.4
53.6
61.2
57.6
60.8
58.4
66.8
53.5
59.0
57.3
w/o mapping Stage
41.6
35.2
40.4
46.8
51.6
57.6
60.4
57.2
61.2
66.4
39.1
57.3
51.8
MSVAMP
Bn
Th
Sw
Ja
Zh
De
Fr
Ru
Es
En
Lrl.
Hr..
Avg.
MindMenger (M2M100)
51.7
56.0
55,2
57.2
57.7
63.1
62.1
60.2
63.4
64.3
54.3
61.1
59.1
w/o mapping Stage
34,1
39.9
32.4
53.7
53.6
58.6
57.5
55.6
57.9
64.6
35.5
57.4
50.8
MindMerger (m155)
53.4
59.0
61.7
64.0
65.0
67.7
63.5
60.4
w/o mapping Stage
40.4
42.2
43.7
54.2
54.2
60.3
57.8
55.4
60.0
62.7
42.1
57.8
53.1
B.1 Ablation of Mapping Stage
The complete experimental results are shown Table [3] decreased by 6.0% and 5.5% the MGSM dataset, and by 8.3% and 7.3% the MSVAMP dataset, demonstrating the effectiveness of this stage. The mapping stage is particularly crucial for enhancing the capabilities of low-resource languages. After ablating this stage, the average accuracy for low-resource languages drops 20.2% and 14.4% on the MGSM dataset, and 18.8% and 11.0% on the MSVAMP dataset. The mapping stage can also be helpful for high-resource languages. After removing the mapping stage, on the MSVAMP dataset, the average accuracy of high- languages dropped by 3.7% and 5.7% on the two MindMerger-Soll implementations.
The average accuracy of MindMerger-SS
Table 14:
Ablation of augmentation stage. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. Referring to Shi et al. 20033. we regard Bn, Th, and Sw as low-resourse languages, and regard the remaining languages high-resource languages.
MGSM
Bn
Th
Ja
Zh
De
Fr
Ru
Es
Lrl.
Hrl.
Avg.
MonoReason MindMerger (M2M100)
7.6
5.6
5.2
34.0
45.2
54.0
56.8
51.6
58.8
65.5
6.1
52.3
38.4
49.6
52.4
53.2
52.8
54.4
60.0
56.4
60.0
58.0
66.0
51.7
58,2
56.3
w/o augmentation Stage
25.6
23.2
26.4
37.2
40.8
51.2
51.6
49.2
$6.0
64.0
25.1
50.0
42.5
MindMerggr (mT
50.4
52.8
57.2
54.4
53.6
61.2
57.6
60.8
58.4
66.8
53.5
59.0
57.3
w/o augmentation Stage
26.8
26.8
28.8
31.2
39.6
54.0
52.4
46.0
55.2
65.2
27.5
49.1
42.6
MSVAMP
Bn
Th
Jaa
De
Es
Lrl.
Hrl.
Avg.
MonoReuson MindMerger (M2M100)
15.0
17.1
15.4
51.9
60,9
62.2
59.3
63.3
15.8
46.2
51.7
56.0
55.2
57.2
57.7
63.1
62.1
60.2
63.4
64.3
54.3
61.1
59.1
w/o augmentation Stage
29.4
27.6
28.2
51.2
52.5
60.4
60.6
58.2
58.8
64.3
28.4
58.0
49.1
Mindderrger (mn
52.0
$2.0""""3.""
54.0
59.0
61.7
64.1
64.0
63.3
65.0
67.7
53.1
63.5
60.4
w/o augmentation Stage
37.5
35.2
37.5
52.8
53.2
61.3
61.7
57.6
63.3
36.7
59.1
52.4
B.2 Ablation of Augmentation Stage
The complete results are shown Table 14 Even without the mapping stage, MindMerger-Sofft solely training in general bilingual pairs can still improve the performance of LLMs in low-resource languages. For the MGSM and MSVAMP datasets, the average accuracy of MindMerger-Soff in low-resource languages increases by MonoReason 19.0% and 12.6% based on M2M100-1. 1010000-1..... and increases by MonoReason 21 21.4% and 20.9% based on mT5-xl. With the help of the augmentation stage, the reasoning capabilities of MindMerger-SSn Soft in low-resource languages are further enhanced. Comparing to the MindMerger-Soff without augmentation stage in the low-resource languages, augmentation stage brings improvements of 26.6% and 25.9% based on M2M100-1 1.22B, and 26.0% and 16.4% based on mT55xl. Moreover, augmentation stage is also valueable for high-resource languages, bringing improvements of 3.1 1% based on M2M100-1.22 .2BB and 4.4% based on m55-xl for the VSVAMP datasct in the average accuracy across high-resource languages.
18

---

Table 15: Comparison between replacement (w/o T) and augmentation. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. Referring to Shi al. 2002) regard Bn, Th, and Sw low-resourse languages, and regard the remaining languages as high-resource languages.
MGSM
Bn
Th
Sw
Ja
Zh
De
Fr
Ru
Es
En
Url.
Hrl.
Avg.
MindMerger-Soff (M2M100)
49.6
52.4
53.2
52.8
54.4
60.0
56.4
60.0
58.0
66.0
51.7
58.2
56.3
Replacement (w/o T)
50.8
44.4
50.8
54.4
50.4
57.2
55.2
58.0
61.2
63.2
48.7
57.1
54.6
MindMerger-Soft (mTS)
50.4
52.8
57.2
54,4
53.6
61.2
57.6
60.8
58.4
66.8
53.5
59.0
57.3
Replacement (w/o T)
52.2
53.6
54,8
46.8
52.0
59.2
58.0
58.4
60.0
63.6
53.5
56.9
55.9
MSVAMP
Bn
Th
Sw
Ja
Zh
Fr
Ru
En
Ur.
Hrl.
Avg.
MindMerger-Soo (M2M100)
$1.7
56.0
55.2
57.2
57.7
63.1
62.1
60.2
63.4
64.3
54.3
61.1
59.1
Replacement (wlo T))
50.0
48.5
52.3
55.7
57.2
60.4
59.7
55.9
58.5
50.3
57.9
55.6
MiniMMrngerSöfft (m)
$2.0
53.4"
34.0
5400
61.7
64.1
64.0
63.3
65.0
67.7
53.1
63..""
"6044""""""
Replacement (w/o T)
51.7
54.7
54.3
54.8
56.5
61.1
59.7
58.5
61.1
60.8
53.6
58.9
57.3
B.3 Comparing Replacement and Augmentation
The complete experimental results are shown in Table 15 It can be observed that the augmentation strategy is more significantly helpful in improving high-resource language reasoning capabilities than low-resource languages. Compared with the replacement version of MindMerrer-Soo augmentation strategy improves the average accuracy by 1.1%, 2.1%, 3.2%, and 4.6% across high- resource languages in the four experimental results. This suggests that, in contrast the replacement strategy that neglects these capabilities, the augmentation strategy effectively takes advantage of them, thereby unlocking the full potential of L.Ms.. For low-resource languages, the augmentation strategy also brings improvements of 3.0% and 4.0% for MindMerger-Sor based on M2M100-1 10-1.2 but on par with the replacement strategy for MindMerger-Sofft based on mT5-xl. It could be the multilingual capabilities of mT5- on low-resource languages are significantly stronger than LLMs that it can completely replace LLMs, while the slightly weaker multilingual capabilities of M2M100-1.20 still need to be complementary to LLMs.
the
B.4 Complete Experiments on the MGSM Dataset
Table 16: Results the MGSM dataset. Lrl., Hrl., and Avg. represent the average accuracy across low-resource languages, high-resource languages, and all languages, respectively. We regard Te, Bn, Th, and Sw low-resourse languages, and regard the remaining languages as high-resource.
Avg.
XCoT Chai al. 2024
42.8
40.4
49.2
48.4
50.0
50.0
47.2
47.2
47.2
48.8
48.4
45.2
49.1
47.2
Langlingge Yoon ar. 2024
34.8
42.8
50.4
412
40.0
45.2
50.8
52.4
56.4
58.0
63.2
42.8
52.3
48.8
MindMergerSSoll
52.8
52.0
59,2
56.8
51.2
55.2
61.2
55.2
61.6
62.4
66.0
55,2
59.0
57.6
In the above, we followed the previous work [Chen et al.) 2023 Zhu et af. 2024) and experimented with ten languages on the MGSM dataset. However, it important note that the MGSM dataset contains a total of eleven languages, including Telugu (Te). Therefore, we presented the evaluation results of MindMerger on the complete MGSM dataset in Table T6 It can be observation that MindMerger--Soft is still the best model on Te, achieving a 10.0% improvement over xCoT CChai al. 2024] and a 20.0% improvement over LangBridge on that language, as well as an average 10.4% improvement over xCoT and an 8.8% improvement over LangBridge across all languages.
C Limitations
Although our experimental results show that merging external and built-in language understanding capabilities can help improve the model's multilingual reasoning capabilities, is unclear the effective boundaries of this mechanism. On one hand, a multilingual model with significantly weaker language capabilities compared to LLMs might not be able to assist LLMS effectively. On the other hand, LLMs may not be able to provide additional built-in knowledge for a multilingual model whose language capabilities are much stronger. Therefore, more experiments are needed to explore the effective boundaries of MindMerger.
19

---

D Other Tables
Table 17: Dataset statistics. # Train, # Test, and # Lang refer to the size of query translation training set per language, the size of multilingual test set per language, and the number of language evaluated for each dataset, respectively.
Dataset MGSM
Task
# Train
Test
# Lang
Math
30,000
250
10
MSVAMP
Math
30,000
1,000
10
X-CSQA
Commonsense
8,888
1,000
16
XNLI
NLI 2,490
5,010
Table 18: The prompts of mathematical reasoning task.
MonoRResoon & MultiReason & QAlign
Below is an instruction that describes task. Write response that appropriately completes the request. ### Instruction: [query) t** Response: Let's is think step by step.
Translate-En
Below an instruction that describes a task. Write a response that appropriately completes the roquest. ### Instruction: (translated query) #a# Response: Let's think step by step.
MindMerger-Hane
Below is an instruction that describes task. Write a response that appropriately completes the request. #6B Instruction: (query) } ### Translated instruction: (translated query) ana Response: Let's think step by step.
Table 19: Examples of training data.
Mapping Stage
Input (Zh):
i3 MEEEE Lt
Output: will be shown the structure tree as a top mode when there are no elements belonging this group the document.
Augmentation Stage (Math)
Input (Zh): f1145 ME
ti to Ma -ME 3. sis 4 1
S.. 11. 2 a 1050
Input (En): Bert fills out the daily crossword puzzle in the newspaper every day. He uses a pencil to fill out the puzzles every two weeks. On average, it takes him 1050 words to use a pencil. How many words are each crossword puzzle on average?
Output: Bert uses up a pencil fill the puzzles every weeks and it takes him 1050 words up pencil, then he must be filling out 1050 words of crossword puzzles every two weeks. To find out how many words are in each daily crossword puzzle, we need to divide the total number of words (1050) by the number of days in two weeks (14). So, there are 1050/14= = words in each daily crossword puzzle on average. #### The answer is: 75
Augmentation stage (X-CSQA)
Input (Zh): i + A ET 2 22 (A) Ji (B 1FB (D) FALI
(F) 25
Input (En): What someone operating vehicle likely accused of after becomiing
inebriated? (A) punish (B) arrest (C) automobile accidents (D) talking nonsense (E) drunk
driving
Output: E
Augmentation stage (XNLI)
Input (Zh): Premise: t9 A E Hypothesis: #.EE F
Input (En): Premise: She doesn't 't really understand. Hypothesis: Actually, she docsn't it get it.
Output: Entailment
20

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state our main claims in the abstract and outline our methods and the most important experimental results. In the introduction, discuss motivations, discuss our main claims, and point out the experimental results that support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification:
We discuss the limitations in Appendix C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peeiiiainnn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work experimentally discovered a valuable mechanism in large language models, and has yet involved theory proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report the detail of experimental setting in $ 4.1 And our code and data are included in the supplementary material and will be made publicly available after the paper is accepted.
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code and data are included in the supplementary material and will be made publicly available after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiips.cc/puulic/guui
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
23

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We report the detail of experimental setting in $ 4.1 And our code and data are included in the supplementary material and will be made publicly available after the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We calculate the statistical significance for the main experimental results in $ 4.3 our method significance outperform all compared baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the compute resources in 4.1

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read the NeurIPS code of ethics and conducted in the paper conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work only involves pure research data and has no social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
25

---

Answer: [NA]
Justification:
Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our experiments use open data sets and strictly cite and comply with licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
26

---

For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer:
[NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "chPRQEx5zN", "reviewer": "Reviewer_iF4J", "summary": "The paper suggests incorporating an embedding block using external multilingual models to improve the models' understanding. Additionally, comprehensive experiments are conducted to demonstrate its efficacy.", "questions": "The details of the two training stages lack clarity, specifically regarding the utilization of translation and parallel data, as well as the precise training tasks involved.", "limitations": "The authors suggest a limitation on the integration of external multilingual models with the model's inherent understanding capabilities, a concern partly explored through the ablation study.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a straightforward and easily implementable method to enhance models' multilingual capability.\n2. Experimental results validate its effectiveness.", "weaknesses": "The paper lacks a clear understanding of why the method can work. \n1. To my understanding, the Embedding Layer of the model merely converts the query from text space to the token/embedding space. This transformation may not be interpreted as \"understanding the input\" since the information hasn't been operated and combined by the attention layer. However, $X$ after the multilingual model has been \"understood\" by the model. Therefore, it does not make much sense to concatenate these two embeddings.\n2. As $\\tilde{X}$ contains the information of query q, $T$ includes duplicated information. This may be why the method can surpass baselines such as QAlign and Translate-En. It would be more convincing if baselines also input duplicated information.\n3. This is further proved by Figure (4b), which illustrates that the mapping layer tends to obtain a unified representation for different languages and this is the same as QAlign. Therefore, it is confusing why this method can outperform QAlign.\n4. The paper uses Llama2 as the base model. However, its multilingual performance is far from sota. Until submission, I believe Mistral, Gemma, and Llama3 have been released, whose multilingual performance is much better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to provide reviews. However, we would like to clarify some misunderstandings as follows:\n\n**W1. To my understanding, the Embedding Layer of the model merely converts the query from text space to the token/embedding space. This transformation may not be interpreted as \"understanding the input\" since the information hasn't been operated and combined by the attention layer. However, \nafter the multilingual model has been \"understood\" by the model. Therefore, it does not make much sense to concatenate these two embeddings.**\n\n- Our model is based on the finding that providing LLM with inputs from two representation spaces can improve LLM's multilingual reasoning capability, where one representation comes from the native space of LLM's embedding layer, and the other representation is the mapping of multilingual queries to unified space provided by the multilingual model. \n   \n- The two different types of representations are the key to boost better understanding of the query in subsequent layers of LLM.\n\n\n**W2. As contains the information of query q, includes duplicated information. This may be why the method can surpass baselines such as QAlign and Translate-En. It would be more convincing if baselines also input duplicated information.**\n\n- We duplicated the input for all baselines during training and inference. As shown in the below table, **the performance of 4 out of 6 baselines decreased with duplicated input**.\n- Note that our model outperforms all baselines by at least 6.7% and 8.0% across all languages\nand low-resource languages, which far exceeds the improvement of the baseline by duplicated input. Therefore, it can be considered that duplicate input is not the mechanism for improving the performance of our model.\n\n| MetaMath-Llama-7B, MGSM             | Bn    | Th    | Sw    | Ja    | Zh    | De    | Fr    | Ru    | Es    | En    | Lrl   | Hrl   | Avg.   |\n|-------------------------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|\n| MonoReason                          | 6.8   | 7.2   | 6.8   | 36.4  | 38.4  | 55.2  | 54.4  | 52.0  | 57.2  | 68.8  | 6.9   | 51.8  | 38.3   |\n| MonoReason (duplicated input)       | 8.4   | 7.2   | 4.8   | 33.2  | 42.8  | 53.6  | 53.6  | 53.6  | 52.0  | 63.2  | 6.8   | 50.3  | 37.2   |\n| MultiReason-Lora                    | 29.6  | 35.2  | 28.0  | 52.0  | 54.8  | 59.6  | 58.4  | 62.4  | 59.6  | 64.8  | 30.9  | 58.8  | 50.4   |\n| MultiReason-Lora (duplicated input) | 28.8  | 43.2  | 33.6  | 50.4  | 54.8  | 57.6  | 57.6  | 61.2  | 62.0  | 62.8  | 35.2  | 58.1  | 51.2   |\n| MultiReason-SFT                     | 33.2  | 40.0  | 42.0  | 42.0  | 42.0  | 45.2  | 44.8  | 45.2  | 48.0  | 52.0  | 38.4  | 45.6  | 43.4   |\n| MultiReason-SFT  (duplicated input) | 31.2  | 41.2  | 39.2  | 42.0  | 47.6  | 49.6  | 52.0  | 49.6  | 48.8  | 58.8  | 46.0  | 49.8  | 46.0   |\n| QAlign                              | 32.4  | 39.6  | 40.4  | 44.0  | 48.4  | 54.8  | 56.8  | 52.4  | 59.6  | **68.0**  | 37.5  | 54.9  | 49.6   |\n| QAlign (duplicated input)           | 30.8  | 38.0  | 40.4  | 44.4  | 46.4  | 55.6  | 55.2  | 56.8  | 57.6  | 65.2  | 49.0  | 54.5  | 49.0   |\n| LangBridge                          | 42.8  | 50.4  | 43.2  | 40.0  | 45.2  | 56.4  | 50.8  | 52.4  | 58.0  | 63.2  | 45.5  | 52.3  | 50.2   |\n| LangBridge (duplicated input)       | 34.0  | 42.8  | 42.0  | 33.2  | 35.6  | 53.6  | 52.0  | 51.2  | 57.2  | 61.2  | 39.6  | 49.1  | 46.3   |\n| Translate-En                        | 48.4  | 37.6  | 37.6  | 49.2  | 46.8  | 60.4  | 56.4  | 47.6  | 59.6  | 65.5  | 41.2  | 55.1  | 50.9   |\n| Translate-En (duplicated input)     | 47.6  | 33.6  | 38.4  | 48.4  | 48.8  | 56.0  | 49.2  | 44.8  | 54.8  | 63.2  | 39.9  | 52.2  | 48.5   |\n| **MindMerger**                          | **52.0**  | **53.4**  | **54.0**  | **59.0**  | **61.7**  | **64.1**  | **64.0**  | **63.3**  | **65.0**  | 67.7  | **53.1**  | **63.5**  | **60.4**   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b9SaDKaQ7D", "reviewer": "Reviewer_q66P", "summary": "The paper propose a way to improve multilingual reasoning in LLM in own native languages without relying on pivoting methods such as translating to English. The method assume a hypothesis that LLMs have built in knowledge and reasoning abilities in a lower-resource language and not just common language like English. The method plugs in LLMs with the external multilingual models as embedding layers and introduce a training scheme build those new knowledge it. Experiments on MGSM show better accuracy across languages.", "questions": "NA", "limitations": "The paper discussed limitations", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The method is pretty novel in ways of incorporating multilingual encoders as additional embedding layers to augment existing embedding layers of the LLMs.\n* The experiments are done with many languages comprehensively, including many low-resource languages, and ablation studies, which are good.", "weaknesses": "* The method is incomplete and ambiguous on the generation side. It makes sense that injecting multilingual embeddings at the bottom for the input may help the LLM understand the context better, but that does not necessarily make the LLM **generate** in those native language better, especially if the LLMs are weak in producing content in native languages.\n   * It is no surprise that the tasks presented in the paper are all \"understanding\" tasks where the generation workload is light and trivial. Other \"generation\" tasks, such as summarization, writing, translation... may not fair well with this method. I would highlight if the authors could show good results for the method for these tasks.\n\n* Missing baseline comparison: https://arxiv.org/abs/2306.11372 , https://aclanthology.org/2023.findings-emnlp.826.pdf\n\n* Confusing word choices, such as \"mapping stage\", \"merging\" and \"augmentation stage\". These words often lead to different understanding rather than what are described in the paper.\n\n* \"Query translation task\" and \"query translation data\" and many parts of the methodology are not explained clearly. It would be better if there are explicit visual examples or diagrams.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews. \n\nThe main concern is the performance of our model on the generation task. We believe that adding experiments on the generation task will help further expand the scope of our work. \n\nWe are grateful for the feedbacks on some writing improvement and we will revise them in the next version.\n\nThe specific responses are as follows:\n\n**W1. The method is incomplete and ambiguous on the generation side. It makes sense that injecting multilingual embeddings at the bottom for the input may help the LLM understand the context better, but that does not necessarily make the LLM generate in those native language better, especially if the LLMs are weak in producing content in native languages.**\n\n\n- Our model can be a key step towards improving LLM multilingual generation capabilities, since **language understanding is the basis of language generation**.\n\n- To response this concern, we added an experimental comparison on the translation task based on Flores-101 dataset. We implemented four settings: X-En, En-X, X-Zh, and Zh-X. In each setting, we compare fully fine-tuned Llama2 (Llama-SFT) with MindMerger using 100 K and 200 K training samples per language, respectively.\n\n- The experimental results are shown in the following four tables. **MindMerger consistently outperforms Llama-SFT in translation quality** under all comparison settings.\n\n- Considering that the top of MindMerger is frozen and not trained, the improvement in generation quality comes from the enhancement of query understanding, which may be a valuable insight for future research on multilingual generation tasks.\n\n| X-En                 | Bn    | Th    | Sw    | Ja    | Zh    | De    | Fr    | Ru    | Es    | Lrl.  | Hrl.  | Avg.   | \n|----------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|\n| Llama-SFT  (100 K)   | 14.8  | 9.6   | 22.7  | 16.4  | 14.6  | 28.7  | 26.3  | 25.0  | 20.7  | 15.7  | 22.0  | 19.9   |   \n| MinderMerger (100 K) | 18.6  | 16.7  | 32.9  | 23.3  | 22.2  | 37.4  | 32.6  | 31.7  | 25.0  | 22.7  | 28.7  | 26.7   |   \n| Llama-SFT (200 K)    | 16.0  | 18.8  | 27.9  | 20.2  | 23.6  | 38.1  | 39.3  | 30.4  | 28.7  | 20.9  | 30.1  | 27.0   |   \n| MinderMerger (200 K) | 27.1  | 24.5  | 40.2  | 26.0  | 28.9  | 42.5  | 42.4  | 34.1  | 30.1  | 30.6  | 34.0  | 32.9   |   \n\n\n| En-X                 | Bn    | Th    | Sw    | Ja    | Zh    | De    | Fr    | Ru    | Es    | Lrl.  | Hrl.  | Avg.   |  \n|----------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|\n| Llama-SFT            | 17.1  | 8.2   | 20.8  | 13.1  | 6.2   | 18.9  | 19.7  | 12.6  | 16.7  | 15.4  | 14.5  | 14.8   |   \n| MinderMerger         | 20.2  | 11.9  | 30.1  | 19.2  | 12.4  | 27.6  | 32.2  | 23.2  | 23.7  | 20.7  | 23.1  | 22.3   |   \n| Llama-SFT (200 K)    | 23.9  | 15.6  | 26.6  | 18.9  | 16.7  | 25.1  | 35.6  | 21.2  | 24.3  | 22.0  | 23.6  | 23.1   |   \n| MinderMerger (200 K) | 24.4  | 17.4  | 32.6  | 23.0  | 20.7  | 31.6  | 40.8  | 26.6  | 26.1  | 24.8  | 28.1  | 27.0   |   \n\n\n| X-Zh                 | Bn    | Th    | Sw    | Ja    | De    | Fr    | Ru    | Es    | En    | Lrl.  | Hrl.  | Avg.   |  \n|----------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|\n| Llama-SFT (100 K)    | 7.0   | 5.7   | 6.9   | 12.9  | 15.7  | 13.5  | 14.3  | 11.6  | 16.3  | 6.5   | 14.1  | 11.5   |  \n| MinderMerger (100 K) | 10.1  | 8.9   | 10.3  | 16.0  | 18.7  | 18.0  | 17.4  | 15.8  | 21.2  | 9.8   | 17.9  | 15.2   |  \n| Llama-SFT (200 K)    | 8.5   | 7.5   | 8.2   | 13.2  | 14.7  | 14.4  | 13.6  | 11.1  | 13.1  | 8.1   | 13.4  | 11.6   |  \n| MinderMerger (200 K) | 11.8  | 11.7  | 12.0  | 17.3  | 20.5  | 20.0  | 19.0  | 17.0  | 23.9  | 11.8  | 19.6  | 17.0   |  \n\n\n| Zh-X                 | Bn   | Th   | Sw   | Ja   | De   | Fr    | Ru    | Es    | En    | Lrl.  | Hrl.  | Avg.   |   \n|----------------------|------|------|------|------|------|-------|-------|-------|-------|-------|-------|--------|\n| Llama-SFT (100 K)    | 9.9  | 3.6  | 7.3  | 12.9 | 11.5 | 14.7  | 9.6   | 11.0  | 12.7  | 6.9   | 12.1  | 10.4   |   \n| MinderMerger (100 K) | 14.1 | 7.8  | 13.9 | 19.4 | 18.8 | 24.8  | 17.5  | 17.7  | 25.9  | 11.9  | 20.7  | 17.8   |   \n| Llama-SFT (200 K)    | 14.2 | 7.8  | 11.3 | 16.5 | 17.7 | 20.5  | 14.1  | 15.7  | 18.6  | 11.1  | 17.2  | 15.2   |   \n| MinderMerger (200 K) | 16.2 | 11.4 | 16.2 | 20.7 | 20.2 | 26.6  | 18.7  | 18.3  | 27.2  | 14.6  | 22.0  | 19.5   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dydU0fyBrd", "reviewer": "Reviewer_VBLh", "summary": "This paper proposes a framework which merges LLMs with the external language understanding capabilities from multilingual models to improve multilingual reasoning performance. Specifically, the authors introduce a two-step training scheme:  they (i) train the framework to embed the multilingual model into the LLM by using translation data, and (ii) further train the framework to collaboratively utilize the built-in capabilities of LLM and the embedded external language capabilities.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a new method to boost the multilingual reasoning of LLMs, which embeds external language understanding capabilities from multilingual models into LLMs.\n- Extensive experimental results and analyses on several datasets demonstrate the effectiveness of the proposed method.", "weaknesses": "- In Section 3.2, the authors mention that they use translation data and query translation task data generated from public translation models for the two-stage training; while in lines 182-192, it shows that they also use several English task datasets, such as MetaMathQA. It is unclear how they use these data. \n- Although the author mentioned that only a small number of parameters are trained during two-stage training,  the paper lacks a comparison with baselines in this apsect. Particularly, the proposed method requires the external multilingual model, which is expected to affect the inference efficiency.\n- I know asking for new results might be unrealistic at this stage, but I am curious what would happen if you only used two-stage training for LLM without including an external multilingual model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable reviews. We have added some experiments and will include them in the next version of paper.\n\n**W1. In Section 3.2, the authors mention that they use translation data and query translation task data generated from public translation models for the two-stage training; while in lines 182-192, it shows that they also use several English task datasets, such as MetaMathQA. It is unclear how they use these data.**\n\n- For all models, we first use English task datasets to fully fine-tune LLM in order to obtain the reason capabilities, and then use other data to train. \n- Taking the mathematical reasoning task as an example, both our model and all baselines are based on MetaMath-Llama (denote as MonoReason in our paper) as the base model, and MetaMath-Llama is fully fine-tuning Llama on the MetaMathQA dataset.\n\n**W2. Although the author mentioned that only a small number of parameters are trained during two-stage training, the paper lacks a comparison with baselines in this apsect. Particularly, the proposed method requires the external multilingual model, which is expected to affect the inference efficiency.**\n\n- **Compared with the parallel encoding input query, the serial generation time is much longer.**\n- As report in the follow table, compared with the single LLM model QAlign under batch size 1, the average inference speed for **MindMerger even faster than QAlign**, where MindMerger spends an average of 3.65s inferring a sample while QAlign uses 3.81.\n\n| Inference time per sample (s) | Bn    | Th    | Sw    | Ja    | Zh    | De    | Fr    | Ru    | Es    | En    | Avg.   |\n|-------------------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|\n| QAlign                        | 3.77  | 4.21  | 3.57  | 4.09  | 4.13  | 3.84  | 3.61  | 3.62  | 3.79  | 3.48  | 3.81   |\n| MindMerger                    | 3.88  | 3.57  | 3.64  | 3.53  | 3.71  | 3.49  | 3.47  | 3.94  | 3.74  | 3.55  | 3.65   |\n\n- This phenomenon is caused by the length of the text generated by QAlign is slightly larger than that of MindMerger as reported as belows:\n\n| The average length of generated text (token) | Bn      | Th      | Sw      | Ja      | Zh      | De      | Fr      | Ru      | Es      | En      | Avg.     |\n|----------------------------------------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|----------|\n| QAlign                                       | 126.30  | 139.94  | 118.34  | 135.86  | 135.94  | 126.52  | 120.20  | 119.76  | 128.24  | 117.60  | 126.87   |\n| MindMerger                                   | 122.32  | 121.42  | 121.94  | 121.78  | 126.74  | 117.56  | 116.30  | 133.22  | 125.84  | 120.34  | 122.75   |\n\n- As for the cost of the external multilingual model, it is very insignificant. As reported in the below table, due to its small number of parameters, the time used to encode the query of external model is only half of QAlign (0.02s vs 0.04s). \n\n| Encoding time per sample (s)         | Bn    | Th    | Sw    | Ja    | Zh    | De    | Fr    | Ru    | Es    | En    | Avg.   |\n|--------------------------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|\n| QAlign                               | 0.05  | 0.04  | 0.04  | 0.03  | 0.03  | 0.03  | 0.04  | 0.03  | 0.03  | 0.03  | 0.04   |\n| Multilingual Encoder + Mapping Layer | 0.02  | 0.02  | 0.02  | 0.02  | 0.02  | 0.02  | 0.02  | 0.02  | 0.02  | 0.02  | 0.02   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OZjBab4e15", "reviewer": "Reviewer_uQ8B", "summary": "The paper introduces MergeMinds, a method that integrates LLMs with multilingual models to enhance reasoning capabilities across multiple languages. Through a two-step training scheme, MergeMinds effectively boosts performance in multilingual reasoning and language understanding tasks, particularly excelling in low-resource languages, as demonstrated by significant accuracy improvements on various datasets.", "questions": "1. It appears that the Te language is not included in the MGSM. Could you explain the reason for this omission?\n2. I noticed that some work similar to xcot was referenced but not included in the comparative table. What was the rationale behind this decision?\n3. In my view, does this work adapt the paradigm from InstructBlip's Qformer and LLaVA's two-stage alignment training to a multilingual setting?\n4. Missing Reference [1-3]\n\n[1] MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment-as-Preference Optimization\n\n[2] The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights\n\n[3] Multilingual large language model: A survey of resources, taxonomy and frontiers", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This work propose MergeMinds method offers a novel solution by merging LLMs with external multilingual capabilities and wo-step training process, enhancing the robustness of the model.\n2. The method shows substantial improvements in multilingual reasoning and language understanding tasks, particularly with notable gains in low-resource languages, demonstrating the effectiveness of the approach.", "weaknesses": "1. From my perspective, the motivation of MergeMinds is extremely similar with LangBridge, both of which ultilize the mapping layer to map multilingual encoder to exsisting MLLM.\n2. In fact, the work of this paper has already been explored on multi-modal large models a long time ago, such as InstructBlip's Qformer and LLaVA's two-stage alignment training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouragement of our work. We reply to your questions as follows.\n\n**W1: From my perspective, the motivation of MergeMinds is extremely similar with LangBridge, both of which utilize the mapping layer to map multilingual encoder to existing MLLM.**\n\n- In related works about MLLM, LLM does not have visual capabilities at all. Compared with this, the most significant difference in our research is that LLM has built-in multilingual capabilities, . \n- Based on this difference, we need to explore whether there is a more efficient training solution than MLLM to effectively activate the multilingual capabilities built-in LLM.\n- LangBridge is a solution like the works in MLLM that only considers integrating encoder to obtain capabilities that are not available in LLM. LangBridge does not consider boosting the built-in multilingual capabilities of LLM, which results in its performance being significantly lower than MindMerger. It lags behind MindMerger by 8.0% and 7.1% in low-resource languages and all languages in the MGSM dataset. Therefore, different from the existing work of MLLM, with MindMerger as a starting point, there are still many challenges to explore in the future to stimulate the multilingual as well as other bulit-in capabilities of LLM.\n\n\n**W2, Q3: The work of this paper has already been explored on multi-modal large models. Does this work adapt the paradigm from InstructBlip's Qformer and LLaVA's two-stage alignment training to a multilingual setting?**\n\n- Because the focus of our work is on how to combine multilingual capabilities from external model and LLM itself, we face the challenge of avoiding over-reliance on the capabilities of the external multilingual model during training. Similarly, we also face the challenge of avoiding over-reliance on the multilingual capabilities built into LLM during training.\n- Moreover, our goal is to activate the multilingual capabilities of the LLM rather than to increase new capabilities it does not have, it is possible to explore a more efficient training solution than the MLLM. It can be observed from Table 6 in the paper that MindMerger has surpassed all baselines under the using of only 5,000 training samples per language at the second stage.\n- In addition, the research on integrating multilingual models into LLM is still under-explored and different from the existing work on MLLM. For example, what data to use for training, what structure to use for integration, etc. still need more insightful discussions.\n\n| MGSM       | Te    | Bn    | Th    | Sw    | Ja    | Zh    | De    | Fr    | Ru    | Es    | En    | Lrl   | Hrl   |\n|------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| xCoT       | 42.8  | 40.4  | 49.2  | 48.4  | 50.0  | 50.0  | 47.2  | 49.6  | 50.0  | 48.8  | 48.4  | 45.2  | 49.1  |\n| LangBridge | 34.8  | 42.8  | 50.4  | 43.2  | 40.0  | 45.2  | 50.8  | 52.4  | 56.4  | 58.0  | 63.2  | 42.8  | 52.3  |\n| MindMerger | **52.0**  | **54.0**  | **54.0**  | **55.6**  | **51.6**  | **54.8**  | **60.0**  | **59.6**  | **57.6**  | **63.2**  | **65.2**  | **53.9**  | **58.9**  |\n\n\n**Q1: It appears that the Te language is not included in the MGSM. Could you explain the reason for this omission?**\n\n- This is not a technical issue, since the QAlign and MathOctopus work we followed did not compare Te, so we ignored Te as well.\n- We added the Te experiment in the table above. MindMerger is still the best model on Te, with a 9.2% improvement over xCoT.\n\n**Q2: I noticed that some work similar to xCoT was referenced but not included in the comparative table. What was the rationale behind this decision?**\n\n- We selected the two strongest methods for each of the two categories of baselines: relearning-based and replacement-based. \n- xCoT is a relearning-based method, but its performance is lower than MultiReason (MultiReason-Lora) and QAlign. Considering the page limit of the paper, we did not include it in the comparison.\n- We show the performance of xCoT in the table above, and we consider adding it to the appendix.\n\n\n**Q4: Missing Reference.**\n- Thank you for the reviewers’ kind reminder, we will cite them in the next version of paper.\n\nBest wishes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a method to improve multilingual capability of LLMs by providing backbone LLMs the undecided query representation from multilingual model to LLM that is more optimized for reasoning. The proposed merging recipe includes two stages (1) mapping multilingual model into the backbone LLM and (2) teaching models to perform task given mapped multilingual model’s representation and backbone LLM’s representation. 

The reviewer (iF4J) have asked for more thorough evaluation against with the baselines, making input setting more equivalent and trying different base LLMs, and the proposed method consistently shows gains in these new settings. The approach is evaluated throughly on various benchmarks, including generation benchmark (suggested by reviewer q66P) and efficiency comparison (suggested by reviewer VBLh). These results strengthens the value of the proposed approach.
Decision: Accept (poster)