Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models
Authors: Lucas Bandarkar
Benjamin Muller
Pritish Yuvraj
Rui Hou
Nayan Singhal
Hongjiang Lv
Bing Liu
Venue: ICLR 2025 Spotlight (2025)
Keywords: model souping
model merging
cross-lingual transfer
multilingual
math
mathematical reasoning
LLM
SFT
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We transfer math skills to non-English languages simply by swapping in a few layers from a model fine-tuned on those languages into a model fine-tuned on math.

Abstract: Model merging, such as model souping, is the practice of combining different models with the same architecture together without further training. In this work, we present a model merging methodology that addresses the difficulty of fine-tuning Large Language Models (LLMs) for target tasks in non-English languages, where task-specific data is often unavailable. We focus on mathematical reasoning and without in-language math data, facilitate cross-lingual transfer by composing language and math capabilities. Starting from the same pretrained model, we fine-tune separate "experts" on math instruction data in English and on generic instruction data in the target language. We then replace the top and bottom transformer layers of the math expert directly with layers from the language expert, which consequently enhances math performance in the target language. The resulting merged models outperform the individual experts and other merging methods on the math benchmark, MGSM, by 10% across four major languages where math instruction data is scarce. In addition, this layer swapping is simple, inexpensive, and intuitive, as it is based on an interpretative analysis of the most important parameter changes during the fine-tuning of each expert. The ability to successfully re-compose LLMs for cross-lingual transfer in this manner opens up future possibilities to combine model expertise, create modular solutions, and transfer reasoning capabilities across languages all post hoc.

Published as a conference paper at ICLR 2025
LAYER SWAPPING FOR ZERO-SHOT CROSS-LINGUAL TRANSFER IN LARGE LANGUAGE MODELS
L Lucas Bandarkar
Benjamin Muller
Pritish Yuvraj Bing Liu
Rui Hou
Nayan Singhal
Hongjiang Lv Univeesity of California, Los Angeles
Meta GenAl
ABSTRACT
Model merging, such as model souping, is the practice of combining different models with the same architecture together without further training. In this work, we present a model merging methodology that addresses the difficulty of fine- tuning Large Language Models (LLMs) for target tasks in non-English languages, where task-specific data is often unavailable. We focus on mathematical reasoning and without in-language math data, facilitate cross- lingual transfer by composing language and math capabilities. Starting from the same pretrained model, we fine- tune separate "cxperts" on math instruction data in English and on generic instruc- tion data in the target language. We then replace the top and bottom transformer layers of the math expert directly with layers from the language expert, which consequently enhances math performance in the target language. The resulting merged models outperform the individual experts and other merging methods on the math benchmark, MGSM. by 10% across four major languages where math instruction data is scarce. In addition, this layer swapping is simple, inexpensive. and intuitive, as based on an interpretative analysis of the most important parameter changes during the fine-tuning of each expert. The ability to success- fully re-compose L.MMs for cross-lingual transfer in this manner opens up future possibilities to combine model expertise, create modular solutions, and transfer reasoning capabilities across languages all post hoc.
INTRODUCTION
Instruction fine-tuning Large Language Models (LLMs) is necessary to customize pre-trained mod- els for real-world applications. This fine-tuning is especially critical in multilingual settings because most popular open-source LLMs have been pretrained on highly English-centric data (Jiag et al.) 2023 Llama al.) 2024 Yang et al., (0244)). Although recent LLMs such as LLAMA 3 and QWEN2 have seen more than a trillion non-English tokens due to the sheer scale of their pretraining datasets, these tokens are still heavily concentrated in just few languages, resulting in limited capabili- ties for most other non-English languages (Llama et al.) 2024; Yang et al. 2024a). Furthermore, the scarcity of high-quality labeled data available for post- training in non- -on-English languages and the -raaninntttaslating English post-training data into target languages- a typical solution has significant computational overhead and often leads to datasets of unreliable quality (Khanuja et al. [2024). Numerous efforts have set out to annotate or assemble high-quality datasets for lower-resource languages (Khan et al. 2024 Singh et al. 20244b Tonja et al. 2024), but massive gap still exists in many tasks and domains, such as math. As a result, developers are forced to largely rely on cross-lingual transfer the generalization of learned capacities from high- high-resource languages to lower ones ones-but the rate of such transfer is low for most languages (Philippy et al.) 2023).
tremendous cost of procuring it further exacerbates the inequality. Even
In this paper, we present novel solution that merges two LLMs together in order to transfer math reasoning capabiliities to lower-resource languages during supervised fine-tuning (SFT). In the ab- sence of in-language math data, we fine-tune two variants of the same pretrained model: one with English math samples and the other with generic instruction data in the target language. Provided
"Correspondence lucasbanda ka reca ucla.edu

---

Published as a conference paper at ICLR 2025
Language experts
outpuls
Top layers from language expert
Upper "transition" zone zone
SFT ganeric insloctionn data in the torget longuoge
Middle loyers expert
base model
English
Lower "tronsition" zone
Bottom layers from language expert
Math
Our merging method which swaps in top and bottom transformer layers from language expert into a math expert, buffered by transition zone.
Figure 1:
these variants, which we refer to as "experts", we combine their learned language and task ca- pabilities by re-composing an LLM with a mix of parameters from each while avoiding negative interference. Notably, the top and bottom few transformer layers are selected from the language expert, and the middle transformer layers selected from the math expert. The intuition behind this simple, yet strategic, method is informed by our analysis of the SFT updates that led to the ex- perts. We find that the learned math capabilities especially in the second half of the model. Meanwhile, we find the enhanced language skills to have come from parameters closest the input and output, in line with previous literature stating that this is where the most language-specifet representations are concentrated (Wendler et al. 20244] Alabi al. 2024; Tang al., 2024). Therefore, our methodology takes the most important layers from each expert and transfers math capabilities to the target language.
concentrated in the middle transformer layers,
The resulting layer-swapped models deliver strong performance across numerous target languages (Swahili, Telugu, Bengali, Japanese) MGSM (Shi et al.) 2023). the manually- version of the Grade School Math benchmark (Cobbe al. 2021). all these major languages, where math SFT data is not readily available, layer swapping outperforms basclines, the individual experts, and model souping (Wortsman et al., 2022)) by 10% on average. For Swahili, layer swapping exceeds mixed Swahili and math SFT dataset when evaluated.
MGSM performance of models tnnetuned Therefore, this methodology provides a simple and effective way to improve the capabilities of LLMs without the need for any task-specific data in the target language. In addition, it is inexpensive and fully post hoc, meaning that has the potential to be practical in many settings. Fundamentally. the success of this method also provides empirical evidence for cross-lingual patterns in the latent structures of LLMs that can be further interpreted and exploited.
We discuss relevant literature in the following section. We then explain the analysis that led to our methodology and present layer swapping in detail in Sections 3 and 4 Next, we show empirical
results and discuss findings in Sections and 6 Finally, we propose future work prompted by the success of this methodology in Section
2 RELATED WORK
2.1 MODEL MERGING
While the use of weight averaging to reduce noise in training well predates instability challenges in deep neural networks (Breiman 1996). the of model merging to combine trained model
2

---

Published as a conference paper at ICLR 2025
checkpoints is emerging research space in deep learning. Similar to ensembling model outputs (Dietterich] 2000), aggregating model weights also improves model robustness and generalization Izmailov et al. 2018), even if trained on the same data (Ramé et al.) 2022). Numerous studies seek to improve the pre-merging conditions, either via linearizing fine- tin-ttnning (Ortiz-Jimenez et al., [023) or aligning weights (Ainsworth et al. 2023). Wortsman et al. (2022) develops empirical method to average, or soup, model variants together to increase the search space during hyperparam- eter tuning. However, simple weight averaging, is vulnerable to negative transfer, or interference, between variants. To address this, methods have been proposed to selectively combine models at the individual weight level, using either the magnitude and direction of the fine-tuning deltas (Ilharco et al. 2023; Yadav et al., 2023 Davari & Belilovsky] 2024 Yu et al. 2024) or leveraging informa- tion theory (Matena & Raffel| | In parallel, sparse fine-tuning methods have been developed to create fine-tuned models with small proportion of weights changed (Guo et al. 2021; Sung et al.)
2021 Xu et al. 202I). which then allows adding together fine-tuning updates with less overlap. Overall, model merging, especially inexpensive model souping, is very common in practice because it improves training stability, model robustness and generalization, and performance by increasing the search space combining expertise in multi-task settings (Yang et al. 2024b).
2.2 LLM MULTILINGUALITY
The inability for a language model to learn more languages without undermining other capabili- ties-the curse of mulrilinguality (Conneau et al. 2020 Pfeiffer et al. (022) a heavily studied
problem in encoder models. Beyond increasing vocabulary capacity (Zheng et al.) 2021 Liang
ct al. )023). numerous works attempt to understand the quantity and location of language-specific
parameters (Wang et al. 2020 Muller et al. 202T, Choenni al. 2023) in multilingual encoders.
parameters were found in the top and bottom layers (Chang jt al. 2022, Choenni et al. [024), language specialization occurs throughout the model. And while recently model scaling has mitigated the limitation from parameter quantity, the massive amount of labeled data needed to train LLMS presents a new challenge. In encoder models, cross-lingual trans- fer was enhanced by aligning cross-lingual representations (Ouyang al. 2021 Patra et al. 2023 Gaschi et al. 2023). but this is difficult in decoder-only models (Jain et al. 2023). To boost transfer in post-training, several data augmentation solutions have been proposed, both for SFT (Qin et al.
While higher rates of language-specing
2023 Chai et al. 2024) or reinforcement learning from human feedback (RLHF) (Dang et al. 2024
She et al. [2224] Lai et al. 2024). Recent work finds that English-centric LLMs process multilinual text by mapping them to and from English representations in the first and last transformer layers to
take advantage of English-based capabilities (Kojima al. 2024 WWndder et al. 20244 Tang al.)
2024 Alabi et al. (2024). This leads to benefits from prompting such LLMs to "think in English",
thereby promoting this internal mapping (Shi al. 2023 Zhang et al. 2024b).
2.3 MODEL MERGING FOR PROSS-LINGUAL TRANSFER
To address the limited representational capacity of multilingual models, many solutions have been proposed to strategically share or split parts of the model. This could be major blocks, as in mixture- of-experts (Fedus et al. 2022 NLLB et al. 2022), or a few parameters in each layer, as cross- lingual adapters (Peiffeer et al. 2020 2022). Ansell et al. (002)) combines modular and sparse fine-tuning approaches in a two-stage SFT, where first task and language experts are fine-tuned and all weights that did not change more than a threshold are masked. Next, the experts are fine-tuned again from the pretrained model with this mask, creating sparse task vectors that can be composed together with lower rates of parameter overlap. This composable sparse for larger decoder-only LLMs (Ansell et al.] 2024).. In this work, we develop a simpler and more flexible merging method for cross-lingual transfer that does not require fine-tuning more than once.
also adapted
PRELIMINARY ANALYSIS
3.1 SETUP
We start by training numerous math and language "experts" by fine-tuning LLAMA 3.1 8B (Llama et al. 2024). We perform SFT runs using next token prediction with 30-40k labeled samples with
3

---

Published as a conference paper at ICLR 2025
varying hyperparameter and for each type of expert, select the three best checkpoints. The math experts were fine-tuned on English math word problems from the Orca-Math synthetic dataset (Mi- tra et al., (024). In order to select the three experts, we use results the English splits of MGSM, as well as the average across languages. For the language experts, we select Swahili, Bengali, Telugu, and Japanese as the target languages. These are languages present in MGSM and other benchmarks discussed below, but where LLAMA 3.1's performance lags behind the top languages like Spanish. In addition, the lack of in-language math instruction data in these languages moti- vates the need for effective methods for zero-shot cross- pooss-lingual transfer. For each, we mix together samples from available instruction datasets in that language to create experts with enhanced general language and instruction-follvvii capabilities. The resulting datasets contain many different types of tasks-such as translation, NER, and runs on these "generic" multi-task datasets, the three checkpoints for each language are primarily selected based off their performance on the target language splits on BELEBELE (Bandarkar et al,, 2024) and FLORES (NLLB et al., (002). These simple tasks, reading comprehension and trans- lation, are strong indicators of basic language understanding and generation capabilities. We also ensure slight improvement MBPP (Austin et al.] [02T), MMLU (Hendrycks et al. 202T). and
no math? After numerous SFT
MGSM as secondary measures of language improvement
3.2
PARAMETER ANALYSIS OF SFT
A.becsoniic
A
Modeer Mine
Adroww R.C Aber R.
Záe W
31054
427%
C3050
03000
00000
024000
83687
6.991
1600
L200
1001
1000
30415
1000
09002
10099
1500
1088
696
1009
1006
69004
1086
1998
1000
13000
3000
1908
3002
1500
1906
1950
1788
30H0
530%
(2A) Japanese expert #1
(2B) Math expert #1
Figure 2:
This visualization displays the location with more significant magnitude of change dur- ing SFT for two representative experts across transformer layers and parameter types. In detail, this shows the percentage of rows for each 2-dimensional parameter in the 32 transformer layers of LLAMA 3.1 8B where the mean absolute value is above a threshold (1.9 x 10-5 and 1.0 x 10-5, re- spectively). Darker green shading represents parameters changing more significantly, relative to the others. Larger versions of these images, as well as for more experts, can be found in Appendix A.5
For these experts, we first investigate where the parameters are being updated the most and where they remain unchanged during fine-tuning. Using similar notation to Ilharco et al.] (2023). let 0 Opre; Oft be the set of weights for the pretrained model and fine-tuned expert, respectively. We generalize the delta of fine-tuning as Wa W W pre for all one- two-dimensional param- eters W E 0. To compare deltas across parameter tensors with different shapes, we represent the
Training specifics such as hyperparameters are provided in Appendix A.2
2 Dataset details are provided in Appendix A.1 See results of chosen experts in Appendix
4

---

Published as a conference paper at ICLR 2025
magnitude of change at the row level using the mean absolute value (MAV) of the difference W (for dimediimalional parameters, this is simply the MAV across the vector).
We observe highly consistent patterns in the WA magnitudes among math experts and, separately, among language experts. The latter is particularly notable given the varying mixtures of tasks and data samples for each language. In Figure we show a visualization for representative language expert and a representative math expert. For the language expert, we find that the attention param- eters (left four columns in the visualization, referring Wo, WK Wv, and Wo, respectively) are getting updated most significantly, notably in the first couple layers (bottom) and the last couple lay- ers (top). The beed-forward layers (right three columns, referring to W1. W3, and W2, respectively) do not change at all until the last few layers. In comparison, the math experts follow a different pat- tern. In these runs, the first half (bottom 16 layers) remain largely unchanged. In this second half, the attention parameters are being updated significantly, similar to language experts, yet the feed- forward layers are also getting changed quite bit. To demonstrate the consistency across training runs and languages, we show the same visualization for more experts in Appendix A.S
3.3 SPARSIPYING UPDATES
We then attempt to create sparse model updates for increased composability similar to Ansell et al. (022), but without retraining. We attempt to utilize the magnitude of the deltas to selectively up- date the model without undermining performance gains. Concretely, whether to apply the update or revert the value to the original value in the pretrained LLAMA 3.1. This is done row-level granularity in order not partially modify linear transformations. How- ever, in our analysis across the math experts, we find that more than 70-80% of model parameters are required to be updated for the increase in performance remain equivalent. Such a small rate of sparsification would not significantly reduce interference between the math and language experts.
thresholds determine
We next attempt location-based sparsification. This means leveraging our intuition of what patterns occur during fine- fine-tuning to select specific parameter tensors to merge or not. A breakthrough was made when we revert the first five and last two transformer layers of our math experts to their original values. Despite undoing the SFT updates, the math experts maintain their increased performance. This is significant when considered alongside our intuition that the first few and last few transformer layers contain the most important language-specifer parameters. This is based on our above analysis and is supported by previous multilingual interpretability research on both encoder-decoder models (Chang et al.) 2022) Choenni et al. 2024) and decoder-only models (Tang al. 2024) Zhang al. 2024c). Our discovery that SFT updates to these same layers are not critical for mathematical reasoning led us experiment with swapping them in from the language expert.
METHODOLOGY
4.1
LAYER SWAPPING
The layer swapping methodology takes two experts, one fine-tuned the target language and the other on the target task in English, and re-composes a single model with the top and bottom trans- former layers from the language expert and the middle from the math expert.
As displayed in Figure D we additionally design transition zone in between the off-the-shelf layers from each expert. These buffer layers are weighted averages of the respective layers from each expert that ensure that the output of one layer does not directly input into a layer fine-tuned separately. While intuitively these transition zones seem necessary, we do not find empirical evidence that they provide statistically significant benefit over replacing them with the math expert's 's layers. However, as discussed in Section 3.T our experts were not fine-tuned for very long and therefore the latent representation spaces after each layer would not have diverged very much from each other. This explains, in theory, why transition zones were not helpful in our setting. We conjecture that the experts had been trained further (or simply with higher learning rates), such buffer zone would be necessary. We therefore still present it as a central component of our layer swapping methodology.
The implementation of layer swapping is simple as iterating through the state dictionary of the math expert and for each parameter, either: (1) keeping as is, (2) replacing its value with that of the language expert, (3) averaging that of the math and language expert (see Algorithm
5

---

Published as a conference paper at ICLR 2025
Algorithm Layer Swapping
Input:
task expert Brask. language expert Blang lower layers to swap b, upper layers to swap u, lower transition layers la, upper transition layers lu weight each expert Otask Alang number of model layers L
Output: Merged model merged
1: for parameter name in models parameters do
2: f- layer number of n, N/A if n not attention or feedforward parameters
3:
if < b or I L -
then
4:
0merged [ -- Olang (n)
top bottom layers from language expert
5:
else if > b 1 + l. or < L - u - ta then
6: O meerged (n: ttask {n}
middle layers from task expert
7: else
8:
O meerged (n² + (Artask * Dtask {n} + Olang * (n)n + Mlang) # transition zone
9:
end if
10:
end for
11:
Return Omerged
4.2 CONFIGURATION
Layer swapping has several components that can be configured in various ways. Most notably, the number of layers to swap at the top and bottom and the number of layers to include in the respective transition zones. We tested how to configure these swapped layers and transition zones to most effectively merge these models, empirically. We find, however, that there wide range of possible configurations in which this methodology is still very effective. Note that all these experiments conducted 32-layer LLAMA 3.1 transformer model. For model of this size, find that the desired configuration of each component is in the ranges listed below, with our default values underlined:
1. The number of bottom layers to swap b E (3,4,5)
2. The number of top layers swap u E (0,
3. The number of layers in the lower transition zone La E {Q, 1,2]
4. The number of layers in the upper transition zone tu E {0, 1,
5. The transition zones averages of the layers from both (i.c. soups) that can be unweighted or magniudc-adjuutedd weighted averages (Ottasks (lagg).
6. The non-transformer parameters (input token embeddings, output layers, etc.) work best they are also averages of the two experts, as opposed to simply from the language expert.
Top
longuage
"brasefun""
Upper
Middle from moth
Middle layors from moth expert
expart
Lewer
Lower *pendlien" sone
"trandtion"
Bottor layars from language expert
language
(3A) Possible config. with layers swapped in.
(3B) Possible config. with less layers swapped in.
Figure 3: The comparison of the maximum (left) and minimum (right) swapping setups that we find effective empirically, Note on the right, there are no upper layers directly from the language expert.
Although the results were largely equivalent within this range, some patterns did exist. We find that for languages where performance lower (e.g. Telugu), the higher performing configurations typ-

---

Published as a conference paper at ICLR 2025
ically had more layers from the language expert. This is perhaps because the boost from improving the language capabilities was more important relative to languages LLAMA 3.1 was already better at. For example for languages such Japanese, the best configuration has close to the minimum layers from language expert, similar to the minimum swap illustrated in Figure 3 The math expert where the most cross-lingual transfer occurred naturally during SFT (that is, with the highest scores on all four target languages), named "math expert #2", performed understandably better with fewer layers from the language expert swapped in. Generally, conjecture that lower-resource languages tend to require more layers from the language expert.
EMPIRICAL RESULTS
5.1 EXPERIMENTS ON SWAHILI, TELUGU, JAPANESE, AND BENGALI
As discussed in Section 3.1 we launched SFT runs with different hyperparameters and on five dif- ferent datasets: math data in English and generie instruction data Bengali, Swahili, Telugu, and Japanese. For each dataset, selected three checkpoints, striking a balance between top perfor- mance and sufficient diversity to ensure robust experimental results. For each language, the three language experts could pair with any of the three math experts, which gives nine merged models to evaluate. To understand the effectiveness of layer swapping, we present the MGSM scores of the individual experts and the base model, LLAMA 3.1, to determine whether layer swapping construc- tively combines the learned task and capabilities. In addition, we present results from the classic model soup (Wortsman al. 2022). Model souping the most practice, used in pre- and post-traine post-training for purposes such as boosting model robustness and gener- alization, but also combine capabilities of experts (Yu et al., 2024). In vanilla model souping, the merged parameters are simple averages of the parameters of the input experts. For each language, we additionally evaluate adjusting for the magnitude of change from the base model to the expert using weighted average. Using notation from Section 3.2 the resulting weight of each expert is inverse of the average MAV for all rows all WA for that expert. For all model soup results presented, we select the highest MGSM score amongst these possible configurations. Given the number experts and merged pairs in our evaluations, we present two aggregate metrics: the mean demonstrates the consistency of each method, the
model merging method in
and maximum of evaluations. While the maximum demonstrates the ceiling of the method, which is often more desirable given the ability to iterate through numerous settings during model training.
Table I: MGSM 8-shot results of layer swapping across four languages compared to the indi- vidual experts and model souping. Note that for aggregate statistics of the individual SFT runs, we select best checkpoints from numerous training runs with periodic checkpointing. The merging methods aggregated over the 9 pairs (3 language experts 3 math experts), which means the min, avg, and max measures are not perfectly comparable.
Setting
LLAMA 3.1 8B
language expert
math expert
mudel soup
layer swap
layer swap
Details
lop training
lop runs
training
config, pairs
defauli config, 9 pairs
config, pairs
Swahili avg
24.8
24.7
29.5
29.3
32.4
32.8
Swahili max
24.8
25.6
32.8
32.0
36.4
37.2
Telugu avg
12.0
20.0
20.1
20.9
22.7
23.0
Telugu max
12.0
22.4
24.0
26.4
27.6
27.6
Bengali avg
29.2
33.5
38.3
36.8
37.1
38.7
Bengali max
29.2
35,2
44.4
38.4
40.4
45.2
Japanese
33.6
35,9
42.7
38.7
38.5
40.1
Japanese max
33.6
36.8
44.8
40.0
40.8
43.2
As displayed in Table I find that layer swapping consistently outperforms these baselines MGSM. For Swahili and Telugu, the default layer swapping configuration scores higher than both the individual experts across all 9 pairs. This provides tremendous evidence that this merging method

---

Published as a conference paper at ICLR 2025
prevents negative interference. For Bengali, where the math expert already performs significantly better than for Swahili and Telugu, numerous laaprng configurations consistently surpass the math expert's 's average score. In addition, the best layer swapped model performs higher than the best math expert on its own, For Japanese, here we find a lower average performance compared to the individual math experts. However, note that Japanese experts were perhaps the weakest; the increases in performance across BELEBELE, FLORES, MBPP, and MMLU after SFT were minimal and with the data and hyperparameters tested, we were unable do better. Furthermore, MGSM performance of both the base LLAMA 3.1 and the math experts was already decent in Japanese, prior to merging. Compared to model souping, layer swapping consistently outperforms it in terms of both maximum and average performance in all languages. The model soup results typically lie between the individual results of the experts.
We further evaluated these layer-swapped models on BELEBELE, FLORES, MBPP, and MMLU to ensure no over-optimization to the MGSM benchmark and find that the results are on par, usually even higher, than the base model and the individual experts. In addition, sample of text generations to further ensure the quality of the resulting model. Given that there are several language and math experts, we evaluate combining layer swapping with model souping. We do so by souping the three experts for each the languages before layer swapping with a soup of the three math experts and present results in Table 6 in the Appendix. We find that these language and math soups perform on par with the individual pairs when layer swapped together. The feasibility in these settings shows that layer swapping can further extend the search space for fine-tuned models.
manually checked
Given these results, we conjecture that perhaps layer swapping provides the biggest benefit for lower-resource languages (e.g. Swahili, Telugu), although such a conclusion would require evalu- ating more than four languages. This conjecture would be explained by the difficulty of improving language capabilities with low-data SFT when LLAMA 3.1 was already pretrained on more tokens in that language. Additionally, for higher-resource languages, cross- cross-lingual transfer occurs more naturally when fine-tuning on English math data.
5.2 FURTHER EVALUATIONS SWAHILI
Given that here, model souping appears to be susceptible to negative interference when merging, additionally evaluate TIES mer-ing (Yadav et al. (023) for Swahili. This merging method resolves interference between experts by sequentially trimming conflicting values, determining the sign of each weight, and then merging the resulting weights. In addition, we present an alternative for Swahili where, instead of post hoc model merging, mix the language and math datasets and do a single fine-tuning run with samples from both. Instead of training two experts, each on 30-40k samples, the joint SFT is over the union (80k samples). Identical to the training of experts, launch many SFT runs with different hyperparameters and select the best three checkpoints.
Table MGSM 8-shot results of layer swapping for Swahili more detail and with two additional comparisons, VIES-merging and dataset merging. We display the minimum performance in Swahili, as well as the average across all 9 languages MGSM and in English.
Setting
LLAMA 3.1 SB
Swabili expert
math expert
swh&math joint SFT
model soup
TIES- merging
layer swap
layer swap
Details
top train ing nins
3 train- ing runs
training runs
best config. pairs
best config, pairs
default config, pairs
best config. 9 pairs
Swahili min
24.8
23.6
27.2
31.6
25.6
25.2
29.6
29.2
Swahili avg
24.8
24.7
29.5
32.1 1
29.3
29.5
32.4
32.8
Swahill max
24.8
25.6
32.8
32.8
32.0
32.4
36.4
37.2
English avg
56.0
55.7
66.2
64.3
62.0
60.1
64.7
64.4
All langs avg
37.7
37.5
45.4
46.0
43.0
41.8
44.1
44.4
Table 2 displays these more detailed results for Swahili. We find that TIES SHEE-merging, similar to model souping, consistently underperforms compared to layer swapping. For mixing the Swahili and math data prior to fine-tuning, we are able to achieve results comparable to layer swapping on average, with the average MGSM score for this joint SFT being less than point lower. How-
8

---

Published as a conference paper at ICLR 2025
ever, the maximum performance of these checkpoints lagged the best layer-swapped pairs by 4.4 percentage points. This means that the ceiling for cross-lingual transfer is significantly higher with this methodology than simply mixing datasets together. This is significant because our method of merging two variants fine-tuned on separate datasets proves to be more effective than an extended fine-tuning on the combined datasets, consistent with Aakanksha et al. (2024).
We note that by swapping in layers from the Swahili expert, MGSM performance English and other non-Swahili languages decreases from the math expert. This expected given that we opti- mize for a different language, yet the decrease is relatively small nevertheless. For comparison, we see that performance drops much more for the other merging methods on these non-target languages.
6 DISCUSSION
Layer swapping is simple, yet effective method for merging models for cross-lingual transfer. The success of this post hoc method prompts numerous key insights.
In settings where in- in-language task data is unavailable or rare, such as labeled math samples in Tel- ugu, layer swapping provides a very effective way to create capable model with simply English task data and general target language data. For lower-resource languages, such a constrained sce- nario is extremely common in practice, especially when it comes to instruction fine-tuning data. All other baselines that we evaluate in such a scenario are not able to combine language and math capabilities as consistently and effectively, as discussed in Section 5
The surprising effectiveness of this method can be explained by the observation that English-centric LLMs do most of their internal thinking in English, which aligns with recent literature (Wendler cet al. 2024; Zhang et al. 2024b; Tang et al. 2024 Kojima et al. 2024). Therefore, our method exploits the finding that the most important manguage-specifii parameters are those in the top and bottom transformer layers that map multilingual input to and from English representations. Po- tentially, we can reinterpret the functionality of these transformer layers as, to certain degree, natural language interfaces to broader model intelligence. Our method preserves the updates to the middle layers, as we hypothesize that this is where most of the improvements to the LLM's mathe- matical reasoning are concentrated. Consequently, this separability of multilingual parameters from other model capabilities could motivate a new generation of modular solutions for multilinguulism in decoder-only LLMS akin to MoE or adapter-based approaches designed for encoder models (Pfeif- fer et al., 2020, 2022 Liu et al. 2023). Regardless, this work suggests that further interpretability research into L.M multilingualiy could lead to more cost-effective solutions to boost non-Engie -nnlishl capabilities.
Model souping is popular because of its flexibility, convenience, and ability to expand the model search space. In this regard, layer swapping provides a more effective alternative for the multilingual setting with the same advantages. This method can be implemented fully post hoc between any number of checkpoints using simple parameter arithmetic, with no further fine-tuning. Because of how inexpensive it is implement, it enables the ability to quickly iterate through and test many configurations (e.g. the number of swapped transformer layers). Similarly to souping, the simplicity of this method means that it has the potential work any stage of training (pretraining, fine- tuning, preference tuning, etc.). In multi-task fine-tuning scenarios, this could allow for non-English capabilities being boosted on the side, separately from other reasoning capabilities-such as math, multi-hop reasoning, coding, safety, etc.-and then models are merged back together post hoc to combine the learned skills.
In terms of model merging, our analysis indicates that weight-level techniques are perhaps not as effective in reducing interference when merging parameters together, as compared to parameter-level or even layer-level merges. A possible explanation is that modifying individual weight values may disrupt the linear dependence within the transformations defined by the weight tensors. Conversely, the recent success of self-speculative decoding in LLMs (Zhang al. 2024a Elhoushi et al. 2024) suggests that the token representation spaces most layers across the model are very similar. so, keeping entire layers in tact may alleviate the risk of undermining newfound learning. Another reason could be that using very granular magnitude measures may be too noisy to properly indicate which weight updates important and which not.

---

Published as a conference paper at ICLR 2025
7 LIMITATIONS AND FUTURE WORK
The positive results of this layer swapping methodology largely raise many questions about the extent to which it is effective and in what settings. We propose here many further investigations that will build broader understanding of the method's practical scope.
Freezing parameters before training experts: Instead of re-composing a model with layers from separate models, logical next evaluation would be to freeze model layers from the beginning. Intuitively, this would only be more effective from a performance standpoint, as there is no ad hoc merging of experts from disparate training jobs. However, a significant benefit of our solution is that it is all post hoc and flexible. Parameter freezing would require knowing which configuration would work prior to training or require numerous training runs to find the best configuration.
LLMs with different pretraining: An explanation for why this method works so well is that English- centrie L.MSs "think English", which is shown in more than Ilama 3. It must be studied whether LLMs with balanced multilingual pretraining have similarly consolidated representations, and if not, whether layer swapping breaks.
Different model sizes: LLMs with more parameters, either from more transformer layers or from larger hidden dimensions, would clearly require new merging configurations. addition, remains to be seen how related attributes such sparsity, redundancy, and the quantity of cross-lingual shared parameters, would influence the performance of layer swapping.
Lower-resource langs: Among the limited languages evaluated, the lower-resource ones benefited most from layer swapping. However, the model needs minimum capabilities in the target language for zero-shot cross-lingual transfer. It is unclear how well the LLM needs to understand the target language for this to be effective.
Other reasoning tasks: This work only tackles mathematical reasoning, but it is conceivable that other complex reasoning capabilities are concentrated in parameters separate from language capa- bilities in similar manner. That being said, the configuration required for layer swapping may be completely different and may require a preliminary analysis as in Section 3
Parameter-efficient fine-tuning: Since layer swapping treats transformer layers unified blocks, it would be equivalent if the model were fine-tuned using parameter-efficient SFT methods such as
LoRA, which consists of inserting adapter weights into the transformer layer (Hu al. 2022). Mod-
ifying other model merging methods for such adapters is simple (Zhang et al. (2023), and therefore layer swapping has the potential to be effective in parameter-efficient fine-tuning settings.
Different training stage: We limit the focus of this work on low-data SFT. Model souping, itself, is implemented in practice at all stages (e.g- pretraining. CPT), and layer swapping has the potential to be effective there as well. Similarly, model souping is effective even when the checkpoints have been fine-tuned significantly. It is unclear what would be the point for our methodology where the experts have diverged too significantly that they would no longer recombine well. Either way, it could still enable a multi-step training where experts are iteratively fine-tuned and merged successively, analogous to gradient aggregation in data-distributed training.
8 CONCLUSION
In this paper, we address the difficulty of training LLMs perform tasks in languages where labeled task-specific data does not exist. We first trained separate experts on English math data and generic instruction data in numerous target languages. An analysis of the importance of different parameter updates during fine-tuning led to the development of the layer swapping method which swaps in the top and bottom layers from language experts into the math expert. Surprisingly. this simple model merging provides one of the highest performing methods to fine-tune an LLM for math in a target language without the presence of target language data. The strong intuition behind this solution and its flexibility, simplicity, and effectiveness provide vast potential to be practical in many other settings. This method can be potentially adapted for new pretrained models, target tasks, target languages, training stages, training setups, and more. In addition, this method indicates better model interpretability of multilinguality can lead to more efficient methods for transferring English capabilities to lower resource languages.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
To reproduce the fine-tuning of our "expert" models, we describe the process Section 3.1 and pro- vide further details the data and fine-tuning hyperparameters in Appendix A..
and A.2 Further
details for our analysis described Section 3.2 can found Appendix AA.5 We provide pseu-
docode of the layer swapping algorithm defined in Section in Algorithm I All of our experiments
for which results are presented are thoroughly explained in Section
ACKNOWLEDGMENTS
The authors acknowledge the crucial support provided by Mostafa Elhoushi, Tanmay Parekh, Jiabao Ji, and Chloe Bi that made this project possible.
REFERENCES
Aakanksha, Arash Ahmadian, Seraphina Goldfarb-Tarrant, Beyza Ermis, Marzich Fadace, and Sara Hooker. Mix data merge models? optimizing for performance and safety in multilingual contexts. In Neurips Safe Generative A/ Workshop 2024, 2024. www.htttss:/openevvieeee
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models mod- ulo permutation symmetries. In The Eleventh International Conference on Learning Representa- tions, 2023. URL URL httttt://ppeeeeiieeeee
Jesujoba Alabi, Marius Mosbach, Matan Eyal, Dietrich Klakow, and Mor Geva. The hidden space of transformer language adapters. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6588 6607, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL URL http:/ttt:/// /ttp:///aaaahhhaoo/a ooyy org/ 2024 acl long. 356
Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulié. Composable sparse fine-tuning for cross-lingual transfer. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1778-1796, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 121116633/1122222llllla
125. URL ht
acl-longe -oog..g.... 125
Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M. Ponti. Scaling sparse fine- tuning to large language models, 2024.
16405
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.0777 2021.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The bele- bele benchmark: a parallel reading comprehension dataset in 122 language variants. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Asxociation for Compulational Linguistics (Volume 1: Long Papers), pp. 749-775, Bangkok, Thailand, August 2024. Association for Computational Linguisties. URL https: 44
//aclanthoooyy.rrr///
L. Breiman. Bagging predictors. Machine Learning, 24:123-140 1996. URL https://appl semanticschon htt.::/cccc..oaaooooooororppaeae httpaaroor..cco......11..1111.1
Linzheng Chai, Jian Yang. Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang. Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. xcot: Cross-lingual instruction tuning for lingual chain-of-tien of-thhtthtt reasoning, 2024. URL htttps:// /tta:////xxiv.. org/ orgg.bs///ab....... 07637
11

---

Published as a conference paper at ICLR 2025
Tyler Chang, Zhuowen Tu, and Benjamin Bergen. The geometry of multilingual language model representations. In Yoav Goldberg. Zomitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 119-136, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 00.18653/112022.emmlppp
URL https: pattps://aclll antheloogy.ogg/g/.
emnlp ma main.9
Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. How do languages influence each other? studying tross-lingual data sharing during LM fine-tuning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 13244-13257 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v11111 222/33/112222... 1ttp//2222222mm www.:2233.mmmmmmmm.mm.m. m1....1... URL https: 2023. emn 1 p-mmalpmm---m mm0-mmai00 i 818
org/
Rochelle Choenni, Ekaterina Shutova, and Dan Garrette. Examining modularity in multilingual LMs via language-specialize subnetworks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 287-301, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18443/1/2024. findings naacl.
madingg-naacl..2 URL htw.lttts///// htt.pss:/ac.aattooooyyoooooooo//
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: 14168, 2021.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 1011865611/2020000 acl-main.777. URL 2020. 0020.ac1-10 ama-
www./yg..ooroooooooooooooooo org/
ma i....aainnn......... 747.
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Rlhf can speak many languages: Unlocking multilingual preference optimization for Ilms, 2024. URL lattps://arxa org/ abs/ / /t/p///a////2..////// /ttp//2477..5523 02552
MohammadReza Davari and Fugene Belilovsky. Model breadcrumbs: Scalable upcycling of fine- tuned foundation models via sparse task vectors merging. In ICMI 2024 Workshop on Foundation Models in the Wild, 2024. Utthtttps:/opeeerriiie
www.oorrummiiiiudo upip
Thomas G. Dietterich. Multiple classifier systems. In Lecture Notes in Computer Science, 2000. URL https: http://pp..maaniiiacccaa www.eteiischooooooeeee.e org/ Cttp:/ooouu::11111111
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Layerskip: Enabling early exit inference and self-speculative decod- ing. Meta Research, 2024. URL /tttts://atatttt meta com hctpa///eeeeeere////ooooccttoooo Layerskip-enabling--ee
ieettt-infeeenne and htt.::/eeeee..oaaaaoeoooeeeeeeaa
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1), January 2022. ISSN 1532-4435.
Felix Gaschi, Patricio Cerda, Parisa Rastin, and Yannick Toussaint. Exploring the relationship be- tween alignment and cross-lingual transfer in multilingual transformers. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Findings of the Association for Computational Lin- guistics: ACL 2023. pp. 3020 3042, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v112000 1/2023. findings-acl.cc 189. URL https: http://taaaannooggo/.//oo oxr/
2023. f nd www-saslll.. .18
12

---

Published as a conference paper at ICLR 2025
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Chengqing Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 188844896, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/202 111.
URL
20022222 acl long. 378
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer-
ence on Learning Representations, 2021. URL https: /opp//eeeeeeiieeeeeeeeeeeee net forum? d=
d7KB imI 3GmQ,
Masanori Hirano, Masahiro Suzuki, and Hiroki Sakaji. Ilm-japanese-dattsse vO: Construction of japanese chat dataset for large language models and its methodology, 2023. URL https://
arxiv.org/abs. /tlaaabb//30...0 2305. 2370.12200
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL / http::///ppeeeeeeeeeeee wwweeeervewwweeeeeeeeeeeeeeeeeee net / forum?
id=nZevKeeFyf9
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, 2023. URL 6tCKff8 jrrj
net www./eeet///rneeee
Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, 2018. URL otc. wow./ooro/ro.i
semanticscho
3833416
Nihal Jain, Dejiao Zhang, Wasi Uddin Ahmad, Zijian Wang, Feng Nan, Xiaopeng Li, Ming Tan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Xiaofei Ma, and Bing Xiang. Contra- CLM: Contrastive learning for causal language model. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6436-6459, Toronto, Canada, July URL
2023. Association for Computational Linguistics. doi: 1/216/////12223aaalaoao https: http:///alaanthhooy..oooo 220/:/22332222 acc. 201...........n....... 355
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv.orq//////e..... wrt..bgg///o.......oooo...o....s 2310. 06825
Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G. Varun Balan G. Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra. Indicllmsuite: A blueprint for creating pre- training and fine-tuning datasets for indian languages. arXiv preprint arXiv: 2403.06350 2024.
Simran Khanuja, Srinivas Gowriraj. Lucio Dery, and Graham Neubig. DeMuX: Data-efficient multilingual learning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7423-7436 Mexico City, Mexico, June 2024. Association for Computational Linguisties. doi: 10.18653/v1/0024 naact-long 412. URL (RL.httts:../ http:///aacaaatoloyoooooo org/2024 www.aaaallaaaa long. 412
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. On the mul- tilingual ability of decoder-bart based pre- pre-trained language models: Finding and controlling language- specific neurons. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies (Volume I: Long Papers), pp. 6919-6971, Mexico
13

---

Published as a conference paper at ICLR 2025
City, Mexico, June 2024. Association for Computational Linguistics. doi: 20.18653/v112000 naacl-long.384 URL htt.h:ttp///// httt/t:s://aaaaanne attp:////ccaatthooroo.y.. wtwp:ooyy..ogg2222nnaaoea
384
Wen Lai, Mohsen Mesgar, and Alexander Fraser. LLMs beyond English: Scaling the multilingual capability of LLMs with cross- cross-linguall feedback. In Lun- Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Findings of the Association for Computatioonal Linguistics ACL 2024, pp. 9186-8213, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL httpt::////..... / acc antho wwtthhllg...oo..o ogy. ory. 2024 f Find nns-anll...... 4488
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettle- moyer; and Madian Khabsa. XIMMVV Overcoming the vocabulary bottleneck in multilingual masked language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 13142-13152, Singapore, December 2023, Association for Computational Linguistics, doi: 10. 10.18653/1 1/2023. emnlp-main.8. 813. URL http::ttp////m pttLittts/ccaaaaaolollo/
ort. wtt.:/2022..mmmmmmmmmmmmmmm
www.emmmmpp.....m. www.eenwwammaa.... 813
Meizhen Liu, Xu Guo, He Jiakai, Jianye Chen, Fengyu Zhou, and Siu Hui. InteMATs: Integrating granularity- specifff multilingual adapters for lings-lingual transfer. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5035-5049, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1011116611122200 10t6://///22....nnneeeem//////// findings-emmlp.3355. URL https:tpp//.. //acaalana aataanoooogee org/2023.
Findings emn.p.sseemip...... 335
Team Llama, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian andAhmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and et al. The Ilama 3 herd of models. Meta Research, 2024.
Michael Matena and Colin A Raffel Merging models with fisher-weighted averaging. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17703-17716. Curran Associates, Inc., 2022. URL
www.ips........ cc/ wwwppppapp.... ieerii. wow.appprr-oooccenee.e.m pdf
File/
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math, 2024.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning, 2022.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. First align, then predict: Understanding the cross-lingual ability of multilingual BERT. In Paola Merlo, Jorg Tiede- mann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chupter of the Association for Computational Linguistics: Main Volume. pp. 2214-2231. Online, April 2021. Association for Computational Linguistics. doi:
URL http//222.eaalaain... https: antho wwwphhhlllg.ooogg//000 www.://0/2...lommaa.... eacl 189.
Team NLLB, Marta R. Costa-jussà, James Cross, Onur Çelebi. Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan. Dirk Rowe, Shan- non Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Saftyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling liman-centered machine translation. Meta Research, 2022.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tan- gent space: Improved editing of preitrained models. In seienh Conference Neural Information Processing Systems, 2023. URL https:/ https:://penzeviies wwt.forummfiiiie
9 2ZDGW
14

---

Published as a conference paper at ICLR 2025
Xuan Ouyang. Shuohuan Wang, Chao Pang. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang- ERNIE-M M: Enhanced multilingual representation by aligning cross-lingual semantics with mono- Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 27-38, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi:
lingual corpora. In
1/203/111/2111// wttp0//ee.m.mmmmaaaaaaaa //aclanthologys org 22011.2221... emnlp main.3
URL https:
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaud- hary, and Xia Song. Beyond English-centric bitexts for better multilingual language represen- tation learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15354-15373, Toronto, Canada, July 2023, Association for Computational Linguis- 1atp://////222.lllooooo 856. URL https: aataaaaaanthhlccoo org/2023.
tics. doi:
acl- oog.. 856
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Bonnie Webber, Trevor Cohn, Yulan Hc, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 7654-7673, Online, November 2020. Association for Computa- tional Linguisties. doi: 10. 20.11663//1/2020.com org/ emn em main 617
www.annnnlmmmmiiiimmmmmmmmm main.617. URL https::://caaaattllll..
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Marine Carpuat, Marie- Catherne de Marneffe, and Ivan Vladimir Meza Ruiz (eds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3479-3495, Seattle, United States, July 2022. Associa- tion for Computational Linguisties. doi: 10. N...8863//1/2222..m www.naaaaaaamaa5555555 URL https: 7 //aclanthology.erz 2022.022222222222 paaaacl-maiin..
Fred Philippy, Siwen Guo, and Shohreh Haddadan. Towards a common understanding of contribut- ing factors for tross-lingual transfer in multilingual language models: A review. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 5877-5891, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10211865//122223.cllbbbob 323. /tthhttps: /ttp:://acaatttt httaaaaaannnooooooooooo org 2023. acl--onn 323
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompt- ing: Improving zero-shot chain-of-thought reasoning across languages. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 2695-2709, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2023..oo eet.:3emmllpaaat 166.. URL https: //aclanthol ogy. wtg.oyy..org/233 emn lommnl-mmiin 163
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In Interna- tional Conference Machine Learning, 2022. URL https:///app..... wtt.ppppssmaaataassacaaeecceeeee
org/ /C:ppos1D:2548777
Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. aataral-alliiimeeet----eeeeeeeee
MAPO: Advancing multilingual reasoning through mization. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10015-10027, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL httts::///aacc anthho wwwaaanntoooooooooooooo ogy- wog..org//ooooo 2024 2024 acl - long. 5339
opti-
Freda Shi, Mirac Suzgun, Markus Freitag. Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Lan- guage models are multilingual -ottuughttthhttttth reasoners. In The Eleventh International Confer- ence on Learning Representations, 2023. URL /ttpps://penzzevew net/ ffrummidd
FR33GCk- IXp
15

---

Published as a conference paper at ICLR 2025
Shivalika Singh, Freddie Vargus, Daniel D' D'souza, Börje Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O'Mahony, Mike Zhang. Ramith Hettiarachchi, Joseph Wilson. Marina Machado, Luisa Moura, Dominik Krzemifiski, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sehastian Ruder, Surya Guthikonda, Emad Alghamdi, Sehastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadace, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 11521-11567, Bangkok. Thailand, August 2024a. Association for Computational Linguistics. URL https:://aclaattooo ogy. org.
2024.acl-les 620.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel. Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024b.
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, 2021. URL httph::tt/psssseeeeee
net/ forum?id
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji- Ji-Rong Wen. Language- specific neurons: The key to multilingual capabilities in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pp. 5701-5715, Bangkok, Thailand, August 2024. Association for Computational Linguisties. org/ 2024. acl- long. 309
URL htt.ttts://aaaaatttii
Atnafu Lambebo Tonja, Bonaventure FP Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Pe- ter Wairagala, Aremu Anuoluwapo, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, et al. Inkubalm: A small language model for low-resource african languages. arXiv preprint arXiv:2408. 17024, 2024.
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual mod- els: Findings and a meta-learning treatment. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4438-4450, Online, November 2020, Association for Computational Linguistics. doi: 10.18653/v wwt.00eeelllmmiin33... URL htttps://aclaanthlooo. orq/ 2020. www.eennnlma...e maa...mmaii.......mmmmm 359
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Ilamas work En- glish? on the latent language of multilingual transformers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 15366-15394 Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: http://acaanntcoo..ooroooee
acl long. 820
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lpess, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Confer- ence on Machine Learning, volume 162, pp. 23965-23998. PMLR, 9 2022. URL https:
//proceedings mlr. www..prres...... prw.s /veees/vv222 62 /wo/://wrtt// http://wwraaaaa....mm 2man22a. htmml
Rumxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei
Huang. Raise a child in large language model: Towards effective and generalizable fine- tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih
16

---

Published as a conference paper at ICLR 2025
(eds.). Proceedings of the 2021 Conference Empirical Methods in Natural Language Pro- cessing, pp. 9514-9528 Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguisties. doi: 10.18653/w1/2221.ccm
749. URL https:
//acl paaclanthologie org/2021 emn p- ma in. 749
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. mies-nggging: Re- solving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint vxXiv:2407.10611 2024a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities, 2024b. URL httph:ttttsxiiii org abs/ 07666
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In International Conference on Machine Learning. PMLR, 2024.
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient mod- ules with arithmetic operation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
nwt. forum? 1aeammmddd-55eee22e IGy
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11263- 11282, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL https: www.llanthhlo...orgggo... aonoooo..
Zhihan Zhang, Dong-Ho Lec, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. PLUG: Leveraging pivot language in cross-lingual instruction tuning. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 7025- 7046, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. URL https::/accaantee logy. wwwpor//222222 2000244.clloongg 379
Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. Unveiling linguistic regions in large language models. In Lui Ku, Andre Martins, and Vivek Srikumar (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6228 6247, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. URL htttps::/aclaatthllott
200/g//222..co 2w1.024..1-1.... oog../oooo...............oa-o--. 338
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. Allocating large vocabulary capacity for cross-lingual language model pre- training. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3203-3215, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi: 10.186633/1/20211emmllp h/aclannthology.orrg//// emn commlp-main...
257. URL https:
17

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 FINE-TUNING DATASETS
Table Datasets used for supervised-fin--tuiing (SFT) in this project
Category
Datasets
URL
Math
Orca Math word problems dataset from Microsoft (Mitra et al. 2024)
https://hugingffaee.
co/aasasee.//mirrssott//
Aya Dataset from Cohere for AI (Singh 2024a)
httes//auggingiee..oootttaee
Telugu
et al.
ConereeorA//333_att:3e
NLLB English-Telugh translation data
htta://bugingfsss.oottaass allcnai/nllb
from FAIR (NLLB al. (022)
English instruction dataset, machine trans- lated Telugu
Aya Dataset by Cohere for Al (Singh et al. 2024a)
httes://uggingfsss.co//tasss//
CoherreForr//ayaa_aaaaastt
Bengali
English-Bengal translation data from NLLB (NLLB et al. 2022)
http://hgggingssc..octtaacet
allcnai/n11b
IndicShareLlamm dataset from A14Bharat
htta://bugpinffss..ooootssssss
(Khan et al. 2024)
afsbharat/Endicraalggn
BongChat dataset from Lumatic Al
httpa:/huugingfssc.c//aaaat
Tmmailoaa///onnghh---112223
Aya Dataset by Cohere for Al (Singh et al. 2024a)
httas:/huggingsss..o/tttsss//
Conneeeoors/aay__aaaaattt
Swahili
English-Swahili translation data from NLLB (NLLB et al. (022)
http://huggingaac..//aaaattss
allcnai/nllb
Inkuba dataset from Lelapa (Tojja et al.
https://agginnaae..coaatassss
2024)
Islapa/Inkub-iisttrutt
xP3 MT dataset from BigScience, with
http:://brgingfaae.cc//aasssss
FLORES samples removed (Muennighofi
bi gsccience/xXP33 XP 3m
et al. 2022)
Aya Dataset by Cohere for Al (Singh et al. 2024a)
htpps//huppiiniae...o//aaaaas
CohereeorrAl/aaa__tttssse
English-Japanese translation data from NLLB NLLB et al. 2022)
htte://ugginijee..o/tttasss// allenai/niib
Japanese
LLMM-Japanese dataset from Izumi Lab
htpp://huggingfee..o//ttssssss
(Hirano et al. 2023)
izumi 1amm=11aa/1lnmapppsssseaaaaaaa
Ichikara dataset from RIKEN AIP
http:///agglingaae..c//aatttttt
platacy/ichlkaaaa--lottrtooooo
Dolly dataset from Databricks, machine translated to Japanese
18

---

Published as a conference paper at ICLR 2025
A.2 SUPERVISED FINE-TUNING DETAILS
While many hyperparameters were tried, below is listed the hyperparameter configurations that led the three best checkpoints (the "experts") for each category. Note that in all runs, do check- pointing every 5000 samples and use different random seed for data sampling.
Table 4: Hyperparameters for the training runs that led to each of our "experts"
Expert
Learn Rate
Batch Size
Seq. Length
weight decay
clip, max norm
sched.
warmup ;92
steps
math #1
2.0
2048
0.1
1.0
Linear
1000
0.99
10
math #2
1.0
2048
0.01
0.5
Linear 1000
0.99
10-7
math #3
4.0
2048
0.1
1.0
Linear
500
0.999
10-8
jpn #1
1.0
1024
0.1
0.5
WSD
1000
0.995
10-7
jpn #2
1.0
1024
0.1
0.5
WSD
1000
0.99
10-7
jpn #3
2.0
1024
0.01
0.5
WSD
1000
0.99
10-7
swh #1
7.0
1024
0.1
0.5
WSD
1000
0.99
10-8
swh #2
2.0 10-7
8
1024
0.05
0.5
WSD
1000
0.99
swh #3
1.0
8
1024
0.1
1.0
WSD
1000
0.999
10-7
tel #1
7.0
8
1024
0.1
0.5
WSD
1000
0.99
10-8 8
tel #2
5.0
8
1024
0.05
0.5
WSD
300
0.99
10-8
tel #3
1.0
2048
0.1
0.5
WSD
1000
0.99
10-7
ben #1
7.0
8
1024
0.1
0.5
WSD
1000
0.99
10-8
ben #2
5.0
8
1024
0.05
0.5
WSD
300
0.99
10-8
ben #3
1.0
4
2048
0.1
0.5
WSD
1000
0.99
10-7
19

---

Published as a conference paper at ICLR 2025
A.3 PERFORMANCE OF INDIVIDUAL EXPERTS
Table 5: Comprehensive results for each of the set of experts on tasks in the target language. Each number represents average over the 3 selected experts. The "10-lang avg" column presents the sults across 10 languages (English, German, French, simplified Chinese, Russian, Spanish, Japanese, Bengali, Swahili, Telugu).
Average across 3-experts
Task
Language
LLAMA 3.1 8b
Math
Japanese
Bengali
Swahili
Telugu
MGSM
English
0.560
0.663
0.561
0.563
0.557
0.561
em majimj 8-shot
Japanese
0.336
0.427
0.359
Bengali
0.292
0.383
0.335
Swahili
0.248
0.295
0.247
Telugu
0.120
0.201
0.200
10-lang avg
0.311
0.454
BELEBELE results
English
0.871
0.877
accuracy, 5-shot
Japanese
0.760
0.754
0.766
Bengali
0.649
0.654
0.683
Swahili
0.553
0.557
0.584
Telugu
0.546
0.557
0.602
FLORES results
Eng-Jpn
20.6
20.5
20.5
BLEU, 0-shot
Jpn-Eng
28.5
28.5
28.9
Eng-Ben
18.8
18.8
17.9
Ben-Eng
30.5
29.7
29.0
Eng-Swh
12.5
12.9
13.0
Swh-Eng
31.5
31.5
30.2
Eng-Tel
15.7
15.2
16.5
Tel-Eng
29.2
29.0
27.6
MBPP result
Japanese
0.464
0.470
0.462
pass @ 1, 0-shot
Bengali
0,438
0.436
0.462
Swahili
0.402
0.407
0.407
Telugu
0.410
0.412
0.437
MMLU result
Japanese
0.520
0.518
0.518
accuracy, 5-shot
Bengali
0.422
0.420
0.436
Swahili
0.419
0.420
0.425
Telugu
0.415
0.412
0.427
20

---

Published as a conference paper at ICLR 2025
A.4 RESULTS FROM COMBINING WITH SOUPING
Table MGSM 8-shot results when combining layer swapping with model souping of the experts for each category. "Multilingual soup" refers the uniform soup of all 12 language experts.
Setting
model soup, pairwise
layer swap, pairwise
layer swap,
layer swap, multilingual soup
language soup w/
math soup
w/ math soup
Details
default config, avg of 9 pairs
default config, avg of 9 pairs
default config. version
default config, version
Swahili
29.3
32.4
32.8
29.6
Telugu
20.9
22.7
22.0
21.2
Bengali
36.8
37.1
40.4
36.0
Japanese
38.7
38.5
39.2
41.6
A.5 PARAMETER-LEVE VISUALIZATIONS THE EXPERT VECTORS
Larger parameter-level visualizations for number of the experts, each visualization is the same as described in Figure 2
attertioon weight attention.wkk.meiih.
attention..ww..eight attention ao weight
fewd_forward.wt wwt.://ww......wigtttt.....
fewc__frrwardww3..wwli feed_forward weight
0.411865
0 0606608
0 840620
0.629863
0.480867
0.526611
0.374442
2.813477
0.659424
0.459836
0.078172
0.153320
0.902129
0.371338
3,508788
0.485352
28
0.001963
0.001708
0.703857
0.610596
0.210205
0.485361
0.507812
0.001600
0.007324
0.977226
0.884466
0.119229
0.441022
0.482033
0.000000
0.009277
07700066
0.609819
0.064180
0.919586
0.994992
24
0000000
0.002444
0 07869037
0.568306
0.044888
0.292480
0.374289
0.000732
0,202644
0.711914
0.668730
0.020006
0.328867
0.607645
C.01709
0.010354
a 0.004297
0.483643
0.028799
0.288300
0-404307
0.004418
0.006616
0.436791
0.401611
0.020508
0,286840
0.800640
0.001709
0.107422
00000000
0.000000
0.099801
0,104092
0.007588
0.009654
0.500293
000000
0.000000
0.088105
0M42334
0.001221
0.042480
0.24241
000000
0.00244
0,04439
0.100201
0.000244
0.009977
0.000617
14
000000
0.000000
0.000977
0.024008
0,000000
0.004488
0.040179
0,000000
0.000488
0.018822
0,000000
0.000244
0.020169
0.000000
0.001444
0.000088
11
80000000
0.000000
0.000044
0.007280
0.000000
0.000000
0.008417
000000
0000000
00000000
0.017762
0.000000
0.000006
0.040004
0000732
000000
0.000000
0,000000
0.000244
0,000000
0.000000
0.001186
0000000
0.000000
0.000000
0.001485
0,000000
0,000000
0.000977
6,000000
0.009277
0.000000
0.001955
0.0000on
6,090000
0,000000
0,809838
80000000
0.00:44
0.699833
0,008809
0.000000
0.000000
0.604398
000000
0000000
0.001221
0064643
0,000000
0.000000
0.00906
0.000000
0.005465
0.000000
0.016849
0.044244
0.014611
0,088118
0.000000
0.000000
0.041166
0L032227
0034668
0.095331
0.042236
0.001700
0.092197
0J03884
Figure Visualization of the magnitude of change during SFT for "math expert #1"
21

---

Published as a conference paper at ICLR 2025
attention. ang neight
whoeggnn attentann.wraeiggh stterdor.ww..eight und_Sarward.nr arianeghh feed, www.arddddaligiee...oo Reed, wwwward weligg
0751799
0035142
D423220
083345
0.251214
0.201300
0 c55322
0016601
C.990234
0.138279
0/20771
0.340000
20
042400
2200370
CA52743
2375607
0.242477
0350224
0.323730
00000001
200053
04672216
0/11128
0.000000
0.276432
6.044091
0-944077
000000
04454000
0339227
0014844
0 1986667
0179476
0033300
0016113
0648414
0.064482
0.014555
083922
23
00009901
wesscon
0.221997
0.773426
0012200
0.11-902
0-420410
0100010
2012000
C.002242
0106116
6.901064 064
0.090000
0010346
18
0000266
0 200000
0 06.0003
0009996
0.00006
16
8000968
08657788
0 880000
8-181967
0000662
0/007110
5 5966447
001220
0000444
5103/89
0.00007
0C46367
0003003
20014055
00025115
0143005
0.000000
0033067
000993
54
2000300
0010000
020000
0000344
C CO0000
50003000
0000-000
D001010
C.000000
O 000048
0833783
6000300
0000083
0.000003
0000000
000030
0,000000
0001000
10
5000300
600700
0000003
0.000000
000000
2209428
0 0234002
0,000003
2.000044
0,000000
C 000000
0000007
50000000
0C00014
0.000003
0.000000
5.000000
000000
00000057
0.000322
5.000000
0.000000
C.000000
C.000000
0000000
000000
50003000
0.014021
0.00000
5.000244
5.000000
000030
0C0095
5,000000
00003777
0,000000
0,000000
0,000000
C 000000
000000
50003000
0000000
0.000003
2,000155
5,000000
D8222400
0085440
0024193
2.009300
5.000000
0.000000
000092
0.78570
0.3027TH
00315M
0023930
C.004118
0.003300
0.003322
Figure 5: Visualization of the magnitude of change during SFT for "math expert #2"
wheight
0.159425
0862/554
0.526904
0.700199
0:209334
0:105644
0.375344
20
51177030
0. 0.425293
G 0.40337
2/453122
2.003944
0.004436
0.2070000
0220145
0320633
DA020M
2810149
0038821
0.000005
0.155922
0014845
109027
00004455
0 0000000
24
0021720
01437790
CJ70002
GEBOSN
0002590
00006030
0C00920
3099792
011044
715000
2811320
0
00004033
0.009122
8290086
0264600
0:00700
0000000
0546007
0127328
0335000
00009400
0002800
75
5007335
1.
DA475TT
20007322
18
0044104
0249827
0197541
20007722
0841000
0037100
a137000
2000003
220000
0000000
1003000
1
0000000
0600000
16
3 5770000
0000944
14
0040071
0000044
68444000
0003277
1200090
0000044
0300000
0000000
00000000
5001009
0000300
00 0000
10
0.201221
5007204
00020000
00000000
0032907
06000000
0000000
0000000
0070400
0028732
0003000
0001903
0099040
00000000
00000000
00000000
0822007
0200mee
0000732
2010006
00000000
0.000000
0.000233
3.009930
5011649
0003733
0009941
0000000
0030000
0.260003
0500085
0009104
60007992
0027944
0000000
0000000
0.009997
0C00005
0018551
0.300000
0/024459
0030000
0030000
0000992
D10747077
0.623779
0.146632
2.01242
00000000
0.000000
0.002232
0LE04455
0684500
06775333
0324218
0208311
6437000
001479
Figure 6: Visualization of the magnitude of change during SFT for "Swahili expert #1"
22

---

Published as a conference paper at ICLR 2025
sttertianarrasiigh
sttedaraasasigge
2260354
04762700
CD30379
5797852
0.122291
0.155003
1187206
20
2001335
0.000000
3001300
0.00008
0.776000
0.11263
000803
5002100
0.00076
0.6mma
5.087755
0000095
0395440
0.00/00
0000911
0008809
0010005
0,202003
807933
2.007322
0022441
D009272
24
6020904
0.340701
00411000
2668210
24007722
0001465
0010204
0000000
0049400
0.60001
0.160077
5.007700
000040
00012
0500015
0056590
50007702
0
0023207
0.158832
0.744131
D4010877
2.007722
0000222
0214432
10
3202019
0094223
5300206
20007222
2000011
20000
0030732
6000723
0 600 Ga
6-122281
0.007222
0.00:221
0030077
000174
6081769
0,99666
50000792
0001000
0042726
5000702
0035112
0000000
0000-14
0.00000
5.000332
5000732
2000
14
6033732
0.30TM0
0300000
0017334
22000722
2000713
6036733
0000400
0.003000
0001005
5.000222
0026611
0028068
0000400
0042946
54000702
50000002
0020732
0.034771
0.06544
00445997
6033732
0001025
00007334
2042400
22000722
0008733
5010400
0.001221
0C95346
0.000000
09C0000
0030044
0.000264
00028990
03003500
003/542
3200244
0077222
0,000005
0015207
0000000
0000000
0,22922
3.00033
0.001005
0.300002
0024056
0.000000
0.000000
00024100
0.400093
01366000
0:171509
00300000
00300000
0034644
33399908
808124
0888929
2.07007
03599311
0/022709
0021279
Figure 7: Visualization of the magnitude of change during SFT for "Japanese expert #1"
3227924
0.162892
0:00000
0 605/700
0:144501
0:911001
0229957
25
3.506055
2647717
c 570654
0010043
0000005
01154403
0.c0000
800017
0.335900
13711137
0.c0mm0
2.02357
600970
09245100
0024886
0014661
26
0022217
115600
c 868027
3 681690
0000044
000999
0002199
24
0014893
O 050590
C s04
0.607255
0003480
0039324
0.007000
#500e1
b800001
0003440
0
0015004
0424014
0003449
20
0210991
0011643
o 69006)
04906000
0000065
5000997
0016055
15
5009789
0000227
04380422
0003485
00007722
0020732
0000000
0.esone
37995544
0000044
5000792
0223399
0033332
0002006
0246779
2.007332
3.000077
0034110
15
6032807
08002488
A377704
2000001
0000000
00X2W7
00000000
0:002999
0000085
0000000
00000022
0220248
0000099
0517544
C207842
2790006
0032579
C040527
0002007
0000000
0300753
00238603
0247763
01107781
10
0C00403
0002000
0240522
0.010033
2,002117
0150000
0000000
00034408
0030000
50009977
00000077
0009774
2.011022
0000400
0110174
0000044
0.008371
0240375
0C00403
0.012211
o 3300748
0137306
8003403
00000077
0241223
6036733
6003441
5/472117
5.007332
50072266
0031600
5/005500
0033000
0,212991
04541990
0038133
0100381
00004003
0002400
0.500347
0.279422
0,442571
03354345
0.345700
0000197
0002036
0.229925
Figure 8: Visualization of the magnitude of change during SFT for "Telugu expert #1"
23

---

Published as a conference paper at ICLR 2025
attention.ae and weight
atentton woonoigt
attentlasswwwweight attentass.wo.wdiitt feed, forward ar Leeight
feed, arwardaraaeiigtt feed, w2.egght
06422502
6:009912
00093355
6.377422
0448378
0433607
0.111144
D092547
0148995
28
0.19075
0040005
eX77079
3802051
0.001000
0.039115
000000
1200338
0030003
0.0000
3000100
668920
0000486
0.000300
0.000222
0.00004
60M0009
0/700446
0-000722
000996
0021196
8000906
0 000069
6007700
0600176
0.000722
0 D0n955
0.0 8954
0.000032
00002229
0.22670
40004400
0.009002
024-004
19
6032107
0000/004
0.207701
0.222722
0000244
0000032
CO20011
0300039
0000000
0280296
3.00030
0.000022
U2205/4
14
0033107
0000014
0222042
2.002441
CO00300
C.137005
0.001221
00000000
0.090822
0100001
CC00086
0000000
0379044
010000422
00000C4
0209645
WG28005
01002994
0226240
0.0099
0.000077
0.911004
C.000000
C.002001
2000244
0020000
0.00001
0.103822
DO00344
0.00R01
5000244
0.001997
0016009
2:10221
5,000000
O D00966
0022670
0,076463
0.338135
0.112277
2.000031
5.000000
0.000244
0.007322
0250301
03320003
0573593
0.333342
0007200
2.00922'
C.001122
Figure 9: Visualization of the magnitude of change during SFT for "Bengali expert #1"
24


Reviews:
[{"review_id": "7wEpHwCrsR", "reviewer": "Reviewer_edMm", "summary": "This paper proposes a new approach to perform zero-shot cross-lingual transfer for solving tasks in a new language. The requirement is to have data for the task in a more resourced language (e.g. English) and non-task-specific data for other languages (4 languages in this paper). The idea is to fine-tune the model (Llama 3.1, 8B in the paper) separately to one new language and another copy to the task in English, then compose a new model of layers of these two models. A study is included of how much the models change during fine-tuning, which concludes that math tasks cause changes in the fine-tuned model closer to the middle layers, while tuning on a new language causes changes in the first and last layers. Based on that the composed model takes task-layers from the math-tuned model's middle, and language-layers from the language-tuned model's start and end. Transfer between the layers of two independently tuned models is done by \"souping\" intermediate layers, i.e. averaging the weights of layers of both models. Gains up to 10% are shown, in comparison to the expert models.", "questions": "1. You write that combining layers works because models are fine-tuned for just a little bit: why not tune for longer?\n2. What would the intuitive explanation be behind the findings of your preliminary analysis on which layers are changed by task-tuning and language-tuning, other examples of similar effects in related literature?\n3. If you were to tune for longer, would that affect the results of the preliminary analysis and change more layers, or different layers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Interesting idea and its evaluation on 1 model and 4 languages, with additional experiments\n* Although the setup raises some questions (limited evaluation, why not freeze layers and avoid having to soup the transition layers, etc.), the expanded evaluation on Swahili and the limitations section address most of these\n* excellent writing, justification and presentation", "weaknesses": "* limited evaluation: only 1 model and one set of tasks (math)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKlc7YcDYV", "reviewer": "Reviewer_oaJT", "summary": "The paper introduces a variant of model merging addressing the challenge of adapting LLMs for tasks in low-resource languages. The methodology involves *layer swapping* (parameters being swapped) between a task expert and a language expert, both following the same underlying architecture. The resulting re-composed LLM is said to outperform both the individual experts without the requirement of a task-specific fine-tuning in the low-resource language. The experiments involve evaluation of the proposed methodology on MGSM (a multilingual benchmark of grade-school math problems) on 4 resource-scarce languages: Japanese, Telugu, Swahili and Bengali.", "questions": "1. How does a re-composed model in one language affect model performance in typologically similar languages? In my opinion, an analysis of this kind would highly benefit the work. \n\n2. Would a 2-stage SFT work better than Joint SFT on language and task?\n\n---\n\n**[POST-REBUTTAL UPDATE]**\n\nI have decided to increase the overall rating from **5** (marginally below the acceptance threshold) to **6** (marginally above the acceptance threshold).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The proposed methodology is highly practical in scenarios where one might have publicly available task-specific data in a high-resource language and generic instruction data in the low-resource language. The model parameter adjustments being fully post-hoc eliminate any additional computational overhead apart from the initial fine-tuning required to create task and language experts.\n\n2. *layer swapping* with the best configuration consistently outperforms the individual SFT experts, the base LLM Llama 3.1 and the general model souping approach in three (Swahili, Telugu, Bengali) out of the four languages under this study.  \n\n3. The paper is easy to follow. The authors also take the effort to acknowledge the possible limitations to the work, encouraging future exploration.", "weaknesses": "1. The methodology is evaluated only on Llama 3.1, using MGSM benchmark for 4 selective languages. In my opinion, evaluation of the method on a single model, single benchmark and limited languages makes the conclusion less generalizable. While the languages used in the study are diverse, incorporating more datasets and models (in terms of different architecture or pre-training) can strengthen the conclusion.\n\n2. The assumption of availability of generic instruction data for low-resource languages might not hold for all languages. Task-specific data and generic instruction data in a high-resource language is generally more accessible. An experiment where language expert is fine-tuned using translated instructions would increase the practicality of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F5LIUi21BL", "reviewer": "Reviewer_X6PS", "summary": "The paper introduces a novel layer-swapping methodology for zero-shot cross-lingual transfer in large language models (LLMs) aimed at improving task performance in non-English languages, particularly for mathematical reasoning. The approach tackles the lack of task-specific data in low-resource languages by fine-tuning two separate \"experts\" of the same base model: one trained on English mathematical data and another trained on general instruction data in the target language. The proposed method then selectively replaces the top and bottom transformer layers of the math expert with those from the language expert, buffered by transition zones between these regions. This configuration shows promising performance gains on the MGSM math benchmark across four languages—Swahili, Telugu, Bengali, and Japanese—without any additional in-language math data.", "questions": "1. Can the authors clarify if they have tried layer swapping on other tasks, such as translation, question-answering, or code generation? Evidence of generalizability beyond math would make the approach significantly stronger.\n2. Layer swapping is positioned as an alternative to methods like model souping, but a comparison to more recent modular fine-tuning techniques, such as adapters (Pfeiffer et al., 2020) or LoRA (Hu et al., 2022), could help contextualize its relative strengths. Can the authors either conduct these comparisons (e.g. computational efficiency, performance) or discuss the anticipated performance differences? \n3. Given that this study uses a 8B (32-layer) LLM, how would the authors anticipate the method scaling to models with more layers or parameters? Could they provide guidance on applying layer swapping to larger models, especially in terms of choosing the number of layers to swap? Would the authors still anticipate the same performance gain with layer swapping on larger models? \n4. For Japanese, layer swapping results in lower average performance compared to the individual math experts. The authors mentioned that the Japanese experts were the weakest as performance across BELEBELE, FLORES, MBPP, and MMLU were minimal. Could the authors share the results on these benchmark (before and after SFT) to help better understand the case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces an efficient, innovative layer-swapping method for zero-shot cross-lingual transfer in LLMs, addressing the lack of task-specific data in low-resource languages with simplicity and strong empirical results.\n\n- This technique is particularly notable for its straightforward implementation, allowing effective merging of task and language expertise without complex adjustments, making it a practical alternative to standard methods like model souping. \n\n- Promising experimental gains on math reasoning benchmarks across multiple low resource languages validate the method's effectiveness, showing that layer swapping successfully enhances cross-lingual transfer without in-language task data.", "weaknesses": "- The method is tested only on math reasoning, leaving it unclear if layer swapping generalizes to other tasks. Additional evaluations on tasks like question-answering or translation would strengthen the claims of broad applicability.\n- While the paper mentions different layer-swapping configurations, it lacks in-depth analysis on which configurations work best and why. A more detailed study of these choices would help to better understand the method make it more robust. For example, provide ablation studies on the number of swapped layers or transition zone sizes, or to analyze how performance changes as these parameters are varied.\n- Comparisons to recent modular fine-tuning techniques, such as adapters or LoRA, are missing. Including these would clarify how layer swapping performs relative to other efficient, cross-lingual methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)