Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models

## Abstract

Abstract Model merging, such as model souping, is the practice of combining different models with the same architecture together without further training. In this work, we present a model merging methodology that addresses the difficulty of fine-tuning Large Language Models (LLMs) for target tasks in non-English languages, where task-specific data is often unavailable. We focus on mathematical reasoning and without in-language math data, facilitate cross-lingual transfer by composing language and math capabilities.
Starting from the same pretrained model, we fine-tune separate “experts” on math instruction data in English and on generic instruction data in the target language. We then replace the top and bottom transformer layers of the math expert directly with layers from the language expert, which consequently enhances math performance in the target language.
The resulting merged models outperform the individual experts and other merging methods on the math benchmark, MGSM, by 10% across four major languages where math instruction data is scarce. In addition, this layer swapping is simple, inexpensive, and intuitive, as it is based on an interpretative analysis of the most important parameter changes during the fine-tuning of each expert. The ability to successfully re-compose LLMs for cross-lingual transfer in this manner opens up future possibilities to combine model expertise, create modular solutions, and transfer reasoning capabilities across languages all post hoc .

## 1 Introduction

Instruction fine-tuning Large Language Models (LLMs) is necessary to customize pre-trained models for real-world applications. This fine-tuning is especially critical in multilingual settings because most popular open-source LLMs have been pretrained on highly English-centric data (Llama et al., ; Yang et al., ; Jiang et al., ).
While recent LLMs like Llama 3 and Qwen2 have seen over a trillion non-English tokens due to the sheer scale of their pretraining datasets, these tokens are still heavily concentrated in just a few languages, resulting in limited capabilities for most other non-English languages (Llama et al., ; Yang et al., ).
Furthermore, the scarcity of high-quality labeled data available for post-training in non-English languages and the tremendous cost of procuring it further exacerbates the inequality.
Even machine-translating English post-training data to target languages—a typical solution—has significant computational overhead and often leads to datasets of unreliable quality (Khanuja et al., ). Numerous efforts have set out to annotate or assemble high-quality datasets for lower-resource languages (Khan et al., ; Singh et al., ; Tonja et al., ), but a massive gap still exists in many tasks and domains, such as math. As a result, developers are forced to largely rely on cross-lingual transfer—the generalization of learned capacities from high-resource languages to lower ones—but the rate of such transfer is low for most languages (Philippy et al., ).

Figure: Figure 1: Our merging method which swaps in top and bottom transformer layers from a language expert into a math expert, buffered by a transition zone.
Refer to caption: extracted/5894914/images/training_experts.png

In this paper, we present a novel solution that merges two LLMs together in order to transfer math reasoning capabilities to lower-resource languages during supervised fine-tuning (SFT). In the absence of in-language math data, we fine-tune two variants of the same pretrained model: one with English math samples and the other with generic instruction data in the target language.
Provided these variants, which we refer to as “experts”, we combine their learned language and task capabilities by re-composing an LLM with a mix of parameters from each. Notably, the top and bottom few transformer layers are selected from the language expert and the middle transformer layers are selected from the math expert. In addition, we build 1- or 2-layer “transition” zones in between, which consists of weighted averages of the parameter values of both. This simple, yet strategic, recomposition leads to high performance in math in the target language and avoids negative interference. The intuition behind this *layer swapping* method is informed by our analysis of the SFT updates that led to the experts. We find that the learned math capabilities are concentrated in the middle transformer layers, especially in the second half of the model. Meanwhile, we find the enhanced language skills to have come from parameters closest to the input and output, in line with previous literature stating this is where the most language-specific representations are concentrated (Chang et al., ; Choenni et al., ). Therefore, our methodology takes the most important layers from each expert and transfers math capabilities to the target language.

The resulting *layer-swapped* models deliver strong performance across numerous target languages (Swahili, Telugu, Bengali, Japanese) on MGSM (Shi et al., ), the manually-translated version of the Grade School Math benchmark (Cobbe et al., ). In all these major languages, where math SFT data is not readily available, *layer swapping* outperforms baselines, the individual experts and model souping (Wortsman et al., ) by 10% on average. For Swahili, *layer swapping* exceeds MGSM performance of models on fine-tuned a mixed Swahili and math SFT dataset when evaluated. Therefore, this methodology provides a simple and effective way to improve the capabilities of LLMs without the need for any task-specific data in the target language. In addition, it is inexpensive and fully post hoc, meaning it has the potential to be practical in many settings. Fundamentally, the success of this method also provides empirical evidence for cross-lingual patterns in the latent structures of LLMs that can be further interpreted and exploited.

We discuss relevant literature in the following section. We then explain the analysis that led to our methodology and present *layer swapping* in detail in Sections 3 and 4. Next, we show empirical results and discuss findings in Sections 5 and 6. Finally, we propose future work prompted by the success of this methodology in Section 7.

## 2 Related Work

### 2.1 Model Merging

While the use of weight averaging to stabilize training through noise reduction well pre-dates instability challenges in deep neural networks (Breiman, ), the use of model merging to combine trained model checkpoints is an emerging research space in deep learning. Similar to ensembling model outputs (Dietterich, ), aggregating model weights also improves model robustness and generalization Izmailov et al. (), even when combining models trained on the same data (Ramé et al., ). Numerous studies seek to improve the pre-merging conditions, either via linearizing fine-tuning (Ortiz-Jimenez et al., ) or aligning weights (Ainsworth et al., ). Wortsman et al. () develops an empirical method to average, or *soup*, a subset of available model checkpoints together, in order to increase the model search space during hyperparameter tuning. Simple weight averaging, however, is at risk of negative transfer, or interference, between trained checkpoints. To address this, methods have been proposed to selectively combine models at the individual weight level, either using the magnitude and direction of the fine-tuning deltas (Ilharco et al., ; Yadav et al., ; Davari & Belilovsky, ; Yu et al., ) or leveraging information theory (Matena & Raffel, ). In parallel, sparse fine-tuning methods have been developed to create fine-tuned models with a small proportion of weights changed (Guo et al., ; Sung et al., ; Xu et al., ), which then allows adding together fine-tuning updates with less overlap. Overall, model merging, and notably inexpensive model souping, is very common in practice because it improves training stability, model robustness and generalization, and performance by increasing the search space or combining expertise in multi-task settings (Yang et al., ).

### 2.2 LLM Multilinguality

The inability for a language model to learn more languages without undermining other capabilities—the *curse of multilinguality* (Conneau et al., ; Pfeiffer et al., )—was a heavily studied problem in encoder models. Numerous investigations attempted to understand the quantity and location of parameter sharing versus language-specific parameters (Wang et al., ; Choenni et al., ) in massively multilingual encoder models (e.g. mBERT (Devlin et al., )). While higher rates of language-specific parameters were found in the top and bottom layers, largely because of the proximity to token representations (Chang et al., ; Choenni et al., ), language specialization occurs across the model. In addition, increasing the model vocabulary has proven effective in increasing a model’s language capacity (Zheng et al., ; Liang et al., ; Llama et al., ). And while recently model scaling has mitigated the limitation from parameter quantity, the massive amount of labeled data needed for LLM training presents a new challenge. In encoder models, cross-lingual transfer was enhanced by aligning cross-lingual representations (Patra et al., ; Ouyang et al., ; Gaschi et al., ), but this is difficult in decoder-only models. To boost transfer in post-training, several data synthesis or augmentation solutions have been proposed, both for SFT (Zhang et al., ; Qin et al., ; Chai et al., ) or reinforcement learning from human feedback (RLHF) (Dang et al., ; She et al., ; Lai et al., ).

### 2.3 Model Merging for Cross-Lingual Transfer

To address the limited representational capacity of multilingual models, many solutions have been proposed to strategically share or splits parts of the model. This could be major blocks, as in mixture-of-experts (Fedus et al., ; NLLB et al., ), or a few parameters in each layer, as in cross-lingual adapters (Pfeiffer et al., ; ). Ansell et al. () combines modular and sparse fine-tuning approaches in a two-stage SFT, where first task and language experts are fine-tuned and all weights who did not change more than a threshold are masked. Next, the experts are fine-tuned again from the pretrained model with this mask, creating sparse task vectors that can be composed together with lower rates of parameter overlap. This composable sparse fine-tuning was also adapted for larger decoder-only LLMs (Ansell et al., ). In this work, we develop a simpler and more flexible merging method for cross-lingual transfer that does not require fine-tuning more than once.

## 3 Preliminary Analysis

### 3.1 Setup

We start by training numerous math and language “experts” by fine-tuning Llama 3.1 8B (Llama et al., ). We perform SFT runs with 30-40k labeled samples with varying hyperparameters(^1^11Training specifics such as hyperparameters are provided in Appendix A.2) and for each type of expert, select the three best checkpoints. The math experts were fine-tuned on English math world problems from the Orca-Math synthetic dataset (Mitra et al., ).
In order to select the three experts, we use results on the English splits of MGSM, as well as the average across languages. For the language experts, we select Swahili, Bengali, Telugu, and Japanese as the target languages. These are languages present in MGSM and other benchmarks discussed below, but where Llama 3.1’s performance lags behind the top languages like Spanish.
In addition, boosting cross-lingual transfer to these languages is important because of the lack of in-language math instruction data. For each of the four, we mix together samples from available instruction datasets in that language, to create experts with enhanced general language and instruction-following capabilities.
The resulting datasets contain many different types of tasks—such as translation, NER, and question-answering—but no math(^2^22Dataset details are provided in Appendix A.1). After numerous SFT runs on these generic, multi-task datasets, the three checkpoints for each language are primarily selected based off their performance on the target language splits on Belebele (Bandarkar et al., ) and FLoRes (NLLB et al., ). These simple tasks, reading comprehension and translation, are strong indicators of basic language understanding and generation capabilities. We, however, also ensure slight improvement on MBPP (Austin et al., ), MMLU (Hendrycks et al., ), and MGSM as secondary measures of language improvement.

### 3.2 Parameter Analysis of SFT

Figure: 1A Japanese expert #1
Refer to caption: extracted/5894914/images/jpn_expert_vis_RESIZED.png

For these experts, we first investigate where the parameters are being updated the most and where they remain unchanged during fine-tuning. Using similar notation to Ilharco et al. (), let $\theta_{pre},\theta_{ft}$ be the set of weights for the pretrained model and fine-tuned expert, respectively. We generalize the delta of fine-tuning as $\displaystyle{\bm{\mathsfit{W}}}_{\Delta}={\bm{\mathsfit{W}}}_{ft}-{\bm{
\mathsfit{W}}}_{pre}$ for all one- or two-dimensional parameters ${\bm{\mathsfit{W}}}\in\theta$.
To compare model parameter deltas across tensors with different shapes, we represent the magnitude of change at the row level using mean absolute value (MAV) of the difference $\displaystyle{\bm{\mathsfit{W}}}_{\Delta}$ (for one-dimensional $\displaystyle{\bm{\mathsfit{W}}}_{\Delta}$, this is simply the MAV across the vector).

We observe high consistency in the patterns of these $\displaystyle{\bm{\mathsfit{W}}}_{\Delta}$ magnitudes across our math experts, and separately all our language experts, despite the latter being trained on different datasets and mixes of tasks. In Fig. 2, we show a visualization for a representative language expert and a representative math expert. For the language expert, we find that the attention parameters (left four columns in the visualization) are getting updated most significantly, notably in the first couple layers (bottom) and the last couple layers (top). The feed-forward layers (right three columns) do not change at all until the last few layers. In comparison, the math experts follow a different patterns. In these runs, the first half (bottom 16 layers) remain largely unchanged. In this second half, the attention parameters are being updated significantly, similar to language experts, except the feed-forward layers are also getting changed quite a bit. To demonstrate the consistency across training runs and languages, we show the same visualization for more experts in Appendix A.4.

### 3.3 Sparsifying Updates

We then attempt to create *sparse* model updates for increased composability similar to Ansell et al. (), but without retraining. We attempted leveraging the magnitude of the deltas to selectively update the model without undermining performance gains. Concretely, we use thresholds to determine whether to apply the update or revert the value to the original value in the pretrained Llama 3.1. This is done at row-level granularity, in order to not partially modify linear transformations. However, in our analysis across math experts, we find that more than 70-80% of model parameters are required to be updated for the increase in performance to remain equivalent. Such a small degree of sparsification would not significantly reduce interference between math and language experts.

We next attempted location-based sparsification. This means leveraging our intuition of what patterns occur during fine-tuning to select specific parameter tensors to merge or not. A breakthrough was made when we revert the first five and last two transformer layers of our math experts to their original values. After this mitigation, we find that the math experts perform equally well. This is significant because our intuition—based on our analysis and previous multilingual interpretability research on both encoder-decoder models (Chang et al., ; Choenni et al., ) and decoder-only models (Tang et al., ; Zhang et al., )—is that the first few and last few transformer layers contain the most important language-specific parameters. Meanwhile, our discovery implies the first few and last few transformer layers were not very important for the math experts. This contrast prompted the swapping of these layers from the language expert into the math expert.

## 4 Methodology

### 4.1 Layer Swapping

The *layer swapping* methodology takes two fine-tuned experts, one on the target language and the other on the target task in English, and re-composes a joint model with the top and bottom transformer layers from the language expert and the middle from the math expert.

As displayed in Fig. 1, we additionally design a *transition zone* in between the off-the-shelf layers from each expert. These buffer layers are weighted averages of the respective layers from each expert that ensure that the outputs of one layer does not directly input into a layer fine-tuned separately. While intuitively these transition zones seem necessary, we do not find empirical evidence that they provide statistically significant benefit over replacing them with the math expert’s layers. However, as discussed in Section 3.1, our experts were not fine-tuned for very long and therefore the latent representation spaces after each layer would not have diverged very much from each other. This explains, in theory, why transition zones were not helpful in our setting. We conjecture that if the experts had been trained further (or simply with higher learning rates), such a buffer zone would be necessary. We therefore still present it as a central component of our *layer swapping* methodology.

The implementation of *layer swapping* is as simple as iterating through the state dictionary of the math expert and for each parameter, either: (1) keeping it as is, (2) replacing its value with that of the language expert, or (3) averaging that of the math and language expert (See Algorithm 1).

Figure: Algorithm 1 *Layer Swapping*

### 4.2 Configuration

*Layer swapping* has several components that can be configured in various ways. Most notably, the number of layers to swap at the top and bottom and the number of layers to include in the respective transition zones. We tested how to configure these swapped layers and transition zones to most effectively merge these models, empirically. We find, however, that there is a wide range of possible configurations in which this methodology is still very effective. Note that all these experiments were conducted on a 32-layer Llama 3.1 transformer model. For a model of this size, we find that the desired configuration of each component is in the ranges listed below, with our default values underlined:

- 1.
The number of bottom layers to swap $b\in\{3,4,\underline{5}\}$
- 2.
The number of top layers to swap $u\in\{0,1,\underline{2}\}$
- 3.
The number of layers in the lower transition zone $t_{b}\in\{\underline{0},1,2\}$
- 4.
The number of layers in the upper transition zone $t_{u}\in\{\underline{0},1,2\}$
- 5.
The transition zones are averages of the layers from both (i.e. soups) that can be unweighted or magnitude-adjusted weighted averages.
- 6.
The non-transformer parameters (input token embeddings, output layers, etc) work best if they are also averages of the two experts, as opposed to simply from the language expert.

Figure: 2A Possible config. with more layers swapped in.
Refer to caption: extracted/5894914/images/4l2u2b.png

While the results were largely equivalent within this range, some patterns did exist. We find that for languages where performance was lower (e.g. Telugu), the higher performing configurations typically had more layers from language expert. This is perhaps because the boost from improving the language capabilities was more important relative to languages Llama 3.1 was already better at. For example for languages such as Japanese, the best configuration has close to the minimum layers from language expert, similar to the minimum swap illustrated in Fig. 3. Amongst the math experts, the math expert with the highest scores on all four target languages (named “math expert #2”) performed understandably better with fewer layers from the language expert swapped in. Generally, we conjecture that lower-resource languages tend to require more layers from the language expert.

## 5 Empirical Results

### 5.1 Experiments on Swahili, Telugu, Japanese, and Bengali

As discussed in Section 3.1, we launched SFT runs with different hyperparameters and on five different datasets: math data in English and generic instruction data in Bengali, Swahili, Telugu, and Japanese. For each dataset, we selected three checkpoints, striking a balance between top performance and sufficient diversity to ensure robust experimental results. For each language, the three language experts could pair with any of the three math experts which gives nine merged models to evaluate. To understand the effectiveness of *layer swapping*, we present the MGSM scores of the base model, Llama 3.1, and the individual experts selected to determine whether *layer swapping* constructively combines the learned task and capabilities. In addition, we present an additional baseline; the classic model soup (Wortsman et al., ). Model souping is the most common model merging method in practice, used in pre- and post-training for purposes such as boosting model robustness and generalization but also to combine capabilities of experts (Yu et al., ). In vanilla model souping, the merged parameters are simple averages of the parameters of the input experts. For each language, we additionally evaluate adjusting for the magnitude of change from the base model to the expert using a weighted average. Using notation from Section 3.2, the resulting weight of each expert is the inverse of the average MAV for all rows in all $\displaystyle{\bm{\mathsfit{W}}}_{\Delta}$ for that expert. For all model soup results presented, we select the highest MGSM score amongst these possible configurations. Given the number of experts and merged pairs in our evaluations, we present two aggregate metrics: the mean and the maximum of evaluations. While the mean demonstrates the consistency of each method, the maximum demonstrates the ceiling of the method, often more desirable given the ability to iterate through numerous settings during model training.

**Table 1: MGSM 8-shot results of *layer swapping* across four languages compared to the individual experts and model souping. Note that for aggregate statistics of the individual SFT runs, we select the 3 best checkpoints from numerous training runs with periodic checkpointing. The merging methods are aggregated over the 9 pairs (3 language experts $\times$ 3 math experts), which means the min, avg, and max measures are not perfectly comparable.**
| Setting | Llama 3.1 8B | language xxx expert | math expert | model soup | *layer swap* | *layer swap* |
| --- | --- | --- | --- | --- | --- | --- |
| Details |  | top 3 training runs | top 3 training runs | best config, 9 pairs | default config, 9 pairs | best config, 9 pairs |
| Swahili avg | *24.8* | 24.7 | 29.5 | 29.3 | 32.4 | 32.8 |
| Swahili max | *24.8* | 25.6 | 32.8 | 32.0 | 36.4 | 37.2 |
| Telugu avg | *12.0* | 20.0 | 20.1 | 20.9 | 22.7 | 23.0 |
| Telugu max | *12.0* | 22.4 | 24.0 | 26.4 | 27.6 | 27.6 |
| Bengali avg | *29.2* | 33.5 | 38.3 | 36.8 | 37.1 | 38.7 |
| Bengali max | *29.2* | 35.2 | 44.4 | 38.4 | 40.4 | 45.2 |
| Japanese avg | *33.6* | 35.9 | 42.7 | 38.7 | 38.5 | 40.1 |
| Japanese max | *33.6* | 36.8 | 44.8 | 40.0 | 40.8 | 43.2 |

As displayed in Table 1, we find that *layer swapping* consistently outperforms these baselines on MGSM. For Swahili and Telugu, the default *layer swapping* configuration scores higher than both the individual experts across all 9 pairs. This provides tremendous evidence that this merging method prevents negative interference.
For Bengali, where the math expert already performs significantly better than for Swahili and Telugu, numerous layer-swapping configurations consistently surpass the math expert’s average score.
In addition, the best *layer swapped* model performs higher than the best math expert on its own.
For Japanese, we find here lower average performance compared to the individual math experts. However, we note that our Japanese experts were perhaps the weakest; the increase in performance across Belebele, FLoRes, MBPP, and MMLU after SFT were minimal and with the data and hyperparameters tested, we were unable to do better. Furthermore, MGSM performance of both the base Llama 3.1 and the math experts were already decent in Japanese, prior to merging. In comparison to model souping, *layer swapping* consistently outperforms it in terms of both maximum and average performance in all languages. Model soup results typically lie in between the individual results of the experts.

We further evaluated these layer-swapped models on Belebele, FLoRes, MBPP, and MMLU to ensure no over-optimization to this benchmark and find that the results are on par, usually even higher, than the base model and the individual experts. In addition, we manually checked a small number of text generations to further ensure the quality of the resulting model. In addition, we also evaluate combining *layer swapping* with model souping given that there are several language and math experts. We experiment souping with souping the three experts for each of the languages before *layer swapping* with a soup of the three math experts and present results in Table 5 in the Appendix. We find that these language and math soups perform on par with the individual pairs when *layer swapped* together. This feasibility in this settings shows that *layer swapping* can further extend the search space of fine-tuned models.

Given these results, we conjecture that perhaps *layer swapping* provides the biggest benefit for lower-resource languages (e.g. Swahili, Telugu), although such a conclusion would require evaluating more than four languages. This conjecture would be explained by the difficulty to improve language capabilities with low-data SFT when Llama 3.1 was already pretrained on more tokens in that language. Additionally, for higher-resource languages, cross-lingual transfer occurs more naturally when fine-tuning on English math data.

### 5.2 Further Evaluations on Swahili

Given that here,model souping appears to be susceptible to negative interference when merging, we additionally evaluate TIES-merging (Yadav et al., ) for Swahili. This merging method resolves interference between experts by sequentially trimming conflicting values, determining the sign of each weight, and then merging the resulting weights. In addition, we present an alternative for Swahili where instead of post hoc model merging, we mix the language and math datasets and do a single fine-tuning run with samples from both. Instead of training two experts, each on 30-40k samples, the joint SFT is over the union (80k samples). Identical to the training of experts, we launch many SFT runs with different hyperparameters and select the best three checkpoints.

**Table 2: MGSM 8-shot results of *layer swapping* for Swahili in more detail and with two additional comparisons, TIES-merging and dataset merging. We display the minimum performance in Swahili, as well as the average across all 9 languages in MGSM and in English.**
| Setting | Llama 3.1 8B | Swahili expert | math xxx expert | swh&math joint SFT | model soup | TIES-merging | *layer swap* | *layer swap* |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Details |  | top 3 training runs | top 3 training runs | top 3 training runs | best config, 9 pairs | best config, 9 pairs | default config, 9 pairs | best config, 9 pairs |
| Swahili min | *24.8* | 23.6 | 27.2 | 31.6 | 25.6 | 25.2 | 29.6 | 29.2 |
| Swahili avg | *24.8* | 24.7 | 29.5 | 32.1 | 29.3 | 29.5 | 32.4 | 32.8 |
| Swahili max | *24.8* | 25.6 | 32.8 | 32.8 | 32.0 | 32.4 | 36.4 | 37.2 |
| English avg | *56.0* | 55.7 | 66.2 | 64.3 | 62.0 | 60.1 | 64.7 | 64.4 |
| All langs avg | *37.7* | 37.5 | 45.4 | 46.0 | 43.0 | 41.8 | 44.1 | 44.4 |

Table 2 displays these more detailed results for Swahili. We find that TIES-merging was similarly performing as model souping in our setup, which means consistently worse than *layer swapping*. For the Swahili and math data mixing prior to fine-tuning, we are able to achieve comparable results to *layer swapping* on average, with the average MGSM score for this joint SFT being less than one point lower. However, the maximum performance of these checkpoints lagged the best *layer-swapped* pairs by 4.4 percentage points. This means that the ceiling for cross-lingual transfer is significantly higher with this methodology than simply mixing datasets together. This is significant because our method of merging two variants fine-tuned on separate datasets proves more effective than an extended fine-tuning on the combined datasets.

We note that by swapping in layers from the Swahili expert, MGSM performance in English and other non-Swahili languages decreases from the math expert. This is expected given we optimize for a different language, yet the decrease is relatively small nonetheless. For comparison, we see that performance drops much more for the other merging methods on these non-target languages.

## 6 Discussion

*Layer swapping* is a simple, yet effective method for merging together models for cross-lingual transfer. The success of this post hoc method prompts numerous key insights.

In settings where in-language task data is unavailable or rare, such as labeled math samples in Telugu, *layer swapping* provides a very effective way to create a capable model with simply English task data and general target language data. For lower-resource languages, such a constrained scenario is extremely common in practice—especially when it comes to instruction fine-tuning data. All other baselines we evaluated in such a scenario were not able to combine language and math capabilities as consistently and as effectively, as discussed in Section 5.

Model souping is popular because of its flexibility, convenience, and ability to expand model search space. In this regard, *layer swapping* provides a more effective alternative for the multilingual setting with the same advantages. This method can be implemented fully post hoc between any number of checkpoints using simple parameter arithmetic, with no further fine-tuning. Because of how inexpensive it is to implement, it enables the ability to quickly iterate through and test many configurations (e.g. the number of swapped transformer layers).
Similar to souping, the simplicity of this method means that it has the potential to work at any stage of training (pretraining, fine-tuning, preference tuning, etc.). In multi-task fine-tuning scenarios, this could allow for non-English capabilities being boosted on the side, separately from other *reasoning* capabilities—such as math, multi-hop reasoning, coding, safety, etc.—and then models are merged back together post hoc to combine the learned skills.

The effectiveness of this method indicates that there are meaningful multilingual patterns occurring in these LLMs that researchers can uncover and exploit. Potentially, we can reinterpret the functionality of the top and bottom transformer layers as, to a certain degree, natural language *interfaces* to broader model intelligence. Accordingly, this can potentially give way to a newer generation of modular solutions for multilingualism in decoder-only LLMs, similar to the MoE or adapter solutions that were designed for encoder models (Pfeiffer et al., ; ; Liu et al., ). Regardless, this work suggests that further interpretability research into LLM multilinguality could lead to more cost-effective solutions for boosting non-English capabilities.

In terms of model merging, our analysis indicates that weight-level techniques are perhaps not as effective for reducing interference when merging parameters together, as compared to parameter-level or even layer-level merges. A possible explanation is that modifying individual weight values may disrupt the linear dependence within the transformations defined by the weight tensors. Conversely, the recent success of self-speculative decoding in LLMs (Zhang et al., ; Elhoushi et al., ) suggests that the output embedding space of most layers across the model are very similar. If so, keeping entire layers in tact may alleviate the risk of undermining newfound learnings. Another potential reason is that using very granular magnitude measures may be too noisy to properly indicate which weight updates are important and which are not. By the nature of backpropagation, weight magnitude changes may be more indicative of the magnitude of the original value (larger weights will have larger gradients) or simply the weight’s proximity to the output layer. This is perhaps especially the case for small-scale training runs, where there is more noise.

## 7 Limitations and Future Work

The positive results of this *layer swapping* methodology largely raise many questions about the extent to which it is effective and in what settings. We propose here many further investigations that will build a broader understanding of the method’s practical scope.

*Freezing parameters before training experts*: Instead of re-composing a model with layers from separate models, a logical next evaluation would be to freeze model layers from the beginning. Intuitively, this would only be more effective from a performance standpoint as there is no ad hoc merging of experts from disparate training jobs. That being said, part of the benefit of this solution is that it is convenient and flexible because it’s all post hoc and required no prior planning. Parameter freezing would require knowing which configuration would work prior to training.

*LLMs with different pretraining*: Any transformer model should work similarly, but it is unclear if the English-centric nature of Llama 3.1 enabled the success of *layer swapping* (or conversely inhibited it). It is also possible that in weaker models (e.g. Llama 1), the non-English representations are not developed enough to enable this cheap transfer. Similarly, our analysis may not extend to LLMs with more balanced multilingual pretraining, who may have very different representation spaces.

*Different model sizes*: LLMs with more parameters, from either more transformer layers or larger hidden dimensions, would clearly require new merging configurations. In addition, it remains to be seen how related attributes such as sparsity, redundancy, and the quantity of cross-lingual shared parameters, would influence the performance of *layer swapping*.

*Lower-resource langs*: Among the limited languages evaluated, the lower-resource ones benefited most from *layer swapping*. However, the model needs minimum capabilities in the target language for zero-shot cross-lingual transfer. It is unclear how well the LLM needs to understand the target language for this to be effective.

*Other reasoning tasks*: This work only tackles mathematical reasoning, but it’s conceivable that other complex reasoning capabilities are concentrated in parameters separate from language capabilities in a similar manner. That being said, the configuration required for *layer swapping* may be completely different and may require a preliminary analysis as in Section 3.

*Parameter-efficient fine-tuning*: Since *layer swapping* treats transformer layers as unified blocks, it would be equivalent if the model were fine-tuned using parameter-efficient SFT methods such as LoRA, which consists of inserting adapter weights into the transformer layer (Hu et al., ). The modification of other model merging methods for such adapters is simple (Zhang et al., ), and therefore *layer swapping* has the potential to be effective in parameter-efficient fine-tuning settings.

*Different training stage*: We limit the focus of this work on low-data SFT. Model souping, itself, is implemented in practice at all stages (e.g. pretraining, CPT). This raises the question of in which settings *layer swapping* would be effective for model merging. Similarly, model souping is effective even when the checkpoints have been fine-tuned significantly. It is unclear what would be the point for our methodology where the experts have diverged from each other too significantly that they would no longer recombine well. If it is only effective in small training runs, it could still enable a multi-step training where experts are iteratively fine-tuned and merged successively, analogous to gradient aggregation in data-distributed training.

## 8 Conclusion

In this paper, we address the difficulty of training LLMs to perform tasks in languages where labeled task-specific data does not exist. We first trained separate experts on English math data and generic instruction data in numerous target languages. An analysis of the importance of different parameter updates during fine-tuning led to the development of the *layer swapping* method which swaps in the top and bottom layers from language experts into the math expert. Surprisingly, this simple model merging provides one of the highest performing methods to fine-tune an LLM for math in a target language without the presence of target language data. The strong intuition behind this solution and its flexibility, simplicity, and effectiveness provides vast potential for being practical in many other settings. This method can be potentially adapted for new pretrained models, target tasks, target languages, training stages, training setups, and more. In addition, this method indicates better model interpretability of multilinguality can lead to more efficient methods for transferring English capabilities to lower resource languages.

#### Reproducibility

For reproducing the fine-tuning of our “expert” models, we describe the process in Section 3.1 and provide further details on the data and fine-tuning hyperparameters in Appendix A.1 and A.2. Further details for our analysis described in Section 3.2 can be found in Appendix A.4. We provide pseudocode of the *layer swapping* algorithm defined in Section 4 in Algorithm 1.
Our experiments for which results are presented are all explained thoroughly in Section 5.

#### Acknowledgments

The authors acknowledge the crucial support provided by Mostafa Elhoushi, Tanmay Parekh, Jiabao Ji, and Chloe Bi that made this project possible.

## Appendix A Appendix

### A.1 Fine-tuning Datasets

**Table 3: Datasets used for supervised-fine-tuning (SFT) in this project**
| Category | Datasets | URL |
| --- | --- | --- |
| Math | Orca Math word problems dataset from Microsoft (Mitra et al., ) | [https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k](https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k) |
| Telugu | Aya Dataset from Cohere for AI (Singh et al., ) | [https://huggingface.co/datasets/CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset) |
| NLLB English-Telugu translation data from FAIR (NLLB et al., ) | [https://huggingface.co/datasets/allenai/nllb](https://huggingface.co/datasets/allenai/nllb) |  |
| English instruction dataset, machine translated to Telugu |  |  |
| Bengali | Aya Dataset by Cohere for AI (Singh et al., ) | [https://huggingface.co/datasets/CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset) |
| English-Bengali translation data from NLLB (NLLB et al., ) | [https://huggingface.co/datasets/allenai/nllb](https://huggingface.co/datasets/allenai/nllb) |  |
| IndicShareLlama dataset from AI4Bharat (Khan et al., ) | [https://huggingface.co/datasets/ai4bharat/indic-align](https://huggingface.co/datasets/ai4bharat/indic-align) |  |
| BongChat dataset from Lumatic AI | [https://huggingface.co/datasets/lumatic-ai/BongChat-v1-253k](https://huggingface.co/datasets/lumatic-ai/BongChat-v1-253k) |  |
| Swahili | Aya Dataset by Cohere for AI (Singh et al., ) | [https://huggingface.co/datasets/CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset) |
| English-Swahili translation data from NLLB (NLLB et al., ) | [https://huggingface.co/datasets/allenai/nllb](https://huggingface.co/datasets/allenai/nllb) |  |
| Inkuba dataset from Lelapa (Tonja et al., ) | [https://huggingface.co/datasets/lelapa/Inkuba-instruct](https://huggingface.co/datasets/lelapa/Inkuba-instruct) |  |
| xP3 MT dataset from BigScience, with FLoRes samples removed (Muennighoff et al., ) | [https://huggingface.co/datasets/bigscience/xP3mt](https://huggingface.co/datasets/bigscience/xP3mt) |  |
| Japanese | Aya Dataset by Cohere for AI (Singh et al., ) | [https://huggingface.co/datasets/CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset) |
| English-Japanese translation data from NLLB (NLLB et al., ) | [https://huggingface.co/datasets/allenai/nllb](https://huggingface.co/datasets/allenai/nllb) |  |
| LLM-Japanese dataset from Izumi Lab (Hirano et al., ) | [https://huggingface.co/datasets/izumi-lab/llm-japanese-dataset](https://huggingface.co/datasets/izumi-lab/llm-japanese-dataset) |  |
| Ichikara dataset from RIKEN AIP | [https://huggingface.co/datasets/p1atdev/ichikara-instruction](https://huggingface.co/datasets/p1atdev/ichikara-instruction) |  |
| Dolly dataset from Databricks, machine translated to Japanese | [https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja) |  |

### A.2 Supervised Fine-tuning Details

While many hyperparameters were tried, below is listed the hyperparameter configurations that led to the three best checkpoints (the “experts”) for each category. Note that in all runs, we do checkpointing every 5000 samples and use a different random seed for data sampling.

**Table 4: Hyperparameters for the training runs that led to each of our “experts”**
| Expert | Learn Rate | Batch Size | Seq. Length | weight decay | clip, max norm | sched. | warmup | $\beta 2$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| math #1 | $2.0\times 10^{-8}$ | 4 | 2048 | 0.1 | 1.0 | Linear | 1000 | 0.99 |
| math #2 | $1.0\times 10^{-7}$ | 4 | 2048 | 0.01 | 0.5 | Linear | 1000 | 0.99 |
| math #3 | $4.0\times 10^{-8}$ | 4 | 2048 | 0.1 | 1.0 | Linear | 500 | 0.999 |
| jpn #1 | $1.0\times 10^{-7}$ | 8 | 1024 | 0.1 | 0.5 | WSD | 1000 | 0.995 |
| jpn #2 | $1.0\times 10^{-7}$ | 8 | 1024 | 0.1 | 0.5 | WSD | 1000 | 0.99 |
| jpn #3 | $2.0\times 10^{-7}$ | 8 | 1024 | 0.01 | 0.5 | WSD | 1000 | 0.99 |
| swh #1 | $7.0\times 10^{-8}$ | 8 | 1024 | 0.1 | 0.5 | WSD | 1000 | 0.99 |
| swh #2 | $2.0\times 10^{-7}$ | 8 | 1024 | 0.05 | 0.5 | WSD | 1000 | 0.99 |
| swh #3 | $1.0\times 10^{-7}$ | 8 | 1024 | 0.1 | 1.0 | WSD | 1000 | 0.999 |
| tel #1 | $7.0\times 10^{-8}$ | 8 | 1024 | 0.1 | 0.5 | WSD | 1000 | 0.99 |
| tel #2 | $5.0\times 10^{-8}$ | 8 | 1024 | 0.05 | 0.5 | WSD | 300 | 0.99 |
| tel #3 | $1.0\times 10^{-7}$ | 4 | 2048 | 0.1 | 0.5 | WSD | 1000 | 0.99 |
| ben #1 | $7.0\times 10^{-8}$ | 8 | 1024 | 0.1 | 0.5 | WSD | 1000 | 0.99 |
| ben #2 | $5.0\times 10^{-8}$ | 8 | 1024 | 0.05 | 0.5 | WSD | 300 | 0.99 |
| ben #3 | $1.0\times 10^{-7}$ | 4 | 2048 | 0.1 | 0.5 | WSD | 1000 | 0.99 |

### A.3 Results from combining with souping

**Table 5: MGSM 8-shot results when combining *layer swapping* with model souping of the 3 experts for each category. ”Multilingual soup” refers to the uniform soup of all 12 language experts.**
| Setting | model soup,xxxxx pairwise | *layer swap*,xxxxxx pairwise | *layer swap*,xxxxxx language soup w/ math soup | *layer swap*,xxxxxx multilingual soup w/ math soup |
| --- | --- | --- | --- | --- |
| Details | default config,xxx avg of 9 pairs | default config,xxx avg of 9 pairs | default config,xxx 1 version | default config,xxx 1 version |
| Swahili | 29.3 | 32.4 | 32.8 | 29.6 |
| Telugu | 20.9 | 22.7 | 22.0 | 21.2 |
| Bengali | 36.8 | 37.1 | 40.4 | 36.0 |
| Japanese | 38.7 | 38.5 | 39.2 | 41.6 |

### A.4 Parameter-Level Visualizations of the Expert Vectors

Larger parameter-level visualizations for a number of the experts, each visualization is the same as described in Fig. 2

Figure: Figure 4: Visualization of the magnitude of change during SFT for “math expert #1”
Refer to caption: extracted/5894914/images/math_expert_vis.png

Figure: Figure 5: Visualization of the magnitude of change during SFT for “math expert #2”
Refer to caption: extracted/5894914/images/math_expert2_vis.png

Figure: Figure 6: Visualization of the magnitude of change during SFT for “Swahili expert #1”
Refer to caption: extracted/5894914/images/swh_expert_vis.png

Figure: Figure 7: Visualization of the magnitude of change during SFT for “Japanese expert #1”
Refer to caption: extracted/5894914/images/jpn_expert_vis.png

Figure: Figure 8: Visualization of the magnitude of change during SFT for “Telugu expert #1”
Refer to caption: extracted/5894914/images/tel_expert_vis.png

Figure: Figure 9: Visualization of the magnitude of change during SFT for “Bengali expert #1”
Refer to caption: extracted/5894914/images/ben_expert_vis.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. You write that combining layers works because models are fine-tuned for just a little bit: why not tune for longer?
2. What would the intuitive explanation be behind the findings of your preliminary analysis on which layers are changed by task-tuning and language-tuning, other examples of similar effects in related literature?
3. If you were to tune for longer, would that affect the results of the preliminary analysis and change more layers, or different layers?
soundness: 4
strengths: * Interesting idea and its evaluation on 1 model and 4 languages, with additional experiments
* Although the setup raises some questions (limited evaluation, why not freeze layers and avoid having to soup the transition layers, etc.), the expanded evaluation on Swahili and the limitations section address most of these
* excellent writing, justification and presentation
summary: This paper proposes a new approach to perform zero-shot cross-lingual transfer for solving tasks in a new language. The requirement is to have data for the task in a more resourced language (e.g. English) and non-task-specific data for other languages (4 languages in this paper). The idea is to fine-tune the model (Llama 3.1, 8B in the paper) separately to one new language and another copy to the task in English, then compose a new model of layers of these two models. A study is included of how much the models change during fine-tuning, which concludes that math tasks cause changes in the fine-tuned model closer to the middle layers, while tuning on a new language causes changes in the first and last layers. Based on that the composed model takes task-layers from the math-tuned model's middle, and language-layers from the language-tuned model's start and end. Transfer between the layers of two independently tuned models is done by "souping" intermediate layers, i.e. averaging the weights of layers of both models. Gains up to 10% are shown, in comparison to the expert models.
weaknesses: * limited evaluation: only 1 model and one set of tasks (math)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does a re-composed model in one language affect model performance in typologically similar languages? In my opinion, an analysis of this kind would highly benefit the work. 

2. Would a 2-stage SFT work better than Joint SFT on language and task?

---

**[POST-REBUTTAL UPDATE]**

I have decided to increase the overall rating from **5** (marginally below the acceptance threshold) to **6** (marginally above the acceptance threshold).
soundness: 4
strengths: 1. The proposed methodology is highly practical in scenarios where one might have publicly available task-specific data in a high-resource language and generic instruction data in the low-resource language. The model parameter adjustments being fully post-hoc eliminate any additional computational overhead apart from the initial fine-tuning required to create task and language experts.

2. *layer swapping* with the best configuration consistently outperforms the individual SFT experts, the base LLM Llama 3.1 and the general model souping approach in three (Swahili, Telugu, Bengali) out of the four languages under this study.  

3. The paper is easy to follow. The authors also take the effort to acknowledge the possible limitations to the work, encouraging future exploration.
summary: The paper introduces a variant of model merging addressing the challenge of adapting LLMs for tasks in low-resource languages. The methodology involves *layer swapping* (parameters being swapped) between a task expert and a language expert, both following the same underlying architecture. The resulting re-composed LLM is said to outperform both the individual experts without the requirement of a task-specific fine-tuning in the low-resource language. The experiments involve evaluation of the proposed methodology on MGSM (a multilingual benchmark of grade-school math problems) on 4 resource-scarce languages: Japanese, Telugu, Swahili and Bengali.
weaknesses: 1. The methodology is evaluated only on Llama 3.1, using MGSM benchmark for 4 selective languages. In my opinion, evaluation of the method on a single model, single benchmark and limited languages makes the conclusion less generalizable. While the languages used in the study are diverse, incorporating more datasets and models (in terms of different architecture or pre-training) can strengthen the conclusion.

2. The assumption of availability of generic instruction data for low-resource languages might not hold for all languages. Task-specific data and generic instruction data in a high-resource language is generally more accessible. An experiment where language expert is fine-tuned using translated instructions would increase the practicality of the work.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can the authors clarify if they have tried layer swapping on other tasks, such as translation, question-answering, or code generation? Evidence of generalizability beyond math would make the approach significantly stronger.
2. Layer swapping is positioned as an alternative to methods like model souping, but a comparison to more recent modular fine-tuning techniques, such as adapters (Pfeiffer et al., 2020) or LoRA (Hu et al., 2022), could help contextualize its relative strengths. Can the authors either conduct these comparisons (e.g. computational efficiency, performance) or discuss the anticipated performance differences? 
3. Given that this study uses a 8B (32-layer) LLM, how would the authors anticipate the method scaling to models with more layers or parameters? Could they provide guidance on applying layer swapping to larger models, especially in terms of choosing the number of layers to swap? Would the authors still anticipate the same performance gain with layer swapping on larger models? 
4. For Japanese, layer swapping results in lower average performance compared to the individual math experts. The authors mentioned that the Japanese experts were the weakest as performance across BELEBELE, FLORES, MBPP, and MMLU were minimal. Could the authors share the results on these benchmark (before and after SFT) to help better understand the case?
soundness: 3
strengths: - The paper introduces an efficient, innovative layer-swapping method for zero-shot cross-lingual transfer in LLMs, addressing the lack of task-specific data in low-resource languages with simplicity and strong empirical results.

- This technique is particularly notable for its straightforward implementation, allowing effective merging of task and language expertise without complex adjustments, making it a practical alternative to standard methods like model souping. 

- Promising experimental gains on math reasoning benchmarks across multiple low resource languages validate the method's effectiveness, showing that layer swapping successfully enhances cross-lingual transfer without in-language task data.
summary: The paper introduces a novel layer-swapping methodology for zero-shot cross-lingual transfer in large language models (LLMs) aimed at improving task performance in non-English languages, particularly for mathematical reasoning. The approach tackles the lack of task-specific data in low-resource languages by fine-tuning two separate "experts" of the same base model: one trained on English mathematical data and another trained on general instruction data in the target language. The proposed method then selectively replaces the top and bottom transformer layers of the math expert with those from the language expert, buffered by transition zones between these regions. This configuration shows promising performance gains on the MGSM math benchmark across four languages—Swahili, Telugu, Bengali, and Japanese—without any additional in-language math data.
weaknesses: - The method is tested only on math reasoning, leaving it unclear if layer swapping generalizes to other tasks. Additional evaluations on tasks like question-answering or translation would strengthen the claims of broad applicability.
- While the paper mentions different layer-swapping configurations, it lacks in-depth analysis on which configurations work best and why. A more detailed study of these choices would help to better understand the method make it more robust. For example, provide ablation studies on the number of swapped layers or transition zone sizes, or to analyze how performance changes as these parameters are varied.
- Comparisons to recent modular fine-tuning techniques, such as adapters or LoRA, are missing. Including these would clarify how layer swapping performs relative to other efficient, cross-lingual methods.

Meta-review: The paper introduces a model merging methodology tailored to address the challenge of adapting large language models (LLMs) to perform tasks in non-English languages without task-specific data. The method involves independently fine-tuning "experts" on math tasks in English and generic instruction data in target languages, followed by swapping top and bottom layers from the language expert into the math expert. This approach achieves up to a 10% performance gain on the MGSM benchmark for mathematical reasoning in resource-scarce languages such as Swahili, Bengali, Telugu, and Japanese. Reviewers largely commended the novelty and practicality of this approach, emphasizing its simplicity, interpretability, and effectiveness in enabling post-hoc modularity in multilingual tasks. Strengths include the innovative model recomposition mechanism, compelling empirical results, and clear presentation. Weaknesses noted by reviewers include the limited evaluation scope—restricted to one benchmark, four languages, and one base model (LLaMA 3.1)—as well as reliance on instruction-tuning data for target languages and a lack of detailed ablation studies or comparisons with alternative modular methods like LoRA. The acceptance decision is grounded in the paper's methodological originality, empirical contributions, and potential impact in advancing cross-lingual capabilities for low-resource languages, particularly in settings with limited task-specific data availability.

Decision: Accept (Spotlight)