How does the optimizer implicitly bias the model merging loss landscape?
Authors: Chenxiang Zhang
Alexander Theus
Damien Teney
Antonio Orvieto
Jun Pang
Sjouke Mauw
Venue: ICLR 2026 Poster (2026)
Keywords: loss landscape
mode connectivity
model merging
optimization
implicit bias
Primary area: optimization

Abstract: Model merging combines independent solutions with different capabilities into a single one while maintaining the same inference cost. 
Two popular approaches are _linear interpolation_, which simply averages multiple model weights, and _task arithmetic_, which combines task vectors obtained by the difference between finetuned and base models. 
While useful in practice, what properties make merging effective are poorly understood. This paper explores how the optimization dynamics affect the loss landscape geometry and its impact on merging success.
We show that a single quantity -- the _effective noise scale_ -- unifies the impact of different optimizer components on model merging. Across architectures and datasets, merging success is a non-monotonic function of the effective noise scale, with a distinct optimum. Decomposing this quantity, we find that larger learning rates, stronger weight decay, smaller batch sizes, and data augmentation all independently modulate the effective noise scale and exhibit the same qualitative trend.
Unlike prior work connecting optimizer noise to the flatness or generalization of _individual_ minima, we show that it also affects the _global_ loss landscape, predicting when independently trained solutions can be successfully merged. Our findings broaden the understanding of how optimization shapes the loss landscape geometry and its consequences for model merging, suggesting that training dynamics could be further manipulated to improve model merging.

Published as a conference paper at ICLR 2026
HOW DOES THE OPTIMIZER IMPLICITLY BIAS THE MODEL MERGING LOSS LANDSCAPE?
Chenxiang Zhang", Alexander Theus Damien Teneys Antonio Orvieto', ", Jun Pang' & Sjouke Mauw!
ABSTRACT
Model merging combines independent solutions with different capabilities into a single one while maintaining the same inference cost. Two popular approaches are linear interpolation, which simply averages multiple model weights, and task arithmetic, which combines task vectors obtained by the difference between fine- tuned and base models. While useful in practice, what properties make merging effective poorly understood. This paper explores how the optimization dy- namics affect the loss landscape geometry and its impact on merging success. We show that single quantity the effective noise scale unifies the impact of differ- ent optimizer components on model merging. Across architectures and datasets, merging success is a non mon-oontooiic function of the effective noise scale, with a distinct optimum. Decomposing this quantity, we find that larger learning rates, stronger weight decay, smaller batch sizes, and data augmentation all indepen- dently modulate the effective noise scale and exhibit the same qualitative trend. Unlike prior work connecting optimizer noise to the flatness or generalization of individual minima, we show that it also affects the global loss landscape, pre- dicting when independently trained solutions can be successfully merged. Our findings broaden the understanding of how optimization shapes the loss landscape geometry and its consequences for model merging, suggesting that training dy- namics could be further manipulated improve model merging.
INTRODUCTION
Model merging combines independently trained solutions to fuse their unique capabilities into sin- gle solution. These methods have been applied to either further improve the performance of a single model (Wortsman al., 2022) to combine models with different, but similar, capabilities (Ilharco et al., 2023). Notably, this is achieved without additional computational cost at inference time. Given these practical advantages, merging methods have been applied state-of-the-eee -h- architectures (Ya- dav et al., 2024; Cohere et al.. 2025). However, these approaches rely on extensive rial-and-erron requiring practitioners to train and evaluate multiple models to identify which candidates success- fully merge. A fundamental unresolved question is why some models are compatible for merging while others fail, despite achieving similar individual performance.
The success of model merging depends the mode connectivity phenomenon, the geometric prop- erty that independent solutions can be connected by paths of low-loss models (Draxler et al., 2018; Garipov et al., 2018). Frankle et al. (2020) demonstrated a stricter condition, linear mode connectiv- ity, showing that solutions sharing a common initial optimization trajectory can be even connected by linear path with similar loss. This finding suggests that optimization dynamics, not just final convergence, fundamentally shape the geometry of the loss landscape between solutions. Yet, de- spite this insight, well-studied factors that affect the optimization dynamics, such as learning rate, weight decay, or batch size, are not well understood for the model merging loss landscape.
In this work, we study the role of optimization dynamics on the outcome of model merging. First, we show that different optimizer components (learning rate, weight decay, batch size, and data aug- mentation) control the same underlying factor, the effective noise scale, which captures this magni-
Equual contributions (see Section 7). Correspondence: <chenxiang zhang@unt iaaangeuni...e lu>.
University of I Laxembourg 2ETH Zärich, MPI for Intelligent Systems *Idap Research Institute *ELLIS Insti- tute Tübingen, MPI for Intelligent Systems, Tübingen AI Center

---

Published as a conference paper at ICLR 2026
tude of stochastic fluctuations in the optimization trajectory. Our experiments demonstrate how the noise magnitude determines whether independently trained solutions occupy merging-compatibn regions of the loss landscape. A noise regime that is too small or too large leads to almost no merging benefits, whereas a moderately large noise regime maximizes the benefits. We decompose the noise into individual components, showing that each one exhibits the same qualitative trend. Starting from the learning rate, we find that a larger and stable value can consistently identify so- lutions that merge more effectively than those trained with smaller learning rates, even when both converge to similar generalization performance on the test data. Since decaying any learning rate during the optimization can always lead to a converged solution, it is perhaps surprising that simply starting with larger learning rate can change the merging outcomes. However, beyond classical research showing direct advantage of large learning rates on generalization (Keskar et al., 201 6), recent works presented different implicit biases of larger learning rate, such as sparser activa- tion (Andriushchenko et al., 2023b), a different sequence of pattern learning (Li et al., 2019), and a flatter solution (Andriushchenko et al., 2023a). Our results extend these benefits beyond single-task performance, showing that a larger learning rate unlocks more effective merging.
Similarly, we find that a large weight decay also enables more effective merging, beyond improve- ments in the single model performance. This can be explained with the notion of the effective learning rate (Van Laarhoven, 2017; Hoffer et al., 2018), which suggests that the main role of weight decay is to prevent the (effective) learning rate, and therefore the stochastic noise, from de- caying to zero during training. Additional components, such as batch size, momentum, and data augmentation, also inject noise into the optimization dynamics. A smaller batch size creates noisier gradient estimates since each update is computed from fewer samples, leading to more variation in the optimization path (Keskar et al., 2016; Jastrzebski et al., 2017). Higher momentum smooths gradient updates and reduces oscillations, biasing the optimizer toward broader, flatter regions of the loss landscape (Sutskever et al., 2013). Data augmentation introduces extra randomness into each minibatch (Hanin & Sun, 2021). We validate these findings across different architectures, tasks, permutation symmetries, and in transfer learning setups.
Lastly, we extend the analysis to task arithmetic merging, which defines a different subspace of solutions than simple linear interpolation. We find that the geometry of the loss landscape in task arithmetic significantly changes depending on the initialization. Given a pretrained initialization (e.g. CLIP), larger learning rate identifies solutions with easier merging compatibility (Figure 7). Moreover, the landscape is also flatter compared to a smaller learning rate. Finally, we find that merging solutions trained different tasks with a moderately larger noise has the best performance but at the cost of losing compatibility with models trained with different noise (Figure 8).
2 PRELIMINARIES
Linear interpolation merging (Frankle et al., 2020). Linear mode connectivity refers to a phe- nomenon where two minima with similar performance can be connected by linear path in the parameter space without significant performance degradation along that path. Formally, given two models with parameters 0 A and OB, we can define a linear interpolated model O₁, as:
O (1 a)0 a0g
where a E [0, 1] is the interpolation coefficient. A pair of models exhibits linear mode connectivity if the loss function C(O.) remains relatively low for all values of a along this linear path. Model merging relies on mode connectivity, but instead, it aims to find solutions with lower loss values.
Task arithmetic merging (Ilharco et al., 2023). Given a base model Orose. a finetuned model O, on task t, the task vector is defined as T, = 0, Obase. The task vector 7. captures the parameter change induced by fine-tuning on task t. Interestingly, task arithmetic enables operations such as addition and scaling of different task vectors, creating a merged model ot. as:
Pie Abase Σ
where a, E R is the coefficient that controls the influence of each task vector. For simplicity, is usually the same for all the vectors. In order to succeed, both linear mode connectivity and task arithmetic assume that the loss landscape around the finetuned models 0, is near-convex.
2

---

Published as a conference paper at ICLR 2026
Effective noise scale. Stochastic optimization introduces randomness through minibatch sampling. Writing the minibatch gradient as 9t = V2(),) ) + & with E[ 0 and Cov[st = (x(0))/ B, the SGD update can be viewed with the learning rate and inversely with the batch size (Mandt et al., 2017). In the standard mo- mentum parameterization, this diffusion is further rescaled by (1 - 1u) (Smith et al., 2018). The remaining magnitude is task and data-dependent through the gradient-noise covariance tr 2 (Mc- Candlish et al., 2018). We summarize all these effects by the effective noise scale:
discretized SDE whose diffusion strength scales proportionally
San (1. B, (2; & B(1 -
1]
tr (O.,,
$
7J
(3)
B(1
Note that ΣA depends on data augmentation and diversity A. When is fixed across runs, we can treat tr (O;) approximately constant, so S provides practical proxy for comparing noise levels across runs.
THE OPTIMIZER'S S IMPLICIT BIAS ON LINEAR INTERPOLATION
This section presents our key findings for linear interpolation merging. We begin by presenting the effective noise scale as the unifying implicit bias controlling model merging. Then, we demonstrate that each optimizer component affects merging effectiveness through this noise.
3.1
EFFECTIVE NOISE SCALE AS UNIPYING FACTOR
1.0
1.0
Accurasy gain (%)
0.5
Accurasy gain
0.5
Accuracy gain
0.5
hs16
0.0
bs32
0r2c:01
Ir2e-01
Croccal paitt
bs64
Irle:01
0.5
bs128
Ir5e-02
Ir5a-02
bs256
Irle-02
1.0
le-2
Se-2
1e:1
2e-1
2
2
25
2
Learning rate
Batch
Effective noise
(a) Learning rate (1)
(b) Batch size (B)
(c) Effective noise (S)
Figure 1: Effective noise scale controls the effectiveness of merging. The y-axis reports the test accuracy gain of merged models over single models. In (a) and (b), plotting batch size against learning rate (or vice versa) reveals no clear trend. When reparameterized in terms of (c) effective noise scale, the joint interaction between different components emerges.
As introduced in Section 2, the effective noise scale S captures the joint interaction of learning rate, batch size, momentum, and augmentation of the optimization dynamics. Rather than treating these hyperparameters as independent, we present how S offers unifying view on the compatibility of single models under linear interpolation merging. We train ResNet18 on CIFARI00 using SGD, sweeping jointly the learning rate 7) and batch size B. The weight decay À 5e random flip and crop augmentations are used. To evaluate the merging effectiveness, we define performance gain as g(mmerge @gingle)
fixed, and
(mmerge) f(Ossngle) given an evaluation function f: x D + R. Appendix A. describes the training and merging setup.
Figure illustrates how S affects the merging effectiveness. When the learning rate or batch size are varied independently, there is no clear trend across both dimensions. For example, when increasing the learning rate for a fixed batch size whereas the opposite holds for larger batch size of B 128 or B 256, which is commonly preferred in practice. However, once we reparametrize the --axis in terms of S, capturing both learning rate 1 and batch size B together, the different curves become aligned. Importantly, this curve is non-monotonic: merging effectiveness improves as S increases from small values, reaches "critical point", and then degrades again once the noise grows too large.
16, the accuracy gains monotonically decrease,
In contrast to prior work that mainly links effective noise to properties of single solutions (Chaudhari et al., 2016; Mandt et al., 2017; Jastrzebski et al., 2017), our results show that it also governs its
3

---

Published as a conference paper at ICLR 2026
surrounding solutions. Specifically, S not only biases the optimization dynamics toward particular regions of the loss landscape, but also controls the compatibility of solutions found in different runs under linear interpolation merging. In the following sections, control each optimizer component in isolation to analyze how it contributes to the merging effectiveness.
3.2
LARGE LEARNING RATE IDENTIFIES MORE COMPATIBLE SOLUTIONS
0.6
1.5
1.5
Acturary gain (8a
Acturacy gen
0.4-
Actursty gain
1.0
0.5
0.0
0.0
03
5e
12-02
3e 02
1e:03 1e:02 52 11
2e-01
1e:03
1c-02
02
Se
Learning rate
Learning rate
Learning rate
2.0
02
0.5
Irle-02
#1e-01
29
Ir3022
Accurecy gain (S)
Irse-03
Accuncy gain Pl
ir5e-02
Accurcc gain (N)
1.5
Irle-01
irte 11e-03
1.0
A6
0.0
0.0
B5
70
75
55
60
Accuracy
Accuracy
(a) SVHN MIP
(b) CIFAR100 ResNet18
(c) TinyImageNet/ DenseNet122
Figure 2: Larger learning rate leads more effective merging. (top) The test accuracy gain of all the models. (bottom) Each point represents a single model performance 0^ on the æ-axis and its accuracy gain after merging the y-axis. The opacity indicates the number of training epochs. For each setup, we observe that larger learning rates have higher accuracy gain, even when there is smaller learning rate with equivalent accuracy on the æ-axis. Note, however, solutions with a "too large" learning rate fail to merge (Appendix B.6).
We present empirical results for vision tasks trained from scratch using a simple MLP, ResNet- 18, and DenseNet121 on SVHN, CIFAR-10, CIFAR-100, and TinylmageNet with SGD. We fix = 5e 4, B = 128, j = 0.9, and random flip and crop augmentation are used across the setups. We ensure that all the solutions well-converged by verifying that the training loss is near-zero. Appendix A. describes the training and merging setup.
Under linear interpolation with fixed learning rates 7). Figure 2 shows that the solutions identified with a larger 1) are consistently more compatible to merge than those found with smaller values. For example, in CIFAR100, the solu- tions found using an = 2e 1 achieve +1.2% of the median gain compared to +0.2% gain of 1 = le despite having similar single model performance of Rs 75% (æ-axis). This demon- strates that larger noise 77 induces more effective loss landscape geometry for model merging. The same phenomenon is observed across different datasets (SVHN, CIFAR, and TinylmageNet) and architectures (MLP, ResNet, and DenseNet).
0.5, merge solutions trained with the same
Pascanu et al. (2025) argue that understanding how the implicit bias of an optimizer shapes the final solution, and how to leverage this bias to find better models, is an important problem. We address this directly by showing how 1j biases the model merging process. Additionally, recent works found different implicit biases of training with large 7), such as sparser activation (Andriushchenko et al., 2023b; Sadrtdinov et al., 2024), a different feature- learning order (Li et al., 2019), or a flatter solution (Andriushchenko et al., 2023a). Our results demonstrate an additional benefit: larger learn- ing rate has an implicit bias on the loss landscape, identifying more compatible solutions for model merging up a certain point (see Appendix Figure 15).
4

---

Published as a conference paper at ICLR 2026
3.3
WEIGHT DECAY AND EFFECTIVE LEARNING RATE
1.5
Iccunacy gain (S5)
Accuntty griuucyggiiig (%)
Accuracy gain (N)
0.31
gain
1.0
0.21
0.5
0.5
0.0
0.0
1e.5
1e.4
Se-4
1e:3
1e.5
2e-4
Se-4
16:3
1e.4
Seid
Weight
0.5
wdle 3
wilee 3
wcbe- 4
1.5
1.5
Picuracy gein (h6
Accurary gain (K0)
Accurscy gain (S0)
WeBe-
adle-5
welleS
web
wd0
1.0
wdb
1.0
0.3-
0.5
0.2-
0.0
84
Accuracy single 196)
Accuracy single (16)
Accuracy single (S)
(a) SVHN MLP
(b) CIFAR100/ ResNet18
(c) TinyTmageeee///// SaaiiimaggeeeeDDDDNetaaaaaee 2e1seNet121
Figure 3: Weight decay À has a similar effect as the learning rate 1j for model merging. For CI- FARIO 100 and TinylmageNet,
scale-invariant networks (w/ normalization layers) and observe that a larger weight decay can not only improve the accuracy of the single model, but also the accu- racy gain via the effective learning rate (Van Laarhoven, 2017). For the MLP architecture, there is no trend as it is not scale-invariant.
The traditional understanding of the role of weight decay regularization is that it reduces overfitting by proportionally decaying the weights towards zero, favouring simpler models. practice, this achieved by adding penalty term A| 0 to the objective L(0). However, modern neural network architectures ubiquitously use normalization layers (loffe & Szegedy, 2015; Ba et al., 2016) and are therefore weight scale-invariant as f(x, ox)) f(x, 0). Then, what is the new role of weight decay regularization in scale-invariant networks? Van Laarhoven (2017); Hoffer et al. (2018) prove and demonstrate that weight decay controls the effective learning rate, and thus the optimization dynam- ics during training. In practice, for a scale-invarian network with = 0, its gradient norm decays proportionally to the increase of weight norm || /V112 x 1/1]0||2. which leads to the vanishing of the effective learning rate (Van Laarhoven, 2017).
We hypothesize that weight decay À has similar effect to the learning rate r for model merging. That is, solutions identified with a larger weight decay are easier to merge than those found using smaller or weight decay. Note that this holds only for scale-invariant networks, since the | of a non scale-invariant network is already constrained by the loss without the weight decay penalty (Van Laarhoven, 2017). We use the same experimental setup as in Section 3.2, except that we now sweep across different weight decay values instead of learning rates.
Figure confirms our hypothesis about the implicit bias of weight decay. Larger increases the effective learning rate for scalc-invariant networks during training. affecting the model merging effectiveness. For example, TinylmageNet, the solutions found using a 5e 4 report +1.2% of the median gain compared to a +0.5% of other À values. The same trend is observed in multiple settings. Furthermore, for non scale-invariant architecture MLP trained on SVHN, different values of À regularization have similar impact the merging effectiveness, confirming that weight decay affects primarily when merging ocale-invarian architectures. Our results extend how weight decay affects the loss landscape of an individual minimum (Van Laarhoven, 2017: D' Angelo et al., 2024) to its connection with other minima. Similar to the learning rate, Appendix Figure 16 shows that excessive weight decay leads to failure in model performance and model merging.
5

---

Published as a conference paper at ICLR 2026
Mocuracy gein (S5
Accuracy gain (59)
Accrrscy gain (N)
0.5
0.5
-1.0
0.0
-2.0
256
128
0.0
0.85
1e-07
1e-1
Batch size
Momentum
Learning withiwithour aug.
momo 95
momo
bs64
Accurcy gain 194
momoo 55
ActuRcyy cain (16)
Accuracy gain
1.0
bs128
momo.6
0.0
bs256
1.0
0.5
0.5
Inle-03 aug.
Inte-00
3.0
Inte-02 wlo aug.
4.0
68
74
76
74
76
65
Accuracy single 176)
Acouracyy single (361
Accuracy single (56)
(a) Batch size (B)
(b) Momentum (μ)
(c) Augmentation (Ex)
Figure 4: Batch size, momentum, and data augmentation control the noise during the optimization dynamics. (a, b) A small B or larger μ has more noise and improves the merging effectiveness. (c) Data augmentation improves performance and merging effectiveness.
3.4 BATCH SIZE, MOMENTUM, AND DATA AUGMENTATION
We consider three additional components affecting the effective noise scale: batch size B, momen- tum μ,, and data augmentation ΣA We train ResNet18 on CIFARI00 with SGD. For batch size, we use the same setup as in Section 3.2, except that we sweep across different B and fix the train- ing steps to 200k steps instead of epochs. Similarly, we sweep across different momentum values. For augmentation, we simply turn off the augmentation during the training. Additional results on different datasets and scheduler choices are in Appendices B. B.2 and E.
Batch size. Stochastic gradient of minibatch ğ unbiased but its variance scales as Var(ğ) xx o²/B, where 8² is the par-sample variance and B the batch size. Prior work has emphasized that this inverse scaling underlies the implicit bias of SGD: smaller batch sizes inject more gradient noise, leading to flatter solutions and better generalization (Jastrzebski et al., 2017; Smith & Le, 2018; Keskar et al., 2016). Figure 4 (a) extends this observation to model merging. Solutions obtained with smaller B are more compatible under merging. The smallest B 16 achieves median accuracy gain of +1%, while larger setup with B = 256 has almost no benefit. This suggests that B induced noise improve the merging effectiveness.
Momentum. Optimizers use momentum to accumulate an exponentially weighted moving aver- age of past gradients. Traditionally understood an acceleration mechanism for escaping shallow local minima and traversing flat regions (Polyak, 1964; Sutskever et al., 2013), momentum also al- ters the effective noise characteristics of SGD. Figure (b) shows that models trained with larger = 0.9 exhibit consistently better mergeability than those trained with a lower or no achieving median accuracy gains of +1.0% compared +0.2% gains for low damped trajectories. Together with batch size, larger momentum shapes better landscape for model merging.
Augmentation. Data augmentation can also be viewed as injecting stochasticity into the optimiza- tion process. By applying random transformations to the input, the gradient covariance Σ incor- porates additional variance beyond minibatch sampling. Previous work interprets augmentation as form of implicit regularization (Hernández-Gareia & König, 2018), or an additional source of optimization noise (Hanin & Sun, 2021). Figure 4 (c) show that augmentation not only improves single-mon model accuracy but also retains merging effectivenesss And even without augmentation, larger learning rate can yield positive merging gains. Augmentation induced noise complements the minibatch and leaming rate noises, shaping the local and global loss landscape of merged models.

---

Published as a conference paper at ICLR 2026
3.5 WHAT ABOUT LANGUAGE MODELING?
Now we consider a language modeling task using the TinySto- ries dataset (Eldan & Li, 2023). We train a small GPT Trans- former model with two layers using the AdamW optimizer with a constant learning rate for 200k steps and save checkpoint ev- ery 2k steps, following the setup at Appendix A. Two endpoint models are trained for an additional 20k steps using a decayed learning rate scheduler. We use the loss performance gain to quantify the merging effectiveness.
0.001
Loss gain
-0.002
Learning rate
Figure 5 extends and supports our results from the previous sec- tions to the language domain. Starting from the learning rate v experiments, we fix À O. Figure 5 shows that larger learn- ing rate simplifies the merging process as measured by a lower loss gain, suggesting an implicit bias effect on the loss landscape. Additionally, Appendix Figure 13 (top) shows that a larger learn- ing rate r = le converges faster loss value of 2.20 com- pared to = le - 4. These observations match the previous results from the vision domain. When adding weight decay À to the equation, further merging benefits is observed. We fix the ? = le 3 and sweep across X values. Figure (bottom) shows that a larger À leads to better loss gain than a smaller value. The largest weight decay, such as À = 1e 1, has the best loss gain, but also has slower convergence (Figure 13). The second largest weight decay of A = le 2 has a similar convergence speed as x = 1e 3, but with better loss gain. Lastly, the small- est values have negligible effect as they have similar results as
Loss shin
-0.002
0.004
1c.3 Weight decay
10.2
16.1
Figure Larger learning rate and weight decay enable more effective merging in language modeling. (top) A larger learn- ing rate 1) achieves a better loss gain. (bottom) Adding larger À offers further merging gains.
training without weight decay.
Irle-04
Irle-05 o IrSe-06
o Ir3e-06
3.6 WHAT ABOUT TRANSFER LEARNING?
Ir5e-05
o Irle-06
Ir3e-05
The setups of previous sections train and bifurcate models on single task, performing model merging between the bifurcated endpoints at the end. Now we consider transfer learning, where the pretraining and finetuning tasks differ. We use the vision en- coder CLIP ViT-B/16 (Radford et al., 2021) pretrained on Im- a and finetune it on the WILDS-FMon 2021) dataset using AdamW with cosine scheduler. Since vary-
3.6
Accuracy Cop
3.0
1.5
(Koh et al.,
0.B
ing the learning rate changes the speed of convergence, we tune the number of training epochs for each setup and ensure conver- gence to near-zero training loss (details in Appendix A.2). We train three seeds per configuration and merge each pair, resulting in three merged models per vj. Additional results using the SGD optimizer instead of Adam W are in Appendix E.2.
0.0
0.0
1.0
2.0
3.0
Accuracy of merged model 00D
Accuracy gain ID
62
61
60-
59-
Figure 6 (top) shows that larger learning rate 77 identifies solu-
58
06
tions that are easier to merge. The smallest = 1e 6 lie in a flatter loss landscape region where the performance gain is 4x smaller than the largest y le - 4 when merged. The Pearson 0.981, indicating an almost per-
66
68
70
Accuracy of merged model ID
correlation coefficient is feet linear correlation between accuracy gain and learning rate Note, however, that one should not blindly use the largest learn- ing rate for finetuning. Figure 6 (bottom) shows that the merged models with the best performance are the ones with a moder- ate learning rate π 3e 5, as also observed by (Wortsman et al., 2022). The largest learning rate setup has the largest accu- racy gain, but the worst- perforrerfrming single model. Appendix B.3 presents similar results using different datasets and a pretrained model.
Figure 6: Merging effectiveness in transfer learning for ID and OOD data. (top) Accuracy gain linearly correlates with learning rate. (bottom) However, a larger learning rate leads to a subopti- mal merged model, despite hav- ing the largest accuracy gain.
7

---

Published as a conference paper at ICLR 2026
4 THE OPTIMIZER'S SS IMPLICIT BIAS ON TASK ARITHMETIC
In the previous section, we have seen how the optimizer implicitly biases the loss landscape for linear interpolation merging. We now turn to task arithmetic (TA) interpolation, which defines different subspace of solutions compared to the previous method.
4.1 Loss LANDSCAPE OF TASK ARITHMETIC
Fie-3
MN5 110//00 of morged mode 188
17--51
Acurro of magg0 mode
1304
150 02
H16-
F104
Vioo
15e-3
P58-5
Irl-20
V7e-5
15e-6
12 91
11c
I5e
11
0.2
0.6
0.6
0.8
0.2
0.8
0,8
0.5
1.0
022
0.4
0.6
0.5
10
0.2
0.5
0.5
0.5
10
anihhrmetc
arithmetcc
(a) Transfer learning
(b) Non-transfer learning
Figure 7: Task arithmetic loss landscape based on the model initialization. (a) In transfer learning with a pretrained initialization on ImageNet1K, larger learning rate solutions have a larger accu- racy gain and a smoother loss landscape (i.e. more robust to a-interpolation). (b) In non-transfer learning however, a larger solutions lie in a sharper minima.
Task arithmetic interpolates two models along different subspace compared to linear interpolation merging, identifying functionally different solutions. Here, we apply task arithmetic merging to study how it implicitly bias the loss landscape in two different settings: (a) Transfer learning setup. The pretraining and finetuning datasets do not overlap, same as in Section 3.6. Task arithmetic is applied using the pretrained base model Oasse (ConvNext-Tiny pretrained on ImageNet1 K) and the finetuned models on CIFAR10; (b) Non-transfer learning setup. The pretraining task is the same as the finetuning dataset, same as Section 3.2. Task arithmetic is applied using the model before the bifurcation as the base model Obose and the endpoint models, all trained on CIFARIO. For each configuration, we change the a-interpolation coefficient to traverse the subspace defined by the task arithmetic merging. This measures the performance change as function of a, which captures the shape of the loss landscape geometry. For simplicity, we fix a = 0.5 for the two task vectors when using task arithmetic merging.
Figure shows the robustness of each configuration to task arithmetic interpolation, highlighting a dichotomy between the two settings. In (a), larger learning rate identifies merged solutions that more robust to a-interpolation changes, corresponding to flatter loss landscape (Andriushchenko et al., 2023a). While in (b), the opposite is observed, that is a smaller 7) is associated to a flatter surface. These results highlight the importance of the Obase- A larger learning rate should pair with suitable initialization shape smoother and flatter loss landscape (Wortsman et al., 2022). Lastly, as in linear interpolation merging, we observe that a "too large" learning rate becomes unstable in both settings (a) and (b). Appendix C.3 presents additional results for different datasets.
4.2 LOSs LANDSCAPE MERGING DIFFERENT TASKS
We now consider merging two models sharing the same initialization Obose finetuned on two differ- ent tasks We finetune one 0 basse model CLIP ViT-B/16 on t1 WILDS-FMot and another on to RESISC45 (Cheng et al., 2017). Then, TA/TIES (Yadav et al., 2023) is applied to merge the two models. We measure the loss landscape geometry via a-interpolation. And to quantify the merging success, use the averaged normalized accuracy, which averages the ratio of the merged model performance over each single model performance (defined in Appendix A.3). Further details on the hyperparameters in Appendix A.2. Results are averaged over three random seeds.
Figure 8 shows how the primary optimizer component (7)) impacts the merging effectiveness of models trained on different tasks. Figure (a) presents the loss landscape geometry of the subspace defined by TA/TIES. We study the robustness of models trained using the same hyperparameters when interpolating a. We observe that training with larger learning 7) yields better performance
8

---

Published as a conference paper at ICLR 2026
Task arkhmmllic e 0.5
TIES mer merging o 1.0
Fi
Normatedd aeceerr
60
Fie-5
homsttee BrT-ney
60-
Nemo sard SOO POS
#3
Lee mo rade for
F2n-6
70-
Lee THO PAR
183
768
12.
11.2
12-1
06.1
08.
12. 725
704
015
(6)
0.5
0.3
althmmtic
Learingg rate ford)
Learnigg rate ford)
(a) Loss landscape a-interpolation
(b) Normalized accuracy cross-1)
Figure 8: Merging models trained different tasks using TA TIES merging. (a) Models trained using the same 1) are merged. Larger solutions are more robust to a-interpolation, up to the critical point 3e 5. TIES shapes a wider loss landscape than TA. (b) Merging models trained with different 7). Merging pairs of similar, larger yields the best performance.
compared to smaller ones in both merging methods (except for 1j le 4, which is the upper limit for stability). Moreover, larger 1) are also more robust to changes of a, corresponding to a flatter minima solution. In the Figure 8 (b), merged models are trained with different learning rates 7). and we present the best a per merging method. The highest performance merged models are those combined with similar and moderately larger 1) (across the antidiagonal). Note that merging task vectors from increasingly larger 7 have worse compatibility when merging with other models. For example, the largest learning rate 7) = le 4 is the most incompatible merge with all the others. Ilharco al. (2023) also observed performance degradation when merging models trained with too large 7,. Our results show that merging two moderately larger r) models have the highest accuracy but lose the compatibility property.
Lastly, the results in Figure (b) also suggests that TIES merging can better counteract the noise induced by larger n, yielding +2% improvement compared to task arithmetic at the best setup = 3e (88.0% vs 85.9)). In the loss landscape analysis, TIES also shapes a wider loss landscape compared to TA merging it is more stable across a-interpolation. Appendix F contains similar results with TIES merging models trained on three different tasks.
PERMUTATION SYMMETRY AND FEATURE ALIGNMENT
We show how the effective noise can shape the geometry of a minimum under permutation symme- try, and that merging requires complementary features to achieve performance gains.
Permutation symmetry. Prior work conjectures that two models A and On trained on the same task from different random initializations are equivalent up to permutation symmetries (Entezari et al., 2022: Ainsworth et al., 2023). Accordingly, there may exist a permutation such that the rebased model (822) aligns with 0B. enabling linear mode connectivity. In our experiments, larger effective noise simplifies this rebasing process. Following Section 3.2, we train ResNet18 models on CIFAR1O0 100 with independent random initializations and rebase one model via weight-based match- ing (Ainsworth et al., 2023). To visualize the effect of rebasing, we construct a 2D slice of the loss landscape by fixing three points as in Izmailov et al. (2018). Figure 9 (left, middle) shows that larger n corresponds a wider minimum and flatter interpolation path between g and π (0) indicating that the solutions are more easily alignable under permutation. We observe similar phenomenon for Transformer language models (see Appendix D.I).
Feature alignment. To connect our effective noise results to representation level behavior, we mea- sure feature alignment between the two branched checkpoints using linear CKA (Kornblith et al., 2019) penultimate-layer activations (2048 test samples), and relate it to the accuracy gain from merging. As shown in Figure 9 (right), increasing the effective noise (here induced by larger 11) simultaneously increases merge gains and decreases feature alignment, indicating more diverse fea- tures across branches. Together with the non- mon-monotonic dependence mergeability on effective noise, this suggests that the intermediate/criitcall noise regime promotes the kind of feature diversity that merging can exploit, whereas low-noise training produces highly aligned representations with little merging benefits.

---

Published as a conference paper at ICLR 2026
32
34
37
24
26
29
36
55
1.5
1.0
1e
03
1.0
pe 01
Accurasy gain (56)
1.0
m(
0.5
1r2e-01
Ir5e-02
0.0
Ir1e-03
0.0
* 0
*
0.0
8A
8v
90
92
94
96
96
0.0
0.5
1.0
0.0
0.5
1.0
Festure alignment
Figure 9: Larger learning rate v) identifies a (left, middle) broader basin that simplifies model permu- tation, and (right) more diverse features that achieve higher accuracy gain. The setup uses ResNetl 8 trained on CIFAR100 as in Section
6 RELATED WORK
Model merging. Early works on merging independently trained solutions on the same task can be found on mode connectivity (Garipov et al., 2018; Draxler et al., 2018). Linear mode connectivity has stricter condition such that connecting paths are linear (Frankle al., 2020; Neyshabur al., 2020). When this is not possible, re- re-basin methods can be used to reparametrize the solution and restore the linear connectivity (Entezari al., 2022; Ainsworth et al., 2023; Theus al., 2025). Built upon these results, model merging methods have been developed to increase the performance on single task (Wortsman et al., 2022) or to combine models trained on different tasks into one (Matena & Raffel, 2022; Ilharco et al., 2023). We refer the reader to Yadav al. (2025), which provides comprehensive survey of the latest merging methods.
Optimization dynamics. Standard optimization theory (Garrigos & Gower, 2023) shows that both batch sizes and learning rates drastically affect stability and convergence properties of SGD. In par- ticular, through an analysis of SGD's ' stationary distribution simple quadratic potentials (Jastrzeb- ski et al., 2017), it is possible to evince that, for single model training, the loss statistics at conver- gence only depend on the ratio between batch size and learning rates as also validated empirically by Smith et al. (2020). In turn, either high learning rates or low batch sizes are known to favor flat minima (Keskar al., 2016). While for more sophisticated optimizers, correlations between batch size, learning rates, and generalization might be more complex (Zhang et al., 2019; Malladi et al., 2022), other factors might more severely affect simple relations, such as non- Gaussianttt (Simsekli et al., 2019) of gradient noise and non-convexity (Xie et al., 2021).
7 CONCLUSION
We study how optimizer choices implicitly shape the model merging loss landscape and highlight the effective noise scale as a unifying factor. Learning rate, weight decay, batch size, and data augmentation all modulate this noise, which in turn determines the compatibility of independently trained solutions. This relationship is non-monotonic- - too little noise yields low benefits, too much destabilizes training, but an intermediate "critical point" enables effective merging. In practice, mergeability appears to be primarily determined by effective noise levels, suggesting that hyperpa- rameter search can be simplified by focusing on fewer dimensions.
Our findings extend prior work connecting optimization trajectory noise to flatness and generaliza- tion of individual models, showing that noise also shapes the compatibility of independent solutions. However, many open questions remain. For example, how systematically tune effeetive noise levels, architectural designs, and pretraining strategics to produce models that are not only strong individually but also inherently mergeable with other solutions?
Limitations. No theoretical guarantees are developed, and no truly large-scale experiments are con- ducted due our limited computational resources. We studied only the standard merging methods, which form the foundation of the latest approaches. Our goal was to use a set of simple, diverse, but realistic setups to understand the role of optimization in model merging.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Christopher Gadzinski and Razvan Pascanu for the helpful discussions. Chenxiang Zhang supported by the University of Luxembourg using ULHPC' and MeluXina' computing clusters. Alexander Theus acknowledges the financial support from the Max Planck ETH Center for Learning Systems. Antonio Orvieto is supported by the Hector Foundation.
AUTHOR CONTRIBUTIONS
Chenxiang Zhang led the project, contributing the majority of experiments, infrastructure, figures, framing, direction, and writing. Alexander Theus conceived the effective noise scale framework, designed and ran the experiments validating it, and contributed to figures, framing, and writing. Antonio Orvieto contributed to the framing, direction, and writing. Damien Teney, Jun Pang, and Sjouke Mauw contributed to the framing and direction.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. re-basi::: Merging models mod- ulo permutation symmetries. In International Conference on Learning Representations, 2023.
Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flam- marion. A modern look the relationship between sharpness and generalization. In International Conference on Machine Learning, 2023a.
Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud- Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learn- ing. 2023b.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607v:600.0644 2016.
Pratik Chaudhari, Anna Choromaáska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Tour Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: biasing gradi- ent descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2016.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEFF, 2017.
Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, et Command An enterprise-ready large language model. arXiv preprint arXtv:2504.00698 2025.
Francesco D' Angelo, Maksym Andriushchenko. Aditya Vardhan Varre, and Nicolas Flammarion.
Why do we need weight decay in modern deep learning? In Advances in Neural Information Processing Systems, 2024.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, 2018.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arX1v22305.07759, 2023.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation in- variance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, 2020.
'nttps://hpc-dost docp ionos..un... httpps:/www luxprovide. Tu/me luxina/
11

---

Published as a conference paper at ICLR 2026
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, 2018.
Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprini arXiv:2301.1/235 2023.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Advances in Neural Information Processing Systems, 2024.
Boris Hanin and Yi Sun. How data augmentation affects optimization for linear regression. In Advances in Neural Information Processing Systems, 2021.
Alex Hernández-CGarcia and Peter König. Data augmentation instead of explicit arXiv preprint /806v
2018
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, 2018.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. First Conference on Language Modeling, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015,
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. In Uncertainty in Artificial Intelligence, 2018
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 04836, 2016.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning,
2021.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, 2019.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems, 2019.
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 2022.
Stephan Mandt, Matthew D Hoffmann, and David M Blei. Stochastic gradient descent approximate bayesian inference. Journal of Machine Learning Research, 2017.
Michael S Matena and Colin A Raffel. Merging models with wiightee averaging. Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2026
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06166 2018.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? In Advances Neural Information Processing Systems, 2020.
Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, and James Martens. Optimizers qualitatively alter solu- tions and we should leverage this. arXiv preprint arXiv:2507. 12224,, 2025.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com- putational Mathematics and Mathematical Physics, 1964.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00 00451, 2018
Ildus Sadrtdino,, Maxim Kodryan, Eduard Pokonechny, Ekaterina Lobacheva, and Dmitry P Vetrov. Where do large learning rates lead us? In Advances in Neural Information Processing Systems, 2024.
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise deep neural networks. In International Conference on Machine Learning, 2019.
Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning. 2020.
Samuel I. Smith and Quoe V Le. A bayesian perspective on generalization and stochastic gradient descent. International Conference on Learning Representations, 2018.
Samuel L Smith, Picter-Jan Kindermans, Chris Ying, and Quoe Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In International Conference Machine Learning, 2013.
Alexander Theus, Alessandro Cabodi, Sotiris Anagnostidis, Antonio Orvieto, Sidak Pal Singh, and Valentina Boeva. Generalized linear mode connectivity for transformers. In Advances in Neural Information Processing Systems, 2025.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05330 2017.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, and Simon Kornblith. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference Machine Learning, 2022.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2021.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 2023.
Prateek Yadav, Tu Vu, Jonathan Lai. Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint arXiv:2410.03617, 2024.
13

---

Published as a conference paper at ICLR 2026
Prateek Yadav, Colin Raffel, Mohammed Mugeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recy- cling and routing among specialized experts for collaborative learning. Transactions on Machine Learning Research, 2025.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Computer Vision and Pattern Recognition. 2022.
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. Advances in neural information processing systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
A Experiment setting details
16
A.1 Training and merging setup
16
A.2 Transfer learning experimental setup
16
A.3 Details on metrics
16
B Additional results for linear interpolation merging
17
B.1 Dataset: CIFARIO
17
B.2
Data augmentation: SVHN, CIFARIO, TinylmageNet.
18
B.3 Transfer learning: VIT, ConvNext-T
19
B.4
Language modeling...
20
B.5
Training loss of decayed models
B.6
Merging fails due to high effective noise
22
Additional results for task arithmetic
23
C.I Learning rate, weight decay
23
C.2 Language modeling
24
C,3 Transfer learning: FMoW, RESISC45
25
C.4 Merging different tasks
26
D Additional results loss landscape
27
D.1 Permutation symmetries
27
D.2 Transition phase: hills, flatland, and valleys
29
D.3 Flatness
30
D.4
Landscape vs. effective noise
30
Additional results for training dynamics
31
E,1
Scheduler
31
E,2 Optimizer
32
Additional results TIES merging
33
G Additional results on SWA
34
LLM research assistance
35
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENT SETTING DETAILS
A. TRAINING AND MERGING SETUP
For Section 3.2, Section 3.3, Section 3.4, and Section 3.5, use the following training setup.
We use the warmup-stable-decay (WSD) scheduler (Zhai et al., 2022; Hu et al., 2024). We use the square root decay as Hägele al. (2024). Given single configuration (e.g. Ir 0.1), we use a constant learning rate to train a model for Tatable epochs, saving a checkpoint 0, every i epochs. For each 0,, we use a decay learning rate scheduler and continue the training for Taccay epochs, obtaining two final endpoint models 0i,4 and 0,,-- Finally, the merged model is linear interpolation (Equation (1)) between i,,A and 0g,B with a 0.5.
We provide example. For the CIFARI00 task, we train model using constant learning rate for Tstable 2000 epochs and save a checkpoint 0; every i = 20 epochs. Then, for each checkpoint, we use a decay scheduler and create two endpoint models 0..4 and 0g This means that at the end, there will be Tatable/ / 2000 200 = 100 different merged models.
Note that, account for the different magnitudes of settings (e.g. Ir 0.1 Ir 0.01), a Tstable of one order of magnitude larger than the standard setting to ensure convergence single models. We use Tstable 2000 for CIFAR1O, CIFAR 100, and SVHN and Tttable 1500 for TinylmageNet. We use Tdeeay = 30 for CIFARIO and CIFAR and Tdecay = 20 for SVHN and TinylmageNet.
A.2 TRANSFER LEARNING EXPERIMENTAL SETUP
For Section 3.6 and Appendix B.3, we use the following training setup.
For CLIP V/T-B/16 finetuned on WILDS-FMoW, we discard the language model. We use the AdamW optimizer with warmup-cosine learning rate scheduler. Since varying the learning rate changes the speed of convergence, we carefully tune the number of training epochs for each setup to ensure convergence (e.g. training loss 0). The following hyperparams (epochs, Ir) are used for each setup (20, le-4), (20, 5e-5). (20, 3e-5), (20, le-5), (30, 5c-6), (40, 3c-6), and (100, le-6).
For CLIP VIT-B/16 finetuned on RESISC45, we follow the above configuration. The following hyperparams are used (20, le-4), (20, 3e-5), (20, le-5), (20, 3e-6), and (20, le-6).
For ViT-S/16 pretrained on INIk and finetuned WILDS-FMo's with a warmup-cosint learning rate scheduler. The following hyperparams are used (20, 1e-3), (20, 3e-4), (20, lc-4), (40, 3e-5), and (100, le-5).
the AdamW optimizer
For ConvNext-T pretrained on INIk and finetuned on CIFARIO, we use the Adam W optimizer with warmup-cosine learning rate scheduler. The following hyperparams used (20, 1e-3), (20, 5e-4), (20, 3e-4), (40, le-4), (40, 5e-5), (80, 1e-5), and (80, 5e-6).
Note that, for each setup, have grid searched and used the largest learning rate possible. This means that an even larger learning rate fails to converge.
A.3 DETAILS ON METRICS
Normalized accuracy compares the relative performance metric of the multi-task model to that of single finetuned models:
1 T accuracy(Ax)) accuracy norm T
Σ
accuracy(0.)
where T' the total number of tasks, ON represents the multi-task model and 0, is the single fine- tuned model for the task t,. This metric compares the baseline performance against each task.
16

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RESULTS FOR LINEAR INTERPOLATION MERGING
B.1 DATASET: CIFAR10
Ir2e-01
0.8
0.8
Irle-01
Accuracy gain (96)
Accuracy gain (%)
Ir5e-02
0.6
0.6
Irle-02
Irle-03
0.4
24
0.2
0.0
0.0
1e-03
1e-02
5e-02
1e-01
2e-01
Learning rate
Accuracy single
wdle-3
0.8
0.8
wd5e-4
Accuracy gain (%)
Accuracy gain (%)
wdle-4
0.6
wdle-5
wd0
0.4
0.2
0.0
0.0
1e-5
1e-4
5e-4
1e-3
91
92
93
94
95
Weight decay
Accuracy single (%)
0.8
0.8
bs16
bs32
Accuracy gain (%)
0.6
Accuracy gain (9)
0.6
bs64
bs128
bs256
8
4
0.0
0.0
256
128
64
32
16
92
93
94
95
Batch size -
Accuracy single (%)
Figure 10: Larger learning rate / larger weight decay / smaller batch size all lead to larger perfor- mance gain in CIFARIO dataset.
17

---

Published as a conference paper at ICLR 2026
B.2
DATA AUGMENTATION: SVHN, CIFARIO, TINYIMAGENET
1.0
2.0
Accunscy gein 050
Accuracy gein 0%)
1.0
0.3
gain
0.0
02
1.0
-1.0
2.0
3.0
-0.1
2.0
4.0
16-03
1e-02
1o-03
1a-02
1c:02
16-01
1o-02
1o:01
56-03
5e-02
So-03
Se-02
Learning rate withywithout
Learning rate withwwithout aug.
Learning rate withvwithout aug.
Irle-02 w/ aug.
1.0
2.0
Irle-03
Accurccy gain (80
0.3
irle-02 weo aug.
Accurecy genn 1%)
1.0
Irle-03 weo sug.
gain
0.0
0.2
-1.0
-1.0
Irle-01
2.0
aug.
Irle-02 w/ aug
Ir5e-03 w/ sug.
Irle-01 weo aug.
.3.0
Irse-02 wio aug.
0.1
2.00
aug.
aug.
...0
80
84 Accuracy single (%)
88
90
94
65
Accuracy single (%)
Acturacy single (%)
(a) SVHN ML.P
CIFARIO ResNet18
TinylmageNet/ DenseNet121
Figure 11: Accuracy gain and data augmentation. The merging fails w/o augmentation. However, larger learning rate remains easier to merge than a smaller one.
18

---

Published as a conference paper at ICLR 2026
B.3
TRANSFER LEARNING: VIT, CONVNEXT-T
Accuracy of merged model OOD
Ir3e-04
2.5
Irle-04
Accuracy 9ain 00D
Ir3e-05
2.0
53
Ir3e-05
Irle-05
52
Irle-05
51
50
0.0
0.0
0.5
1.0
1.5
2.0
58
60
61
Accuracy gain ID
Accuracy of merged model ID
Figure 12: Larger learning rate enables easier merging under transfer learning for both ID and OOD datasets. The pretrained architecture is VIT trained on INIk and finetuned on FMoW. The evaluation done on the test set ID and OOD splits.
19

---

Published as a conference paper at ICLR 2026
B.4 LANGUAGE MODELING
-0.0015
-0.0010
0.0020
Loss gain
0.0015
Loss gain
0.0025
-0.0030
wdle-01
Irle-03
wd1e-02
-0.0020
Ir5e-04
00.033
wdle-03
Irle-04
wdle-04
0.0025
Ir5e-05
0.0040
wd0
2.10
2.20
2.30
2.40
2.20 Loss single
Loss single
Figure 13: Larger learning rate and weight decay enable more effective merging in language mod- eling. (left) Different setups at loss single =: 2.20 clearly differ loss gain. (right) Similar phenomenon when tuning weight decay.
20

---

Published as a conference paper at ICLR 2026
B.5
TRAINING LOSS OF DECAYED MODELS
1.0
Ir3e-02
Ir2e-01
Irle-02
0.5
Irle-0
0.8
Ir5e-03
Ir5e-02
Training loss
Irle-03
Training loss
0.4
Irle-02
0.6
Irle-03
0.4
0.2
0.2
0.0
500
1000
1500
2000
1000
1500
2000
Epoch
Epoch
(a) SVHN
(b) CIFARIO
Ir2e-01
Ir5e-02
Irle-01
Ir3e-02
Ir5e-02
3
Irle-02
Training loss
Irle-02
Training loss
Irle-03
Irle-03
1.0
0.5
1
0.0
1000
1500
2000
0
500
1000
1500
Epoch
(c) CIFAR100
(d) TinyImageNet
Figure 14: Training loss of decayed models from Section 3.2. For deep networks trained on CIFAR and TinylmageNet, ensure that different setups reach near 0 training loss. For the simple MLP trained on SVHN, convergence to 0 training loss is slow. However, the largest learning rate 0.03 has the highest accuracy model despite a larger loss.
21

---

Published as a conference paper at ICLR 2026
B.6
MERGING FAILS DUE TO HIGH EFFECTIVE NOISE
2.0
Irle-01
0.8
h12e00
Ir2e-01
1.5
0z
Accurscy Qein
0.6
IrSe-02
Accuracy gain
1.0
IrSe-02
Accuracy gain
Irle-02
Irae 02
02
0.4
0r1e-03
Irle003
0.2
0.0
0.0
0.0
-0.5
90
94
60
65
75
Accuracy single
Accuracy single
Accuracy single
(a) CIFAR1O/ ResNet18
(b) CIFARIO0/ ResNet18
(c) Tiny Imageet DenseNet121
Figure 15: Too large learning rate causes instability/failure in merging.
Accuracy gain
Accuracy gein
Accurecy gain
wdbe-3
wd5e:3
wdle-3
wdle-3
wdle-3
we
wd5e-4
wdSe-4
w
wdle
wd5e-5
wd0
90
95
50
Accuracy single
Accuracy single
Accuracy single
(a) CIFARIO/ ResNet18
(b) CIFARIO0/ ResNet18
(c) Tiny DenseNet121
Figure 16: Too large weight decay causes instabilityffailure in merging.
22

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS FOR TASK ARITHMETIC
C.1 LEARNING RATE, WEIGHT DECAY
Accuracy gain (%6)
Accuracy gain (%6)
-1
-2
-2
Ir2e-01
-3
Ir3e-02
33
Irle-01
Irle-02
Ir5e-02
-4
Ir5e-03
-4
Irle-02
Irle-03
Irle-03
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Task arithmetic
Task arithmetic
(a) SVHN
(b) CIFARIO
Ir2e-01
Ir5e-02
Irle-01
Ir3e-02
Accuracy gain (96)
Ir5e-02
Accuracy gain (%6)
Irle:02
Irle-02
Irle-03
-1
Irle-03
-1
-2
-4
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
Task arithmetic
Task arithmetic
(c) CIFAR100
(d) TinylmageNet
Task arithmetic interpolation robustness of models w/o Pretrained weight from the Sec- tion 3.2. In the absence of pretrained weight, the largest learning rate is the least robust to task arithmetic interpolation.
Figure 17:
23

---

Published as a conference paper at ICLR 2026
Accuracy gain
-1
Accuracy gain
-2
-2
wdle-3
wdle-3
-3
wd5e-4
-3
wd5e-4
wdle-4
wdle-4
-4
wdle-5
-4
wdle-5
wdo
wdo
0.4
Task arithmetic
Task arithmetic
(a) SVHI
CIFARIO
wd5e-4
wdle-4
wd5e-5
Accuracy gain
Accuracy gain
wdo
-1
-2
wdle-3
-2
wd5e-4
wdle-4
-3
-4
-4
wd0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Task arithmetic
Task arithmetic
(c) CIFARI00
(d) TinyImageNet
Figure 18: Task arithmetic interpolation robustness of models w/o Pretrained weight from the Sec- tion 3.3. In the absence of a pretrained weight, the largest weight decay is the least robust to task arithmetic interpolation.
C.2 LANGUAGE MODELING
0.08
Irle-3
0.12
wdle-1
Ir5e-4
wdle-2
0.10
Irle-4
wdle-3
0.06
Loss gain
Ir5e-5
Loos gain
0.0B
wdle-4
wdo
0.04
0.06
0.02
0.02
0.00
0.00
0.2
0.2
Task arithmeticc
Task arithmetic
Figure 19: Task arithmetic loss gain in language modeling for a small GPT on the TinyStories dataset trained for 200k steps. In the absence of pretrained weight, the largest learning rate/weight decay the least robust to task arithmetic interpolation.
24

---

Published as a conference paper at ICLR 2026
C.3
TRANSFER LEARNING: FMOW, RESISC45
Accuracy of merged model (%)
Ir5e-5
Ir3e-5
Irle-5
-10
Ir5e-6
Accuracy gain
60
Ir3e-6
-0
Irle-6
-30
Irle-4
Ir3e-5
-40
Irle-5
50
-50
Ir3e-6
Irle-6
0.6
0.2
0.4
0.6
0.8
1.0
Task arithmetic
Task
Figure 20: Task arithmetic robustness and gain for CLIP ViT-B/16 finetuned on FMoW.
Accuracy merged model (%)
Irle-3
0
Ir3e-4
Irle-
55
Ir3e-5
Accuracy gain
-10
Irle-5
-20
Irle-3
45
30
Ir3e-4
Irle-4
Ir3e-5
Irle-5
35
0.6
0.2
0.4
0.6
0.8
1.0
Task arithmetic
arithmetic
Figure 21: Task arithmetic robustness and gain for ViT-S/16 pretrained on IN1k finetuned on FMoW.
Accuracy of merged model (%)
Irle-4
0
Ir3e-5
Irle-5
Ir3e-6
Accuracy gain
55
Irle-6
-10
Irle-4
Ir3e-5
Irle-5
-15
Ir3e-6
Irle-6
-20
0.2
0.6
0.2
0.4
0.6
0.8
1.0
Task arithmetic
Task
Figure 22: Task arithmetic robustness and gain for CLIP ViT-B/16 finetuned on RESISC45.
25

---

Published as a conference paper at ICLR 2026
C.4
MERGING DIFFERENT TASKS
Task arithmetic
Task arithmetic
Task arithmetic o 0.5
90
80
Worman lzze accurady
68.5
68.9
70.8
75.2
75.8
71.4
72.2
75.4
84.4
79.1
70.5
71.3
73.3
85.9
78.0
Learning rate for
66.1
66.5
68.7
72.8
70.4
76.0
77.2
81.5
85.0
72.7
77.2
78.8
84.3
86.9
71.7
63.0
63.6
65.6
69.9
67.5
76.1
77.8
81.2
81.3
69.9
79.9
81.8
85.6
82.1
68.7
60
61.8
68.3
76.4
79.2
69.3
79.3
83.7
50
Task arithmetic
Task arithmetic
Task arithmetic
1.0
90
76.7
Normalized accuracy
80
69.2
70.2
74.4
06.3
76.4
06.7
67.4
71.3
84.1
70.9
63.7
64.4
67.0
78.3
64.2
Learning rate for
77.1
79.4
85.7
87.0
70.5
74.8
77.5
85.0
83.2
67.1
69.9
72.9
80.6
76.5
61.9
70
81.6
84.0
87.6
81.3
68.5
75.4
80.4
86.0
76.8
65.7
51.7
63.9
77.3
70.9
61.5
60
83.7
79.1
65.5
00.0
61.7
75.7
69.5
61.4
Learning
f
Learning 4
Learning
Figure 23: Task arithmetic merging of two different tasks across values. Similar setups (antidiago- nal) consistently have better merged models. Note that for a = 0.2 smallest learning rate models do not merge well. Same for 1.0, indicating sharper minima defined by task arithmetic subspace, similar as Section 4.1.
TIES merging = 0.2
TIES merging a= 0.4
TIES merging a = 0,5
90
62.1
62.4
63.4
67.3
70.4
64.0
64.2
65.3
71.6
79.1
63.3
63.5
64.4
70.8
80.9
Learning rate for G
62.5
65.9
70.4
70.6
70.5
momalliae accuracy
880
70
67.4
60
53.2
52.9
53.2
53.9
49.3
60.0
59.9
61.3
61.7
55.0
63.5
63.8
63.3
64.6
57.9
TIES merging
0.6
TIES merging a 0.8
TIES merging a
1.0
90
62.5
62.6
63.3
69.7
81.7
61.1
61.3
61.8
67.4
81.3
59.7
59.8
60.2
65.0
78.5
80
Learning rate &
71A
70.4
69.9
67.6
74.7
82.6
69
67.1
67.3
68.6
66.6
59.9
73.5
73.5
73.5
67.6
61.0
78.6
77.9
75.2
66.0
60.0
50
Learning rate for f
Learning rate for
Learning rate for
Figure 24: TIES merging two different tasks across values. Similar setups (antidiagonal) consistently have better merged models.
26

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RESULTS ON LOSS LANDSCAPE
D.I PERMUTATION SYMMETRIES
41
43
50
65
27
32
39
57
24
26
37
55
55
n=1e le
1.0
17
le
02
n= le
01
m(
0.5
0.5
0.5
0.0
*
0
8.
0.0
0A
%
0.0
8.
8.
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(a) Epoch 200
(b) Epoch 200
(c) Epoch 200
35
37
40
47
62
25
31
38
56
> 56
24
26
29
36
55
55
7 le eee--p
1.0
le
02
n= le -01
B
0.5
0.0
8.
0.0
8,
8.
0.0
8g
8.
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(d) Epoch 800
(c) Epoch 800
( Epoch 800
32
34
37
44
61
61
25
30
38
56
> 56
24
26
29
36
55
55
1.0
1e
1.0
f1
1e
02
1.0
n- le 01
ml
ml,))
0.5
0.0
D.
0.0
0.
0g
9.
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(g) Epoch 1600
(h) Epoch 1600
(i) Epoch 1600
Figure 25: Larger learning rate 17 identifies broader basin that simplify model permutation across different training phase. The setup uses ResNet18 trained on CIFAR100 as in Section 3. The x and g-axis are normalized enable visual comparison of the basin size.
In Figure 27, we evaluate the connectedness of small GPT-2 models trained from scratch on WikiText-103 under varying learning rates. All models are 6-layer GPT-2-style decoders (block size 512, d'model = 512, Theal = 8, Ninner 2048), trained with the GPT-2 tokenizer using batch size of 32 for 10 epochs, weight decay 0.01, and learning-rate warmup ratio 0.05. To obtain apply the symmetry-awan merging methods of Theus et al. (2025). We consider three settings: vanilla averaging (no alignment), weight matching (alignment via max- imizing parameter similarity), and learned matching (alignment optimized directly for next-token prediction on WikiText- 103). As in our experiments without symmetry alignment, larger learning rates tend to improve connectivity and reduce loss barriers. However, consistent with prior observa-
optimal neuron alignments,
27

---

Published as a conference paper at ICLR 2026
1.5
Irle-01
Irle-02
Accuracy gain (%)
Irle-03
1.0
0.5
-0.6
-0.4
-0.2
0.0
Accuracy gain w/ perm. invariance (%)
Figure 26: Accuracy gain after permutation invariance for CIFAR100. We validate the following hypothesis: the minima identified with a larger effective noise simplifies the rebasing process. We train two sets models O and On with random initialization using the same setup as in Section 3.2. The weight-based matching is used to rebase π a(0a) On. Then, measure the permutation invariant gain = acc (@A)) acc(@g). A larger gainine value corresponds to a more ad- vantageous basin for merging. We observe clear correlation between the standard merging gain obtained in the shared initialization and branching setup (y-axis) against the merging gainino be- tween independent initialized models (æ-axis). In particular, larger Ir (or effective noise) helps identify "flatter" basins that also enables more effective rebasin.
Itled
Ir2.5c
Loss barrier
1.0
0.2-
Loss
0.5
01
0.0-
0.0-
0.2
0.4
0.8
0.8
m6dp
0.2
0.4
0.6
0.8
m6dpl
6,
0.2
0.4
0.6
0.8 mldu)
Interpolation coefficient (A)
Interpolation coefficient
Interpolation coefficient (A)
2.0
Loss barrerr
0.25
0.15
0.5
10-4
2.504
56-4
0.0
16.4
2.Se-4
S6-4
0.00
16-4
3.544
56-4
Learning rate
Learning rate
Learning rate
(a) Vanilla averaging
(b) Weight matching
(c) Learned Matching
Figure 27: Loss-barrier analysis for small GPT-2 models trained on WikiText-103 under the align- ment methods of Theus al. (2025). Panels (a)-(c) show the complete loss interpolation curves for different learning rates, while the remaining panels highlight the peak (maximum) loss barrier extracted from each trajectory. Lower loss barriers are better.
tions that text-based Transformers trained from scratch do not exhibit linear mode connectivity, see no cases where interpolation reduces loss.
28

---

Published as a conference paper at ICLR 2026
D.2
TRANSITION PHASE: HILLS, FLATLAND, AND VALLEYS
Epochs
Epochs
0.23
5
0.51
15
0.22
20
Test
0.50
40
60-
0.49
100
0.20
100
120
0.19
8/
0.2
0.4
0.6
0.8
80
6A
0.2
0.4
0.6
0.8
96
a Interpolation
Interpolation
(a) SVHN
(b) CIFARIO
Epochs
E Epcchs
1.01
10
2.00
20
40
1.90
60
Test
80
Test
1.80
120
100
1.70
0A
0.2
0.4
0.6
0.8
0.2
0.4
0.6
8h
a Interpolation
Interpolation
(c) CIFAR100
(d) TinyImageNet
The loss geometry of the linear interpolation between two endpoints changes from hill - valley, based on the timing of the bifurcation. Given a training budget T. the legend indicates the bifurcation start epoch Ta, which means the training continues for T T T, epochs with 0 and 0p. The transition phase (dashed line) marks the phase change from hill into valley.
Figure 28:
c10
test
3.5
c100
train
tiny
transition
svhn
2.5
Loss
1.5
0.5
0
20
40
60
80
Epoch
Figure 29: Identifying the transition phase from hill to valley.
29

---

Published as a conference paper at ICLR 2026
D.3 FLATNESS
top8 largest A
top8 largest
600
Ir0.1 1r0.01
600
Ir0.1
1r0.01
500
500
Sharpness
400
Sharpnest
400
300
200
200
100
100
0
0
500
1000
1500
2000
0
500
1000
1500
2000
Epochs
Epochs
(a) CIFARIO
(b) CIFAR100
Figure 30: The flatness measured using the top-8 eigenvalues of the hessian. The larger learning rate solutions lie inside sharper minima.
D.4 LANDSCAPE vs. EFFECTIVE NOISE
Ø Finetured models
(a) Low noise
(b) Optimal noise
(c) High noise
Figure 31: ResNet18 trained on CIFAR100 2D loss slices in the plane spanned by the base model and two fine-tuned models under varying effective noise scales. Low and high noise both yield broad, flat corridors between the fine-tuned solutions, whereas an intermediate (optimal) noise level introduces performance gains between them.
30

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS FOR TRAINING DYNAMICS
E.1
SCHEDULER
Initial phase
Middle phase
Final phase
0.24
0.22
casine
0.21
Test loss
0.21
0.20
0.20
0.19
0.19
0n
Interpolation
Interpolation
Interpolation
Epochs
Epachs
100
110
Epochs
140
(a) CIFARIO
Initial phase
Middle phase
Final phase
1.05
0.98
1.00
1.02
0.98
0.96
Test loss
0.96
0.96
0.92
0.93
8A
0.2
0.4
0.6
0.8
9g
8A
0.2
0.4
0.6
0.8
0g
8A
0.2
0.4
0.8
Interpolation
Interpolation
Interpolation
Epochs
140
(b) CIFAR100
Figure 32: Comparison between WSD and cosine scheduler.
We use the same setup described in Appendix A, varying only the scheduler for the whole training duration. The same training budget (epochs) is used. Figure 32 shows that WSD scheduler enables casier merging, especially when bifurcating in the final phase where the learning rate of cosine is already small.
31

---

Published as a conference paper at ICLR 2026
E.2
OPTIMIZER
Accuracy of merged model OOD
96.0
Irle-03
1.5
95.5
IrSe-04
Accuracy gain OOD
Ir3e-04
1.0
95.0
Irle:04
IrSe-05
94.5
Irle-05
0.5
Ir5e 06
68
93.5
0.0
0.2
0.4
0.6
97.25
97.50
97.75
98.00
98.25 98.50
Accuracy gain ID
Accuracy of merged model ID
(a) Adam
0.3
Accuracy of merged model O0D
Irle-02
95.0
Irše-03
Accuracy gain OOD
0.2
94.8
Irle-03
Ir5e-04
0.1
94.6
Irle-04
94.4
Ir5e-05
-0.1
94.2
0
-0.2
94.0
93.8
-0.3
-0.2
-0.1
0.0
0.1
97.4
97.5
97.6
97.7 97.8 97.9
Accuracy gain ID
Accuracy of merged model
(b) SGD
Figure 33: Optimizer comparison in transfer learning. Using AdamW, larger learning rate enables easier merging for both ID and OOD datasets, while for SGD, benefits are only for ID dataset. The pretrained architecture is ConvNext-1 trained on IN1k and finetuned on CIFAR10. The test set ID is CIFARIO 10 and test set OOD is CIFARIO. (Recht et al., 2018).
We compare how the optimizer affects the merging effectiveness between AdamW and SGD. Note that in this experiment, SGD with small Ir required 20 more steps compared to AdamW for conver- gence to near zero training loss. Figure 33 shows that SGD-trained models have larger performance gain with larger Ir for ID dataset, but not for OOD dataset. Moreover, SGD trained models have lower final performance compared Adam W models (95% vs 96%).
32

---

Published as a conference paper at ICLR 2026
F
ADDITIONAL RESULTS ON TIES MERGING
Normalizee accuracy
80
75
Ir3e-5
Irle-5
Ir3e-6
Irle-6
60
0.5
1.0
1.5
Task arithmetic a
Figure 34: TIES merging of models trained on three different tasks (RESISC45, FMoW, CIFARIO). Results are averaged over three seeds.
We study whether more advanced merging methods can reduce sensitivity hyperparams sug- gested. First, we apply TIES directly to the existing setting in Section with two tasks (RESISC45, FMoW). We use TIES keep 70% of the values and "mean" aggregation.
TIES merging can partially counteract the high noise. We extend the setting in Section 4.1 by ap- plying TIES to three tasks (RESISC45, FMoW, and CIFARI0) and measure normalized accuracy across interpolation. Figure 34 shows that at small a < 0.5, a larger Ir trained models have the highest performance, while at larger a > 0.5, larger Ir becomes unstable. This follows the same trend observed in the main sections.
33

---

Published as a conference paper at ICLR 2026
G ADDITIONAL RESULTS ON SWA
1.5
1.5
Accuracy gain (%)
Accuracy gain (%)
1.0
1.0
Ir2e-01
Ir2e-01
0.5
Irle-01
0.5
Irle-01
Ir5e-02
Ir5e-02
Irle-02
Irle-02
0.0
0.0
Irle-03
0rle-03
1.0
1.5
1.0
1.5
Accuracy gain w/ SWA (%)
Accuracy gain w/ SWA (96)
3
(h) A: 10
Ir2e-01
Ir1e-01
Ir5e-02
Accuracy (%)
Accuracy (%6
Irle-02
76
76
Irle-03
Ir2e-01
Irle-01
Ir5e-02
Irle-02
Irle-03
73
73
74
76
77
78
74
76
77
78
Accuracy w/ SWA (%)
Accuracy w/ SWA (%)
(c) k 3
(d) k 10
Figure 35: Accuracy gain when applying stochastic weight averaging (SWA).
Using the models trained in Section 3.2, we apply stochastic weight averaging (SWA) the last checkpoints of the branched models, obtaining 0z and Ogre, which are merged into gewa We define gain (cc(O 0.5 (aco(Oxua) +aco(07 (0)a)) to measure the accuracy gain of SWA models after merging.
Figure 35 (top) shows that SWA endpoints can also benefit from merging the branched models Oawa and Ogwa. However, the merge gains are lower compared to the standard setting w/o SWA. This is because SWA already incorporates the benefit of large (noise) to explore wider valleys by merging the models along the same trajectory, while merging combines models from different trajectories. Figure 35 (bottom) shows that the final accuracies are comparable, and the methods are complementary. These results support the conclusion that effective noise governs mergeability, including SWA.
34

---

Published as a conference paper at ICLR 2026
H LLM RESEARCH ASSISTANCE
We use LLM to assist this research project in the following tasks: manuscript polishing and retrieval of related work. For both tasks, we make mild use of LLM for the manuscript writing phase. In particular, polishing has been used only to improve the flow of the sentences, while the retrieval of contents has been used to find related works.
35


Reviews:
[{"review_id": "EOIvgjCnei", "reviewer": "Reviewer_e6Pv", "summary": "This paper investigates how optimizer choices shape the mergeability of independently trained models. The central claim is that a single quantity—the effective noise scale $\\tilde{S} \\approx \\eta / B(1-\\mu)^2$ (augmented by data-augmentation–induced covariance)—organizes the effect of learning rate, batch size, momentum, weight decay, and augmentation on merging success. Empirically, mergeability (via linear interpolation and task-arithmetic) is non-monotonic in $\\tilde{S}$: there is a “sweet spot” between too little and too much noise. The study spans CNN/MLP/Transformer/GPT, from-scratch, transfer learning (CLIP ViT-B/16), and small language modeling (TinyStories). Key reported regularities: (i) larger LR or WD (on scale-invariant nets) tends to produce more mergeable solutions; (ii) smaller batches help; (iii) augmentation helps; (iv) for task arithmetic, benefits of larger LR depend strongly on having a pretrained initialization.", "questions": "1. Can you disentangle selection vs creation of mergeable minima? For example, if you freeze the early trajectory (same prefix) and only vary \n$\\tilde{S}$ after bifurcation, do you retain the same non-monotonic curve?\n2. What breaks at larger scales (if anything)? Are there plans of extending the scale of the experiments or are there already preliminary adapter-level or LoRA-head results on larger backbones?\n3. Given a target mergeability level, how would practitioners choose LR/WD/B/augmentation?\n4. How does “tune noise” differ from, or complement, SWA/model-soups’ push toward wide valleys?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Unifying lens**: Framing optimizer and data choices through a single effective noise control for cross-solution compatibility (not just single-solution generalization) is a crisp, novel angle.\n- **Systematic sweeps**: Clear ablations across LR/WD/B/augmentation, with consistent procedures and multiple architectures/datasets.\n- **Cross-domain evidence**: Vision from scratch, transfer learning, and (albeit small) language modeling make the phenomenon less likely to be an artifact of one setup.\n- The story flows well from preliminaries → unified factor → component ablations → task arithmetic → transfer.\n- **Bridges two communities**: Brings SGD-as-stochastic-process insights into model-merging practice, with implications for soups, adapters, and FL-style workflows.", "weaknesses": "- Much of the study is effectively in shared-trajectory (same init + branched checkpoints). Mergeability here may reflect trajectory alignment more than global landscape bias. Also, permutation symmetries are not addressed.\n- Flatness alone doesn’t explain mergeability; representation overlap matters. One way to address this could be reporting centered-kernel alignment between penultimate-layer features of the two models; correlate CKA with merge gain at fixed $\\tilde{S}$.\n- The connection to SWA / soups / rebasin / SANE / KnOTS / PMA is noted but not dissected. It would be interesting tocCompare “tune noise” against (i) SWA style training aimed at wider valleys, (ii) permutation-aware merging, and (iii) subspace alignment (e.g., SVD/CCA-merge). Show complementarity or superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4prtHlv2gw", "reviewer": "Reviewer_8mKb", "summary": "The authors show that the effective noise scale, which aggregates noise across data covariance, learning rate, and momentum, has an inverted U relationship to model mergeability; too little or too much noise makes model merging difficult.", "questions": "- I didn't understand what you actually did here (line 395; this should probably also be rephrased for clarity): Models w/o pretraining weight from Section 3.2 (i.e. pretraining dataset is the same as the finetuning dataset). Task arithmetic is applied to a base model and two task vectors. For each checkpoint, the base model θbase is the checkpoint itself, and the task vectors are obtained from the endpoint models τA = θA − θbase and τB = θB − θbase.\n- I'm a bit unsure about exactly what's going on in the figures (e.g. Figure 2). Does each point on the figure correspond to a checkpoint during training? And, if I'm understanding correctly, two checkpoints are merged — do both checkpoints show up on the graph as separate points?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The inverted U relationship to effective noise and mergeability is interesting and intuitive; too little noise gives weird solutions, too much noise moves models too far apart to merge\n- The authors show the relationship with the effective noise holds for both vision and language models.\n- The authors try a few common (and simple) model merging techniques", "weaknesses": "- The biggest weakness is that the contribution seems quite incremental; it's already known that solutions located on relatively flat areas of the loss landscape have better generalization behavior and that the effective noise during training helps find these flatter solutions (the authors acknowledge this). It seems reasonable to assume that it's easier to merge models on a flat part of the loss landscape. But, someone should probably check that this idea holds up and the inverted U behavior of the effective noise is interesting, so that's why I recommend a weak accept.\n- I assume that the flatness of the loss landscape/effective noise perspective probably holds for other model merging techniques. It would be nice to know if that's true for some more complicated techniques as well (e.g. Fisher-weighted averaging). There's always more techniques that can be tried, so I don't think this is a major issue.\n\nTypos/Grammar:\n- Line 250: should be \"ubiquitously\"\n- Line 252: should be \"demonstrating\"\n- Line 268: Could use a rephrase: \"Lastly, similarly as the learning rate, Appendix Figure 16 shows that a too large weight decay leads to failure in model performance and model merging.\"\n- Line 408: Grammar: \"Furthermore, as in linear interpolation merging, a “too large” becomes unstable.\"\n- Caption for Figure 8: Could use a rephrase: \"Merging pairs of similar and larger learning rates has the best performance.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7vXonFhPx0", "reviewer": "Reviewer_h7WG", "summary": "This paper investigates how optimization dynamics affect model merging success by introducing the effective noise scale as a unifying factor. The authors demonstrate that optimizer components (learning rate, weight decay, batch size, data augmentation) collectively modulate this noise scale, which exhibits a non-monotonic relationship with merging effectiveness. Through extensive experiments across multiple architectures (MLP, ResNet, DenseNet, Transformer, GPT) and datasets (SVHN, CIFAR, TinyImageNet, WILDS, TinyStories), the paper shows that intermediate noise levels enable optimal merging, while too little or too much noise degrades compatibility. The findings apply to both linear interpolation and task arithmetic merging approaches.", "questions": "- Very important question and reason I gave a 2 instead of a 4: Section 3.6 starts with \"In the previous sections, we analyzed settings where models are trained from scratch.\" Indicating that for all experiments in earlier sections the models were trained from scratch (from the same or different random initializations?). This is very surprising to me since the accuracy gain is positive in multiple experiments from these earlier sections which contradicts a large body of work on LMC and merging for models trained from scratch. [1] established that even networks trained on the same tasks aren't stable to SGD noise at initialization (i.e. no LMC for models trained from scratch) and there are multiple papers, such as [2, 3, 4], trying to permute / align the neurons of models trained from scratch to be able to merge them without a catastrophic drop in accuracy. Based on these works, successful merging of models trained from scratch shouldn't be possible, therefore I suspect there might be something wrong with the experimental framework. Perhaps the models weren't trained from a random initialization but from a pre-trained checkpoint, this would be fine, I don't think it removes much from the paper's value. **If this question is properly addressed and a valid justification is given I will gladly raise my score from a 2 to a 4 (and even further if more experiments are provided, see weaknesses).** The text also needs to me modified accordingly.\n- Typos: line 296 end \"+1?%\"; line 357 \"Figure 6 on the right\" but the subplots are one over the other\n- How does the effective noise scale interact with model capacity and dataset complexity? Are the optimal noise levels dataset/architecture-dependent?\n- Have you measured the actual gradient-noise covariance Σ_A to validate the proxy Ŝ?\n- Can you provide guidelines for practitioners to identify the \"sweet spot\" without extensive hyperparameter search?\n\n**References**\n- [1] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis\n- [2] Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa. Git Re-Basin: Merging Models modulo Permutation Symmetries\n- [3] Sidak Pal Singh, Martin Jaggi. Model Fusion via Optimal Transport\n- [4] Stefan Horoi, Albert Manuel Orozco Camacho, Eugene Belilovsky, Guy Wolf. Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Unifying framework: The effective noise scale provides an elegant unification of multiple optimizer hyperparameters, offering practitioners a single lens through which to understand merging compatibility.\n- Comprehensive experimental validation: The paper presents thorough empirical evidence across diverse architectures, datasets, and modalities (vision and language), demonstrating the generality of findings.\n- Systematic ablations: Each optimizer component is carefully isolated and studied with proper controls ensuring convergence.\n- Practical relevance: The work addresses the important practical problem of model merging, with direct implications for model soup and task arithmetic methods. While a lot of effort has been dedicated in recent years to designing better model merging methods, little to no attention has been given to how to training procedure itself affects mergeability despite clearly playing an important role.\n- Clarity: The paper is very well written and easy to follow. The claims are clear and the experiments are simple yet well explained and in direct support of the claims.", "weaknesses": "- Lack of theoretical analysis: The paper is primarily empirical without theoretical guarantees explaining why effective noise controls merging or why the relationship is non-monotonic. The connection to loss landscape geometry could be formalized more rigorously.\n- Limited scale: The authors acknowledge computational constraints prevented large-scale experiments. The largest models tested are relatively small by modern standards, raising questions about scalability. This is problematic since model merging techniques are often used to combine relatively larger models.\n- Proxy measurement: The experiments use the normalized proxy Ŝ = η/(B(1-μ)²) rather than directly measuring the full effective noise scale including Σ_A(θ_t) from Equation 3. This approximation's validity isn't thoroughly validated.\n- Non-actionable insights: The non-monotonic relationship with a \"sweet spot\" makes it difficult to provide clear prescriptive guidance. Practitioners still need to search for optimal hyperparameters rather than having a principled selection method. A more detailed analysis of *how* to find this sweet spot would be very useful.\n- Limited scope of experiments: While the experiments are extensive in that they adequately support the paper's claims regarding learning rate, weight decay, etc. they have a relatively narrow scope. There are so many interesting parameters to play with here which are also important training choices such as the actual **optimizer** used (the authors used AdamW for almost all experiments it would be valuable to compare against Adam or SGD in the same set-up); the effect of **momentum**, how does momentum affect the effective noise scale? stronger momentum minimizes the effect of each individual batch, potentially lowering gradients variance; and last but not least, the **learning rate scheduler** is also a very important factor which can have a huge effect on training and performance.\n- Limited merging settings: On the merging side, there have been many recent methods proposed (e.g. TIES, DARE, etc.), it would be a very valuable contribution to look at how the merging method interacts with the effective noise scale. Can certain merging methods counteract the effects of sub-optimal optimizer hyper-parameters? For example, is the learning rate less crucial if the merging method uses some sort of pruning mechanism? Also, the settings used are a bit simple by model merging standard, i.e. only two models are merged at the time and the models were trained either on the same, or similar tasks. In practice more than two models trained on tasks that can be very different are merged.\n\nIn summary, I like the use of the effective noise scale as a unifying framework to analyze the effect of the optimization procedure on mergeability, I think the effect of training on merging has been understudied. I also think the paper is very well written and clear and that the experiments accurately support the claims. However, there is limited theoretical analysis of this framework and the experiments are somewhat limited in scope (see Weaknesses). Also, while the unifying framework is interesting and is justified by the experiments, the insights are somewhat non-actionable. I like the paper but I think it would need either (1) a more in-depth theoretical analysis, (2) more experiments to help with practical impact either of other training factors (effect of optimizer, momentum, learning rate schedule) or (3) an analysis of the interplay between effective noise and merging method in more realistic merging scenarios, OR (4) more clear actionnable insights as to how to find the sweet spot without just running a large hyper-parameter sweep which practitioners already do on new tasks. I'd be happy to increase my score if at least one or two of these 4 points are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)