Generalized Linear Mode Connectivity for Transformers
Authors: Alexander Theus
Alessandro Cabodi
Sotiris Anagnostidis
Antonio Orvieto
Sidak Pal Singh
Valentina Boeva
Venue: NeurIPS 2025 oral ()
Keywords: Neural Network Merging
Linear Mode Connectivity
Model Re-basin
Parameter Space Geometry
Transformer
Permutation Invariance
Model Fusion
Primary area: deep_learning
TL;DR: We propose a unified framework for model merging that leverages multiple symmetry classes to enable low- and zero-loss interpolation between independently trained Transformer models, including Vision Transformers and GPT-2.

Abstract: Understanding the geometry of neural network loss landscapes is a central question in deep learning, with implications for generalization and optimization. A striking phenomenon is $\textit{linear mode connectivity}$ (LMC), where independently trained models can be connected by low- or zero-barrier paths, despite appearing to lie in separate loss basins. However, this is often obscured by symmetries in parameter space—such as neuron permutations—which make functionally equivalent models appear dissimilar. Prior work has predominantly focused on neuron reordering through permutations, but such approaches are limited in scope and fail to capture the richer symmetries exhibited by modern architectures such as Transformers. In this work, we introduce a unified framework that captures four symmetry classes—permutations, semi-permutations, orthogonal transformations, and general invertible maps—broadening the set of valid reparameterizations and subsuming many previous approaches as special cases. Crucially, this generalization enables, for the first time, the discovery of low- and zero-barrier linear interpolation paths between independently trained Vision Transformers and GPT-2 models. Furthermore, our framework extends beyond pairwise alignment, to multi-model and width-heterogeneous settings, enabling alignment across architectures of different sizes. These results reveal deeper structure in the loss landscape and underscore the importance of symmetry-aware analysis for understanding model space geometry.

Generalized Linear Mode Connectivity for Transformers
Alexander Theus" atheus@ethz.co ch
Alessandro Cabodi acabodi@ethz.co ch
Sotiris Anagnostidis sanagnos@ethz. ch
Antonio Orvieto 345 antonioStue. ellis.eu
Sidak Pal Singh contact@siddkkall.co com
Valentina Boeva 6. vboevaβethz.co ch
Abstract
Understanding the geometry of neural network loss landscapes is central ques- tion deep learning, with implications for generalization and optimization. striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-barrier paths, despite appearing to lie in separate loss basins. However, this is often obscured by symmetries in make functionally equiv-
parameter space-such as neuron permutations- alent models appear dissimilar. Prior work has predominantly focused on neu- ron reordering through permutations, but such approaches are limited in scope and fail to capture the richer symmetries exhibited by modern architectures such as Transformers. In this work, we introduce a unified framework that captures four symmetry classes-permuneee permutations, semi peniittemutations, orthogonal transfor- mations, and general invertible maps brondebin the set of valid reparameteri- zations and subsuming many previous approaches special cases. Crucially, this generalization enables, for the first time, the discovery of low- and zero-barrier linear interpolation paths between independently trained Vision Transformers and GPT-2 models. Furthermore, our framework extends beyond pairwise alignment, multi-model and width-heterogeneous settings, enabling alignment across chitectures of different sizes. These results reveal deeper structure in the loss landscape and underscore the importance of symmetry-aware analysis for under- standing model space geometry. Our code is available here.
Introduction
Understanding the geometry neural network loss landscapes is central both theoretical and practical advances in deep learning. A key observation driving this line of research is that inde- pendently trained models, despite converging to different points in parameter space, can sometimes be connected by low-loss paths, suggesting unexpected interrelation between seemingly isolated loss minima [Freeman and Bruna, 2016, Garipov et al., 2018, Tatro et al., 2020]. When these con- necting paths are approximately linear and maintain low loss throughout, the phenomenon is known as Linear Mode Connectivity (LMC) [Frankle et al., 2020, Entezari et al., 2021].
Equal advising. FTT Zürich, 2MPI for Learning Systems, $ELLIS Institute Tübingen, MMP for Intelligent Systems, TTübingen AT Center, "Swiss Institute of Bioinformatics, UUniversité Paris Cité, Institut Cochin, INSERM U1016
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LMC challenges the naive view of loss landscapes as consisting of distinct basins separated by high barriers. Instead, it hints at a repuratterrzatoonnndeee -inuucd redundancy: multiple minima that appear distant in weight space may correspond to functionally similar solutions, made to appear dissimilar due to symmetries the parameterization. Recovering LMC between models thus requires align- ing their parameters into commeetr--qquivalee configurations that reveal their underlying functional equivalence.
Several state-of-the-art approaches leverage discrete neuron permutations to achieve such align- ments, demonstrating LMC for relatively simple architectures like shallow multilayer percep- trons (MLPs) and, under certain conditions, VGG networks and ResNets [Singh and Jaggi, 2020, Ainsworth et al., 2022]. Such works highlight the central role of symmetries in understanding neural network landscapes and inspired the development of so-called model re-basin techniques: transfor- mations that port independently trained networks into a common valley the loss landscape.
Although foundational, permutation symmetries alone do not capture the full range of symmetries exhibited by modern architectures such Transformers. Our empirical findings show that relying solely on discrete reordering often fails to reveal low-loss paths, as persistent barriers can remain even after permutation alignment [Verma and Elbayad, 2024].
In this work, we broaden the symmetry lens. We introduce a unified framework that formalizes four classes of transforma- tions: permutations,
remiipermutations. orthogonal transfor- mations, and general invertible maps which, if appropriately used, can produce valid reparameterizations under which the original model functionality is preserved, while achieving low barriers. This generalization subsumes several existing align- ment techniques as special cases and provides the means un- cover LMC in Transformers by allowing to identify and lever- age their richer symmetry classes. Furthermore, this formal- ization also accommodates alignment between heterogeneous Transformers with differing architectural widths.
X
nn
Figure 1: By considering network symmetries beyond permutations, we can teleport two independently trained Transformers to the same loss basin. θ is projected into functionally equivalent representa- tion T(OB).
Our empirical results demonstrate for the first time low- and zero-loss linear connections between independently trained Vision Transformers and GPT-2 models, even in multi-model settings. These findings suggest that the Transformers' loss landscape is more connected than previously thought, provided the symmetries play are adequately modeled and exploited (see Figure 1). We discuss broader implications for ensem- bling, federated and continual learning, adversarial robustness, and the role of positional encodings in Appendix F.
This paper advances the understanding of loss landscape geometry and model alignment through the following key contributions:
Unified symmetry framework: We formalize a broad class of parameter transformations including permutations, semi-permutations, orthogonal transformations, and general invertible maps that preserve model functionality. This unifies and extends prior approaches under single theoretical lens and enables alignment across both homogeneous and heterogeneous architectures (Section 3).
ii LMC for Transformers:
We demonstrate, for the first time, low- and zero-loss linear paths between independently trained Vision Transformers and GPT-2 models using richer symmetry classes (Section and 5).
ii Multi-model mode connectivity: Our framework extends to the moilti-model setting, revealing that several independently trained transformers can be merged while maintaining near-zero interpolation barrier (Section and 5).
iv Soft alignment via continuous symmetries: We show that relaxing exact equivalence through differentiable, non-discrete transformations may improve interpolation outcomes (Appendix D).

---

2 Generalizing Linear Mode Connectivity
Let O be network parameters and (-) the induced function. Let (3, y) be the loss incurred by f[0] on data point (z, y). For a dataset D = {(11, M= define the empirical risk
[I)
Σ (T,
Definition (Linear Mode Connectivitty.. Consider two models 0 0g. both pre-trained on the same task (for simplicity). They are linearly mode connected if (()))(D) = [[) and the interpolation barrier
0x Onl (D) = A|A A + + (1- x)0n - (AL[OA]( + (1 [[On](D))
is near-zero for all À E []. The empirical barrier is:
0 (n) sup By On](D)
(1)
xc
and LMC is observed when A 0 (D) 0.
Because neural networks admit non-unique parameterizations [Li al., 2023], especially under different initializations, we seek invertible, function prettiin-rresrriing alignment mappings π O - θ that, when applied A and 0g. encourage LMC:
min ( To (OB))(D) s.t. f(TA((A)) fl0x). f(=B(OB) f[0 NA.8
These mappings need not be mere permutations. As discussed in Section 3, modern architectures (e.g., Transformers) exhibit richer, component-specific symmetries that can more effectively re- duce-op or eliminate- barrier (Appendix E). This broader view is not confined to networks with identical architecture and naturally extends to enable alignment across models of differing widths.
Extension to multi-model connectivity. the multi-model barrier
For M independently trained models {0 JM m=1 define
1 I(D) Bi(0m) m mmmmmll
sup AEAMLL
C
Am pm (D)
Σ Am _(
(2)
where A-1 = E R-Y Em Am 1}. Multi-model linear connectivity holds when (T(0))(2) O, indicating shared low-loss basin. As in the pairwise case, search mappings { satisfying f[Tm(0m). T (0..)) forming the basis for the merging procedure in Section 4.3.
for
f(0m] that minimize
3 Network symmetries under the generalized framework
To perform alignment and enable meaningful interpolation between independently trained mod- els, we must first understand the underlying symmetries that govern neural network parameter spaces. While prior work has focused primarily on discrete permutations, these represent only a slice of the broader symmetry landscape. In this section, we introduce hierarchy of network sym- metries persstattennnnnn semi pemmiieemmttaionn orthogonal, and invertible each allowing progressively more flexible identify where it arises neural architectures, and conclude by showing how these symmetries manifest in Transformer models, enabling their effective alignment and merging. More theoretical grounding and formal analysis of these symmetry classes can be found Appendix E.
transformations, summarized in Table 1. We define each class,
3.1 Symmetry classes
Permutation. Permutation symmetry refers to transformations that reorder inputs while preserv- ing the network's function. arises when components treat each input dimension independently, allowing neuron reordering without affecting the output. This symmetry is characteristic of elemen- twise operations such as GELU, sigmoid, softmax, tanh, where output values correspond directly to input positions.

---

Table 1: Hierarchical organization of symmetry classes neural network components, illustrating their associated transformation structures and representative examples. Each class is strict subset of the one below it.
Hierarchy
Class
Structure
Examples
S
Permutation
Permutation matrices
GELU, sigmoid, softmax, tanh RELU, LayerNorm, MHA
Sa
Semi-permutation
Sparse, stochastic matrices
Ss
Orthogonal
Orthogonal matrices (O)
RMSNorm
C S4 Invertible
Full-rank matrices
Lincar layer
"Sermi-permutation" symmetry extends permutation symmetry by allowing sparse, weighted mixing of input dimensions. is defined by matrices P E M where M 2 N, each column stochastic vector, and each contains most one positive entry. This symmetry arises in components that are linearly decomposable- that is, their functional output satisfies the following identity:
Semi-permutation.
f(x) f (ax) f ()1-a) Va [0, 1],
which holds for piecewise- linear functions such as RELU, PRELU, and the absolute value. These functions permit structured, non-permutative mixing of input channels while preserving functional- ity. As permutations are subset of cemi-permutations, many existing works focus on permutation symmetries of RELU based activation networks.
Orthogonal. Orthogonal symmetry allows transformations that preserve vector norms and an- gles-such rotations and reflections-withutt altering the network's behavior. This symmetry arises in components that normalize inputs across dimensions, irrespective of their orientation in space. RMSNorm is key example, remaining invariant under orthogonal transformations.
Invertible. Neural network components that preserve linearity admit functional equivalence under invertible transformations - the most general symmetry class in our hierarchy. This class includes layers whose learned transformations orthogonality. A key example is the attention mechanism, where the QK and OV circuits [Elhage al., 2021] (i.e., the projection weights for queries, keys, and values) be reparameterized via invertible maps without altering model behavior.
unconstrained by structural restrictions such sparsity
Approximate invariance and soft symmetries. The strict requirement of functional equivalence (x) (X) can be relaxed to approximate equality ['0'((]] (X), allowing continuous transformations to serve as soft symmetry operations. In this setting, soft permutations represented by doubly stochastic matrices, computed via entropic optimal transport Sinkhorn- based projections. Such relaxations enable more flexible neuron alignments- c.g., many-to-one or one-to-many mappings- and can improve test-time performance (see Appendix D).
3.2
Symmetries in Transformers
3.2.1 Feed-forward layer
The Transformer's feed-forward layer applies two linear projections with nonlinearity in between:
FF(x) We ò x + b,) + b2.
where ø is elementwise activation function, typically GELU RELU. When is not linearly decomposable- as GELU-the layer only admits a strict permutation symmetry. If ó instead is piecewise linear (c.g., RELU), the layer also admits broader semi pemi-permutation symmetry as described in Section 3.1. For any permutation matrix Per E R the reparameterizztion
W P W W
bj Ppbb
yields equivalent function:
FF (x) FF(x).

---

Acthatie
IRN
Wafw
Actinated
P,W,,o
HMSNiin
RM5Nkm
Atrezton
Atterice
Wx
W.
Mali-Hcall
W.
OO WaloFi
o O'W. P.
oo w.fo P.
Malt-Baai Alccioo
Pi w,
Iopax
Iepers
(a) Transformer layer.
(b) Transformer layer after projection.
Figure 2: (a) illustrates standard Transformer layer, and (b) shows its function- peeetiion--reevinngg equiv- alent after structured weight transformations. Attention heads are semi-permuted via blockwise multiplication with Pn using the operator, feedforward weights are permuted via Prr. and resid- ual stream weights orthogonally transformed via O. The figures inspired by Ashkboos al. [2024].
3.2.2 Multi-head attention
The multi-head attention mechanism of Transformers is defined as:
H
(QW?))KW))
MultiHead Q, K. V)
softmax
(VW.) w
vdk
head,(Q. VVK.V))
Intra-head. Each attention head exhibits two invertible symmetries, formally characterized in the QK and OV circuits by Elhage al. [2021]. The first arises from the product W.9 (W: which governs the attention scores (QK-circuit). The second appears in W. W which maps val- ues to outputs (OV-circuit). Because any invertible transformation applied within the query, key, value, output projections can be algebraically multiplied out, the behavior of the attention mech- anism uniquely determined by these two products. Consequently, we omit explicit invertible reparameterizatioos and treat the QK and OV circuits as canonical representations of each head.
Inter-head. Multi-head attention exhibits semi- permuttemmttaiion symmetry across heads. Since heads summed independently, their order is irrelevant (permutation symmetry). Moreover, heads can be decomposed linearly:
head X; QK, OV) = head X; QK, a OV) + head X: QK, (1 a) OV),
for any E R. This enables structured reweighting and mixing of heads via sparse, stochastic matrices PH. placing multi-head attention in the cemi-permutation class.
3.2.3 Residual
Recently, Ashkboos et al. [2024] observed that Transformers exhibit an orthogonal symmetry along the residual path, where the only non-linear component is the normalization layer. When RMSNorm employed, the model exhibits orthogonal symmetry directly (see Section 3.1). In contrast, when LayerNorm is used, it can be reformulated in terms of RMSNorm as follows:
LayerNormi Z) = RMSNorm ZM) diag(a) Ix
where and B are learnable scale and offset parameters, respectively, specific to each LayerNorm instance. The matrix M In 1,1 centers each row of Z by subtracting its mean. The matrix M and diag(a))) D can then be absorbed in preceding and subsequent linear layers.

---

Moreover, since the orthogonal transformation O can be chosen as a rectangular matrix with or- thonormal columns (O RMxN, M .). this symmetry enables width--epanding transforma- tions that preserve functionality. We provide detailed derivation of this property in Appendix E, and demonstrate its empirical utility in Section
Figure 2 illustrates how applying any orthogonal matrix O to the residual stream of a Trans- former-after RMSNorm reparameterizzaioon -ields functionally equivalent model.
4 Method
Building on the symmetry framework from Section 3.1, we now describe methods for aligning two independently trained Transformer models. The goal is to find canction-preservinn transformations constrained to the relevant symmetry classes- that bring the models into structural and representa- tional agreement.
Concretely, given a Transformer with layers, hidden dimension d,, residual embedding size d,, and H attention heads per layer, alignment involves estimating a set of commetry-constrainge matri- ces: (i) a global orthogonal matrix O E Rd- xd. for the residual stream; (ii) L permutation matrices PFF e Rdh xd, for aligning neurons in the feed-forward layers; and (iii) L semi- permuttantttooon ma-
trices PH E #xH for aligning attention heads across layers.
We consider three strategies for estimating these transformations. Activation matching aligns layers by comparing intermediate activations on a shared dataset. Weight matching aligns parameters di- rectly by minimizing distance under the symmetry constraints. Learned matching treats alignment as optimization problem, learning the symmetry-aware paraameterizations end-to-end. For ac- tivation matching, we use the method introduced by Verma and Elbayad [2024]. See Appendix for an ablation study of our proposed algorithms.
4.1 Weight matching
We adapt the weight-based alignment strategy introduced in Ainsworth al. [2022), which formu- lates alignment as an optimization problem over permutation matrices that maximize weight simi- larity across networks. The core intuition is that if two units across models have similar incoming and outgoing weights, they will likely implement similar functions and thus be aligned.
For Transformer
layers, adopt layerwise version of the "sum of bilinear assign- ments problem" (SOBLAP) proposed by Ainsworth et al. [2022]. Given weights W(4) and w in layer 4, we search for a permutation P!T that maximizes alignment:
ph
arg max (W(A), (W(B)) PES,
(B) OT) (W(A). (1 OW B
)F.
where S₂ is the set of permutation matrices, and O is the orthogonal matrix for the residual stream. This objective NP-hard, but can be approximated using coordinate descent strategy where each PFF is updated by solving linear assignment problem conditioned on adjacent layers.
For attention layers, we exploit the fact that QK and OV circuits are invariant under invertible repa- rameterizations [Elhage et al., 2021]. We define the QK circuit and OV circuit for each head i as:
QK, = (o T (OTT). OV, O w. WP O.
We then define cost matrix for aligning heads between models A and B using the Frobenius norm:
head = |[QK! (4) QK! BB) | + [[ov(4) o
We solve a linear assignment problem to obtain the head-level permutation matrix PHl
For the residual stream, we estimate one global orthogonal matrix O by solving the Procrustes problem:
arg min
|| (4) R O
x O 0=1
where (4) and R are the weights along the residual path collected from both models. The closed-form solution is given by the SVD of R (B)TR(A)

---

4.2 Learned matching
While activation and weight matching rely on static alignment criteria, learned matching directly optimizes the alignment parameters using task loss supervision. Rather than aligning weights or activations explicitly, we treat the symmetry transformations themselves as trainable parameters, to be learned end-to-end (see Algorithm 1).
Algorithm Learning matching via task loss
Require: Base models A Os: Dataset D; Iterations No:: Adam optimizer (1r = 11)..
I: Initialize latent matrices Zrn, Zu. Zo using weight matching.
2: for t to Ne do
Project: (Phr, Pi, 0 + [PROIPERM( ZFr) PROOOORTI)
4: Align: Oaligmed (0p: Ptr, 0).
applies transformations
5: Sample: X ~ 20.44, 0.6)
6: Sample batch B {(X,, Y.) Y.) from D.
7: Interpolate: IINTERP +- À - 0 + -) 0 galianed
Objective: J 1
B Exx, YeB CeNEONNERR: X, Y).
Gradients: (92z: 92-) +- V Z11 Zo)J.
b STE for Zre Zu
10:
Update: For (FFHHO), Z& Adam Ze. 92,, (1).
11: end for
12: Final projections: PA PROSPERM Zr), Pa PROIPERM (Z)), O' PROJORTH(Zo) (Zo).
13: return Psr PE,O".
We introduce unconstrained latent matrices Zrr, Zni and Zo, which are projected to the respective symmetry classes at each forward pass:
PFF (ZFF). Ри PROIPERM Zn), O PROJORTH(ZO).
(5)
We initialize these latent matrices using the weight matching procedure described in Section 4.1. For the permutation matrices, PROJPERM projects each matrix to the nearest permutation via the Hungarian algorithm; we use a straight-through estimator allow gradients to flow through the relaxed Z parameters. The orthogonal projection PROJORTH computes UVT from the SVD of Zo UΣV This operation is fully differentiable.
At each step, we interpolate between model A and the reparameterized model using a uniformly randomly sampled interpolation coefficient X (00.4, 0.6):
PINTERP A A (1 A) ग (0
where x--) denotes alignment of model B. We then compute the original training loss on OINTERP and backpropagate through the alignment transformation.
This approach encourages Transformer similarity through task performance, rather than explicit similarity of parameters or activation vectors, thus enabling the joint optimization all symmetry- aware transformations over all network layers.
4.3 Multi-model merging
Universe matching. Following Crisostomi et al. [2024], we build a shared universe U that serves as a common reference for all models. Given trained models ......, OMs initialize ( O, with any seed model (s {1....,My M)). For each iteration t 1:N.
M
T - ALIGN(O,,.. U(-1)) Vm,
U
T (0m)
M3
1===
7

---

Table 2:
Loss barrier (lower better) for each alignment method. Reported values are mean standard error, with rank in parentheses. Models are width-homogeneous. and only two models are aligned with each other. Rows highlighted in color correspond our methods, using the same colors as in subsequent figures; bold text indicates the hest performance for each dataset.
VIT
GPT-2
Method
CIFAR-10
CIFAR-I00 100
Tiny ImageNet
Tiny Shukespeare
BookCorpus
Vanilla averaging
1.69 0.07
2.46 0.04 (5)
2.84 0.02
2.02 +002000 0.12
4.34 = 0.09
Activation matching
1.27 - 0.13 (1)
2.11 0.17 (4)
1.86 4 0.10 (4)
1.43 0.16
1.05 = 0.13 (4)
Weight matchhnng (ours)
(3)
(2)
Learned matching (permutatioos)
0.45 + 0.02 (3)
0.53 = 0.07 (2)
0.29 0.02 (2)
0.63
0.177
1.60 1 0.04 (3)
Learned matching (ours)
0.00 (1)
0.00
0.00
0.00 = 0.00 (1)
0.00 (1)
0.42 + 0.01 (1)
7.5
5.9
- Leareed
-a- Leamndd
- Weight + Varik
- Weight
5.8
7.0
5.7
Loss
6.5
5.6
6.0
5.5
5.3
0,
0.2
0.4
0.6
0.8 (l
6..
0.2
0.4
0.6
0.8
m16y))
8.
0.2
0.4
0.6
0.8 n10
Interpolation confficient (A)
Interpolation confficient
Interpolation confficient (A)
(a) Tiny ImageNet.
(b) Tiny Shakespeare.
(c) Tiny Shakespeare.
Figure 3: Loss along the interpolation path for (a, b) width- hodth-homogeneouss and (c) width- heterogeneous alignment. In (c) O2 denotes the smaller model (reduced embedding dimension; the reduction ratio is indicated the label), which is aligned to the larger one (0,, with embedding dimension 512). Learned matching was used for the width-heterogeneous results.
Each ALIGN step estimates a function- peretiin-reeeevingg transformation Tm constrained to the symmetry classes Section 3.1. The evolving anchor (()) aggregates aligned parameters into unified coor- dinate system, and after N iterations the resulting {mm (N)) approximately minimize the multi-model barrier in Eq. (2).
Learned refinement. We refine { mm extended to M-way mixtures. Sampling X Dirichlet(aamm) with 0.1, minimize
(N) using the learned matching method from Section 4.2,
M
MM
J J=Ex
Am mm (0m) (
Σ m=m
Am Cl (lTTm(0... ( (0..) I(D)
which directly drives [m (((mmm(m (DD) toward zero. Gradients are backpropagated through m.. via the projection operators of Section 4.2.
5 Results
We evaluate the proposed model alignment methods on two Transformer architectures: Vision Transformers (ViTs) and GPT-2, spanning vision and language tasks. To measure LMC, we com- pute the loss barrier between two models O and O as defined in Equation on the test split. A lower barrier indicates better connectivity; the optimal value is B 0. See Appendix C and D for further results.
Two-way model alignment. Table 2 and Figure 3 summarize alignment between two indepen- dently trained models. Our learned matching method consistently outperforms all alternative ap- proaches, achieving zero near-zero barriers every dataset except BookCorpus. The weight- matching variant-fulle unsupervised and training-free-rr yields substantial reductions and of- ten surpasses permutation-only learned matching (which is akin to the STE-based approach in

---

(a) Vanilla averaging.
(b) Weight matching.
(c) Learned matching.
Figure 4: Linear mode connectivity surface between three CIFAR- 10 models for different alignment strategies. Colors show the relative loss deviation from the linear interpolation baseline across the simplex spanned by π ())), ()))), and (Oc).. The dashed contour (e 0) marks points where the loss equals the linear baseline.
Git Rebasin). The degree of connectivity among Transformer minima is striking: for compari- son, achieving zero barrier for ResNet-20 CIFAR-10 in Git Rebasin required 32x width increase [Ainsworth et al., 2022]. Figure 3e further considers width-heterogennous alignment, align- ing larger model with embedding dimension 512 to smaller counterparts. Despite the architectural mismatch, the interpolation paths remain at or near zero barrier, indicating connected regions across Transformer sizes.
While image tasks reliably attain R O-including Tiny ImageNet-the language experiments, particularly BookCorpus, exhibit higher barriers. This may reflect imperfect alignment, gen- uinely disconnected minima. Juneja et al. [2023] show that, for NLP classifiers, fine-tuning can yield multiple basins associated with distinet generalization strategies (e.g., lexical-overlap vs. syn- tactic cues). Thus, while models using the same strategy might be linearly connected, linear paths across strategies exhibit barriers. This suggests the non-zero barriers BookCorpus may reflect fundamentally different solutions rather than merely suboptimal alignment.
Multi-way model alignment. Figure 4 extends the analysis to aligning three independently trained CIFAR- 10 models. We visualize the loss over the simplex spanned by T(),). T(O and n (Oc). Relative vanilla averaging, both weight matching and learned matching flatten the surface toward the linear-interpolation baseline, with learned matching producing the broadest re- gion of near-zero deviation. These results suggest that procedures connect only pairs of solutions but also carve out a shared low-loss manifold spanning multiple models.
Understanding the gap between weight- and learned matching
Weight matching an attractive, data-free alignment method: it operates directly on parameters, requires no training data, and is computationally tings such as federated learning where data sharing is limited. However, typically yields higher interpolation barriers than leamned matching, which optimizes alignment with task-loss supervision. This raises a key question: where does the gap arise, and can parameter-only methods close it?
ffficient-making it practical for set-
3r
To locate the gap, we analyze the learned transformations. Across runs, the permutations of attention heads and feed- forward blocks found by weight matching remain essentially unchanged under learned matching. The difference con- centrates the orthogonal map that aligns the residual stream. As shown in Fig. 5, the angen-angles of Own are broadly distributed over [0, 2m), indicating nearly arbitrary rotations/reflections. whereas the relative correction Oan OLм Ow concentrates near mod 2x, i.e., learned matching makes small, targeted refinements.
Weight matching
Learned matching
Figure 5: Eigenvalue spectra of Own and Oair = OLMOW The latter indicates small refinements from learned matching.

---

In short, learned matching primarily refines the orthogonal alignment provided by weight matching. This suggests a promising avenue: better, data-free estimation of O (e.g,, with structural priors or spectral objectives) could narrow most of the performance gap without full supervision.
7
Related work
Linear Mode Connectivity (LMC)., LMC describes the existence of low-loss linear paths between independently trained networks. Early work on mode connectivity [Garipov et al., 2018, Draxler al., 2018] identified nearly constant-loss non-linear paths, suggesting SGD solutions lie on connected manifold. LMC focuses on linear interpolations: Entezari et al. [2021] conjectured that permutation symmetries account for the observed disconnection, and once resolved, SGD solutions lie in a single basin. While a formal proof remains open, empirical evidence increasingly supports this view (see below). Recent work also links global LMC to layer-wise linearity [Zhou al., 2024, Adilova et al., 2023].
Model merging and symmetry alignment. Several approaches have exploited the symmetry struc- ture of neural networks to enable one-shot model merging. OT fusion [Singh and Jaggi, 2020] optimal transport (OT) problem, computing the Wasserstein barycen- ter between corresponding layers based on activation or weight similarity. This enables data-driven model fusion that outperforms naive averaging and can approximate ensemble performance after moderate finetuning. Related methods include Liu et al. [2022], Akash et al. [2022]. Subsequently, Git Re-Basin [Ainsworth et al., 2022] considers three alignment methods (two of which are highly similar to prior work of [Singh and Jaggi, 2020]): activation matching, weight matching (WM), and a dearning-basee variant using staight-throus estimators (STE) [Bengio et al., 2013]. WM data-free but loss landscape-agnostic: STE backpropagates through soft permutations and depends critically on WM for initialization. This approach achieves zero-barrier interpolation for modified (widened) ResNets with LayerNorm, result later extended to BatchNorm networks via statistical recalibration [Jordan et al., 2022].
casts neuron alignment
Ito et al. [2025] show that WM aligns dominant singular directions without altering singular values, thereby enabling LMC while preserving functionality. Consequently, exhibits higher transitivity than STE, which may overfit local loss geometry. Sinkhorn Re-Basin [Peña et al., 2022] instead optimizes relaxed permutations using Sinkhorn operator and implicit differentiation [Eisenberger et al., 2022] to reduce interpolation barriers.
Transformer-speciffe extensions have also emerged: Imfeld al. [2023] adapts OT fusion to handle multi-head attention and residual structures using Sinkhorn- soft alignment, while Verma and Elbayad [2024] uses correlation-base matching to align BERT models. Both show reduced loss barriers compared to naive averaging, though non-zero barriers remain.
Beyond merging identical-task models, Stoica et al. [2024] tackles multi-task model merging by aligning both inter- and intra-model features, revealing redundancy in neural representations. Cycle- consistent alignment across multiple models is explored in Crisostomi et al. [2024], enforcing con- sistency of neuron permutations to support multi-way merging.
8 Conclusion
We introduced unified framework for symmetry-aware model alignment that captures a broad class of transformations- permuuttiinn,,nininnonnooo perrmutatto,o orthogonal, and general invertible maps. This generalization subsumes prior re-basin techniques and enables, for the first time, the discovery of low- and zero-loss linear interpolation paths between independently trained Vision Transformers and GPT-2 models. Our empirical results show that broader symmetry classes are essential to un- covering the connectedness of modern neural network loss landscapes. These findings highlight the importance of modeling and leveraging richer symmetries in reparameterization to advance our un- derstanding of neural network geometry, with potential implications for model ensembling, transfer, and interoperability. A more extensive discussion of the theoretical implications and broader im- pact of these results provided in Appendix F, and the practical limitations of our framework are summarized in Appendix A.
10

---

Acknowledgements
Alexander Theus and Sidak Pal Singh acknowledge the financial support from the Max Planck ETH Center for Learning Systems. Antonio Orvieto acknowledges the financial support of the Hector Foundation.
References
Linara Adilova, Asja Fischer, and Martin Jaggi. Layerwise linear mode connectivity, arXiv preprint arXiv:2307.06966 2023.
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209. 04836, 2022.
Aditya Kumar Akash, Sixu Li, and Nicolás Garefa Trillos. Wasserstein barycenter-based model fusion and linear mode connectivity of neural networks, 2022. URL htttt://pa/xxiii org/ abs/ 2210.06677
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL wwgpeiii.rrrraaeeoeoo org/
attps://arxiv..
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 138.3v:330.33322 2013.
Donato Crisostomi, Marco Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodolà. c²m²: Cycle-consistent multi-model merging, 2024. URL https: /tapxi///axxvv...///// orgo aaaaabas/555 17897.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1309-1318. PMLR, 10-15 Jul 2018. URL https: hatpa://ppoceeding/. mlr. wwwpspresseeeeee /ataa//vee/draeer111eee 1a. 1a..8a.htmtt html.
Marvin Eisenberger, Aysim Toker, Laura Leal- Taixé, Florian Bernard, and Daniel Cremers. A uni- fied framework for implicit sinkhorn differentiation, 2022. URL https: //axxvvvvvvv wweexivv.or//...o org/ /bs// 2205. 06688.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1): 2021.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint 1rXiv:2110.0620 2021.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference Machine Learning, PMLR, 2020.
pages
Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint arXiv: 161.05511 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information pro- cessing systems, 31, 2018.
Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport, 2023.
Akira Ito, Masanori Yamada, and Atsutoshi Kumagai. Analysis of linear mode connectivity via permutation-based weight matching: With insights into other permutation search methods, 2025. URL https:/ :/ /arxiv.... http://b//22220555... 04051.

---

Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur, Repair: Renor- malizing permuted activations for interpolation repair. arXiv preprint arXiv:2211.08401, 2022.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity
reveals generalization strategies, 2023. URL https :/tpppa:///// //tpxaaxvvvv////////////a///aaaa orgg www.///g//////o//e//..///..//ee/ aaaaabaa//22/ 11241...
Weishi Li, Yong Peng. Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309. 2023.
Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications model ensemble and federated learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 13857-13869. PMLR, 17-23 Jul 2022. URL https://prooeeedngg mlr wwwppp/sssessp//peeeeeeeep vttess//v12/li222::11 htmll.
Fidel A. Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation, 2022. URL
https://arxiv wwwp/iiivrrgeeeeooe
12042.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Infor- mation Processing Systems, 01220045-22055 2020.
George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training, 2024. URL htttt:s/p///xiii org/ abs/2305. 03053.
David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima, 2021. URL https: www.ii.oog//bba
04448.
N. Joseph Tatro, Pun-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Opti- mizing mode connectivity via neuron alignment, 2020. URL 02439.
org/abs/2003
Neha Verma and Maha Elbayad. Merging text transformer models from different initializations, 2024. URL httts:://aaxiiiii hwtp/abb/2403.0000
Zhampeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Table 3: Configuration and training details for Vision Transformer (ViT) and GPT-2 across two
datasets each.
Component
VIT
GPT-2
CIFAR-
Tiny ImageNet
Tiny Shakespeare
BookCorpus
Transformer layers
Attention heads
Embedding dimension
256
384
256
512
MLP hidden dimension
512
768
1024
2048
Patch size
4 4
8 8
Sequence length
512
Training epochs
150
150
100
Batch size
128
128
32
64 AdamW
Optimizer
Adam x 10 14
Adam
Adam
Learning rate
x 10
x 10
2.5 x 10 0.01
Weight decay
x 10-3
0.05
0.01
Learning rate schodule
Cosine annealing RTX 2060
Cosine (warmup) Ix RTX 4090
Cosine (warmup) Ix RTX 4090
Cosine (warmup) 4x RTX 4090
Hardware
A Limitations
While the research introduces unified framework for symmetry-aware model alignment in Trans- formers. it presents some limitations and areas for future exploration. Our methodology introduces generalized notion of LMC that, in some cases, requires reparameteriization (c.g.. RMSNorm repa- rameterization or multiplying out antra-head dependencies). Although it preserves functional equiv- alence, it alters the underlying network architecture. The empirical results for language models were based on a smaller version of GPT-2 language models with reduced parameters due to re- source constraints (Appendix B), indicating the need for evaluation on larger, more contemporary language models. The current scope focused on aligning pairs of models that have been pretrained on the same task, and further investigation could extend these methodologies to models trained on (partially) different tasks [Stoica et al., 2024]. Additionally, while additional results ( (Appendix D) demonstrate that soft permutations can improve test-time performance of the merged model, more work is needed to fully refine soft relaxations of symnetry operations to improve the performance of aligned models. Finally, due to computational constraints, the study utilizes standard benchmarks such as CIFAR- 10/100 and Tiny ImageNet for Vision Transformers and SinyShakespeare and Book- Corpus for GPT-2; exploring performance on broader or more complex range of benchmarks could further validate the findings.
B Experimental details
We provide implementation and training details for the Vision Transformer (ViT) and GPT-2 mod- els used in our experiments. Table 3 summarizes the key architectural parameters, optimization settings, and hardware configurations for both models. Additional details on data preprocessing, augmentation, and training protocols are provided in the following subsections.
B.1 Vision Transformer (VIT)
We trained two Vision Transformer (ViT) configurations, one for CIFAR-10/100 and another for Tiny ImageNet. For CIFAR-10/100, the model consisted of 6 transformer layers and 8 attention heads, with an embedding dimension of 256 and feedforward (MLP) hidden dimension of 512. Input images were divided into non-overlapping patches of size 4 x 4.
For Tiny ImageNet, the model was scaled up to 8 transformer layers with the same number of atten- tion heads (8). The embedding and hidden dimensions were increased to 384 and 768, respectively, and the patch size was enlarged to 8
accommodate hegler-resolution inputs.
All ViT models were trained for 155 epochs using the AdamW optimizer with an initial learning rate of 3 10-1, For CIFAR-10/100 we applied weight decay of 1 x 10-3, while for Tiny ImageNet
13

---

we used 0.05. Both configurations used cosine learning rate scheduling; the Tiny ImageNet setup additionally employed a short warmup phase at the start of training.
For CIFAR-10/100, standard data augmentation was applied, including random cropping with padding (crop size 32, padding 4), horizontal flipping, and color jittering (brightness, contrast, and saturation adjustments of 0.4, and hue variation of 0.1), For Tiny ImageNet, more advanced augmentation pipeline was used, consisting of random resized cropping to 64 x 64 (scale range (0.8, 1.0), aspect ratio (0.9, 1.1)), random horizontal flipping with p = 0.5, and the AutoAugment policy for ImageNet. All images were normalized using the -ataset-specific mean and standard deviation.
Additionally, for Tiny ImageNet, we applied post-hoc temperature scaling rescale the model logits and improve confidence calibration.
Training for the CIFAR-10/10 model was performed on single NVIDIA GeForce RTX 2060 GPU, while the Tiny ImageNet model was trained on a single NVIDIA RTX 4090 GPU. On the Tiny ImageNet test dataset, the models achieve an accuracy of 44.19 + 0.17 and a calibrated loss of 2.54 + 0.02. For the CIFAR-10 test dataset, they obtain an accuracy of 83.81 + 0.44 and a loss of 0.57 + 0.01.
B.2 GPT-2
Two scall-scale GPT-2 models were trained: one on the Tiny Shakespeare corpus and another on the BookCorpus dataset.
For Tiny Shakespeare, the model consisted of 6 transformer layers with 4 attention heads, embed- ding dimension of 256, and an MI.P hidden dimension of 1024. Sequences were truncated or padded to 256 tokens. Training was performed for 100 epochs with batch size of 32 using the AdamW optimizer. The learning rate was set to x 10-4 with a cosine learning rate schedule and warmup phase, and a weight decay of 0.01. Additionally, early stopping was performed with patience of 5 epochs to prevent overfitting. Training was conducted on single NVIDIA RTX 4090 GPU, The model achieves
loss of 5.28 0.00.
For BookCorpus, the model used 6 transformer layers and attention heads, with an embedding dimension of 512 and MLP hidden dimension of 2048. Tokenized sequences were limited to 512 tokens using the GPT-2 tokenizer, with -ndou--seueence tokens used for padding. The model was trained for 5 epochs using the AdamW optimizer with initial learning rate of 2.5 10-1, 5% warmup ratio, and weight decay of 0.01. Mixed-precisse (fp16) training was enabled to improve throughput. This model was trained across four NVIDIA RTX 4090 GPUs with an effective batch size of 64 (16 per device). On this dataset, the models obtain a loss of 3.55 + 0.01 on the test split.
B.3 Merging
To merge the ViT and GPT-2 models, use the same setup as for training the individual models.
C Ablation study
C.1 Learned matching
C.1.1 Coefficient sampling
In Section 5, we already ablated the effect of using learned permutations in place of orthogonal maps, highlighting the importance of capturing more general alignment symmetries. Here, further investigate how the choice of interpolation coefficient sampling strategy influences performance (see Line 5 in Algorithm 1). Specifically, we compare four sampling schemes:
Fixed interpolation (A = 0.5): A deterministic strategy where is always set to 0.5, representing balanced average of the two models.
Uniform sampling [0.4, 0.6]: A narrow uniform distribution centered at 0.5, introducing small random perturbations around equal weighting.
14

---

0.63
samping
Uniform somplnng BJ,, ud)
Reed ineepolation wd
Loss barrier
Loss
2.20
Orthegonal matching Permutatt oo matebing
0.55
0.30
e(Modal
4
12
Interpolation coefficient (A)
Iteration
Figure 6: Loss curves for different strategies of sampling the interpolation coefficient A. Each line shows the mean loss, with shaded areas de- noting standard deviation. Experiments were ried out on ViTs trained on CIFAR-10.
Figure 7: Loss barriers over multiple iterations of weight matching, comparing orthogonal per- mutation matching for the residual weights. Ex- periments were carried on ViTs trained on CIFAR-10.
Train Loss (WM Init)
Test Loss (WM init)
0.80
Train Loss (Identity init)
Test Loss (Identity init)
0.70
Loss
0.60
0.50
0.40
0.30
0.20
5
10
15
Epoch
Figure 8: Comparison of train and test loss over epochs for learned matching, using weight matching ("WM init") versus identity initial- ization ("Identity init"). Using weight matching point yields consistently lower loss. Model: VIT. Dataset: CIFAR-10.
initialization
Uniform sampling [0.0, 1.0]: broad uniform distribution over the entire interpolation range, allowing any convex combination of the models.
Gaussian sampling (0055, 0.1): A stochastic strategy that samples from a Gaussian cen- tered at 0.5 with standard deviation 0.1, clipped to the interval [0. 1].
We visualize the impact of these sampling strategies in Figure 6, where we report the mean and stan- dard deviation of loss values along the interpolation path for ViTs trained on CIFAR-10. Notably, both uniform and Gaussian sampling result in relatively high loss barriers, indicating unstable inter- polations, particularly around À 0.5. In contrast, narrow uniform sampling and fixed interpolation produce lower loss near the midpoint. However, fixed interpolation exhibits significantly higher loss the surrounding regions, leading to elevated barriers for certain random seeds and greater variance overall. Based on these observations, we adopt narrow uniform sampling in our implementation, as it offers more consistent performance across different random seeds.
C.1.2 Initialization
The results reported in Section 5 use weight matching to initialize the function-preserviin permuta- tion and orthogonal transformations. In this section, we evaluate the effectiveness of weight match- ing an initialization strategy by comparing it to a baseline with no prior matching.
15

---

1.20
Hard Permutations
Permutations Uni
Soft Permutations U2-Ua
Loss
0.80
0.60
Model A
<Modee B)
Interpolation coefficient (A)
Figure 9: Loss barrier across the interpolation path using soft permutations. Plotted are mean test loss and shaded regions representing +1 standard deviation over seeds. Model: VIT. Dataset: CIFAR-10.
We train the learned matching procedure for 15 epochs and report both training and test losses for ViT' trained on CIFAR-I0 across four random seeds (see Figure 8). Initializing with weight matching leads significantly lower training and test losses. Notably, models reach zero test loss barriers within just one to two epochs of training. In contrast, training without any prior matching results in consistently higher losses, with positive loss barriers remaining even after 15 epochs.
Evidently, learning the transformations alone is insufficient; a strong initialization provided by weight matching is essential for achieving low loss barriers and faster convergence.
C.2 Weight matching
Our proposed iterative weight matching algorithm, while not outperforming learned matching, re- quires significantly fewer computational resources and has the added advantage of being completely data-free. Nevertheless, several questions remain. In particular: (1) Does orthogonal matching pro- vide improvements over permutation matching, observed in the learned variant (see Table 2)? (2) How many iterations are needed for convergence?
To address these questions, we evaluate the loss barrier across different numbers of iterations for both permutation and orthogonal matching. Results for ViTs trained on CIFAR-10 are shown in Figure 7. A stark contrast emerges: for orthogonal matching, the loss barrier steadily decreases, converging after five iterations with a substantially reduced barrier. In contrast, permutation-bant matching shows significant improvement across iterations.
These findings confirm that orthogonal matching provides a more effective path to convergence than permutation matching in the data-free setting, reinforcing its role in our proposed algorithm.
D Soft-permutations
Soft permutations provide continuous relaxation of hard (i.e., exact) permutations by allowing convex combinations of layer units. Formally, we define them as doubly stochastic matrices (i.c., matrices with non-negative entries whose rows and columns each Birkhoff polytope, whose vertices correspond to the set of hard permutation matrices. This relax- ation enables mappings that go beyond strict one-to-one neuron correspondences, offering greater flexibility in aligning network representations. We show results in Figure 9.
one) lying within the
This section details the methodology for learning such soft permutation matrices to align the pa- rameters of two models, 0, and 03. Unlike hard permutations Algorithm which use a Straight-Through Estimator (STE), here soft permutations are derived from learnable latent param- eters and made doubly stochastic using differentiable Sinkhorn normalization. The latter approach performed better than STE in our experiments.
16

---

D.1 Objective
The primary goal to learn a set of layer-wise latent matrices (Zi}.. These latent matrices are transformed into soft permutation matrices (Pi} (where Pt Sinkkorn Zi))) which are then used to construct transformation æ. This transformation aligns one model to another, e.g., yielding galigned л B: {P.)). The optimal latent matrices [ } are those that minimize the empirical risk (loss) of interpolated model over batch B drawn from the dataset D:
[ = arg min
[AEE (10 + (1 - X) <(0s; (Sinkhorn(xxpl Z.)) (X,Y) (X
(Zi)
B
where À is an interpolation coefficient, typically sampled uniformly at random (c.g.. À 04 (10.4.000 0.6] as in Algorithm ~ L[00, 1)). The optimization is performed with respect to latent parameters Z
D.2 Parametrization and initialization of latent matrices
Parametrization from latent matrices. The soft permutation matrices Pt (which must be doubly stochastic) are optimized directly. Instead, we optimize underlying unconstrained latent matrices Zi To obtain Pt from Zr:
1. First, non-negative matrix P1 is generated using an element-wise exponential map:
P exp( Z1)
This ensures all entries are positive, a requirement for the Sinkhorn algorithm, and allows unconstrained optimization Zj as gradients can flow back through the exp function.
2. Second, Pt is projected onto the Birkhoff polytope B using the differentiable Sinkhorn- Knopp normalization (detailed below in the learning process) to yield the doubly stochastic soft permutation matrix P..
Thus, P. = -Sinkhomnexxp(22 Z:)), and Zi are the parameters learned via gradient descent.
Initialization of latent matrices Z The latent matrices Zi are initialized by first constructing target matrix PP, which represents a desired initial state for exp(Z - Sinkhorn normaliza- tion.
A baseline for random noise is established using Xavier-scheme variance: let o² fan-in san-in+tien in an--ous
A scaling factor for noise is defined as =v33, where is coefficient tuning the amount of noise be injected. random noise matrix prund is then sampled, with entries, for example, prand) Uniform(0, 2a). Note one can add small positive constant to P! ensure all entries are strictly positive.
The target matrix PM is constructed based on one of the following strategies:
Random initialization: The target matrix is formed directly from the random noise:
pp
prand
From pre-computed permutation: If an initial hard permutation matrix Piiit is available (e.g., from weight matching, as used for initializing ZFF, ZH Algorithm I), the target matrix is formed by perturbing this known permutation:
P. = Pinit prand
The initial latent parameters Z are then set by inverting the exponential parametrization, i.e.. zí log( (P)), applied element-win wise to the strictly positive target matrix PP. This ensures that exp) = P at the start of the learning process.
17

---

(a) Obtain the non-negative matrix from the current latent parameters: P. exp( Zi). D.3 Learning process
The latent matrices Z for all relevant layers are optimized over Naer iterations. In each iteration t, using the Adam optimizer with learning rate
1. Soft permutation matrix computation:
For each layer I:
(b) Project P1 onto the Birkhoff polytope using iterations of the Sinkhorn- Knopp algo- rithm to get the soft permutation matrix P. Let Q = P.. For l = I..... K:
Q (k) diag
Q
Q
(k-1)
(Normalize rows)
1
Qi
e1(h) diag
1
(Normalize columns)
Q
(k)
The resulting soft permutation Pt ((()) This Sinkhorn normalization process differentiable with respect to Pt.
2. Model alignment and interpolation:
Align model Og using the computed soft permutations {Pth: Sample an interpolation coefficient (c.g.. À 4(0.4, 0.6] as Algorithm I, from U(0, 1] based on desired outcomes, see discussion below). Form the interpolated model: nigned
O + T00;; (Pi)).
OINTERP A + (1 X
3. Loss computation and parameter update:
Compute the empirical cross cross-entropy loss I sampled batch B {(X,, YD) from dataset D:
I J+
LCE(OINTER;; X Y) (X,Y)EB
Σ
Compute gradients of the loss with respect to the latent parameters: (..., V V2,J,.... Up- date each latent matrix Zi using the Adam optimizer:
Z Adam Z₁, I,,
Since the exponential mapping and Sinkhorn normalization are differentiable, gradients flow directly back to the latent parameters Zz.
After Nierr training iterations. the final optimized latent matrices {Zí} are used to yield the learned soft permutation matrices PP- = Sinkhorn (exp Z;))}.
D.4 Remarks.
In explorations, these learned soft permutations applied to align components within Trans- former architectures, specifically targeting attention heads and MLP layers between independently trained models. Empirically, this approach often yields meaningful improvements in the test-time performance of the merged (interpolated) model, particularly at the midpoint the interpolation path (A As 0.5). However, key issue of using soft permutations is that exact functional equivalence the aligned endpoint (i.e., for galigned compared to the original 0g) is generally not maintained. We observe that the degree of functional equivalence at the endpoint can be improved by modifying the sampling strategy the interpolation coefficient À during the learning process for instance, by sampling À uniformly from the entire [0, 1] range, which allows for more direct optimization of the transformation the aligned endpoint. Nevertheless, this adjustment typically involves a trade-off: while endpoint functional equivalence may improve, the performance gain observed at the midpoint of the interpolation path might be reduced compared to when sampled more narrowly (e.g., from 24 (0.4, 0.6)). We leave further exploration of this approach to future work.
18

---

E
General symmetries of network components
Consider a feed-forward network with layers, where the 1-th layer computes activation vector of (W with X being the input. The full parameter set is O (Wi) Such network implements the following composed mapping:
Y WW C00 WL o.... WiX
with:
W1
0
0
0
W
0
0
W
We define transformation the alignment that reparametrizes the network to account for its inher- ent symmetries, mapping the original parameters 0 to an equivalent (or approximately equivalent) set 0' = (W7) with the goal of achieving linear mode connectivity- -ie. maintaining low loss barrier along the interpolation path.
The reparamcterization imposed by is typically achieved in practice by defining set of (ap- proximately) invertible matrices { where S E IRdx acts as change of basis for the di-dimensional hidden representation ag. We fix the input and output bases by setting So La and Sy = Idz. The transformed weights are then given by:
w = S for E {1....,
The aligned network function becomes:
[()'(xx= oz (W. (W 11 W (S (SX
Exact functional invariance, XX) = (x) is guaranteed if S = I and the activation functions d are equivariant with respect their corresponding transformations S,, i.e. Sia oi(S, for E fl...., L- -- 1}.
A common ensuring such equivariance is when o; is element-wiss activation function (e.g., RELU)-as seen in Section 3. and S; is permutation matrix Pt. In this scenario, the set of orig- inal parameters O be represented
seen
block diagonal matrix dang diag( (Wi W,.... Wt). The transformation O'lng (fding with blocks W P.W.P. be expressed by defining block diagonal transformation matrices:
P
Io
0
P
0
P
:
Plt
P Pright
0
0
I
P
Ł-1
Then, the transformed parameters are obtained by block-wise operations:
(0 ding )it
(Peea)u( (Ouiag (f and Wi, = = -W
(P This results in W P,W, W/ P.W.P. for I f2.....L 1}.
It is important to note that the transformation matrices S are not fundamentally restricted to per- mutations. Many neural network components possess inherent symmetries richer than permutation symmetry, and modern architectures, such as Transformers, are often designed to effectively lever- age such components. Recognizing and exploiting broader symmetry classes, when available, ex- pands the set of valid reparameterization mappings. This, in turn, enhances alignment strategies by increasing the likelihood of discovering the low- or zero-barrier interpolation paths, thus allowing to achieve LMC
19

---

For instance, Root Mean Square Layer Normalization (RMSNorm) component inherently possesses orthogonal symmetry (see Section 3.2). This property particularly significant architectures such as Transformers, where RMSNorm is commonly applied as standalone preceding major components like the attention or MLP blocks, without an immediately follow- ing element-wise activation. The absence of such non-linearity preserves the richer orthogonal symmetry, which would otherwise be reduced to permutation symmetry. We can leverage such an architectural design to exploit the full orthogonal symmetry of RMSNorm for alignment purposes. We now examine the symmetry properties of RMSNorm more detail.
-yppiallyyy
Consider network block defined as follows:
Wo X +
Y Wi RMSNorm Wo X + -
= Wi
+
Wo X bll2 + eN
Introduce an orthogonal matrix O E ERNN (M 2 N. 0 I). Since ||Oz|12 || inserting O gives
OWW X +) + +03
Y = W10
OOW, X b)ll3 6N
To express the normalization over M elements, define
v Mr EM
N M
Then
|| 1| +
M |=|112 +ea
the block can be rewritten
O(W x+0)
1 103
Y Win
1WW X b)112 EM
c 7
Hence the network remains functionally equivalent when written as
Y W/' RMSNorm (W; X + +b;; 8 EME),
(6)
with transformed parameters
W = Wi OT,0
=WOwW.
b'= Ob,
and normalization constants
1 103 c
M N 7
08
Y 1M:
8
This derivation for RMSNorm underscores a critical point: the nature of component's 's inherent symmetries dictates the of valid transformation matrices S, that can be used for reparameteriza- tion while preserving its functionality. Let Sj denote the set of allowed transformation matrices for a given layer of dimension d, If layer exclusively admits permutation symmetry, then S = Pa, the finite set of di di permutation matrices. However, if a component, such as RMSNorm blocks Transformers, exhibit orthogonal symmetry, the set permissible transformations expands to S, = (dd), the orthogonal group. For components allowing even more general transformations, this could = GL(d, R.,, the general linear group of invertible matrices. These sets of trans- formations nested, with Pa. O(d)) GL(d,, R), where Pa, is finite group, while Odd)
20

---

and GL(d, R) are continuous, significantly larger Lie groups. The overall alignment transforma- tion for the entire network is constructed by selecting an appropriate S, E S for each layer or component. Consequently, by identifying and utilizing the richest symmetry class available for each network component for example, employing transformations from (dl) for an RMSNorm layer rather than being restricted to the smaller set Pa, (which might be the case if its orthogonal sym- metry were immediately broken by a subsequent clement-wise activation function) we drastically expand the search space of valid, sunction-preserving larger search space offers more degrees of freedom in the alignment process, thereby substantially increasing the potential to discover configurations O' (aligned parameter vector) that lie on low-loss linear paths and thus achieve Linear Mode Connectivity between independently trained models.
reparameterizations π T()). This significantly
An additional important consequence of this symmetry is that, since the matrix O can be rectangular with M 2 .N, the transformation can expand the dimensionality of the layer while preserving exact functional equivalence. In other words, if is possible to increase the width of certain components (such as RMSNorm layers and their connected blocks) while maintaining their outputs unchanged, thanks to the orthogonal symmetry. This enables reparameterizations that not only align models of identical architecture but also align models of differing widths-expande of alignment strategies. More generally, by exploiting symmetries such as (dz), we can define valid mappings T()) that traverse across architectures within a family of functionally equivalent models. This further enlarges the search space for alignment and opens the door to architecture- aware alignment methods and interpolation across model scales.
the expressive power
F Discussion
Geometry of Transtormer minima. Establishing zero-barrier LMC in Transformers reveals surprisingly connected geometry in the loss landscape, where symmetries resolve apparent barriers and enable smooth interpolation between independently trained models. This extends prior obser- vations for simpler architectures (e.g., MLPs and CNNs Draxler et al. [2018], Garipov et al. [2018]) while opening several avenues for future research. Below, we outline implications of particular importance.
Loss landscape insights. Our results support the view that Transformer solutions reside in broad, connected basins once functional symmetries (e.g., permutations and orthogonals) accounted for, challenging the notion of isolated minima in high-dimensional parameter spaces. Prior work con- jectured such connectivity primarily for SGD-trained MIPs and CNNs Entezari et al. [2021]; we empirically extend this to Transformers. Moreover, these findings complement the Lottery Ticket Hypothesis Frankle al. [2020], which posits that dense networks contain sparse, trainable subnet- works (" ("winning tickets"). Our results suggest that such subnetworks may reside within the same connected regions, yielding more navigable landscapes for pruning, distillation, and efficient train- ing from scratch.
Federated and continual learning. In federated learning, heterogeneous client models often hin- der aggregation (e.g., under FedAvg). Symmetry-based alignment can pre-process weights re- solve these discrepancies, reducing variance and improving convergence under non-lID data. In continual learning, merging task-specific updates with previous model states preserves connectiv- ity across sequential minima barriers drop to zero post-alignment posttaligmmet--fffrringg parfmeter-efficient alternative to replay buffers or regularization-bate methods.
Adversarial robustness directions. Adversarially robust models are typically associated with flat- ter minima, which enhance generalization by enlarging basins of attraction Stutz et al. [2021]. How- ever, adversarial training often incurs a trade-off between clean and robust accuracy. A promising extension of our framework could involve transporting high-accuracy (non-robust) model into the basin of a robust counterpart via symmetry-aligee properties from both. While robustness is not evaluated here, the existence of zero-barrier paths provides a principled mechanism for such basin hopping, motivating future exploration.
interpolation, potentially inheriting beneficial
Role of positional encodings. Positional encodings further modulate the symmetries admissible in language models. Absolute Positional Encodings (APEs), as employed in our GPT-2 setup,
21

---

are additive at the embedding layer and do not alter internal block invariances: permutation matri- ces P E S1 in feed-forward networks (FFNs), semi-permutation matrices P E S2 in multi-head attention (MHA), and orthogonal matrices O E S3 residual connections remain valid symme- try elements. By contrast, Rotary Positional Encodings (RoPE) embed rotations directly within query-key projections, enforeing block diigonaa constraints on the residual orthogonals O E S3- This decomposition breaks the full orthogonal group, necessitating specialized alignment proce- dures that respect these subgroup structures. Developing such RoPE-aware symmetry mappings is promising direction for extending our approach to modern large-scale language models.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are empirically verified in Section 5, in particular Table 2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We mention our limitations in Appendix and in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [es]]
23

---

Justification:
We proof the symmetry network components in Appendix E.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification:
We disclose all information needed for reproduction in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
We will provide data, code, and instructions for the final paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nigg cc/
public/ Sealic/guiess//e wttpe//giide//ooeuuuie hates/CCoeSuubmiise oessionnlliyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.c.. cc/ wwwpcpcwlcc//ccccccccccccccollcl wwtp/ggdge///ooooooooooooooo
ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification:
WE provide experimental details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide error bars in Table 2 and Figure 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information about hte computer resources used in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::///euuuppeeeeeeeee httpppp...c.pppppteettteeeetaaae
Answer: [Yes]
Justification: We have read and understood the code of ethics and have made every effort adhere to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Our work contributes a better understanding of the Transformer loss land- scape. It does not impact society at large.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We do not release any models datasets. Our work uses standard, publicly available resources-CIRR CIFAR-10 10 and GPT-2 pretrained on Book and does not pose additional risk of misuse beyond their existing public availability.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly available datasets (CIFAR-10, BookCorpus) and pretrained models (GPT-2), all of which properly cited in the paper. We adhere to the licenses and terms of use as specified by their original creators.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
If assets are released,
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We will release codebase along with detailed README files. The release does not include source code copied from external libraries; all code provided is our own.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects Question:
For crowdsoureing experiments and research with human subjects, does the pa- per include the full text instructions given participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not use crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does use crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
28

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We used LLM solely for editing assistance.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https:/ /ttp://eee.pp//eeeeee/eeeee ww/Conneeeenceeeeeeeeeeeeeeeeeee / 222// for what should or should not be described.
29


Reviews:
[{"review_id": "xZzuzIsZ2X", "reviewer": "Reviewer_nSkR", "summary": "Previous studies on linear mode connectivity (LMC) in deep neural networks have primarily focused on symmetries arising from neuron permutations. This limits the applicability of these studies to modern architectures, such as Transformers. To address this, the authors propose a unified framework covering four classes of symmetries: permutations, semi-permutations, orthogonal transformations, and general invertible maps. This framework generalizes existing approaches. Using this framework, the authors demonstrate the existence of low- or zero-barrier linear interpolation paths between Vision Transformer and GPT-2 models that were trained independently.", "questions": "- Is there a specific reason for selecting the interpolation coefficient $\\lambda$ from the interval $U(0.4, 0.6)$? For instance, would a wider range such as $U(0.3, 0.7)$ also be feasible?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The overall writing is clear and easy to follow.\n- The diagrams in the paper are clearly presented and easy to understand.\n- The newly proposed symmetry classes are novel and are presented with clear mathematical formulations.\n- The authors empirically demonstrate the existence of linear mode connectivity with sufficiently low loss barriers in Transformer architectures.\n\n**Weaknesses**\n\nThe main limitation of the paper is that the empirical results are not sufficiently extensive.\n\n- The experiments on Vision Transformers are primarily conducted on relatively simple datasets such as CIFAR-10/100, and the GPT-2 experiments are limited to the comparatively small BookCorpus dataset. Additional evaluation on more complex datasets, where Transformers are more commonly applied, would help validate the generality of the findings.\n- The experiments are restricted to ViT and GPT-2 models. Including results on a broader range of Transformer architectures would further support the practicality and general applicability of the proposed method.", "was_revised": true, "final_justification": "I find the results showing that LMC exists in Transformer architectures interesting. I appreciate that the authors addressed my question adequately. I would like to see more diverse experiments included in the camera-ready version. I will maintain my original score.", "rebuttal": "We thank the reviewer for their positive and constructive feedback. We appreciate the recognition of our framework's novelty, the clarity of the presentation, and the significance of our findings.\nOur work provides the **first demonstration of Linear Mode Connectivity (LMC) in standard Transformer architectures** - a foundational result that we believe constitutes a meaningful contribution in its own right. Notably, prior work has struggled to achieve zero-barrier interpolation even in simpler settings (e.g., Git Rebasin only achieves zero barrier by increasing layer widths on convolutional networks trained on CIFAR). That said, we fully agree that additional empirical validation would further strengthen our claims. We have already conducted new experiments and will include more extensive large-scale results in the camera-ready version.\n### **Additional Results**\n* **Tiny-ImageNet**: We train Vision Transformers (ViTs) from the ground up on Tiny-ImageNet using a cosine decay schedule with linear warmup. The resulting models achieve around 44.6% test accuracy. Upon merging two separately trained models, we observe no increase in test loss—achieving 44.57% accuracy at the interpolation midpoint. This **absence of a loss barrier** indicates that the independently trained ViTs converge into the same region of the loss landscape. Plots illustrating the interpolation behavior and loss surfaces will be provided in the camera-ready version.\n\n* **Three-way model merging**: We also generalize our analysis to merging **three models simultaneously**. In this setting, two interpolation coefficients are drawn from a Dirichlet distribution to explore the interior of the simplex. We evaluate this approach on ViTs trained on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and find that the merged model at the uniform average point $\\tfrac{1}{3} \\Theta_A + \\tfrac{1}{3} \\Theta_B + \\tfrac{1}{3} \\Theta_C$ continues to lie on a **barrier-free path**, further confirming that these models reach a common solution basin during training.\n\nWe will incorporate experiments on the larger-scale ImageNet-1k dataset, along with a more diverse set of architectures, in the camera-ready version to further validate the generality of our framework.\n\n## Questions\n**Q1: Is there a specific reason for selecting the interpolation coefficient \\$\\lambda\\$ from the interval $\\[0.4, 0.6]\\$? For instance, would a wider range such as $\\[0.1, 0.9]\\$ also be feasible?**\nThis touches on an important aspect of our optimization strategy.\nOur choice to sample \\$\\lambda\\$ from the narrower $\\[0.4, 0.6]\\$ interval is deliberate. Empirically, the loss barrier tends to peak around the midpoint of the interpolation path (i.e., \\$\\lambda \\approx 0.5\\$). By focusing our optimization on this most challenging region, we are able to more effectively reduce the **maximum barrier height**, which is the critical quantity in establishing LMC.\n\nWhile a broader range such as $\\[0.1, 0.9]\\$ is technically feasible, it would divert optimization efforts away from the region that most directly contributes to barrier elimination. Indeed, in Figure 5 of the Appendix we investigate the performance of different sampling strategies, including a broader range of $\\[0.0, 1.0]\\$, which we empirically find to deteriorate performance around the midpoint. Thus, our narrower focus provides a better efficiency–effectiveness trade-off."}, {"review_id": "N7stS9GpIl", "reviewer": "Reviewer_DXBv", "summary": "The paper discusses linear mode connectivity and introduces a unified framework capturing richer symmetries than in the present literature, including permutations, semi-permutations, orthogonal transformations, and invertible maps that explain these connections and enable discovery of smooth linear paths between ViTs and GPT-2 models. These findings highlight the importance of symmetry-aware analysis for understanding neural network model space geometry.", "questions": "Please see weaknesses. I'm ready to revise my score once these are carefully addressed.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors do a great job carefully analyzing different layers and activation functions. They show low / zero-loss linear interpolation paths between independently trained ViTs and GPT-2 models, establishing deeper connectivity in Transformer loss landscapes than what has been known so far. \n- The proposed unified treatment of permutations, semi-permutations, orthogonal, and general invertible maps goes beyond prior permutation-only approaches, capturing richer symmetries in modern architectures. The paper explores multiple alignment strategies (activation, weight, and learned matching), with learned matching demonstrating strong empirical performance by fully eliminating interpolation barriers in experiments.\n- Results on CIFAR-10, CIFAR-100, and BookCorpus empirically support the main claims.\n\nWeaknesses:\n- The paper reports the barrier in terms of the test loss. It would also be interesting to see the corresponding accuracy / perplexity (in particular, how much is 0.41 loss barrier between two GPT-2 networks?)\n- Add missing citations, e.g., the definition of LMC, the methods such as activation matching and weight matching, etc. Please be generous as to where specific concepts and methods were introduced, as you build on those in your work.\n- Algorithm 1 implements a straight-through estimator. Please cite the literature following a similar approach. How does your approach compare to existing STE-based methods? What new insights or technical contributions does your broader symmetry framework introduce beyond extending to more transformation classes? How does the proposed method compare to OT-fusion?\n- The reviewer is missing a well thought discussion about the implication of LMC between transformer networks. What do the proposed (very impressive) findings enable?", "was_revised": true, "final_justification": "The authors have addressed my concerns in their rebuttal. I believe the paper provides a solid contribution to the literature and have revised my score accordingly.", "rebuttal": "We thank the reviewer for their thorough and constructive feedback. We appreciate the recognition of our contributions, including the unified symmetry framework and the empirical demonstration of zero-loss paths in Transformers. We agree that contextualizing results with performance metrics, adding citations, clarifying comparisons, and discussing implications will strengthen the paper. We will revise accordingly, incorporating these changes into the camera-ready version.\n\nBefore delving into the reviewer's insightful questions, we would first like to highlight some additional results:\n\n## Additional Results\n* **Tiny-ImageNet**: We trained Vision Transformers (ViTs) from scratch on Tiny-ImageNet, utilizing a cosine learning rate schedule with a linear warmup phase. The resulting models reach a test accuracy close to 44.6%. When interpolating between two such models, the combined model achieves 44.57% test accuracy and shows a **smooth, barrier-free loss trajectory**. This supports the idea that separately trained ViTs on Tiny-ImageNet converge to a shared region in parameter space. We will include loss landscape and interpolation visualizations in the final version.\n\n* **Merging three models**: We expand our experiments to include **simultaneous merging of three models**. To do this, we sample two mixing coefficients from a Dirichlet distribution, allowing us to explore convex combinations beyond pairwise interpolation. This method is evaluated on ViTs trained on CIFAR-10, CIFAR-100, and Tiny-ImageNet. Notably, the interpolated model at the equal-weight point $\\tfrac{1}{3} \\Theta_A + \\tfrac{1}{3} \\Theta_B + \\tfrac{1}{3} \\Theta_C$ also lies on a **zero-barrier path**, indicating that all three models share a common low-loss region in the optimization landscape.\n\n## Weaknesses\n\n**W1: Contextualizing loss barriers with accuracy and perplexity.** We appreciate the reviewer’s feedback and agree that the paper would benefit from clearer presentation of model performance. On CIFAR-10, the individual trained models achieve a test accuracy of approximately 85%, with a corresponding test loss around 0.55. The interpolated model at the midpoint exhibits a slightly lower test accuracy of ~84%, but a reduced test loss of ~0.49. This decrease in test loss, despite similar accuracy, suggests **improved calibration** — a finding supported by the interpolated model’s higher Expected Calibration Error (ECE) score. \n\nFor GPT-2, the baseline model achieves a test loss of approximately 3.55, corresponding to a perplexity of 34.8. In comparison, the merged model shows a slightly higher test loss of ~3.91, resulting in a perplexity of 51.9.\n\nWe will include detailed performance metrics for both individual and merged models in the camera-ready version, along with calibration plots that highlight the improved calibration observed in interpolated models.\n\n**W2: Add citations.** We thank the reviewer and we agree on the importance of crediting prior work and will extend our citations accordingly. Currently, we cite Frankle et al. [1] for LMC; Ainsworth et al. [2] for Git Re-Basin; Tatro et al. [3] for activation matching; Sing & Jaggi [4] for weight-matching/OT-Fusion; Verma et al. [5] for the first work on exploring LMC for Transformers; Ashkboos et al. [6] for uncovering orthogonal symmetries in Transformers.\n\n**W3: Relation to STE-based Methods and OT-Fusion.**  We agree that it is important to provide a clear distinction from STE-based methods, including foundational STE work.\n\n* **STE Comparison**: Our Algorithm 1 uses STE for backpropagation through soft latent matrices, similar to Git Re-Basin. However, our core contribution is not in how permutation matrices are learned, but in *extending beyond permutations* to identify and leverage broader *transformer symmetries* for *function-preserving alignment*. While Git Re-Basin [2] and Verma et al. [3] align residual weights via permutations (using OT or STE), we go further by aligning with *orthogonal transformations* derived via SVD. This richer symmetry perspective reveals *zero-loss barriers* between independently trained Transformers—unreachable with permutations alone. We empirically validate this: restricting to permutation learning via STE (as in Git Re-Basin’s “Learned matching (permutations)”) consistently leads to high loss barriers (see Tables 2 and 3), underscoring the need for richer symmetry exploitation.\n\n* **OT-Fusion Comparison**: OT-Fusion [4] introduced activation- and weight-matching to align MLPs and CNNs, but extending these techniques to Transformers is non-trivial. Verma et al. [3] adapt OT-Fusion’s *activation matching* to Transformers, a method we include in our results as “activation matching,” which our method consistently outperforms (Tables 2 and 3). OT-Fusion’s weight matching resembles Git Re-Basin’s first iteration. In Appendix B.2, we evaluate Git Re-Basin’s multi-iteration version (“permutation matching” in Figure 5) and find it performs significantly worse than our method, failing to reduce loss barriers after the first iteration.\n\n**W4: Implications of LMC in Transformers.** We thank the reviewer for highlighting the need to discuss implications more thoroughly — our results indeed uncover profound insights into Transformer landscapes. We will add a discussion section, incorporating these points for greater clarity and impact. \n\nEstablishing zero-barrier LMC in Transformers reveals a surprisingly connected geometry in the loss landscape, where symmetries resolve apparent barriers, allowing smooth interpolation between independently trained models. This extends prior observations from simpler architectures (e.g., MLPs and CNNs, as in Draxler et al. [7] and Garipov et al. [8]) while opening exciting avenues for future research. Below, we outline the key implications with enhanced technical detail:\n\n**Loss Landscape Insights:** These results support the view that Transformer solutions reside in broad, connected basins once functional symmetries (e.g., permutations and orthogonals) are accounted for, challenging the notion of isolated minima in high-dimensional spaces. Prior work conjectured such connectivity primarily for SGD-trained solutions in fully connected or convolutional networks (e.g., Entezari et al. [9]); we empirically extend this to Transformers. Furthermore, this complements the Lottery Ticket Hypothesis [1], which posits that dense networks contain sparse, trainable subnetworks (\"winning tickets\"); our LMC findings suggest these tickets may lie within the same connected regions, enabling more navigable landscapes for pruning, distillation, and efficient training from scratch.\n\n**Federated and Continual Learning:** In federated learning, heterogeneous client models often hinder aggregation (e.g., via FedAvg); alignment algorithms could preprocess weights to resolve symmetries, reducing variance and improving convergence on non-IID data. For continual learning, merging task-specific updates with prior states mitigates catastrophic forgetting by preserving connectivity across sequential minima, as barriers drop to zero post-alignment—offering a parameter-efficient alternative to replay buffers or regularization methods.\n\n**Adversarial Robustness Directions:** Adversarially robust models are associated with flatter minima, which enhance generalization by enlarging basins of attraction (Stutz et al. [10]). However, adversarial training often trades off standard accuracy for robustness. A compelling extension of our work could involve transporting a high-accuracy (non-robust) model into the basin of a robust counterpart via symmetry-aligned interpolation, potentially inheriting benefits from both (e.g., improved clean and adversarial accuracy). While we do not evaluate robustness empirically here, the zero-barrier paths provide a foundational mechanism for such \"basin hopping,\" inspiring future investigations.\n\n## References\n\n[1] Frankle et al. (2020), *Linear Mode Connectivity and the Lottery Ticket Hypothesis*.\n\n[2] Ainsworth et al. (2023), *Git Re-Basin: Merging Models modulo Permutation Symmetries*.\n\n[3] Tatro et al. (2020), *Optimizing Mode Connectivity via Neuron Alignment*.\n\n[4] Singh and Jaggi (2023), *Model Fusion via Optimal Transport*.\n\n[5] Verma and Elbayad (2024), *Merging Text Transformer Models from Different Initializations*.\n\n[6] Ashkboos et al. (2024), *SliceGPT: Compress Large Language Models by Deleting Rows and Columns*.\n\n[7] Draxler et al. (2018), *Essentially No Barriers in Neural Network Energy Landscape*.\n\n[8] Garipov et al. (2018), *Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs*.\n\n[9] Entezari et al. (2021), *The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks*.\n\n[10] Stutz et al. (2021), *Relating Adversarially Robust Generalization to Flat Minima*."}, {"review_id": "0yQZBBX7P3", "reviewer": "Reviewer_HV2f", "summary": "The paper presents a thorough accounting of the (invariant) symmetries present in modern transformer architectures. These go beyond the permutations symmetries found in convolutions networks and include richer mathematical structures, including orthogonal and invertible transforms. Using a carefully constructed optimizer, the authors are able to apply symmetries to bring two models into the same “loss basin,” exhibiting zero loss barrier between the models for the first time.", "questions": "1. For language models, how do common forms of positional encoding affect the symmetries? \n2. This is subjective, but I find the title to be a bit misleading. It is not the connectivity that is being generalized but rather the symmetries which enable that connectivity. This is an important distinction and I think the current title undersells the papers results.", "limitations": "The main limitation (along with weakness 1) is that the loss barrier persists in the GPT-2 results. This may suggest that either the enumeration of symmetries is incomplete for those models, that symmetries are insufficient to give linear mode connectivity, or that the optimization procedure is not finding a good solution. Some discussion here would be very useful to indicate directions for future improvement.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths \n\n1. The presentation of the transformer symmetries is really excellent.\n2. The achievement of zero loss barrier for transformers is significant. This is a natural next step following the near zero barrier results in Git Re-Basin [1]. However it has been several years and this paper is the first to actually achieve the result!\n\n[1] Ainsworth, Samuel, Jonathan Hayase, and Siddhartha Srinivasa. \"Git Re-Basin: Merging Models modulo Permutation Symmetries.\" The Eleventh International Conference on Learning Representations.\n\n## Weaknesses \n\n1. The best results are obtained by directly optimizing over the various symmetries to minimize the barrier. This is (potentially) computationally expensive and it is not clear to me when this expense would be justified.\n2. Limited evaluation. The main ViT results study one architecture and two datasets (CIFAR 10 and 100). It would be interesting to see merges for more difficult datasets such as ImageNet or CLIP models, where ViTs are commonly used.", "was_revised": true, "final_justification": "This work is the first to demonstrate zero barrier for independently trained transformers, which is a substantial achievement. Additionally the presentation of the symmetries in the paper is excellent and I believe it will be of great resource to those interested in transformers.", "rebuttal": "We thank the reviewer for their thoughtful review and positive assessment of our work's presentation and significance. We are pleased that the reviewer recognizes the value in our demonstration of zero-loss barriers for Transformers. Below, we address the points raised, incorporating clarifications and new experimental evidence, where relevant.\n\n## Weaknesses\n\n**W1: Computational costs.**\nWe understand the reviewer's concern about the cost incurred by our Learned Matching (LM) method, which optimizes over symmetries to minimize barriers. This approach does involve optimization over symmetries to minimize the barrier, which is more computationally intensive than data-free methods like Weight Matching (WM). \n\n* **The relationship between model heterogeneity and alignment difficulty.** Our primary aim is to demonstrate linear mode connectivity in Transformers for the first time. We acknowledge the reviewer’s concern about practical applicability. There is indeed a tradeoff: **the more divergent the models, the harder they are to align and merge**. In our case, the models were trained independently from different random initializations and seeds, making alignment nontrivial without learning it. However, such divergence is less pronounced in practical settings. For instance, in **federated learning**, models are typically trained from the same initialization and aggregated periodically. Weight matching could mitigate divergence in this context, enabling less frequent aggregation and thus faster training.\n\n* **Understanding the gap between weight- and learned matching.** A first step towards bridging the gap between weight- and learned matching is understanding how their alignment solutions differ. Interestingly, we observe that the only difference between the solutions of weight- and learned matching is the orthogonal matrix that aligns the residual path. Thus, the key to unveiling zero-loss paths without requiring any data or finetuning is better alignment of the residual path. We will add this ablation study to the camera-ready version.\n\n**W2: Limited evaluation:** Our work provides the **first demonstration of Linear Mode Connectivity (LMC) in standard Transformer architectures** - a foundational result that we believe constitutes a meaningful contribution in its own right. Notably, prior work has struggled to achieve zero-barrier interpolation even in simpler settings (e.g., Git Rebasin only achieves zero barrier by radically increasing layer widths on ResNets trained on CIFAR). That said, we agree that additional empirical validation would only further strengthen our claims.\n\n* **Tiny-ImageNet**: We train Vision Transformers (ViTs) from scratch on Tiny-ImageNet, employing a cosine learning rate decay and linear warmup. Our models attain a test accuracy of approximately 44.6%. When we merge two independently trained models, we observe a **barrier-free interpolation path** in the loss landscape, with the merged model achieving 44.57% test accuracy—essentially matching the original models. This suggests that the independently trained ViTs converge into the same optimization basin. We plan to include visualizations (loss surface and interpolation curves) in the camera-ready version.\n\n* **Merging more than two models**: We further explore merging **three models at once** by sampling two interpolation weights from a Dirichlet distribution. This extends our study to a broader merging space. We apply this approach to ViTs trained on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and observe that even the centroid point $\\tfrac{1}{3} \\Theta_A + \\tfrac{1}{3} \\Theta_B + \\tfrac{1}{3} \\Theta_C$ remains on a **zero-loss path**, reinforcing our finding that ViT training trajectories consistently arrive in a shared solution basin.\n\nWe commit to including larger-scale ImageNet-1k experiments and a broader range of architectures in the camera-ready version, to further confirm our framework's applicability.\n\n## Questions\n\n**Q1: Positional encodings and symmetries:**  We thank the reviewer for this question, which raises an interesting point worthy of analysis. We will include additional pointers in the paper, as these may help inform future research.  \n\n- **Absolute Positional Encodings (APEs):** As used in our GPT-2 setup, these are added at the input embedding layer and do not impact the internal symmetries of Transformer blocks (e.g., permutations in FFNs, semi-permutations in attention, or orthogonals in residuals).  \n- **Rotary Positional Encodings (RoPE):** These embed rotations directly into attention, likely imposing new constraints like block-diagonal structures on orthogonal symmetries. \n\nWe are excited to add a section on the symmetry of different positional embeddings in the camera-ready version.\n\n**Q2: Title Feedback:**  We value the reviewer's perspective that the title could better spotlight the role of richer symmetries in enabling LMC between Transformers - this was indeed our goal but our title choice may not have been entirely on point. To address this, we are considering revisions such as:  \n- *Generalized Symmetries Enable Linear Mode Connectivity in Transformers*  \n- *Unlocking Linear Mode Connectivity in Transformers with Richer Symmetries*  \n- *Beyond Permutations: Achieving Linear Mode Connectivity in Transformers with Richer Symmetries*\nThese aim to more precisely convey our contributions while maintaining focus on linear mode connectivity. \n\n## Limitations\nWe thank the reviewer for noting the contrast between ViTs and GPT-2 regarding Linear Mode Connectivity. This difference aligns with findings from Juneja et al. [1], who show that NLP models often converge to functionally distinct basins, even with the same initialization and training data. These basins correspond to different generalization strategies, and interpolating between them leads to loss barriers, breaking LMC. In contrast, vision models such as ViTs tend to generalize in more homogeneous ways, which facilitates the emergence of LMC. Our observations support this interpretation. We will add a Discussion section to the camera-ready version addressing this.\n\n## References\n[1] Juneja et al. (2023), *Linear Connectivity Reveals Generalization Strategies*."}, {"review_id": "jRU9Ot1GKG", "reviewer": "Reviewer_My19", "summary": "This paper extends permutation symmetries in neural networks, as mostly discussed in the literature, to other 3 types of symmetries. Building on this symmetry framework they describe methods for aligning two independently trained Transformer models. These mathod range from weight matching, to activation matching and learned matching (with gradient).", "questions": "1) The proposed Learned matching is a reminiscnet of \"Learning Neural Network Subspaces\" by Wortsman et al. What are the diifferences, highlighting novelty?\n2) One important motivations to remove barriers is to have them averaged to improved accuracy. Did you try to make ensembles given the zero barrier in CIDARs?\n3) This is a great start, but only limited to ViTs to CIFARs for zero barriers. How about more complex tasks, as stated in literature, ImageNet?\n4) experimental design: Which ViTs did you use (B/L, etc) and how well the models are optimized for CIFAR tasks? what is the top1 accuracy? and how important it is to have a overfit model on the downstream task?\n5) Given the promising results of learned matching, did you try to remove the barrier for two non-ViT based models on ImageNet? We need some pointers to compare the effectiveness of the best method here this with some baselines.\n6) What do you think of applying learned matching to DiTs?", "limitations": "- some of the claims are not clear to me: Is this work the first to look into matching of ViTs or the first to make the barrier zero in CIFARs?\n- experiments are limited to proof of concepts like ViT on CIFARs and one GPT\n- some other limitations include (that are also correctly reflected in Appendix): For example the paper does not provide results for more than two models, and I wonder why.\n- remaining are my notes in the questions", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ a fundamental problem yet important direction to improve current SOTA models\n+ well written, structured and formulated, easy to understand and follow\n\n- limited experiments and results (more in questions)", "was_revised": true, "final_justification": "After reading the discussions and responses to my questions/concerns I increase my score to 4. I hope that this line of work motivate future works on applications on generative DiTs too.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of our work's foundational importance and clear writing. Below, we address the reviewer's questions and limitations point by point, incorporating new experimental evidence where relevant to strengthen our responses.\n\nWe agree with the reviewer that the paper can only benefit from more extensive experiments. For this reason, we have added experiments on *Tiny-ImageNet*, and for merging *more than two models* with each other, achieving *zero-loss barriers* in both settings (see response to *Q3*). We commit to include even larger-scale results (e.g., on ImageNet-1k) in the camera-ready version.\n\nWe hope these additions clarify and strengthen the paper. If they address the reviewer's concerns, we would appreciate the reviewer's consideration in updating their score.\n\nBefore delving into the reviewer's valuable questions and remarks, we would like to shortly add some context to our obtained results.\n\n## Context on results\nWe emphasize that our core contribution is foundational: prior to our work, **Linear Mode Connectivity (LMC) had not been demonstrated for Transformer architectures — not even on simple benchmarks like MNIST**. Even for MLPs and CNNs, which have been the primary architectures for studying LMC, achieving zero-loss paths often required substantial architectural modifications. For example, Git Re-Basin [1] demonstrates LMC for ResNet-20 on CIFAR-10 only after widening the model by a factor of 20.\n\nIn contrast, our work is the first to demonstrate *zero-loss* linear interpolation between independently trained Vision Transformers and GPT-2 models — without any width expansion or architectural changes. **This is a striking and somewhat surprising result**: despite their greater depth and complexity, standard Transformers exhibit LMC even more readily than CNNs. This provides the first direct evidence that LMC extends to modern Transformer-based architectures.\n\nWe agree that evaluating on larger-scale models and datasets like CLIP or ImageNet would strengthen our claims. At the same time, we believe that starting with benchmarks like CIFAR and Tiny ImageNet is essential: prior to our work, LMC had not been observed in Transformers at *any* scale. These controlled settings allow us to rigorously establish this phenomenon and provide a solid foundation for scaling to more complex models.\n\n## Questions\n**Q1: Difference between \"Learning Neural Network Subspaces\" by Wortsman et al. and our method.**\n\nWe understand how these studies might appear similar at first glance, as both involve low-loss subspace. To clarify, the key distinctions are in their goals and technical approaches:\n\n* **Our paper**: Our learned matching is a *post-hoc alignment technique* for merging independently trained models. Our objective is to learn *function-preserving transformations to connect existing models with low-loss linear paths*, keeping the original weights frozen. This enables the analysis of the geometry of loss landscapes and the connectedness of Transformers.\n* **Wortsman et al. [2]**: Their method is a *training procedure* that optimizes an entire subspace (e.g., a line of models) from scratch during a single training run. It updates the boundary model weights by backpropagating losses from sampled subspace points, aiming to produce robust models or cheap ensembles—*not to align pre-trained models*.\n\n**Q2:** We thank the reviewer for the insightful comment. While our primary focus is on LMC in Transformers, we agree that the behavior of interpolated models is of interest. We typically observe reduced test loss at the interpolation midpoint, but no improvement in test accuracy. For instance, on CIFAR-10, ViTs achieve ~85% accuracy and ~0.55 loss, whereas the midpoint interpolated model has ~84% accuracy and ~0.49 loss. This suggests improved calibration, which is supported by notably higher Expected Calibration Error (ECE) scores in interpolated models.\n\n**Q3: Extension to more complex tasks.**\n\nWe agree that extending our experiments to more challenging datasets and merging settings enhances the paper's contributions. In response, we include the following new results:\n\n* **Tiny-ImageNet**: We train ViTs on Tiny-ImageNet from scratch, using a cosine learning rate schedule with linear warmup, achieving test accuracy of ~44.6%. The merged model achieves a **zero-loss barrier**, and a test accuracy of 44.57%, demonstrating that independently trained ViTs on Tiny-ImageNet converge to the same loss basin. Loss landscape plots and interpolation figures will be added to the camera-ready version.\n\n* **Multi-model merging**: We extend our experiments to merging **three models simultaneously**. Rather than sampling a single interpolation coefficient at random, we now sample two coefficients from a Dirichlet distribution. We evaluate this setting for ViTs on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and discover zero-loss barrier at the midpoint $\\tfrac{1}{3} \\Theta_A + \\tfrac{1}{3} \\Theta_B + \\tfrac{1}{3} \\Theta_C$.\n\n**Q4: Experimental setup of ViTs.**\nWe did not use standard ViT-B/L variants, as our study requires training from scratch with multiple random initializations to probe loss landscape connectivity. Instead, we used custom ViTs tailored for CIFAR to allow controlled, repeatable experiments.\n\n* **Architecture details**: The full architectural configuration is included in Appendix A.1 of the paper. Our ViTs use: patch size of 4, embedding dimension of 256, MLP dimension of 512, 6 layers, and 8 attention heads.\n\n* **Optimization and Performance**: The models are well-optimized, achieving ~85% top-1 accuracy on CIFAR-10. This indicates effective learning without severe under- or overfitting—no significant train-test gap was observed. These results align with prior work on similar custom ViTs.\n\n**Q5: Removing barriers for non-ViT based models.**\nWe appreciate the interest in extending our method beyond Transformers. Our core claim is that Transformers exhibit richer symmetries than earlier architectures—extending beyond simple permutations. While permutation-based methods (e.g., Git Rebasin) suffice for CNNs like ResNets, they fail in Transformers, where normalization layers and attention mechanisms introduce orthogonal and invertible symmetries. These are absent in CNNs, where nonlinearities like ReLUs break such structure. Consequently, applying our framework to CNNs reduces to permutation-only alignment—already well-explored in prior work we build upon.\n\nNevertheless, these richer symmetries are not entirely unique to Transformers. For example, **MAMBA** exhibits orthogonal invariance along its residual path, highlighting an exciting direction for future work on linear mode connectivity in MAMBA.\n\n**Q6: Learned matching for DiTs.** We thank the reviewer for this forward-looking question. Diffusion Transformers (DiTs) share core components with the ViTs and GPT-2 models we studied, including Transformer blocks with multi-head attention, residual connections, and normalization layers. Our identified symmetries and approach to align them should directly translate to DiTs. \n\nIt would be highly interesting to see results for interpolated DiTs trained on different datasets (e.g., landscapes vs. portraits). This could lead to **hybrid generations** — mixing styles or domains in novel ways.\n\n## Limitations\n**L1: Unclear claims.** We are the **first method to achieve zero- and low-loss barriers between independently trained Transformers**. We are not the first paper to look into matching Transformers. For example, Verma et al. [3] match Transformers by permuting neurons according to their activations, referred to as \"activation matching\" in our Methods and Results chapter. \n\n## References\n[1] Ainsworth et al. (2023), *Git Re-Basin: Merging Models modulo Permutation Symmetries*.  \n\n[2] Wortsman et al. (2021), *Learning Neural Network Subspaces*.\n\n[3] Verma & Elbayad (2024), *Merging Text Transformer Models from Different Initializations*."}]

Author rebuttal:

Meta-review: The recommendation is based on the reviewers' comments, the area chair's evaluation, and the author-reviewer discussion. 

This paper studies linear mode connectivity in transformer models, including vision transformers and GPT-2 models, by considering a broader class of weight symmetry. All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of reviewers. 

In the post-rebuttal phase, all reviewers were satisfied with the authors’ responses and agreed on the decision of acceptance. Overall, I recommend acceptance of this submission. I also expect the authors to include the new results and suggested changes during the rebuttal phase in the final version.

Given that this work provides a solid demonstration of the generalized linear model connectivity in transformer models, I believe the insights to understanding the geometry of the loss landscape of transformer models and the associated applications are significant, and I recommed for an oral presentation.
Decision: Accept (oral)