Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Authors: Binchi Zhang
Zaiyi Zheng
Zhengzhang Chen
Jundong Li
Venue: ICML 2025 spotlightposter (2025)
Keywords: Rotation Symmetry
Parameter Matching
Model Fusion
Transformers
Primary area: deep_learning->attention_mechanisms

Abstract: Symmetry in the parameter space of deep neural networks (DNNs) has proven beneficial for various deep learning applications. A well-known example is the permutation symmetry in Multi-Layer Perceptrons (MLPs), where permuting the rows of weight matrices in one layer and applying the inverse permutation to adjacent layers yields a functionally equivalent model. While permutation symmetry fully characterizes the equivalence set for MLPs, its discrete nature limits its utility for transformers. In this paper, we introduce rotation symmetry, a novel form of parameter space symmetry for transformers that generalizes permutation symmetry by rotating parameter matrices in self-attention layers. Unlike permutation symmetry, rotation symmetry operates in a continuous domain, thereby significantly expanding the equivalence set for transformers. Based on this property, we propose a theoretically optimal parameter matching algorithm as a plug-and-play module to enhance model fusion. We evaluate our approach using pre-trained transformers across diverse natural language and vision tasks. Experimental results demonstrate that our rotation symmetry-based matching algorithm substantially improves model fusion, highlighting the potential of parameter space symmetry to facilitate model fusion. Our code is available on https://github.com/zhengzaiyi/RotationSymmetry

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Binchi Zhang" 1
Zaiyi Zheng"
Zhengzhang Chen?
Jundong Li!
Abstract
2021; Simsek al., 2021: Entezari et al., 2022; Grigsby et al., 2023; Lim et al., 2024b; Ziyin et al., 2024; 2025). One of the most studied forms of parameter space symmetry is permutation symmetry (Ainsworth et al., 2023; Entezari et al., 2022). For instance, in a two-layer MLP, permuting the rows of the weight matrix in the first layer and applying the corresponding inverse permutation to the second layer results in functionally equivalent model, i.e., the outputs of the original and permuted models remain identical for any given input (Ainsworth et al., 2023). All functionally equivalent models corresponding to weight permutations form an equivalence set, which provides theoretical insights into neural network optimization, such as the linear mode connectivity of loss landscapes (Entezari et al., 2022; Zhou et al., 2023e; Ferbach et al., 2024). In addition, permuta- tion symmetry has also proven helpful in advancing neural network applications, such as model fusion (Singh & Jaggi, 2020; Ainsworth et al., 2023) and optimization (Zhao et al., 2024; Zamir et al., 2025).
Symmetry in the parameter space of deep neural networks (DNNs) has proven beneficial for var- ious deep learning applications. A well-known example is the permutation symmetry in Multi- Layer Perceptrons (MLPs), where permuting the rows of weight matrices in one layer and applying the inverse permutation to adjacent layers yields a functionally equivalent model. While permutation symmetry fully characterizes the equivalence set for MIPs, its discrete nature limits its utility for transformers. In this paper, we introduce rotation symmetry, a novel form of parameter space sym- metry for transformers that generalizes permuta- tion symmetry by rotating parameter matrices in self-attention layers. Unlike permutation symme- try, rotation symmetry operates in a continuous do- main, thereby significantly expanding the equiv- alence set for transformers. Based on this prop- erty, we propose a theoretically optimal parameter matching algorithm as a plug-and-play module to enhance model fusion. We evaluate our approach using pre-trained transformers across diverse nat- ural language and vision tasks. Experimental results demonstrate that our rotation symmetry- based matching algorithm substantially improves model fusion, highlighting the potential of param- eter space symmetry to facilitate model fusion. Our code is available on https: /ttg/tttttt titbbb. com/ http://hnnzai//Rotaaaaee ion.rraattnnnymmnyy......
Although parameter space symmetry has been extensively studied in classical neural network architectures, such as MLPs and CNNs, the understanding of its application in transformers (Vaswani et al., 2017) remains limited. Trans- formers have seen rapid advancements in recent years, achieving remarkable success in a wide range of applica- tions (Yun et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Clark et al., 2020; Zhou et al., 2021; He et al., 2024; Zhu et al., 2024; Zheng et al., 2024; Zhang et al., 2025; Wei & Liu, 2025). The transformer architecture is built upon two primary submodules: feedforward networks and self- attention layers. The feedforward network, which is struc- turally similar to MLPs, naturally inherits the permutation symmetry that has been extensively studied in the existing literature. Self-attention layers, on the other hand, involve unique attention mechanism powered by matrix products of queries, keys, and values, which introduce additional potentials for symmetry beyond permutations, Permutation symmetries limit the equivalence set of neural networks to discrete operations, which aligns well with MLPs due to their element wise activations (e.g., ReLU (Glorot et al., 2011)). In contrast, the continuous nature of the matrix operations in self-attention layers necessitates more flexible operations fully characterize their equivalence set.
1. Introduction
Parameter space symmetry is an intriguing property of neu- ral networks that has gamered increasing attention in recent years (Du et al., 2018: 8; Armenta & Jodoin, 2021; Kunin et al.,
Equal contribution 'University of Virginia *NEC Labo- ratories America. Correspondence to: Jundong Li <jun- Long@virginia.cduu
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
In this paper, introduce rotation symmetry, a novel
We introduce a novel rotation symmetry for the atten- tion mechanism in transformers, extending the concept of symmetry to continuous space.
form of parameter space symmetry for self-attention layers in transformers. Specifically, we analyze the query and key matrices jointly and demonstrate that applying a rotation to the query matrix, followed by the corresponding inverse rotation to the key matrix, preserves the query-key product. Additionally, we find that the same rotation rule can be ap- plied to the value and oulput matrices. Our findings provide novel insights into the functional invariance of the attention mechanism and extend the permutation symmetry (Singh & Jaggi, 2020; Wang et al., 2020; Tatro et al., 2020; Entezari et al., 2022; Ainsworth et al., 2023; Imfeld et al., 2024) from discrete spaces to continuous spaces, which signifi- cantly extends the scope of parameter space symmetries for transformers.
Building on rotation symmetry, we propose theoretically optimal parameter matching algorithm that improves the effectiveness of model fusion in transformers.
Through extensive experiments, we validate the efficacy of our proposed parameter matching algorithm, demon- strating its potential to advance model fusion through parameter space symmetry.
2. Preliminaries
To facilitate further discussion, we begin by defining the notations. Let A (i, :], A [:. and A[i.j] denote the i-th row vector, the j-th column vector, and the (i, j)th element of the matrix A, respectively. Additionally, let 1 denote all- ones vector. We now illustrate the concept of permutation symmetry in neural networks, using an MLP as an example. Consider an L-layer MLP model defined as:
To further demonstrate the benefits of rotation symmetry for transformers, we explore its utility in model fusion. The goal of model fusion is to merge multiple well-trained end models in the parameter space to produce a single merged model with improved overall utility. Model fusion is widely adopted across various settings, such as hyperparameter tun- ing (where end models are trained on the same benchmark) and multi-task learning (where end models are trained with different tasks) (Jin et al., 2023). Unlike ensemble learn- ing (Dietterich, 2000; Lakshminarayanan et al., 2017; Sagi & Rokach, 2018; Dong et al., 2020), model fusion can work in a data-agnostic manner, making it suitable for privacy- sensitive scenarios such as federated learning (Yurochkin et al., 2019; Wang et al.. 2020).
Jw (X) = (4)) z = (Z (1-1) (W()) + +b0), (1)
where ( = X is the input feature matrix, 6(4) is the bias vector (b(3) is a row vector and can be seen as broadcast over all rows in Equation (I)), W = [[W()) (()) Jt=1.... collects all learnable parameters, and (c)) stands for non- linear activation function, such as ReLU (Nair & Hinton, 2010). A loss function [(fw (X), Y) is used to measure the distance between the model prediction and the ground truth label Y.
The existing literature has demonstrated that the perfor- mance of model fusion is closely tied to the distance be- tween the end models (Wortsman et al., 2022). Inspired by this finding, we propose a parameter matching algorithm that selects functionally equivalent end models from the equivalence class determined by rotation symmetry. This approach ensures that the selected representative models closer in parameter space, resulting in smaller inner distance. To achieve this, we formulate the problem of parameter matching as an optimization problem with or- thogonal constraints. Leveraging the continuous nature of rotation symmetry, we propose a closed-fone solution to this problem. Our parameter matching algorithm is highly efficient, easy to implement, and can be seamlessly incor- porated as a plug-and-play module for model fusion. To evaluate its effectiveness, we conduct extensive experiments with pre-trained transformers on real-world NI.P and vision tasks. The experimental results demonstrate that incorporat- ing rotation symmetry into parameter matching improves model fusion effectively and efficiently. Furthermore, ad- ditional experiments reveal that even matching a subset of parameters can lead to notable performance improvements, highlighting the practical utility of our approach. Our con- tributions are threefold:
To analyze the permutation symmetry in the MLP model, let P P be permutation matrix, where PÍi E +(00 1] and P(i, 1 P[t.j]] 1 for any i, j. All permutation matrices are orthogonal (Strang, 1976), satisfying PT
p Consequently, for layer 1 and ( + 1, we have:
z (++1 =o (o(z"") + +0 j( (+yT +b(+1)
w(yh + 6 p)cW(H3)) b(+1))
The third equal sign holds because the element activa- tion function a is decoupled from column permutation (i.e., being multiplied by P). Based on Equation (2), it follows that for layers/ and I + 1, the mappings ( - P ( (()) + 60P,. (I+1) + ((++)) preserves the output ((+1) for any input z For each pair of adjacent layers, similar mapping exists independently based on specific permutation matrix, denoted p(l) for layers I and 1 + 1. Consequently, for the entire L-layer MLP, there exists a mapping that preserves the model's predic- tions for any input X: W(() + (P(")) ()() p((-1), ((t) + (0) Equivalently, if we define W' = (T())T(()) p-)))))))))))))))))))))))))))))) 1=1,,, for any p()) E P
2

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
111,.... L 1) and p(o) P(L) I, then we have (x) fw(() for any X. This phenomenon is re- ferred to as the permutation symmetry of the parameter space (Godfrey et al., 2022; Hecht-Nielsen, 1990; Navon et al., 2023; Rossi et al., 2023; Simsek et al., 2021). Lever- aging permutation symmetry allows us to identify an equiv- alence class of functionally equivalent model parameters, which is known as permutation invariance (Ainsworth et al., 2023; Entezari et al., 2022; Lubana et al., 2023). We denote the equivalence relation induced by permutation invariance as т, where, for example, W = x (W) represents the equivalence between the original parameters W and the permuted parameters W
3.2. Rotation Symmetry of Self-attention Layers
We then focus on the self-attention layers. In this paper, we introduce the rotation symmetryof self-attention layers. For MLPs, we have to switch the order of multiplying P and passing the activation function (the third equal sign in Equation (2)), requiring the matrix P to be a permutation matrix. In contrast, the self-attention layer does not contain an element wise activation function, enabling a wider range of the matrix P in self-attention layers. The self-attention layer can be written
ATT = LN (Cattin {x [xxxv) } Wo + +bo + x), +x)
XºKV S.fmar X (x(()) da xi.
3. Parameter Space Symmetry of Transformers
where Cal stands for the operator concatenating the out- puts of multi-head attention, fftmz(() denotes the softmax operator, H denotes the number of multi-heads, and the subscripts Q, K, V, and O denote Query, Key, Value, and Output, respectively.
Transformers (Vaswani et al.. 2017) have revolutionized deep learning with their ability to handle sequential data ef-
fectively, particularly in natural language processing (NLP) and other fields (Brown al., 2020; Devlin al., 2019; Liu et al., 2021; Radford et al., 2021). Their success is primarily driven by two key components feedforward net- works and self-attention layers (Vaswani et al., 2017). To better understand the parameter space symmetry of trans- formers, we examine these two core modules individually in the following sections.
We first transform the query and key matrices as
xx _ = (X((W)) + (X T + +b))"
=(X(W)) + + RRR (X(W)) + + = =(X(R W + bO xXXX w + b% R)
where R is a rotation matrix, i.e., RR I. It is worth noting that each multi-head corresponds to specific rota- tion matrix R. Let Wo [W] W3 ... W and we then rewrite the concatenating operation (with the product by Wo) as Σ S(-.. )X (W+) + (W))) Sim- ilar to the Q-K case, we can transform the value and out- put matrices as (X(W)) T + by (W?))
3.1. Permutation Symmetry of Feedforward Networks
We first look at the feedforward networks. The feedforward network adopted in the transformer blocks is a two-layer MLP model which can be written as
(W))) + by RR (W)) = (R W T (W for each multi-head h, where R a rotation matrix. Finally, we derive an equivalence class of the parameters in self- attention layer determined by rotation invariance
FFN(X)= = LN + b.).W. - b. X), (3)
where LN denotes the Layer Normalization operator (Ba et al., 2016). Different from (Vaswani et al., 2017), we include the residual connection (He et al., 2016) and layer normalization modules into the formula of the feedforward network (and the self-attention layer mentioned later). Ac- cording to Equation (2) and the analysis in Preliminaries, the feedforward networks have the permutation symmetry prop- erty and the equivalence class determined by permutation invariance is defined as
W (Rtx)) "W. ba bb) Ri
R (R W by - bb Ri W - Tw),
by bb R... W W Ri. bo + bo.
(5)
where Rd and R are rotation matrices for h 1, H. The progress from permutation to rotation extends the sym- metry of transformers to a continuous space and enhances our understanding of the parameter space symmetry of at- tention mechanism. The denseness of the symmetry allows for the choice of better invariant models to analyze trans- formers' loss landscapes. For better clarity, we provide an intuitive illustration of the rotation symmetry in Figure 1.
W, + W., b. b.P. W. W.P, b, - b,, (4)
where P E P is a permutation matrix. It is worth noting that the permutations of different feedforward networks in transformer are independent due to the scalable modular design. Consequently, we are able to flexibly compute the permutation invariance equivalence class of each FFN module in a transformer model.
4. Symmetry for Model Fusion
We follow the architecture in the original transformer pa- per (Vaswani al., 2017).
In this section, we explore the benefit of the rotation symme- try of transformers in model fusion (Li et al., 2023; Matena
3

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Oatpee
RR"
Rouse
Rooue back
Model Fuxion
(A'K2) vd
Altetinn
Sfteer
(10) (K'W)
CR
CR
Qinceer
Kitrerr
Viiieer
Equivalent funciion fo ouution man.
Parameter Matching
Figure 1. The rotation symmetry of self-attention layers.
& Raffel, 2022; Wortsman et al., 2022; Yadav et al., 2023; Jin et al., 2023; Daheim et al., 2024; Yang et al., 2024). Model fusion is proposed to merge multiple given end mod- els trained in different settings (e.g, upon different datasets and hyperparameter settings) in the parameter space to im- prove model utility and robustness. Compared with en- semble learning (Dietterich, 2000; Sagi & Rokach, 2018; lakshminarayanan et al., 2017), model fusion has a lower inference-stage complexity without requiring access to the training data. Most existing methods of model fusion con- duct a weighted averaging of different end models, e.g., direct averaging (Wortsman et al., 2022), Fisher- weighted averaging (Matena & Raffel, 2022), and regression- mean averaging (Jin et al., 2023). We next show the potential of exploiting the permutation and rotation symmetry plug- and-play module to improve the model fusion techniques.
Figure 2. An intuitive example of the usage of parameter space symmetry for model fusion. The background shows the contour map of the loss landscape in the model parameter space. A and B are the original end models to be merged, AB is the result of naive model fusion, and AB' is the result of model fusion with parameter matching.
mately flat or convex curve (Entezari et al., 2022; Ainsworth et al., 2023). This property is called linear mode connectiv- ity and is regarded as weak form of convexity (Ainsworth et al., 2023). As a result, parameter matching can also help improve the convexity of the objective the area adjacent to the end models. We showcase an intuitive example of using parameter space symmetry to improve model fusion in Figure 2 to illustrate the intuition behind parameter match- ing. We can observe that the merged model after parameter matching (AB') has a lower loss value, i.e., better utility than the naive merged model (AB).
4.1. Background and Motivation
Let Wi...., We denote & different end models (after train- ing or pre-training) with the same architecture. The goal of model fusion is to merge the given k end models in the parameter space and obtain a single model. If the given models are trained over different datasets, we can expect the merged model to have better utility and out-of-distribution robustness (Jin et al., 2023). The theoretical results in pre- vious literature (Wortsman et al., 2022) have shown that strong convexity and closer end models can boost the utility of direct model fusion. Consequently, the primary advan- tage of permutation (and rotation) symmetry applying to model fusion is that we can substitute the end models with corresponding carefully chosen equivalent models (in the equivalence class determined by permutation and rotation invariance) to make the selected end models more concen- trated, i.e., closer to each other. This step is usually called parameter matching (Singh & Jaggi, 2020; Wang et al., 2020; Ainsworth et al., 2023), aka parameter or neuron alignment. Parameter matching, which aims to reduce the distance between end models, can naturally yield closer end models and improve model fusion performance. On the other hand, different end models lie in the basins of different local optimums regarding the highly non-convex nature of transformers. Previous studies have verified that parameter matching can merge different end models toward a single low-loss basin, i.e., the loss value along the linear interpolation between matched models shows an approxi-
4.2. Parameter Matching
Next, our primary goal is to develop a practical parameter matching algorithm to minimize the distance between dif- ferent end models. We begin by merging two models W1 and W2.
Matching two FFNs. Let 5, bi.. Wo, bog 1= de- note the model parameters in the two feedforward networks to be merged where k denote the index of the networks. We have already derived the equivalence relation of parameters in the feedforward networks determined by permutation in- variance as Equation (4). Based on Equation (4), we can formulate parameter matching as an optimization problem as follows.
argmin p. ES PP W., Ps W 13
(6)
+ b|,,, Pi
bi Ps 1 + W. P W., Pal
As shown in Equation (6), the goal is to find the func- tionally equivalent models from the equivalence classes π (W1) and л (W2) determined by P1 and Pz, which has the smallest 6-2 distance in the parameter space. Follow- ing the method (Ainsworth et al., 2023), we reformulate the optimization problem shown in Equation (6) as a linear assignment problem:
argmax W, W. + b W W (7)

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Linear assignment problems such as Equation (7) have been
Algorithm 1 Matching two self-attention layers.
well-studied in previous literature (Martello & Toth, 1987;
Input: Model parameters of the attention layers (WQ Qu
Burkard & Cela, 1999) and can be solved precisely by Hun- garian Algorithm (Martello & Toth, 1987). After calculat- ing the value of P. we substitute P = P.P back to Equation (4) and let P2- = I (for simplicity), then we ob- tain the matched parameter as Wi: + P W bi + b,, P. W., + W., P. bo, + bo. Parameters of the sec- ond model (W big: W., o bo,] } remain unchanged, which makes the second model an anchor model for matching.
W b., K2 Kk, W b W,,,,b 10,, 1==12: Output: The optimally matched parameters from source parameters (k = 1) to anchor parameters (k = 2. maintain unchanged).
bb Bo.
1: QK solution: R = U (V)) R 2k2
T,
I, where + W ( )
T 9k1
gk gk.
gk
(v
gk
W (W Q2 )
K1 K2
+
(bg,) bb + (6 Dbkr.
2: QK matching: W + (R Rb1
W
Qr
W
Ki
+
Matching two ATTNs. Let the parameters in the two self-attention layers to be merged be (W Qu: b/. W k, bk2 W V. byg Wo, Wo..bo J.1. where k denote the index of the layers. The equivalence relation of these parameters in the self-attention layers in terms of rotation invariance is shown in Equation (5). Similar to matching FFNs, our goal is to match the parameters of one (source) attention layer to the other (target) attention layer where the matched parameters are supposed to be closest to the target parameters while being functionally equivalent when feeding with any input data, i.e., in the equivalence class determined by rotation invariance. The goal can be formulated as an optimization problem.
(R gk1) ) TW Kg D + bb. R
+ KK R
qkl
vo solution: R
U (Vib)
R
I, where
U E (V:))
Wh (Wh)
(Wo,) W3, +
(b),) TO
4: VO matching: W + RR Why Wo, +
v
W R by, + by R
where is the identity matrix and UEV = WQ, W +
WK, WT, composition.
Kg + b2, bQq + bb, bx, is the result of eigende-
We leave detailed proof of Theorem 4.1 in Appendix A. According to Theorem 4.1, we can obtain the algorithm to match two self-attention layers shown in Algorithm I. Algorithm can be seen as an adaptation of the Kabsch algorithm (Kabsch, 1976; Umeyama, 1991). Without loss of generality, we let the parameters in the second (k- 2) self-attention layer be the anchor and conduct rotation for the first layer (k 1). The denseness of rotation symmetry helps reduce the distance between the end models after parameter matching as Algorithm 1 shows.
(W b2
T
(W
min Re Re
R
ER
gk2
(Wh)) K1
(W
K2)
R
Rijaa
F
(8)
(Wh) vi T
(W
R
bV
R
Wh.
Wht
R
+-1
- R
Rescaling Matching. We find that the rescaling symme- try (Neyshabur et al., 2015; Meng et al., 2019; Godfrey et al., 2022; Kalogeropoulos et al., 2024) can be integrated with our proposed rotation symmetry in the parameter matching algorithm. For instance, for the (simplified) Q-K product Wq WK, we can find a rescaling operation that preserves the functionality of the Q-K product a aWQ _ WW where * 0 is a real number. By adding scalar variable to each parameter matrix, we can formulate the objective of rescaling symmetry as follows (taking the Q-K product as an example).
where denotes the set of rotation matrices. To solve this problem, divide Equation (8) into two separate optimiza- tion problems (the first line in terms of R2 Ri as the first objective and the second line in terms of R Rho as the second objective). Considering the similar formula- tion of these two optimization problems, in this paper, we propose the following theorem to solve both problems.
Theorem 4.1. The following optimization problem has form solution.
min Ri RRER
WT
@=
T
R
2
min ||a W + +labbe, - bi2. 1|2 >| Wk. 12 + || /a bxx.1??
bq
bQ=
-R2
(11)
(9)
W
WT,
R
2
+
K1
K₂
bki
bk₃
R
where a is the rescaling variable and denotes the parame- ters after rotation commetry-based matching. Here, we still set model 2 as the anchor model and conduct the rescaling operation on model align with model We can easily
The solution is given by
RV R₂ = I,
(10)
5

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Table Experimental results of in-domain (Emotion and NER) and out-of-domain (NER-COR CoNLL) model fusion for two base lan-
Table 2. Experimental results of ViT merging over the image clas-
sification task.
guage models over Emotion and NER tasks.
w/o match
w/o rescale
match
Emotion
NER
NER-CONLL
Simple
7.60
10.22
10.19
Roberta Deberta
Roberta
Deberta
Roberta Deberta
Fisher
17.96
18.61
18.58
Simple
35.87
2.99
60.88
27.54
26.86
10.80
Regmean
14.24
15,31
15.35
+match
35.87
2.99
60.88
31.31
26.87
23,30
OT-ACTS-EMD
32.08
32.50
32.53
Fisher
44.02
35.95
54.55
33.20
23,06
12.53
OT-ACTS
61.15
61.23
61.25
+match
44.05
35.98
54.58
33.83
23.05
18.33
OT-WTS
57.11
57.16
57.17
Regmean
35.87
2.99
60.88
27.54
26,86
10.80
+match
39.95
2.99
60.88
31.31
26.87
14.06
5. Experiments
solve the optimality condition of Equation (11) as
In this section, we conduct experiments to evaluate the ef- fectiveness of our proposed parameter matching algorithm based on rotation symmetry. Specifically, we aim an- swer the following research questions: RQI: Can our pro- posed parameter matching algorithm enhance the perfor- mance of model fusion for transformer-based models? RQ2: How does rotation symmetry contribute to parameter match- ing for self-attention layers? RQ3: How does parameter matching influence the loss landscape between indepen- dently trained models? RQ4: As a plugin module, does our algorithm introduce significant additional computational overhead? RQ5: Is matching all transformer layers equally important? Can we improve efficiency without compromis- ing utility by matching only subset of layers?
(IWW,,l 1I + 162,111) at 1|| 1|2 -166k !|? - ((WQ,) W + (t/2,> ) a² + +((W,) Wk.) + (bk, kkk)) (12)
The roots of Equation (12) can be derived using numerical methods as the value of a. We then conduct the rescaling operation to model I as W + aW Q₁ 1/2 + ab'2,' Wk. + W K, b'K, - Lb'K It is worth noting that the rescaling symmetry-based matching is conducted after the rotation symmetry-berd based matching. Using the rescaling operation, we extend the rotation matrices to orthogonal matrices with larger norms, Nevertheless, the end models close to each other in practical scenarios so the value of is usually close to 1.
5.1. Experimental Settings
Platform. Our implementation is based Python 3.10 and Pytorch 1.13. All fine-tuning, parameter matching, and model merging processes are conducted on a cluster equipped with Nvidia A100 80GB GPUs.
Complexity. We next provide brief analysis of the com- plexity of our proposed parameter matching algorithm. We let the hidden dimension of the target transformer be d, the layer of the target transformer be L. and the number of atten- tion heads be H. For the parameter matching of feedforward networks, the linear assignment problem can be solved in Odd) by the Hungarian algorithm (Kuhn, 1955; Martello & Toth, 1987). Additionally, to solve the optimization prob- lem Equation (8), Algorithm requires the complexity of (d)) for eigendecomposition. Hence, the complexity of proposed full parameter matching algorithm of transformer is (d LH), similar to the complexity of the feedforward. It is worth noting that the complexity can be further reduced in two ways. The first way is to match a subset layers instead of all. In this way, the complexity of our proposed parameter matching algorithm is Odd' L,H) where L. denotes the number of selected layers. The second way is match each unit module (a single feedforward net- work or a single attention layer) in parallel. The decoupling of matching differrntt modules makes it easy to implement multiprocessing. Consequently, the overall complexity be- comes to (dd LH/p) where p is the number processes in parallel.
Models. To evaluate the effectiveness of our ap- proach, we use two widely adopted transformer mod- els: RoBERTa (Liu et al., 2019) and DeBERTa (He et al., 2021). We obtain pretrained models of RoBERTa (roberta-bass with 12 attention layers) and DeBERTa- Large (mi crosoft /ddberttaa---aaneeeeeeeee tention layers) from the Hugging Face library. For vision transformers (ViT') (Dosovitskiy et al., 2021), we directly use the pretrained models following (Imfeld et al., 2024). By selecting these three type of models, we assess the ef- fectiveness of our method across different model scales and downstream tasks.
with 24 at-
Finetuning and Matching. In the experiments, each pre- trained language model is fine-tuned for 20 epochs on each dataset individually. We set the learning rate at 1e-5, the batch size at 16, and the warmup ratio at 0.06 for each model. After fine-tuning, we perform parameter matching and model merging, considering both in-domain (pairwise
6

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
ATTN
10.45
wio ATTN
12.21
wio ATTN
11.89
wio FEN
11.15
20.21
FFN
12:94
rescoling
10.22
wil rescaling
18.6
w/ rescaling
15.3
match (Ours)
10.19
match (Ours)
match (Ours)
15.35
10
11
12
10.0
12.5
15.0
17.5
20.0
12
14
16
Simple
Fisher
Regmean
ATIN
$2.08
wio IN-
61.15
wio ATTN-
57.11
wio FFN
28.66
60.07
56.16
wio rescallng
32.5
wil resealing
61.23
wio nescaling
match (Ours)
march (Curs)
match (Ours)
57.17
28
50
52
34
60.0
60.5
61.0
61.5
56.0
56.5
$7.0
57.5
OT-ACTS- B--
OTAATS
OL-WIS
Figure 3. Ablation Study of VIT merging over the image classification task. "ATTN" is short for "attention", We compare our matching algorithm with its three variants (wlo ATTaFFN/nescalingg) and the original performance (w/o match) on all six merging baselines.
VIT Settings. We employ three OT (optimal transportation)--aaed methods (OT-ACTS-EM)) OT-ACTS, and OT-WTS) from (Imfeld et al., 2024) as additional baselines. Two pretrained models are merged, and evaluate their performance the CIFAR- 10 (Krizhevsky et al., 2009) dataset. Table 2 presents the classification accuracy of the merged models. To improve the flexibility of parameter matching, we match each layer separately before conducting model fusion. The merged model achieving the highest validation performance is selected the final test model.
wib march
651.52 $7
Git Re- Basin
625.73 73
OT-Fusion
626.16
w/o ATTN, rescaling
640.0]
w/o ATTN
639.96
w/o rescaling
61.3 3,68
match (Ouns)
613.63
600
610
620
630
640
650
660
670
Distance
Figure 4. The Euclidean Distance of end ViT models after different parameter matching algorithms.
fine-tuned models) and out-of-domail (grouped models) experiments. Notably, we match only the parameters within the attention layers, while the classifier module is directly copied from the model fine-tuned on the corresponding downstream task. For additional details on datasets and baseline methods, please refer to Appendix C.
From the results in Table I and Table 2, we make the following observations: (1) Our parameter matching al- gorithm consistently improves the performance of differ- ent model fusion methods. The reason is that parameter matching helps align distant models, bringing them closer in parameter space. As a result, the merged model is more likely approach an overall minimum. (2) Compared to RoBERTa-base, our method brings larger improvement for DeBERTa-large, suggesting that larger models benefit more from parameter matching. We explain this as larger models have more parameters that can be aligned, and in high- dimensional spaces, models tend to be more spread out. Subsequently, parameter matching helps bridge this gap more effectively. (3) Among all the fusion methods, sim- ple fusion shows the largest improvement after parameter matching. This is likely because simple fusion directly av- erages model weights, so reducing the distance between model parameters brings a clear benefit. In contrast, ad- vanced fusion methods work by aligning outputs of the end models based on input data, which is equivalent to weighted averaging. Unlike direct averaging, these methods benefit from parameter matching in more implicit way.
5.2. Performance of Model Fusion
To answer RQI, we compare the performance of three model fusion baselines with and without our proposed parameter matching algorithm.
In-Domain Settings. We evaluate model fusion perfor- mance on Emotion Classification and Named Entity Recog- nition (NER) tasks, where our approach (+match) is inte- grated as plugin module for three model fusion baselines. For each task, we fine-tune the language models on each in-domain dataset (5 for Emotion, 6 for NER) respectively and merge them pairwise. The first two columns in Table present the average macro F1-score (for Emotion) and miero Fl-score (for NER) of the merged models following (Jin al., 2023).
Out-of-Domain Settings. To assess the generalization ability, we merge models trained on all in-domain NER datasets and evaluate their performance on CoNLL datasets, which serve as out-of-domain (OOD) test sets. The third column in Table reports the OOD performance of the merged models.
5.3. Ablation Study
To answer RQ2, we conduct ablation studies evaluate the contributions of rotation symmetry in matching different model components. These experiments are performed on the same pair of pretrained VIT models used in the previous
7

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
win morch
Setting
10006
ATTN
match 2.295
0.7425
7,0550
5.799
Barriers
wio FFN
match
w/o ATTN 2.331
w/o FFN 2.231
(alAgloo)
0,7464
7.015g
5 Biooe
Oala (uaa)
5.78
w/ match
2.208
7.0550
4.77
0.0
0.2
0.4
0.6
0.8
1.0
0.7488
7.0545
0,7030
OaLA STS-8
1.776
STS-6
Figure 5. Loss landscapes and barriers between the two pretrained ViT models under four distinct matching settings. "LB" short for "Loss Barrier".
7,0540
byyer indes
mondhed laytr
(a) Single-Layer Matching
(b) Tail-Layers, Matching
Figure Evaluation loss of matching a subset of layers.
Table 3. The average runtime in seconds of the fine-tume fine-tuning (top), matching (middle), and merging (bottom) stage.
5.4. Loss Landscape Study
Deberta
Roberta
ViT
Fine-tuning Matching
12983.71
3071.49
To address RQ3, we examine the loss landscape between the two pretrained ViT models introduced in Section 5.2, both with and without parameter matching. Specifically, we first analyze the cross entropy loss values [(O(a)) along the linear interpolation path between the end model parameters 0(a) = 0 + (1-a) -a)-)-0. a E [0, 1J. where Oa and OB denote the end models. Next, we separately apply attention- only matching (w/o FFN), FFN-only matching (w/o A77N), and complete matching (w/ match) to 0p before construct- ing the same linear interpolation path with respect to 0a. The results Figure 5 give rise to the following observa- tions: (1) All model pairs exhibit positive loss barriers, i.e., the range of loss value along the interpolation path, high- lighting the strong inherent non non-convyxity of ViTs (Park & Kim, 2022). (2) Complete matching yields the lowest loss barrier, indicating that aligning parameters through rotation symmetry leads to smoother connectivity between model pairs. (3) While FFN-only matching (w/o ATTN) has been proven effective for MLPs and CNNs (Ainsworth al., 2023), its effectiveness remains limited for Transformers. Their intricate loss landscape brings additional challenges for parameter matching.
1.59
1.71
3,48
Simple Merging
0.13
0.09
0.22
Fisher Merging
197.47
69.57
83.67
Regmean Merging
137.67
36.44
71.02
section. We design three ablation settings: (1) ATTN: Matches only the FFN modules, excluding self-attention modules. (2) w/o FFN: Matches only the self-attention modules, excluding FFN modules. (3) w/o rescaling: Dis- ables the rescaling symmetry in our algorithm. Ablation results in Figure 3 demonstrate the following observations: (1) In Fisher and RegMean, attention matching contributes more significantly to the fusion process. (2) In contrast, OTFusion benefits more from FFN matching. These distinc- tions demonstrate the nuanced interactions between the fu- sion mechanisms and corresponding component alignment strategies. (3) w/o rescaling results in performance degra- dation across most settings, validating the effectiveness of rescaling symmetry integrated with rotation symmetry.
To further quantify the influence of rotation symmetry, we measure the Euclidean distance between the matched model and the anchor model in the parameter space. Instead of ag- gregating distances at the layer or module level, we analyze the entire model holistically to provide a more comprehen- sive assessment of alignment. We also compare our method against two external baselines: Git Re-Basin (Ainsworth et al., 2023) and OT-Fusion (Imfeld et al., 2024). Follow- ing the original paper. OT-Fusion is applied only to FFN modules. The results in Figure 4 indicate that our rotation symmetry-based parameter matching algorithm consistently reduces the distance between end models more effectively methods. This improve- ment can be attributed to the continuous nature of rotation symmetry, which allows for smoother and more precise alignment, especially in self-attention layers. Furthermore, the integration of rescaling symmetry leads to additional dis- tance reductions of end models, further enhancing parameter alignment.
5.5. Complexity Study
To answer RQ4, we evaluate the computational overhead in- troduced by our parameter matching algorithm. Specifically, we measure the average runtime for fine-tuning (per dataset), matching (per model pair), and merging methods (per model pair) from the main experiments, as shown Table 3. For ViTs, we only use pre-trained models following (Imfeld et al., 2024) without fine-tuning. The results show that our parameter matching module incurs an overhead of less than 5% compared to Fisher RegMean merging across all models, with negligible impact on overall complexity.
than permutation symmetry-beat
5.6. Matching a Subset of Layers
To answer RQS, we investigate the effect of matching dif- ferent subsets of layers. We fine-tune DeBERTa models on CoLA and STS-B from the GLUE benchmark and evaluate the performance of matched models under different subsets
8

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
of layers. Specifically, we cross-validate the importance of each attention layer in matching through two settings:
to low-rankness. Meanwhile, Tran al. (2024) leverages the rotation symmetry construct transformer-baset neural functional networks. In comparison, our study focuses on the role of rotation symmetry in model fusion and proposes a theoretically optimal parameter matching approach based on the properties of rotation symmetries.
Single-Layer Matching. In this setting, we match only a single attention layer while leaving all other layers un- changed. The evaluation loss corresponding different matched layer indices is shown in Figure 6(a).
Model Fusion. The goal of model fusion (Li et al., 2023) is to merge multiple available end models (with the same architecture) to obtain stronger model. The scenarios of model fusion can be flexible. When training on the same dataset, model fusion can be used to improve the model utility generalization by merging models trained with dif- ferent configurations or in different stages (Izmailov et al., 2018; Gupta et al., 2020; Cha et al., 2021; Wortsman et al., 2022; Rame et al., 2022; Arpit et al., 2022; Huang et al.,
Tail-Layers Matching. Here, we match a certain num- ber of trailing attention layers (e.g., when matching three attention layers, we align only the last three layers). The evaluation loss for different numbers of matched layers is presented in Figure 6(b).
The experimental results in Figure 6 demonstrate that match- ing head layers yields greater improvements in model util- ity compared to tail layers. Notably, the loss value drops sharply when the first layers are matched. Based on this observation, we can further improve efficiency by dropping tail layers without significantly compromising utility.
2024; Yadav et al., 2024; Hammoud et al., 2024). As a representative method in this setting, ModelSoup (Worts- man et al., 2022) greedily averages the models fine-tuned with different hyperparameter configurations to improve the utility and robustness of the model. In addition, when training on different datasets or tasks, model fusion can be used to improve out-of-domain generalization or mul- titasking of the model (Matena & Raffel, 2022; Choshen et al., 2022; Li et al., 2022; Jin al., 2023; Zhou et al., 2024), especially for language models. A state-of-the-art merging algorithm, RegMean (Jin et al., 2023), success- fully merges language models fine-tuned over different tasks and improves the model's out-of--istribine Moreover, model fusion plays a pivotal role in federated learning (Konečný et al., 2016; McMahan et al., 2017; Wang et al., 2020) when the local updates are collected to make a global update. FedAvg (McMahan et al., 2017) is classical merging algorithm that directly computes the average of the local models as the updated global model. Recent studies propose to incorporate the permutation symmetry to align the neurons of different end models (Wang et al., 2020; Singh & Jaggi, 2020; Ainsworth et al., 2023). However, these methods fail to achieve a desirable performance when tackling transformer- based models (Jin et al., 2023).
6. Related Work
Parameter Space Symmetry. Parameter space symmetry refers to set of models with different parameter values but functionally equivalent. This concept has been exten- sively studied in the context of deep neural networks, as it plays a crucial role in understanding model behavior and training dynamics. Examples of parameter space symme- tries include rescaling symmetry (Neyshabur et al., 2015; Badrinarayanan et al., 2015; Du et al., 2018; Meng et al., 2019), scaling symmetry (Kunin et al., 2021), and trans- lation symmetry (Kunin al., 2021). These symmetries have been identified in conventional deep neural networks to provide deeper insights into training dynamics and to accelerate the optimization process (Zhao et al., 2022; 2023; 2024). Another important type of parameter space sym- metry is permutation symmetry, which has been shown to closely relate to the manifold of global minima and critical points (Fukumizu & Amari, 2000; Brea et al., 2019; Sim- sek et al., 2021; Benton et al., 2021; Entezari et al., 2022; Ainsworth et al., 2023). The permutation symmetry can also be used to align (match) the outputs or model parameters of different end models with the same architecture (Singh & Jaggi, 2020; Wang et al., 2020; Ainsworth et al., 2023; Peña et al., 2023; Imfeld et al., 2024; Navon et al., 2024). Complementary to parameter space symmetry, recent ad- vances in neural functionals and metanetworks (Navon et al., 2023; Zhou et al., 2023b;a; Lim et al., 2024a; Tran et al., 2024) explore
generalization.
7. Conclusion
In this paper, we introduced rotation symmetry as a novel type of parameter space symmetry for transformers, extend- ing the concept of permutation symmetry to continuous spaces. Building on this foundation, we proposed a theoreti- cally optimal parameter matching algorithm to enhance the fusion of transformer models in plug and play manner. To validate our approach, we conducted extensive experiments on real-world NLP and vision benchmarks. The results demonstrated that incorporating rotation symmetry effec- tively and efficiently facilitates transformer model fusion, showcasing our method's 's practical utility.
cntnuuttttioneuivariiiinn functionals that op- erate directly on model weights across diverse architectures. Some concurrent works (Liu, 2024; Tran et al., 2024) in- vestigate similar forms of rotation symmetry in neural net- works. Liu (2024) shows that the mirror symmetry leads

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Impact Statement
tion saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiiv:997.02222 2019.
Our study provides novel insights into the parameter space symmetry of transformers and establishes a practical frame- work for advancing model fusion techniques. Bridging the- oretical innovations with practical applications, this work reveals the potential for leveraging parameter space symme- tries in deep learning research. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Burkard, R. E. and Cela, E. Linear assignment problems and extensions. In Handbook of combinatorial optimization: Supplement volume A. pp. 75-149. 1999.
Acknowledgment
Cha, J., Chun, S., Lee, K., Cho. H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22416 2021.
This work is supported in part by the National Science Foundation (NSF) under grants IIS-2006844, IIS-2144209, IIS-2223769, CNS-21 BCS-2228534, and CMMI- 2411248; the Office of Naval Research (ONR) under grant N000142442636; and the Commonwealth Cyber Initiative (CCI) under grant VV-1Q24-011.
Choshen, L.,, Venezian, E., Slonim, N., and Katz, Y. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044, 2022.
References
Clark, K., Luong, M.-T., Le, Q. V., and Manning, Elec- tra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020.
Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023.
Daheim, N., Möllenhoff, T., Ponti, E., Gurevych, I., and Khan, M. E. Model merging by uncertainty- -based gradi- ent matching. In International Conference on Learning Representations, 2024.
Armenta, M. and Jodoin, P.-M. The representation theory of neural networks. Mathematics, 9(24):3216, 2021.
Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting per- formance in domain generalization. Advances in Neural Information Processing Systems, 35:8265-8277, 2022.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Proceedings of the 2019 Confer-
J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. 2016.
arXiv preprint
Badrinarayanan, V., Mishra, B., and Cipolla, R. Symmetry- invariant optimization in deep networks. arXiv preprint 111:0:451 2015.
Dietterich, T. G. Ensemble methods in machine learning. International workshop on multiple classifier systems, pp. 1-15, 2000.
Benton, G., Maddox,
W., Lotfi, S., and Wilson, A. G. G. Loss surface simplexes for mode connecting volumes and fast ensembling. International Conference on Machine Learning, pp. 769-779, 2021.
Dong, X., Yu, Z., Cao, W., Shi, Y., and Ma. Q. A survey on ensemble learning. Frontiers of Computer Science, 14: 241-258, 2020.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x words: Transformers for image recognition at scale, 2021. URL https: //arxiv. wwgo
Bostan, L.-A.-M. and Klinger, R. An analysis of annotated corpora for emotion classification in text. In Bender, E. M., Derezynski, L., and Isabelle, P. (eds.), Proceedings of the 27th International Conference on Computational Linguistics, pp. 2104-2119, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. ogy. otgg/18--11
URL hwwps:ttph.......... /
Du, S. S., Hu, W., and Lee, J. D. Algorithmic regulariza- tion in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018
Brea, J., Simsek, B., Illing, B., and Gerstner, W. Weight- space symmetry in deep networks gives rise to permuta-
10

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectiv- ity of neural networks. In International Conference Learning Representations, 2022.
Hovy, E., Mareus, M., Palmer, M., Ramshaw, L., and Weischedel, R. OntoNotes: The 90% solution. In Moore, R. C., Bilmes, J., Chu-Carroll, J., and Sanderson, M. (eds.), Proceedings of the Human Language Tech- nology Conference of the NAACI, Companion Volume: Short Papers, pp. 57-60, New York City, USA, June 2006. Association for Computational Linguistics. URL org NO6-2000000000
Ferbach, D., Goujaud, B., Gidel, G., and Dieuleveut, A. Proving linear mode connectivity of neural networks via optimal transport. In International Conference on Artifi- cial Intelligence and Statistics, pp. 3853-3861, 2024.
attps://aclanthol.e
Huang, C.,, Ye, P., Chen, T., He, T., Yue, X., and Ouyang, W. Emr-merging: Tuning-free high-performance model merging. Advances in Neural Information Processing
Fukumizu, K. and Amari, S.-i. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural networks, 13(3)331-3327, 2000.
2000.
Systems, 37:122741 2024.
Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pp. 315-323, 2011.
Imfeld, M., Graldi, J., Giordano, M., Hofmann, T., Anag- nostidis, S., and Singh, S. P. Transformer fusion with optimal transport. In International Conference on Learn- ing Representations, 2024.
Godfrey, C., Brown, D., Emerson, T., and Kvinge, H. On the symmetries of deep learning models and their internal rep- resentations. Advances in Neural Information Processing Systems, 35:1 1893-1199, 2022.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In 34th Conference on Un- certainty in Artificial Intelligence (UAI), pp. 876-885, 2018.
Gower, J. C. and Dijksterhuis, G. B. Procrustes problems, volume 30. OUP Oxford, 2004.
Jin, X., Ren, X., Preotiuc-Pictro, D., and Cheng, P. Data- less knowledge fusion by merging weights of language models. In International Conference Learning Repre- sentations, 2023.
Grigsby, E., Lindsey, K., and Rolnick, D. Hidden symme- tries of relu networks. In International Conference on Machine Learning, pp. 11734-11760 2023.
Gupta, V., Serrano, S. A., and DeCoste, D. Stochastic weight averaging parallel: Large-batch training that generalizes well. In International Conference on Learn- ing Representations, 2020.
Kabsch, W. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffruction, Theoretical and General Crystallog- raphy, 22(5):922-9233 1976.
Hammoud, H., Michieli, U., Pizzati, F., Torr. P., Bibi, A., Ghanem, B., and Ozay, M. Model merging and safety alignment: One bad model spoils the bunch. In Find- ings of the Association for Computational Linguisties: EMNLP 2024, pp. 13033-13046, 2024.
Kalogeropoulos, I., Bouritsas, G., and Panagakis, Y. Scale equivariant graph metanetworks. Advances in neural in- formation processing systems, 37:106800-106844 2024.
Konečný, J., McMahan, H. B., Yu, X., Richtárik, P., Suresh, A. T., and Bacon, Federated learning: Strate- gies for improving communication efficiency. arXiv preprint arXiv:1610.0555 2016.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn- ing for image recognition. In Proceedings of the IEEF conference on computer vision and pattern recognition, pp. 770-778, 2016.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention, 2021.
He, Y., Zheng, Z., Soga, P., Zhu, Y., Dong, Y., and Li, J. Explaining graph neural networks with large language models: A counterfactual perspective on molecule graphs. In Findings of the Association for Computational Linguis- fies: EMNLP 2024, pp. 7079-7096, 2024.
Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83- 97, 1955.
Kunin, D., Sagastuy- J., Ganguli, S., Yamins, D. L., and Tanaka, H. Neural mechanics: Symmetry and bro- ken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021.
Hecht-Nielsen, R. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp. 135. 1990.

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. pre-training for natural language generation. translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pp. 787 1-7880, 2020.
y Arcas, B. Communication-effice efffcientt learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, 2017.
Bart: Denoising sequence-to-sequenn
Meng, Q. Zheng, S., Zhang, H., Chen, W., Ye, Q.. Ma, Z.-M., Yu, N., and Liu, T.-Y. G-sgd: Optimizing relu neural networks in its positively scale-invariant space. In International Conference on Learning Representations, 2019.
Li, M., Gururangan, S., Dettmers, T., Lewis, M., Althoff, T,,, Smith, N. A., and Zettlemoyer, L. Branch-train-merg- Embarrassingly parallel training of expert language mod- cls. In First Workshop on Interpolation Regularizers and Beyond at NeurlPS 2022, 2022.
Nair, V. and Hinton, G. Rectified linear units improve re- stricted boltzmann machines. In International Conference Machine Learning. pp. 807-814 2010.
Navon, A., Shamsian, A., Achituve, I., Fetaya, E., Chechik, G., and Maron, H. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pp.
Li, W. Peng, Y., Zhang, M., Ding, L., Hu, H., and Shen, L. Deep model fusion: A survey, arXiv preprint WiXv:2339.15696 2023.
2023.
Lim, D., Maron, H., Law, M. T., Lorraine, J., and Lucas, Graph metanetworks for processing diverse neural architectures. In International Conference on Learning Representations, 2024a.
Navon, A., Shamsian, A., Fetaya, E., Chechik, G., Dym, N., and Maron, H. Equivariant deep weight space alignment. In International Conference on Machine Learning, pp. 11376-37395, 2024.
Lim, D., Putterman, T., Walters, R., Maron, H., and Jegelka, S. The empirical impact of neural parameter symme- tries, lack thereof. Advances in Neural Information Processing Systems, 37:28322-28358, 2024b.
Neyshabur, B., Salakhutdinov, R. R., and Srebro, N. Path- sgd: Path-normalized optimization in deep neural net- works. Advances in neural information processing sys- lems, 28, 2015.
Liu, Y. Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /900./ 1692, 2019.
Park, N. and Kim, S. How do vision transformers work?, 2022. URL 2202.06700
Peña, F. G., Medeiros, H. R., Dubail, T., Aminbeidokhti, M., Granger, E., and Pedersoli, M. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20237-20246, 2023.
Liu, Z. Symmetry induces structure and constraint of learn-
ing. In International Conference on Machine Learning, 2024.
Liu, Z.,, Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision. pp. 10012-10022, 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. 2021.
Lubana, E, S., Bigelow, E. J., Dick, R. P., Krueger, D., and Tanaka, H. Mechanistic mode connectivity. In Inter- national Conference on Machine Learning, pp. 22965- 23004, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Martello, S. and Toth, P. Linear assignment problems. In North-Holland Mathematics Studies, volume 132, pp- 259-2882. 1987.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averaging for
12

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821-10836, 2022.
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. In International Conference Learning Representations, 2020.
Rossi, S., Singh, A., and Hannagan, T. On permutation symmetries in bayesian neural network posteriors: a vari- ational perspective. Advances in Neural Information Pro- cessing Systems, 36. 2023.
Wei, X. and Liu, L. Are large language models good in- context learners for financial sentiment analysis? arXiv preprint arXiv:2503.0877. 2025.
Sagi, O. and Rokach, L. Ensemble learning; A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8(4): 2018.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference machine learning, pp. 23965- 23998, 2022.
Schönemann, P. H. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
Simsek, B., Ged, F., Jacot, A., Spadaro, F., Hongler, C., Gerstner, W., and Brea, Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp. 9722-9732, 2021.
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. Ties dee-me:ginn: Resolving interference when merging models. Advances in Neurul Information Processing Systems, 36, 2023.
Singh, S. P. and Jaggi, M. Model fusion via optimal trans- port. Advances Neural Information Processing Systems, 33:22045-22055, 2020.
Yadav, P., Vu, T. Lai, J., Chronopoulou, A., Faruqui, M., Bansal, M., and Munkhdalai, T. What matters for model merging at scale? arXiv preprint arXiv:2410.03611 2024.
Strang, G. Linear algebra and its applications. Academic Press, 1976.
Yang, E., Wang. Z., Shen, L.,, Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024.
Tatro, N., Chen, P.-Y., Das, P., Melnyk, I., Sattigeri, P., and Lai, R. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300- 15311, 2020.
Yun, S., Jeong, M., Kim, R., Kang, J., and Kim, H. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
Tjong Kim Sang, E. F. and De Meulder, F. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Sev- enth Conference on Natural Language Learning at HLT- NAACL 2003, pp. 142-147, 2003. URL https:// ac mellanhollle ogy. org/ 2tgg/03--m -03-041-
Yurochkin, M., Agarwal, M., Ghosh, S., Greenewald, K., Hoang, N., and Khazaeni, Y. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pp. 7252-7261, 2019.
Tran, V.-H., Vo, T. N., The, A. N., Huu, T. T., Nguyen- Nhat, M.-K., Tran, T., Pham, D.-T, and Nguyen, T. M. Equivariant neural functional networks for transformers. arXiv preprint arXiv: 2e::::24000044
Zamir, G., Dokania, A., Zhao, B., and Yu, R. Improving learning to optimize using parameter symmetries. arXiv preprint arXiv:2504. /5399, 2025.
2rX.v:2410.04209, 042000 2024.
Zhang, B., Chen, 7.,, Zheng, Z.,, Li, J., and Chen, Resolv- ing däiting-unlearning conflicts: A knowledge codebook framework for large language model updating. arXiv preprint vrXiv:2502.0013 58, 2025.
Umeyama, S. squares estimation of transformation parameters between two point patterns. IEEE Transac- fions on Pattern Analysis & Machine Intelligence, 13(04): 376-380, 1991.
Zhao, B., Dehmamy, N., Walters, R., and Yu, R. Symmetry teleportation for accelerated optimization. Advances in neural information processing systems, 35:16679-16660, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.,, Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Zhao, B., Ganev, I., Walters, R., Yu, R., and Dehmamy, N. Symmetries, flat minima, and the conserved quantities of gradient flow. In The Eleventh International Conference on Learning Representations, 2023.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analy- sis platform for natural language understanding, 2019.
13

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Zhao, B., Gower, R. M., Walters, R., and Yu, R. Improving convergence and generalization using parameter symme- tries. In International Conference on Learning Represen- tations. 2024.
Zheng, Z., Dong, Y., Wang, S., Liu, H., Wang, Q. and Li, Kg-ef: Knowledge graph completion with context filter- ing under the guidance of large language models. In 2024 IEEE International Conference on Big Data (BigData), pp. 805-810, 2024.
Zhou, A., Yang. K., Burns, K., Cardace, A., Jiang, Y., Sokota, S., Kolter, J. Z., and Finn, C. Permutation equiv- ariant neural functionals. Advances in neural information processing systems, 16:24966-24992 2023a.
Zhou, A., Yang, K., Jiang, Y., Burns, K., Xu, W., Sokota, S., Kolter, J. Z., and Finn, C. Neural functional transformers. Advances in neural information processing systems, 36: 77485-77502, 2023b.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Zhou, Y., Song, L.., Wang, B., and Chen, W. Metagpt: Merging large language models using model exclusive task arithmetic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1711-1724, 2024.
Zhou, Z. Yang, Y., Yang, X., Yan, J., and Hu, W. Going beyond linear mode connectivity: The layerwise linear feature connectivity. Advances in Neural Information Processing Systems, 36, 2023c.
Zhu, Y.. Wu, L., Zhang, B., Wang, S., Guo, Q. Hong, L., Simon, L., and Li, J. Understanding and modeling job marketplace with pretrained language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 5143- 5150, 2024.
Ziyin, L., Wang, M., Li, H., and Wu, L. Parameter symme- try and noise equilibrium of stochastic gradient descent. arXiv preprint arXiv:2402.07193, 2024.
Ziyin, L., Xu, Y., and Chuang. I. L. Remove symmetries to control model expressivity and improve optimization. In International Conference on Learning Representations, 2025.
14

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
A. Proof
Theorem 4.1.
The following optimization problem has a closed-form solution.
W
W@:
R
2
WKL
W
Ri
min RRR
Q2
K1
K2
(13)
bq.
bQ=
- R
bk.
bk²
- R
F
The solution is given by
R = UVV RR= I,
(14)
where I the identity matrix and UEVT =Wo, Wo. + WK, WK, + bb. bQz + bis, bxg is the result of eigendecompo- sition.
Proof. We first show the process of converting the optimization problem to an Orthogonal Procrustes problem (Schönemann, 1966; Gower& & Dijksterhuis, 2004). We can obtain that the optimization problem is equivalent to the following one.
min R.. RgER
W o Ri Wo, R |iQq, Ri bq, Rall
R W K1
WK, Rell + ||bx, Rı -5, R2ll.
(15)
Considering that R1, R2 E R and R1, RS are nonsingular matrices, we let R = R2 Ry and convert Equation (15) to the following one:
(1 R R- W Rallz- + 11(bo, R boz) Rall + [1 T Ki R.REER
min
R- W
Rs ((bb, R- ((2-b,,) bxx) Ra 1
(16)
As multiplying R2 preserves the Frobenius norm, we can remove the R2 terms from the objective and obtain an Orthogonal Procrustes problem as follows:
min W R
W 1 lbq. R - boall || W R Wx.ll² |b R - bx
(17)
We take a look at the first term W R
W - and have:
min
W R W
Q.
RER
min
(W R - W W R - -W
RER
max
W.. R. (
(18)
RER
mma (R Wa. w RER
max R. WQ. W
Similarly, we can convert Equation (17) to the following one:
max R. Wo, WT. + WK, WT. + b bq. bb,, bk.)
(19)
RER
We then conduct the singular value decomposition to the matrix Wo, WT. Q: + WK, W K2 + +bg bq. + bb, bx. = UVVT and have:
max (R, We, WT.
Q₂ + WK, WK, + bQ. boz + bb, bks)r
RER
max Lr (VEUT R)
RER
(20)
max ir RER
(EUT RV)
-tr
where U RV- = i.,.e R= UV T Finally, we incorporate R = TVT into R = Subsequently, we have R = UV and R = I as a closed-form solution of the optimization problem in Theorem 4.1.
R=R,R, R and let R2 I for simplicity.
15

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
From the proof, we can observe that the optimization problem in Theorem 4.1 has infinite pairs of solutions {R,, R2} regarding the value of Rg. In this paper, let R2 I for simplicity, which makes model 2 anchor model. However, the value of R2 can be further adjusted to benefit some data-dependent model fusion techniques (Singh & Jaggi, 2020; Jin et al., 2023). We leave this part to future work.
B. Comparison with Related Work
Git Re-Basin (Ainsworth al., 2023) introduces three model fusion algorithms: weight matching, activation matching, and straight-through matching, based permutation symmetry. While these methods are effective for CNNs and MLPs, they do not easily extend to transformers due to the discrete nature of permutation symmetry. In discrete space, the parameter matching problem is equivalent to solving sum of bilinear assignment problems, which is NP-hard (Ainsworth et al., 2023). To address this limitation, we introduce rotation symmetry, which extends the symmetry space to a continuous domain, allowing for a closed-form solution to parameter matching. Unlike permutation symmetry, rotation symmetry provides a more flexible and efficient solution for aligning parameters in transformers.
Additionally, our study emphasizes the advantage of weight matching as a plug-and-play module for model fusion, as preserves model functionality while reducing the inner distance of end models after matching. OT-ACTS (Imfeld et al., 2024) also includes parameter matching step, leveraging optimal transport for model fusion. However, its matching module is restricted to permutation operations, and its fusion method is limited to simple merging. In contrast, our approach generalizes parameter matching to rotation operations and integrates it with more advanced merging strategies, demonstrating the potential of parameter matching to enhance model fusion.
concurrent work by Tran et al. (2024) explores a similar form of rotation symmetry. However, their focus is on constructing functionally equivalent networks for transformers, whereas we focus on leveraging rotation symmetry for model fusion, proposing a theoretically optimal parameter matching algorithm specifically designed to improve fusion performance.
C. Experimental Details
Datasets. Similar with (Jin et al., 2023), We employ emotion classification and named entity recognition (NER) as the tasks for the main experiments. The emotion classification datasets are extracted from (Bostan & Klinger, 2018). Five of them are selected as in-domain datasets (emoint, ssec, electoraltweets, grounded emotions, affectivetext). and five others are good datasets (dailydialog, crowdflower, tec, tales-emotion, isear). For the NER task, we choose OntoNotes (Hovy et al., 2006) for model finetuning and CoNLL (Tjong Kim Sang & De Meulder, 2003) for out-of-domain evaluation. Additionally, in the ablation study and subset layers matching, we employ datasets (STS-B, SST-2, CoLA) from GLUE (Wang et al., 2019) to analyze the module performance.
Baselines.
We compared the performance of our method with two matching baselines:
OTFusion (Singh & Jaggi, 2020) mitigates the positional mismatch of functionally analogous neurons across end models by employing optimal transport to align neurons layer-wise, thus enabling more effective model merging.
Git Re-Basin (Ainsworth et al.. 2023) matches the parameters of end models through learned permutation symmetries in the parameter space, aiming to transform their parameters into shared basin within the loss landscape and facilitate seamless merging.
Meanwhile, to address RQI, we further select six merging methods as baselines:
Simple (Wortsman et al., 2022) directly averages the weight matrices of independently pretrained end models. Fisher (Matena & Raffel, 2022) approximates each end model's posterior as a Gaussian distribution with a precision matrix derived from its Fisher information, thereby casting model merging as a Maximum Likelihood Estimation (MLE) problem over these posterior distributions.
RegMean (Jin et al., 2023) formulates model merging regression problem, aiming minimize the Euclidean distance between the predictions of the merged model and individual end models over a set of input samples.
OT-WTS (Imfeld et al., 2024) extends OTFusion by introducing soft neuron alignment, which replaces the rigid one-to-one neuron matching with a more flexible, weighted correspondence between neurons. Subsequent model merging is then guided by these soft alignment weights. OT-ACTS and OT-ACTS-EMD (Imfeld et al., 2024) are two variants of OT-WTS that match neurons based on their
16

---

Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
activations rather than directly aligning parameter matrices. Specifically, OT-ACTS measures activation similarity using Euclidean distance, while O1-ACTS-EMID employs Earth Mover's Distance for more fine-grained comparison.
The code implementation is based on (Jin et al., 2023)² for NLP tasks and (Imfeld et al., 2024)² for vision tasks.
D. Limitations
In this study, we proposed theoretically optimal parameter matching algorithm based on rotation symmetry. However, the optimality guarantees of our method are subject to several important prerequisites.
First, theoretical analysis in Theorem 4.1 focuses on the binary model matching case, where the optimal solution is independent of the choice of anchor model. This property, however, does not naturally extend to the multi-model setting. When matching multiple models, a straightforward extension involves selecting a single model as an anchor and aligning the others to using pairwise rotation- -baeed matching. Yet, in this case, the global optimality depends on the definition of the overall distance metric and may not preserved. An alternative iterative pairwise matching, this introduces path dependency, i.e., the final result can vary depending on the order of merging. Extending our framework to the optimal multi-model matching problem remains an open challenge for future work.
Second, the scope of our parameter matching algorithm is within the rotation symmetry, as defined in Equation (5). While this serves as tractable and well-motivated instance of parameter space symmetry. we recognize that Transformers exhibit a richer set of structural symmetries beyond permutation, rotation, and rescaling. For example, recent work suggests that geometric properties of normalization and softmax layers may enlarge the symmetry set of previous model layers (Kunin et al., 2021; Lim et al., 2024a). Characterizing the complete symmetry group of transformers remains an open question.
We also considered the possibility of extending rotation matrices to invertible matrices in terms of parameter space symmetries. Although invertible matrices expand the symmetry set, they are less suitable for practical parameter matching. Specifically, our closed-form solution in Equation (9) relies on the orthogonality of the rotation matrices. If replacing them with general invertible matrices, it becomes difficult to derive closed-form solution for Equation (9), leading to suboptimal solution and increased complexity. Additionally, parameter matching requires computation of the inverse matrix of every R, which is impractical for invertible matrices. In contrast, orthogonal matrices benefit from computationally efficient inverses (via transposition) and theoretical simplicity.
Finally, we apply the rescaling symmetry after optimizing for rotation-based matching. However, our sequential approach (first optimizing rotation R. then rescaling weight a:) does not guarantee global optimality for the joint optimization over (R, a). Deriving globally optimal solution would require solving Equation (12) in closed form, which remains ton-trivial. Hence, we consider our method as practical approximation to the joint optimization solution which balances theoretical soundness with computational efficiency. Exploring a fully joint optimization framework for rotation and rescaling symmetries can be a promising direction for future research.
fusion
17


Reviews:
[{"review_id": "7x7dkesBxc", "reviewer": "Reviewer_ydmT", "summary": "The paper extends the concept of permutation symmetry in MLPs to rotation symmetry for the self-attention layer in the transformers. The authors show that due to the inherent design of self-attention layers, each of the query, key, and value vectors can be rotated without changing the functional representation and thus can be used to match two transformer models. Based on this, authors propose a model merging algorithm for transformer-based models and empirically show that by matching two models, merging can be improved.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback. We are honored to have this valuable chance to address your raised concerns and questions. We believe that our paper will be much stronger thanks to your efforts.\n\n**Response to claims and W2**:\nWe thank the reviewer for this insightful suggestion. \nIn the binary case, the optimality of parameter matching does not rely on the choice of anchor models. However, this might not hold for multiple models.\nWhen merging multiple end models, a naive extension of our method is to select a single model as an anchor and align all others to it via rotation symmetry, as done in pairwise merging. \nHowever, the optimality of this approach depends on how the overall distance measure is defined.\nAn alternative approach could involve iterative pairwise merging, but this introduces path dependency issues where the final result depends on the order of merging. \nWe agree that finding optimal alignments across multiple models is an interesting and important direction and plan to explore it in the future work.\nWe will make corresponding adjustments to the claims in the revision of our paper.\n\n**Response to experiments 1**:\nThank you for this valuable suggestion. We've computed the LMC curves of the ViT model with and without our proposed matching technique on the CIFAR-10 dataset. Our analysis reveals two key observations:\n\n1. For most interpolation coefficients $\\lambda\\in[0,1]$, models merged with our matching approach exhibit lower loss values than unmatched models, indicating improved connectivity.\n\n2. We observe a loss barrier from the LMC curve between the end models used in our experiments. We explain this because ViT exhibits a strong non-convexity [1]. Similar results are recorded in previous works [2].\n\nWe will include detailed LMC results in our revised manuscript to fully illustrate these findings.\n\n**Response to experiments 2 and W1**:\nWe thank the reviewer for this insightful question. We've analyzed the LMC curves between end models fine-tuned on different NLP tasks and observed that all model pairs (in our settings) exhibit positive loss barriers, confirming that fine-tuning often does move models into different loss basins. However, the magnitude of these barriers varies across tasks.\n\nImportantly, we observe positive loss barriers between the non-anchor original model and its matched counterpart. This provides evidence that our matching technique achieves model rebasin through rotation symmetry.\n\nRegarding why matching only the first few layers is often sufficient: this suggests that rotational symmetry divergence primarily occurs in early layers during fine-tuning, while later layers maintain more consistent representations. This aligns with observations that early layers capture more task-specific features [3].\n\nWe acknowledge that precisely characterizing loss basin distributions remains an open problem due to the highly non-convex nature of transformer loss landscapes. Even for simpler architectures like MLPs, loss barrier analysis remains challenging. We believe this is a valuable and insightful direction for future research.\n\n**Response to comments**:\nWe believe there might be misunderstanding regarding this sentence. In our manuscript, we mentioned that the **linear assignment problem** can be solved precisely by Hungarian Algorithm, rather than weight matching or activation matching problems.\n\n**Response to questions**:\nWe thank the reviewer for this important point. To our knowledge, our experimental setting differs from the model soup paper [4].\nThe model soup paper primarily studies merging models trained on **the same dataset** with different hyperparameters, where models likely remain in the same loss basin as they optimize for the same objective.\nIn contrast, our experiments merge models fine-tuned on different datasets. Our LMC analysis confirms these models occupy different loss basins, as evidenced by positive loss barriers. \n\nWe extend our best gratitude for your efforts in the rebuttal phase. We highly value the opportunity to improve our paper. We will include the additional results and discussions in the next version of our paper. We sincerely appreciate your time and consideration.\n\n[1] Park, Namuk, and Songkuk Kim. \"How Do Vision Transformers Work?.\" ICLR 2022.\n\n[2] Imfeld, Moritz, et al. \"Transformer fusion with optimal transport.\" ICLR 2024\n\n[3] Tenney, Ian, Dipanjan Das, and Ellie Pavlick. \"BERT Rediscovers the Classical NLP Pipeline.\" ACL 2019.\n\n[4] Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\" ICML 2022.", "claims_and_evidence": "Most of the claims made in the paper are well-supported; however, the authors suggest that the proposed matching algorithm can be used to merge multiple models, though they only experiment with merging with two models. Similar work for CNNs suggests that it may not be possible to merge more than two models simultaneously [1]. \n\n[1]. Sharma et al., Simultaneous Linear Connectivity of Neural Networks Modulo Permutation", "methods_and_evaluation_criteria": "Yes, methods is evaluated correctly with the relevant benchmark datasets.", "theoretical_claims": "Yes, the theoretical claims are correct and easy to understand.", "experimental_designs_or_analyses": "1. I think authors should show that Linear Mode Connectivity improves after the merging. \n\n2. Moreover, it is not clear to me if the fine-tuned models will move out of the loss basin of the original pre-trained models. This could also explain why matching only the first few layers works, as observed by the authors.", "relation_to_broader_scientific_literature": "The paper introduces a new concept of symmetry --- rotational symmetry --- which could match transformers. Previous work has only looked into the permutation symmetry for MLP/CNNs, which limited its application to transformers. However, rotational symmetry is more general symmetry, which can be used with transformers. The authors theoretically explain and show how to obtain this rotation symmetry for transformers; this could spur future research.", "essential_references_not_discussed": "Related works are cited and discussed.", "other_strengths_and_weaknesses": "### Strengths: \n1. The paper is well-written and easy to follow. \n\n2. The paper studies an important problem of symmetry for transformer-based models and successfully extends the previous work to transformer models. \n\n3. The closed-loop form for obtaining $R$ is interesting.\n\n4. Experiments are well-designed, and empirical results demonstrate the efficacy of the method. \n\n### Weaknesses:\n\n1. The models are fine-tuned so that may already be in the same basin. As suggested earlier, study the LMC before and after the merging.\n\n2. Experiments on merging more than two models need to be included, or the claim needs to be readjusted otherwise.", "other_comments_or_suggestions": "Line 226 \n>and can be solved precisely by Hungarian Algorithm (Martello & Toth, 1987).\n\nI believe weight matching and activation matching give an approximate solution; I would refrain from using the word precisely.", "questions_for_authors": "Why do you think fine-tuning the models will move them outside the loss basin of the original model? Previous work suggests they remain in the same basin [2]. \n\n[2]. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dcFetJWPbP", "reviewer": "Reviewer_sNWf", "summary": "The paper introduces rotation symmetry in transformers, extending permutation symmetry from discrete to continuous spaces. It demonstrates theoretically and empirically that rotating query-key and value-output parameter matrices preserves functional equivalence. The main contribution is a theoretically optimal algorithm for matching parameters during model fusion, significantly enhancing performance across NLP and vision benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback. We are honored to have this valuable chance to address your raised concerns and questions. We believe that our paper will be much stronger thanks to your efforts.\n\n**Response to theoretical claims**: \nThank you for this thoughtful point. We would like to clarify that the perfect orthogonality isn't an assumption but rather a property that holds for any SVD decomposition.\nEven in noisy parameter spaces, SVD always provides orthogonal singular vectors. Any parameter matrix obtained through practical training, e.g., standard SGD, differentially private SGD, and other optimization methods, can be decomposed this way.\n\n**Response to questions**: Yes, the correlation between reduced Euclidean distance and improved model fusion performance is backed by previous studies [1]. Their theoretical results show that strong convexity and closer end models can boost the utility of direct model fusion. Additionally, experimental results in [1] and our paper support the correlation from an empirical perspective.\n\nWe extend our best gratitude for your efforts in the rebuttal phase. We highly value the opportunity to improve our paper. We will include the additional results and discussions in the next version of our paper. We sincerely appreciate your time and consideration.\n\n[1] Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\" ICML 2022.", "claims_and_evidence": "The main claim is that rotation symmetry improves transformer model fusion by reducing distances between parameter sets. This is convincingly supported by extensive empirical evidence showing consistent improvement across multiple fusion methods, transformer architectures, and tasks.", "methods_and_evaluation_criteria": "The methods and evaluation criteria (real-world NLP and vision tasks, including Emotion, NER, GLUE benchmarks, CIFAR-10) are appropriate, realistic, and convincingly aligned with demonstrating the advantages of their proposed method.", "theoretical_claims": "The main theorem (Theorem 4.1) provides a closed-form solution for the rotation symmetry optimization problem. The proof is sound and well-structured.\n\nWhile the derivations in the paper are mathematically elegant, there are concerns regarding their reliance on idealized conditions. In particular, the closed-form solution (Theorem 4.1) leverages perfect orthogonality and well-conditioned eigendecompositions - assumptions that may not hold in high-dimensional, noisy parameter spaces encountered during practical training.", "experimental_designs_or_analyses": "Experimental designs and analyses appear sound and valid. Model fusion benchmarks, comparisons against well-established methods, and the analysis of distances in parameter space are well-executed.", "relation_to_broader_scientific_literature": "This work substantially builds upon existing literature on permutation symmetry (e.g., Ainsworth et al., 2023; Entezari et al., 2022) and extends it significantly into continuous rotation symmetry for transformers, providing both theoretical novelty and practical utility. It addresses a clear gap regarding transformer-specific symmetries.", "essential_references_not_discussed": "All related works are cited to my knowledge.", "other_strengths_and_weaknesses": "The introduction of continuous rotation symmetry is original, theoretically insightful, and practically impactful. However, the potential computational overhead in larger transformer models or extremely high-dimensional settings might need further exploration.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The correlation between reduced Euclidean distance in parameter space and improved model fusion performance is assumed to be direct. Are there references that provide a theoretical backing for this correlation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WaEGen0Z1T", "reviewer": "Reviewer_QTZj", "summary": "The paper studies transformer parameter symmetries. Specifically, it explains how to weight space average two attention layers modulo not only permutation symmetries but also rotation symmetries. Experimental results show that considering this extra symmetry leads to better alignment between different trained transformers.\n\n## update after rebuttal\n\nThe authors have reasonably addressed my concerns, and so I increased my rating to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback. We are honored to have this valuable chance to address your raised concerns and questions. We believe that our paper will be much stronger thanks to your efforts.\n\n**Response to comments and Q1**:\nWe thank the reviewer for this important question.\nTo clarify, our choice of end ViT models strictly follows previous works [1], resulting in similar performance degradation when merging models trained on different tasks (Table 1 in [1]).\nWe attribute these low metric values to two key factors: \n1. Task divergence. The end ViT models were fine-tuned on substantially different tasks, leading to specialized parameters that conflict when naively merged.\n2. Non-convexity. ViTs exhibit highly non-convex loss landscapes, making naive interpolation between models challenging compared to simpler models such as MLPs.\n\nActually, merging large models on different tasks remains challenging. This also motivates our parameter matching strategy which improves model merging without access to any training data. We are committed to including the metric values of the end models in the next version of our paper.\n\n**Response to Q2**: We thank the reviewer for providing this insightful comment. We find that all orthogonal matrices in the rotation symmetry can be directly replaced by invertible matrices without any barrier. However, invertible matrices are not applicable for parameter matching. In our parameter matching algorithm, the orthogonal matrix is an important promise for Theorem 1. If using invertible matrices, Eq. (9) can only be solved by gradient descent, losing the guarantee of global optimum. In addition, parameter matching requires computation of the inverse matrix of every $R$, which is impractical for invertible matrices. In contrast, the inverse matrix of the orthogonal matrix is just its transpose. In summary, if we only consider the rotation symmetry for transformers, then yes, the orthogonal matrices in the rotation symmetry can be replaced by invertible matrices without barrier; while if we consider the parameter matching based on rotation symmetry, then no, using invertible matrices results in suboptimal parameter matching and large complexity.\n\nWe extend our best gratitude for your efforts in the rebuttal phase. We highly value the opportunity to improve our paper. We will include the additional results and discussions in the next version of our paper. We sincerely appreciate your time and consideration.\n\n[1] Imfeld, Moritz, et al. \"Transformer fusion with optimal transport.\" ICLR 2024", "claims_and_evidence": "Most claims are well supported.", "methods_and_evaluation_criteria": "The benchmarks seem to align with prior work.", "theoretical_claims": "I haven't checked the proofs. The results seem plausible.", "experimental_designs_or_analyses": "The experimental design seems reasonable.", "relation_to_broader_scientific_literature": "The paper considers a further symmetry than prior work. I am not aware of prior work exploring rotation symmetries in attention layers.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The paper considers rotation symmetries and also rotation symmetries + scaling. One could also straightforwardly consider invertible linear transformations $A$ in general since $QA A^{-1} K^T = QK^T$.", "other_comments_or_suggestions": "1. Include the baseline scores in the tables. I.e the scores of the models that are merged.\n2. Comment on the fact that CIFAR10 accuracies are extremely low after merging. (30%)", "questions_for_authors": "1. Is there any intuition/reason as to why the merged networks do not perform well? E.g. 30% accuracy on CIFAR10 sounds very low.\n2. Would it be possible to extend to general invertible linear transformations $A$ as described under \"Other Strengths And Weaknesses\" above?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MEgOVQ0Wmh", "reviewer": "Reviewer_RtZb", "summary": "In this paper, the authors identify a neural network (NN) parameter symmetry beyond the well-studied permutation symmetry. In particular, they show that the weights of self-attention layers are governed by *rotation symmetry*, i.e. one can transform the query, key, value, output matrices by appropriate rotation matrices/their transposes and maintain the function computed by the attention mechanism intact. They proceed by leveraging this symmetry to improve model fusion for self-attention layers. They devise a new parameter matching algorithm by showing that the optimisation problem of aligning self-attention weights up to rotation admits a closed form solution that requires solving an eigendecomposition problem. They additionally enhance their matching algorithm by another step that accounts for weight scaling symmetries, while in the case of a Transformer, the MLPs are aligned using the parameter matching algorithm of Ainsworth et al., 2023. The method is experimentally tested on fusing language and vision transformers, showcasing improved downstream task performance compared to simple (averaging) fusion and other baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback. We will include additional results and discussions in the revision. We believe that our paper will be much stronger thanks to your efforts.\n\n**Response to claims**: We thank the reviewer for the suggestion, and we will adjust the wording in the next version of our manuscript. Additionally, we would like to clarify that we have included [1] as a concurrent work in the discussion. \n\n**Response to W1**:\nWe thank the reviewer for this suggestion. Our backbone model selection follows previous studies on model fusion [2,3]. Specifically, we deliberately selected models representing different transformer architectures.\n\nWhile experiments on larger models would indeed be valuable, they would require substantially more computing resources. \nDue to computing resource constraints, we focused on models that could be effectively studied within our available infrastructure.\n\n**Response to W2**:\nFor clarity, the three model fusion methods (Fisher, Regmean, and OT) are not baselines in our evaluation, but rather backbone methods that we enhance with our parameter matching algorithm. \nTo the best of our knowledge, our work represents the first parameter matching algorithm specifically designed to enhance model fusion performance\nWe specifically selected these methods because they achieve desirable performance and provide a diverse set of merging strategies to demonstrate the general applicability of our matching algorithm.\n\n**Response to W3**:\nWe first would like to clarify that we follow previous studies [2] to leave classification heads unmerged.\nUnlike attention layers that capture generalizable patterns, classifier layer parameters are highly task-specific in nature.\nEven minor modifications to downstream tasks (e.g., altering label orders in classification tasks or targeting different output distributions) result in entirely different optimal parameter values for classifier heads.\nThis task-specificity makes merging classifier heads conceptually unsound without task alignment information. \n\n**Response to W4**:\nPlease refer to our response to reviewer **ydmT**.\n\n**Response to W5**:\nThank you for this insightful question that helps clarify component contributions. We conducted an ablation study isolating the effects of matching different components:\n\n|  | Fisher | Regmean | OT |\n|  --- | -------- | -------- | -------- |\n| FFN-only | 12.21     | 11.89     | 32.08     |\n| Attn-only | **20.21**     | 12.94     | 28.66     |\n| FFN+Attn | 18.61    | **15.31**     | **32.50**     |\n\nFor Fisher and RegMean, attention matching provides more contributions to model fusion. For OTFusion, FFN matching provides more contributions. These differences highlight how the underlying fusion method interacts with component matching. We will include the results in our revised manuscript.\n\n**Response to references**: We are committed to adding the suggested references to Related Work of our next revision.\n\n**Response to Q1**: \nWe thank the reviewer for this thoughtful question. You raise an important theoretical point about joint optimization of rotation and scaling. It's true that our sequential approach (first optimizing rotation $R$, then scaling $\\alpha$) cannot guarantee global optimality for the joint ($R$, $\\alpha$) optimization problem. Proving that sequential optimization yields the global optimum would be non-trivial, as calculating the general solution of Eq. (12) without knowing the specific values of matrices is complex.\nWe can view our approach (sequentially optimizing $R$ and $\\alpha$) as a practical approximation to the joint optimization problem. Developing an algorithm that jointly optimizes for both rotation and scaling symmetries would be an interesting extension to our method, though it would likely come with increased computational complexity. Our current method balances theoretical soundness with computational efficiency.\n\n**Response to Q2**: We believe there might be no symmetries for Softmax and LayerNorm that require matching since these modules do not contain parameter matrices to be rotated.\n\n**Response to Q3**: We thank the reviewer for this thoughtful question. While our work focuses on rotation symmetries, we acknowledge that transformers may exhibit additional symmetries beyond the permutation, rotation, and scaling transformations we address.\nThe transformer architecture, with its complex interplay of components, likely possesses a rich symmetry structure that extends beyond what we have explored. Characterizing the complete symmetry group of transformers remains an open question. We identify this as an important direction for future research.\n\n[1] Tran, Viet-Hoang, et al. \"Equivariant Neural Functional Networks for Transformers.\" arXiv 2024.\n\n[2] Jin, Xisen, et al. \"Dataless knowledge fusion by merging weights of language models.\" ICLR 2023\n\n[3] Imfeld, Moritz, et al. \"Transformer fusion with optimal transport.\" ICLR 2024", "claims_and_evidence": "- The main claim of the paper is that taking rotation symmetry into account when aligning self-attention weights can improve the downstream performance of fused Transformer models. Although, experimentally this generally seems to be true, I think that the authors have overclaimed in their abstract by mentioning that their \"matching algorithm substantially improves model fusion\", as this is not clear from the experimental results (Tables 1 and 2). In specific, in several cases the matching algorithm seems to be on par or slightly improves simple fusion, therefore the word \"substantially\" does not seem to follow from the results.\n\n- Additionally, the authors mention multiple times that they \"introduce rotation symmetry\" (e.g. Contribution 1), which I believe is misleading, as it has been discussed before in the work of Tran et al., 2024 for weight space learning (neural functionals). I think the phrasing needs to be changed in the text to clarify that rotation symmetry is not a contribution, rather it is taken advantage of to improve model fusion.", "methods_and_evaluation_criteria": "- **Practicality**. The proposed methodology for self-attention weight alignment is technically sound and can be easily implemented in practice. Additionally, I appreciated the fact that the authors include in their framework the MLP alignment, as well as the scaling symmetries of self-attention weights, making their approach practical for fusion of real-world Transformers. \n- See Experimental Designs Or Analyses for my comments on empirical evaluation.", "theoretical_claims": "The optimal solution of the matching algorithm, even though seems straightforward to derive, is an important result and strengthens the credibility of the algorithm (recall that MLP permutation matching algorithms do not admit optimal closed-form solutions beyond 2 layers). The proof is concise and seems correct.", "experimental_designs_or_analyses": "- **Weakness**. The experimental section provides evidence, at least partial, to the claims. However, it was unclear to me how several experimental design choices were made. For example,\n    - Why did the authors choose these particular Transformers (RoBERTa, DeBERTa, ViT)? How would model fusion behave on larger models? Have the authors considered extending their experimental section with more recent architectures, or architectures from different domains? I do not intend to imply that this is necessary, rather than it should be made clear why those particular choices were made.\n  - Similarly, how did the authors choose these particular baselines (apart from the obvious simple averaging one)?\n  - Why do the authors match only the self-attention layers and do not perform merging on the classifier? Perhaps an ablation study on that would help. \n  - Would it be possible to use a variant of the matching algorithm to match more than two models, akin to model soups? This would further strengthen the impact of this work. \n  - How does each component of the overall matching algorithm contribute to the resulting downstream performance (e.g. permutation matching of MLPs compared to rotation matching of self-attention)?", "relation_to_broader_scientific_literature": "Although I have not followed the entire literature on model merging, the paper is mostly well-contextualised: it mentions naive (non-symmetry) merging algorithms and compares against them, while it also discusses permutation symmetry matching, which is the most well-studied one.", "essential_references_not_discussed": "Regarding permutation symmetry matching, I believe that the paper misses two recent algorithms that improve upon Ainsworth et al.,2023:\n- Peña et al., Re-basin via implicit Sinkhorn differentiation, CVPR'23\n- Navon et al., Equivariant Deep Weight Space Alignment, ICML'24. \n\nAdditionally, regarding weight space symmetries, I believe that the authors should have dedicated more space in mentioning the ongoing efforts on neural functionals/metanetworks (e.g. Navon et al., ICML'23, Zhou et al., NeurIPS'23, Lim et al., ICLR'24 etc.), while the have missed two important references on scaling symmetries (which is something that is taken into account in the current work):\n- Kalogeropoulos et al., Scale Equivariant Graph Metanetworks, NeurIPS'25\n- Godfrey et al., NeurIPS'22: this is mentioned, but not in the paragraph concerning scaling symmetries.", "other_strengths_and_weaknesses": "**Strengths**\n- *Importance/Significance*. With the growing availability of trained models, fusing the knowledge embedded in their weights is arguably an important problem, as it reduces the need to train new models (potentially bigger and potentially on new datasets). Since Transformers are currently one of the most popular NN technologies, designing an improved and specialised fusion algorithm for them is a crucial step.\n- *Novelty*. Although the rotation symmetry of Transformers has already been studied, it has not been examined in the context of model fusion. To the best of my knowledge, the rotation symmetry parameter matching algorithm provided by the authors is novel.\n\n**Weaknesses**\n- As previously mentioned, the main weaknesses are that some claims need to be modified and that the experimental choices need to be explained more thoroughly, and potentially extended.", "other_comments_or_suggestions": "-", "questions_for_authors": "Additional questions that I believe need to be discussed in the paper:\n\n1) When the authors take scaling into account, it seems to me that the way it is done, breaks the optimality of the matching algorithm. Could the authors comment on that? Would it be possible to devise a matching algorithm that is optimal w.r.t both scaling and rotation symmetries?\n2) Would it be possible to take into account the symmetries of the rest of the transformer components: e.g. LayerNorm and Softmax?\n3) Are rotation symmetries the only ones in self-attention layers? Perhaps there exist a larger group of transformations to which the layer is invariant. Could the authors discuss this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Leveraging rotation symmetry in self-attention layers, the authors propose a parameter matching algorithm that achieves improved empirical performance across diverse ML tasks.

The reviewers recognized the introduction of rotation symmetry in transformer model fusion as a novel, theoretically sound, and practically valuable contribution. Concerns initially raised about potentially overstated empirical claims and the need for a more comprehensive experimental evaluation were largely resolved through discussions with the authors. Overall, the paper was viewed positively for its conceptual originality and prospective impact.

The authors are encouraged to incorporate the important feedback given by the knowledgeable reviewers.
Decision: Accept (spotlight poster)