The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof
Authors: Derek Lim
Theo Putterman
Robin Walters
Haggai Maron
Stefanie Jegelka
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter symmetry
loss landscapes
identifiability
mode connectivity
Bayesian neural networks
Primary area: deep_learning_architectures
TL;DR: We develop neural network architectures with less parameter-space symmetries, and empirically study the impact of parameter symmetries on various phenomena in deep learning.

Abstract: Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries --- transformations of neural network parameters that do not change the underlying neural network function. These include linear mode connectivity, model merging, Bayesian neural network inference, metanetworks, and several other characteristics of optimization or loss-landscapes. However, theoretical analysis of the relationship between parameter space symmetries and these phenonmena is difficult. In this work, we empirically investigate the impact of neural parameter symmetries by introducing new neural network architectures that have reduced parameter space symmetries. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries.  With these new methods, we conduct a comprehensive experimental study consisting of multiple tasks aimed at assessing the effect of removing parameter symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries; for instance, we observe linear mode connectivity between our networks without alignment of weight spaces, and we find that our networks allow for faster and more effective Bayesian neural network training.

The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof
Derek Lim"
Theo (Moe) Putterman'
MIT CSAIL derekl anfmiti edu
UC Berkeley
moeputtorrman@@eekelleyy.. edu
Robin Walters Northeastern University
Haggai Maron
Stefanie Jegelka TU Munich, MIT
Technion, NVIDIA
Abstract
Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries - transformations neural network parameters that do not change the underlying neural network function. These include linear mode connectivity, model merging, Bayesian neural network inference, metanetworks, and several other characteristics of optimization loss-landscapes. However, theo- retical analysis of the relationship between parameter space symmetries and these phenomena is difficult. In this work, empirically investigate the impact of neural parameter symmetries by introducing new neural network architectures that have re- duced parameter space symmetries. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space sym- metries. With these new methods, conduct comprehensive experimental study consisting of multiple tasks aimed at assessing the effect of removing parameter symmetries. Our experiments reveal several interesting observations on the empiri- cal impact of parameter symmetries; for instance, we observe linear mode connec- tivity between our networks without alignment weight spaces, and find that our networks allow for faster and more effective Bayesian neural network training. Our code available attttps: /ttp:://..bt....../ whthu.ccooo..... com/ hamm//pp////a/ htta///symmmrrriccommm -ettrrsorksssess
Introduction
Neural networks have found profound empirical success, but have many associated behaviors and phenomena that are difficult understand. One important property of neural networks is that they generally have many parameter space symmetries - for any set of parameters, there are typically many other choices of parameters that correspond to the For instance, permutations of hidden neurons in multi-layer perceptron (MLP) induce permutations of weights that leave the overall onput-output relationship unchanged. These parameter symmetries are a type of (not-necessarily detrimental) redundancy in the parameterization of neural networks, that adds much non- -un-uucli structure parameter space.
exact neural network function (24).
Parameter space symmetries appear to influence several phenomena observed in neural networks. For example, when linearly interpolating between the parameters of two independently trained networks with the same architecture, the intermediate networks typically perform poorly [59] [3] However, if we first align the two networks via permutation of parameters that does not affect the network function, then the intermediate networks can perform just as well as the unmerged networks [59 D.. In some sense, this suggests that neural network loss landscapes are more convex
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Standard
W-A --ysymmetric
--symmetrii
W O M
Trained
FIGLU
(20(W1z)
F (1 M)
Fixed
(xx) (F))
Figure (Left) Standard MLP. The hidden nodes (grey hatches) can be freely permuted, which induces permutation parameter symmetries. Black edges denote trainable parameters. (Middle) Our W-Asymmetric MLP, which fixes certain weights to be constant and untrainable (colored dashed lines) to break parameter symmetries. (Right) Our o-Asymmetric MLP, which uses our FiGLU nonlinearity involving fixed matrix (colored dashed lines) to break parameter symmetries.
or well-behaved after removing permutation symmetries. Other areas that parameter symmetries play role in include interpretability of neurons (20), optimization [48] 84 [8], model merging (60),
learned equivariance 77), Bayesian deep learning (3). loss landscape geometry (54), processing
neural network weights as input data using metanetworks [39], and generalization measures [49
To rigorously study the effect of parameter symmetries, we study the effect of removing them. In
particular, introduce two ways of modifying neural network architectures to remove parameter space symmetries (see Figure I
(1) W Asymmetric networks fix certain elements of each linear map to break symmetries in the computation graph.
(2) a- p-Assmmetric networks use new nonlinearity (FiGLU) that does not act elementwise, and hence does not induce symmetries such as permutations.
These two approaches are inspired by previous work, which shows that both symmetries of computa- tion graphs 39] and equivariances of nonlinearities (20 induce parameter symmetries in standard neural networks. We theoretically prove that both of our approaches remove parameter symmetries under certain conditions. Our Asymmetric networks are similar structurally to standard networks and can be trained with standard backpropagation and first-order optimization algorithms like Adam. Thus, they are a reasonable "counterfactual" system for studying neural networks that are similar to standard neural networks, but that do not have as many parameter symmetries.
With our Asymmetric networks, we run suite of experiments study the effects removing parameter symmetries on several base architectures, including MLPs, ResNets, and graph neural networks. We investigate linear mode connectivity, Bayesian deep learning, metanetworks, and monotonic linear interpolation. Through the lenses of linear mode connectivity and monotonic linear interpolation, we see that the loss landscapes of our Asymmetric networks are remarkably more well-behaved and closer to convex than the loss landscapes of standard neural networks. When using our Asymmetric networks as the base model in a Bayesian neural network, we find faster training and better performance than using standard neural networks that have many parameter symmetries. When using metanetworks to predict properties such as test accuracy an input neural network, we that all tested metanetworks standard networks. Overall, our Asymmetric networks provide valuable insights for empirical study and hold promise for advancing understanding of the impact of neural parameter symmetries.
accurately predict the accuracy of Asymmetric networks than
Background and Definitions
Let Ô be the space of parameters of fixed neural network architecture. For any choice parameters 0 E θ, we have a neural network function fo X' + from an input space X' to an output space y. We call function
④ + θ parameter space symmetry if fa(x) = foro) for all inputs æ and parameters A E θ (i.e. if fo and fa(n) are always the same function).
For instance, consider two-layer MIP with no biases, parameterized by matrices -W2, Wi) with elementwise nonlincarity Then fe(x) W W2)(W,x)) Let P be a permutation matrix,

---

and let ((0) = (W2PT, PW1). Then for any input x, (x) = W.P (PW1s) - Po( W20 (2a) - [o
(I)
so o is a parameter space symmetry. A key step is the second equality, which holds because (a(x) o (Px): any elementwise nonlinearity permutation equivariant. Any other equívariance of a also induces a parameter symmetry; for instance, if o = max(0, x) is the ReLU function, then aa(x) o(aar) for any a 0, so there is a positive-scaling--ael parameter symmetry 49
3 Related Work
Characterizing parameter space symmetries. While many works spanning several decades have noted specific parameter space symmetries in neural networks [24 61]. some works take a more systematic approach to deriving parameter space symmetries. Godfrey et al. [20] characterize all global linear symmetries induced by the nonlinearity for two-layer multi-layer perceptrons with pointwise nonlincarities. Zhao et al. [78 study several types of symmetries, and derive nonlinear, data-dependent parameter space symmetries. Lim et al. (39) show that graph automorphisms of the computation graph of a neural network induce permutation parameter symmetries, which captures hidden neuron permutations MLPs and hidden channel permutations in CNNs.
Constraints and post-processing to break parameter space symmetries. Several works develop methods for constraining post-processing the weights of single neural network to remove ambiguities from parameter space symmetries. This includes methods to remove scaling symmetries induced by normalization layers
sesttively-hmooeneeeee nonlinearities such ReLU 6 55
54 36), methods to remove permutation symmetries [55] 54 36], and methods to remove sign
symmetries induced by odd activation functions
Unlike these previous works, we develop neural network architectures that have reduced parameter space symmetries. Our models are optimized using standard unconstrained gradient-descent based methods like Adam. Hence, our networks do not require any non-standard optimization algorithms such manifold optimization projected gradient descent [6] (55) nor do they require post-training- processing to remove symmetries or special care during analysis of parameters (such as geodesic interpolation in Riemannian weight space (54). These methods based constraining weights post-processing have significantly different optimization and loss landscape properties (for instance, linear interpolation is not well-defined general nonlinear parameter manifolds), so they less suitable than our Asymmetric networks for studying phenomena that may generalize to standard neural networks.
Aligning multiple networks for relative invariance to parameter symmetries. One way to reduce the impact of parameter symmetries in certain settings, especially for model merging, is to align the parameters of one network to another. Several methods have been proposed for choosing permutations that align the parameters of two neural networks of the same architecture, via efficient heuristics or learned methods 4 wetpattion-paammtee---ryee
53 47 67]. Other approaches relax the exact constraint or do additional postprocessing to achieve effective merging of models in parameter space [59] 28 29 60, 56). As our Asymmetric networks have
69
63
13
1
removed parameter symmetries, they can often be successfully merged and linearly interpolated between without any alignment.
4 Asymmetric Networks
We develop two methods of parameterizing neural network architectures without parameter sym- metries, both of which are justified by theoretical results. We can prove that our o-Asym networks have permutation and scale symmetries removed, and that our W-Asym networks have permutation symmetries removed. Although we have not formally proven that W-Asym networks have scale symmetries removed, we believe that they do (intuitively, the fixed weights fix scale).
We first focus on the case of fully-connected MLPs with no biases, which take the form fo(x)
(Wr)) ) for weights 0 = (Wi.. ......, Wi) and nonlinearity a. Then in Section 4.3 W W W W a (W
we discuss how we use these approaches to remove parameter symmetries in other architectures (e.g.
CNNs. GNNs) well.
3

---

Standard
W -Asymmmetrie
Linear
W-Asymmetric (Fix entries)
W-Asymmetric (Fix filters)
Standard
Convolution
Figure 2: Depiction of our W-Asymmetric approach removing parameter symmetries. Entries with a black outline are untrained. Note that the W-Asym linear map has nonzeros per row, the W-Asym convolution with fixed entries has fixed entries for its single output channel, and the W-Asym convolution with fixed filters has single input filter fixed. We often use a constant number of fixed entries per row or output channel in our experiments.
4.1 Computation Graph Approach (W (W-Asymmetric Networks)
Our first approach to developing neural networks with greatly reduced parameter space symmetries relies on their computation graph. In particular, tecture as a DAG G = (V, E) with neurons as nodes V and connections between them as edges E. For choice parameters 0 E R! we get function fe from input neuron space to output neuron space [19 49 39]
write feedforward neural network archi-
Lim et al. showed that neural DAG automorphisms é, which are graph automorphisms of the DAG G that preserve types of nodes and weight-sharing constraints, induce permutation parameter symmetries @ that leave the function unchanged: fa Jocøy- Thus, any feedforward neural network architecture that has no permutation parameter symmetries must necessarily have computation graph with no nontrivial neural DAG automorphisms.
To modify MLPs so they have no nontrivial neural DAG automorphisms, we mask edges in the computation graph, by setting certain edge weights to constant values that are not updated during training. For an MLP, we can do this by enforcing that every linear layer T [Rd - Idda takes the form of matrix W E Rddad, xdi, where each row has unique pattern untrained weights. To achieve this, define mask M E {0, 1] such that Wi; is trainable parameter if and only if
My 1, and the rows of M are pairwise distinct binary vectors in {0, 1)dd. We call any neural network with linear maps masked such W-Asymmetric neural network. Appendix B. we show that masking these entries that they are not trained is sufficient to remove all nontrivial neural DAG automorphisms.
Theorem 1. If each mask matrix M has unique nonzero rows, then W W-Asymmetric MLPs with fixed entries set zero have nontrivial neural DAG automorphisms.
In practice, we generate a binary mask M by randomly selecting subset of nax fixed elements for each row. For the fixed entries, we sample them from normal distribution (O, s7) with standard deviation is >> 0 that is hyperparameter that we tune. Our asymmetric linear layer can written as
W' = M W (1 M) F,
where W dRd is matrix of trainable parameters, and xdd is matrix of fixed elements, sampled from (O0 kl). The only trainable parameters are the unmasked entries of M O W. of which there are d2 (dj nix). We empirically find that having x be significantly larger than the standard deviation of typical initializations for weight matrices (e.g. =1 while the trained coefficients have standard deviation about 4/V1000) is important for breaking parameter symmetries.

---

4.2 Nonlinearity Approach (r-Asymmetric Networks)
Another approach for removing parameter symmetries is to change the nonlinearity. As studied by Godfrey et al. [20]. equivariances of the nonlinearity induce parameter symmetries in MLPs with elementwise nonlincarities. Recall that an elementwise nonlinearity acts by using the same function on each coordinate of the input: (x) = (21)...... (1(xa) for some real function IR + R. Any elementwise nonlinearity is permutation equivariant, and hence induces a permutation parameter symmetry.
R d is elementwise if takes the form
Thus, in contrast to most neural network architectures, for Asymmetric networks we must use nonlinearity that does not act elementwise. Likewise, the nonlinearity cannot have any linear symmetry itself, since if a o A = Boo for A, B E GL(d), then for a two-layer network:
W 0 W, = W,B- W1 WW AW,
(3)
So 22 W1) and (W2B-1 1W1) give the same neural network function. Thus, in order to define model class without parameter symmetries, it is necessary for to have linear equivariances, Le. we desire that if a o A = B a for A, B E GL(d), then A = B = For two-layer MLPs with square invertible weights, this is in fact sufficient to remove all parameter symmetries: we prove this in Appendix B.2
Proposition 1. Let the parameter space θ be all pairs of square invertible matrices 0 = (W2, Wi) for 22, W, E GL),), and let fe(z) = W20 W1.). If a has no linear equivariances, then fe. foz fndd only 0f = 02. In other words, there are nontrivial parameter space symmetries.
4.2.1 FiGLU: the Fixed Gated Linear Unit Nonlinearity
Motivated by Proposition] D we define non-clementwise nonlinearity that does not have the equiv- ariances of standard nonlincarities. Letting be the sigmoid function 77xx) = T+e we define our nonlinearity as
o (x) = (Fx)) Or
for a randomly sampled, untrained matrix F. Similarly to W-Asym nets, we sample F as an i.i.d Gaussian matrix with variance that we tune. This nonlinearity is similar Swish / SiLU [57] 26) with an additional matrix F to mix feature dimensions (to break permutation equivariance), and it is also similar a gated linear unit (GLU) with no trainable parameters 9 Thus, we call our nonlinearity FiGLU: the Fixed Gated Linear Unit.
In Appendix B.2.1 equivariances, which are the only equivariances for most elementwise nonlinearities 200.
prove that FiGLU does not have permutation equivariances diagonal
Proposition 2. With probability 1 over the sampling of F, FIGLU has no permutation equivariances or diagonal equivariances.
We call any network with our bymmetry-breaking FIGLU nonlincarity a o-Asymmetric Network.
4.3 Extension to Other Architectures
The graph-based approach (W-Asymmetric Networks) works naturally for neural network archi- tectures with "channel" dimensions, such as convolutional neural networks (CNNs), graph neural networks (GNNs) [22), Transformers [66), and cquivariant neural networks based on equivariant linear maps [6). In these types of networks, permutations of entire channels induce permutation parameter symmetries 33) For such networks, we thus mask entire connections between channels, e.g. entire filters in CNNs. For CNNs, we also experiment with randomly masking some number of entries in each filter (instead of masking entire filters), and find that this also works well in removing parameter symmetries. For neural networks with linear layers that include bias terms, do not modify the biases in any way, as they do not introduce new computation graph automorphisms (39].
The nonlinearity based approach (a-Ammmmetric Networks) can be straightforwardly applied many general architectures as well. Though, the fixed matrix F may have be changed to structured linear map; for instance, in CNNs take F to be ID convolution.
5

---

MLP MNIST
ResNet width CIFAR-10
GNN ogbn-arXiv
Scandand Rabadt
Standard
2.55
Gardur
Bezadh
2:25
0.25
0.22
Texs
1.75
130
122
35-
1.00
0.5
0.2
0.6
0'5
1.0
5.0
02
0.4 Interpalation
65
3.5
3.0
0.0
0.2
0.5
26
0.5
3.5
Interpolation
Interpoiation
Figure 3: Linear mode connectivity: test loss curves along linear interpolations between trained networks. (Left) MLP on MNIST. (Middle) ResNet with 8x width on CIFAR-10. (Right) GNN on arXv.. W-Asmmeetrie networks interpolate the best, followed by networks aligned with Git-Rebasin, then a-Asymeetric networks, and finally standard networks.
4.4 Universal Approximation
Our two approaches remove parameter symmetries from standard neural networks, but still intu- itively retain much of the structure of standard networks. One important property of widely-used neural network architectures is universal approximation - for any target function of a certain type, there exists a neural network of the given architecture that approximates the target to an arbitrary accuracy 88 25 42 75). In Appendix B.3] we show that --ssmmeetric MLPs retain this property:
Theorem 2 (Informal). For nix o(n)), where is the hidden dimension, W W-Asymmetric MLPs are Universal Approximators with probability 1 over the choice of hardwired entries.
We have not been able to prove a similar result for A-yymeetrii networks. Classical universal approximation results for standard neural networks do not apply to o-Asym nets, they tend to assume elementwise nonlinearities.
5
Experiments
5.1 Linear Mode Connectivity without Permutation Alignment
Background. Under certain conditions, neural networks have been found exhibit linear mode connectivity, which is when all networks on the line segment in parameter space between two well-performing trained networks are also well-performing. Starting with Frankle et al. [18), who coined the term and provided the first in-depth analysis, many works have studied this phenomenon 44 13 76 [4]. When the two networks are randomly initialized and trained independently, linear mode connectivity generally does not hold [3] M. However, if one of the two networks is permuted with a parameter symmetry that does not change its function, but that aligns its parameters with the other network, then linear mode connectivity empirically and theoretically holds for many more model / task combinations [13 I 79 [4]. In fact, Entezari et al. [33 conjectures that all permutation symmetries are accounted for, then linear mode connectivity generally holds. Since our Asymmetric networks remove parameter space symmetries, we may expect linear mode connectivity to hold, without any post-processing or alignment step.
Hypothesis Asymmetric networks are more linearly mode connected than standard networks, and do not require post-processing or alignment of pairs of networks before merging,
Experimental Setup. We consider several networks and tasks: MLPs on MNIST, ResNets on CIFAR- 10, and Graph Neural Networks [73] on arrv-arXiv [27]. For each architecture and task, we compute the midpoint test loss barrier: 6 + ( 2 (L(0,) + L(B2)). This measures how much worse the interpolated network with parameters 1 + 02 is than the original networks with parameters 0, and 0z. We measure this barrier for standard networks, pairs of networks aligned with Git-Rebasin []] and networks with our two approaches (o-Asym and W-Asym) applied. To be clear, whenever we interpolate between the weights of two Asymmetric networks, they have the same exact fixed weights (for both W-Asym and A-yyy)) and the same exact mask M (for W-Asym).
Results. Figure plots interpolation curves and Table T displays midpoint test loss barriers of various methods. Our Asymmettri approach lowers the test loss barrier compared to standard networks, but falls short of the alignment approach of Git- Rebasin. On the other hand, our W Asymmetriic

---

approach achieves strong (and sometimes perfect) interpolation, and interpolates better than standard networks aligned via Git-ReBasin. This may be caused by failure of the Git-ReBasin approaches to find the optimal permutations, importance of other parameter symmetries besides layer-wise permutations, or other properties of W W-Asymmetric networks.
Table I: Test loss interpolation barriers at midpoint: L( + (,2) 3 (L((1) L(02)) We use
different methods of breaking symmetries in each column; from left to right: no symmetry breaking. Git- Git-Rebasin [ our o-Asym approach, and our W-Asym approach. We report mean and standard deviation of the barrier across at least 5 pairs of networks, and bold lowest barriers.
Standard
Git-ReBasin
o-Asym (ours)
W-Asym (ours)
MLP (MNIST)
0.188 + .12
-0006 .00
0.117 = .01
-0.012 = .00
ResNet (CIFAR-10)
3.287 + .32
2.041 + .21
2.521 = .46
0.934 = .72
ResNet 8x width (CIFAR-10)
2.640 + 224
0.509 + .45
1.492 = .15
0.031 = .05
GNN (ogbn-arXiv)
1.475 1 24
0.269 1 .02
0.901 : .11
0.095 - .03
5.2 Bayesian Neural Networks
Background. Bayesian deep learning is promising approach to improve several deficits of main- stream deep learning methods, such as uncertainty quantification and integration of priors 33 5T]. However, parameter symmetries are problematic in Bayesian neural networks, as they are major source of statistical nonidentifiability [30 Parameter symmetries introduce modes in the posterior p(0 that make the posterior harder approximate
2 34 72.. sample from (5 71), and otherwise analyze [6.. For instance, one common technique for training Bayesian neural networks is varia- tional inference via fitting a Gaussian distribution the true posterior (() This approach suffers because the Gaussian distribution has only one mode, whereas the true posterior has at least one mode for every parameter symmetry. As such, some approaches treat kernel matrices are random variables, which has less symmetries than treating features or weights as random variables, and allows better approximation by unimodal posteriors [74] 43). Instead, we consider traditional, commonly- Bayesian deep learning techniques applied on the features or weights of our Asymmetric networks.
Table 2: Bayesian neural network results. Reported loss is the negative log likelihood loss. All results (except for last column) are after 50 epochs of training. W-Asymmetric networks tend to improve over their standard counterparts, especially early in training. 16-layer MLPs fail to train, but 16-layer W --Asmmeetric MLPs successfully train. Standard or Asymmetric networks better than their counterpart by standard deviation are bolded.
Model
Train Loss
Test Loss
BCE 4
Test Ace ?
Test Ace (25 Epochs) f
CIFAR-10
MLP-8
1.31 + .00
1.24
+ 001
.039 +: .009
56.37 + 331
52.87 +: 0.2 54.15
W-Asym
1.31
.01
.042 .009
57.08
M6.P-
2.29 .02
2.28
+ .03
.026 + .017
13.54
2.0
13.34 + 2.7
W-Asym MLP-16
1.39 -
1.27 = 001
.045 = .009
55.16 = .44
51.42 = 0.3
CIFAR- 100 CIFAR-10
ReaNec20
.96 + .01
.535 + .03
.015 + .007
81.98 + 1.2
72.37 + 1.0
Asysym ResNec20
1. .02
.535
.044 .004
73.64 1.5
.803 1: .08
.706
+ .08
.052 1 .007
75.71
1 2.8
59.85 1: 3.9
W-Asym Ro
.745 +
.658
.06
.049 = .004
77.40
+ 2.4
63.20 - 3.0
ReaNe120 (BN)
1.68 + .03
1.57 + .20
.078 = .004
56.83 + .62
46,80 + 0.9
W-Asyan ResNe120 (BN)
1.62
.03
.076
58.40 =
ResNer20 (LN)
1.97
.02
1.88 1. .02
.090 .007
50.02
1 .54
37.24 1 1.1
W-Asym ResNer20 (LN)
1.91 + .03
1.82
.02
.086 - 0006
51.20 - .47
39.03
Hypothesis. Using Asymmetric networks as the base model improves Bayesian neural networks, as the posterior will have less modes.
Experimental setup. We train Standard Bayesian and Asymmetric Bayesian Networks for image classification using variational inference. We use the method of [64] for variational inference, which fits Gaussian approximate posterior with diagonal plus low-rank covariance. We train 10 instances of each model and then report train loss, test loss, test accuracy, and Expected Calibration Error (ECE) 45 which is measure of calibration.
Results. See training curves in Figure 4 and quantiative results in Table Using W-Asymmetric networks as a base for Bayesian deep learning improves training speed and convergence. Most
7

---

strikingly, Bayesian MLPs of depth 16 cannot train at all. while W-Asymmetric Bayesian MLPs train well. In general, the W-Asymmetric approach improves training and test accuracy across the several models (MIPs, ResNets of varying sizes, and ResNets with either batch norm or layer norm).
MLP MNIST
ResNet110 CIFAR-10
ResNet20 CIFAR-100
20"
2 2
Loss
Train Lock
205
20
20
Epoch 40
30
10
20
30
40
10
20
55
Epoch
Epoch
Figure 4: Bayesian neural network training loss over time for depth 8 MLPs MNIST (left), ResNetl10 on CIFAR-10 (middle), and ResNet20 with BatchNorm on CIFAR-100 (right), W- Asymmetric networks train more quickly, and achieve lower training loss.
5.3 Metanetworks
Table 3: Metanetwork performance for predicting the test accuracy of small ResNets and our W- Asym ResNets. Each row is a different metanetwork. Reported are R² and Kendall 7 on the test set higher is better.
ResNet
W-Asym ResNet
R²
T
R²
T
MLP
.330 +: .04
.389 + .03
.594 + .12
.864 + .01
DMC
.950 + .01
.787
.02
.967 +
.911 00
DeepSets 77
.855 + 001
.617
+ .03
.936 + .00
.858 - .00
StatNN 65
.966 + .00
.866 + .00
.988 + .00
.935 = .01
Background. Metanetworks 39 also referred deep weight-space networks 46 58).
meta-models (35), or neural functionals [81
are neural networks that take as inputs the parameters of other neural networks. Recent work has found that making metanetworks invariant or equivariant to parameter-space symmetries of the input neural networks can substantially improve
82 83
metanetwork performance [46 8T 39 32).
Hypothesis. Asymmetric networks are casier to train metanetworks on because they do not have to explicitly account for symmetries.
Experimental setup. We experiment with metanetworks on the task of predicting the CIFAR- 10 test accuracy of an input image classifier, which many metanetworks have been tested on 665 12 81 39 We use metanetworks based on simple MLPs, ID-CNN metanetworks [12] and metanetworks that are exactly invariant to permutation parameter symmetries: DeepSets 77 and StatNN 65 We train two separate datasets of 10,000 image classifiers: one dataset small ResNet models, and one dataset of W-Asymmetric ResNet models. More information on the data, metanetworks, and training details are in Appendix F3
Results. In Table 3 we see that metanetworks are signficantly better at predicting the performance of our W-Asymmetrie ResNets than standard ResNets. Interestingly, simple MLP metanetworks, which view the input parameters as a flattened vector, can predict the test accuracy of Asymmetric Networks quite well, but fail on standard networks. Also, the permutation equivariant metanetworks (DeepSets and StatNN) both improve on W-Asym ResNets compared to on ResNets, even though the permutation symmetries of standard ResNets do not affect these metanetworks; thus, it may be possible that other symmetries in standard ResNets (but not Asym-ResNets) harm metanetwork performance, or they may be other factors besides symmetries that improve metanetwork performance for Asym-ResNets. ResNets.
8

---

Standard ResNet
G-Asymmetric ResNet
W-Asymmetric ResNet
23
201
Train Loss
154
1.4
0.5
5.0
0.6 Interpalation
0.0
1.0
00
02
C5
0.0
1.0
0.0
0.2
0.4
0.5
0.0
15
Interpolation
Interpolation
Figure 5: Train loss against interpolation coefficient a for the interpolation (1-a)00-- + a0r between initial parameters Oo and trained parameters 0r. Trajectories for the 20 (0o, 0r) pairs of lowest train loss for each architecture are plotted. The trajectories for Asymmetric ResNets appear significantly more monotonic and convex.
Table 4: Monotonic linear interpolation: properties of linear interpolations between 300 pairs of initialization and trained parameters. Arrows denote behavior that is more similar to convex optimization, e.g. there is a downarrow (4) next to A because convex objectives have nonpositive A, while nonconvex can have positive A. For both types of Asymmetrie networks, all differences from Standard ResNets are statistically significant (p < .001) under a two-sided T-test: Asymmetric networks have significantly more monotonie and convex linear interpolations from initialization.
A
Percent Monotonic t
Local Convexity f
Global Convexity 1
Standard ResNet
.079 + .09
26.3%
.548 + .139
.823 + 229
o-Asyy ResNet
.004 + .047
87.3%
.675 + .143
.976 + .098
W-Asym ResNet
.02 .06
100%
.769 + .165
1.00 .000
5.4 Monotonic Linear Interpolation
Background. One common method of studying the loss landscapes of neural networks is by studying the one-dimensional line segment of parameters attained by linear interpolation between parameters at initialization and parameters after training. Many works have observed monotonic linear interpolation (MLI), which is when the training loss monotonically decreases along this line segment 21 17 4T 68), Loss landscapes of convex problems have this property as well, so presence of the monotonic linear interpolation property has been used rough measure of how well-behaved the loss landscape is. However, with many types of models, tasks, or hyperparameter there is large plateau where the loss barely changes for much of the line segment [7 770: neither of these properties can happen for convex objectives trained to completion. To the best of our knowledge, there has been little work on the role of parameter symmetries - or lack thereof - in monotonic linear interpolation (besides one minor experiment in [4] Appendix C.9). Nonetheless, since removing parameter symmetries substantially improves linear interpolation between trained networks (Section 5.1 one may expect removing parameter symmetries to improve monotonic linear interpolation.
settings, monotonic linear interpolation does not hold 41 68
Hypothesis. The training loss along the line segment between initialization and trained parameters is more monotonic and convex for Asymmetric networks.
Experimental setup. For the learning task, we follow the setup used for creating the dataset of image classifiers in Section 5.3 In particular, we train 300 standard ResNets and 300 W- Asymmetric ResNets with varying hyperparameters sampled from the same distributions as used for the dataset of image classifiers (see Appendix Table 13). For each of these networks, we linearly interpolate between its initial parameters Og and its final trained parameters 0y: (1 - a)0 + a&r for 25 uniformly spaced values 0 = 0i Q2 < ... a25 1. To measure monotonicity, we record the maximum increase between adjacent networks A = max (()4())))1)))))))))))))))))))(() - L(a:)), )), and the percentage of networks that have A < 0 i.e. the percentage of networks that satisfy monotonie linear interpolation. To measure convexity, we consider a local convexity measure (the proportion of a; where the centered difference second derivative approximation is nonnegative) and global convexity measure (the proportion of a; such that L(a:) lies below the line segment between the endpoints, i.e. L(a:) (1 (O) + o; L()).
Results. Table shows the measures of monotonicity and convexity for standard, o-Asymmetric, and W- A--ymmmmttiic ResNets. Remarkably, every single one of the 300 W-Asymmetric ResNets satisfies

---

monotonic linear interpolation and has a trajectory that lies underneath the line segment between the endpoints. Qualitatively, we can see Figure W-Asymmetric ResNets do not have any clear loss barriers from initialization, nor any loss plateaus that indicate nonconvexity. In contrast, the majority of standard ResNets have non-monotonic trajectories, and the monotonic trajectories seem to be more nonconvex. o-Asymmetric network trajectories are signficantly more convex and monotonic than standard network trajectories, but there are some non-monotonic or nonconvex trajectories still.
5.5 Other Optimization and Loss Landscape Properties
In Appendix A we note other interesting differences in optimization and loss landscape properties of Asymmetric and standard neural networks. These can be summarized as:
1. Even though Asymmetric networks interpolate much better than standard networks, the parameters of trained Asymmetric networks are often basically the same distance away from each other in weight space as standard networks.
2. Asymmetric networks do not tend to overfit as much: the difference in train performance and test performance can be substantially lower than that of standard networks.
3. Asymmetric networks can take longer to train, especially when choosing hyperparameters that make them more dissimilar to standard networks.
Discussion
While many properties of Asymmetric networks in line with our hypotheses and intuition about the impact of removing parameter symmetries, there are many unexpected effects and unanswered questions that are promising further investigate. For instance, did not extensively explore Asymmetric networks in the context of model interpretability, generalization measures in weight spaces, or optimization improvements, all of which are known be influenced some extent by parameter symmetries. Further studying the properties in Section 5.3 the dependence of hehavior on the choices of Asymmetry-in -inducit--iduuingg induuiing hyperparameters, and other design choices in making networks asymmetric could also bring more insights into parameter space symmetries.
Also, it is interesting that our A-Asmmeetric networks do not appear to break parameter symmetries as well as our W-Asymmetric networks. We have run preliminary empirical tests on several variants of o-Asym networks, such multiplying the gate, using cosine instead of sigmoid, squaring instead using sigmoid, and putting LayerNorm in the nonlinearity. However, none of these approaches worked well. We believe that such failures may be because o-Asymmetry breaks symmetries at the activation / neuron level, whereas W-Asymmetry breaks symmetries in the larger space of weights (for more evidence, Appendix D where we show that fixing biases at neurons also fails to effectively remove symmetries). These curiosities provide interesting directions for future work.
F
the nonlinearity, sparsifying F, adding instead of
All in all, we believe that future theoretical and empirical study of Asymmetric networks could garner many insights into the role of parameter symmetries in deep learning.
Acknowledgements
We would like to thank Kwangjun Ahn, Benjamin Banks, Nima Dehmamy, Nikos Karalias, Jinwoo Kim, Mare Law, Hannah Lawrence, Thien Le, Jonathan Lorraine, James Lucas, Behrooz Tahmasebi, and Logan Weber for discussions at various points of this project. DL is supported by an NSF Graduate Fellowship. RW is supported in part by NSF award 2134178. HM is the Robert J. Shillman Fellow, and is supported by the Israel Science Foundation through a personal grant (ISF 264/23) and an equipment grant (ISF 532/23). This research (MURI ML-SCOPE), NSF AI Institute TILOS (NSF CCF-2112663) NSF
supported in part by Office of Naval Research
grant 1800014-20-1-220 award 2134108, and the Alexander von Humboldt Foundation.
References
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL
10

---

[2] Laurence Aitchison, Adam Yang, and Sebastian W Ober. Deep kernel processes. In International Conference on Machine Learning, pages 130-140. PMLR, 2021.
[3] Gul Sena Altintas, Gregor Bachmann, Lorenzo Noci, and Thomas Hofmann. Disentangling linear mode-connectivity. arXiv preprint urXiv:23/2.098822 2023.
[4] Stephen Ashmore and Michael Gashler. A method for finding similarity between perceptrons by forward bipartite alignment. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE, 2015.
[5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, Layer normalization, arXiv preprint arXiv.1607.06450, 2016.
[6] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. arXiv preprint 151 2015.
[7] Georg Bökman and Fredrik Kahl. Investigating how rel.U-networks encode symmetries. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https
//openreview //pppareriiere.ee/foouraaaaaaaa
[8] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, (4)):303-114 1989.
[9] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933-941. PMLR, 2017.
[10] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
[11] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
[12] Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: dissecting the weight space of neural networks. arXiv preprint arXiv:2002.05688 2020.
[13] Rahim Entezari, Hanic Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference Learning Representations, 2022. httpttttt////eeeeee
net/forummid WNitimiddddddiiytteee dNigytemkl.
[14] Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving linear mode connectivity of neural networks via optimal transport. In International Conference Artificial Intelligence and Statistics, pages 3853-3861, PMLR, 2024.
[15] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric.
arXiv preprint arXiv: 19030008, 2019.
[16] Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International conference on machine learning, pages 3318-3328. PMLR, 2021.
[17] Jonathan Frankle. Revisiting "qualitatively characterizing neural network optimization prob- lems", 2020.
[18] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning, pages 2259-3269 PMLR, 2020.
[19] Cedric Gegout, Bernard Girau, and Fabrice Rossi. A mathematical model for feed-forward neural networks: theoretical description and parallel applications. PhD thesis, Laboratoire de I'informatique du parallélisme, 1995.
[20] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35:1 1893-11905, 2022.
[21] Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. ICLR, 2015.
[22] William L Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020.
11

---

[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[24] Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129-135. Elsevier, 1990.
[25] Robert Hecht-Nielsen, Theory of the backpropagation neural network. Neural networks for perception, pages 65-93. Elsevier, 1992.
[26] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint /606. 08006. 15, 2016.
[27] Weihua Hu, Matthias Fey, Marinka Zitiik Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118- 18-22133, 2020.
[28] Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719, 2023.
[29] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations, 2023. URL httt::::/opeerevveee www.wviwwwwiineonnieeeeeeee net/forum?
id-gU5sJ6ZeggX
[30] Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Ben- namoun. Hands-on bayesian neural networks a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29-48, 2022.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412 2014.
[32] Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efstratios Gavves, Cees GM Snoek, and David W Zhang. Graph neural networks for learning equivariant representations of neural networks. arXiv preprint 2rXiv:2403.122/1 2024.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images, 2009.
[34] Richard Kurle, Ralf Herbrich, Tim Januschowski, Yuyang Bernie Wang, and Jan Gasthaus. On the detrimental effect of invariances in the likelihood for variational inference. Advances in Neural Information Processing Systems, 35:4531-45422 2022.
[35] Lauro Langosco, Neel Alex, William Baker, David John Quarel, Herbie Bradley, and David Krueger. Towards meta-models for automated interpretability, 2024. URL https
7/openreview wwwp:/eeeooooomiaaeeeeee...
[36] Olivier Laurent, Emanuel Aldea, and Gianni Franchi. A symmetry-aware exploration of bayesian neural network posteriors. In The Twelfth International Conference on Learning Representations, 2024. URL URL /tttp::/ wwwpeereeieeenee/ffooroee
[37] Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Alek- sander Madry. Ffev: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[38] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 -3244, 1998.
[39] Derek Lim, Haggai Maron, Mare T. Law. Jonathan Lorraine, and James Lucas. Graph metanet- works for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations, 2024. URL httpttttt:::::treeeetee wttp://www...eeo..m//i/.iiiii.e
[40] Nlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018 2018.
[41] James Lucas, Juhan Bae, Michael R Zhang, Stanislav Fort, Richard Zemel, and Roger Grosse. Analyzing monotonic linear interpolation in neural network loss landscapes. arXiv preprint 2rXiv:2104.11044, 2021.
12

---

[42] Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. International conference on machine learning, pages 1363-4371. PMLR, 2019.
[43] Edward Milsom, Ben Anson, and Laurence Aitchison. Convolutional deep kernel machines. In The Twelfth International Conference on Learning Representations, 2024. URL https
/
/oopnneviee. /
[44] Seyed Iman Mirzadch, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In Interna- rional Conference on Learning Representations, 2021. URL htttps://pperreviee net/
[45] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
[46] Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces, In International Conference on Machine Learning, pages 25790-25816. PMLR, 2023.
[47] Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment. arXiv preprint arXiv:2310.13394 2023.
[48] Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: aath-normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015.
[49] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on learning theory, pages 1376-1401. PMLR, 2015.
[50] Theodore Papamarkou, Jacob Hinkle, M Todd Young, and David Womble. Challenges in markov chain monte carlo for bayesian neural networks. Statistical Science, 37(3):425-444 2022.
[51] Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan
Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, Aliaksandr Hubin, et al. Position paper: Bayesian deep learning in the age of large-scale ai. arXiv preprint arXiv:2402. 00809,00000... 2024.
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[53] Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20237-20246. 2023.
[54] Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riceardo Zecchina. Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry. In International Conference on Machine Learning, pages 17781- PMLR, 2022.
[55] Arya A Pourzanjani, Richard M Jiang, and Linda R Petzold. Improving the identifiability of neural networks for bayesian inference. In NIPS workshop on bayesian deep learning, volume 4, page 31, 2017.
[56] Xingyu Qu and Samuel Horvath. Rethink model re- re-basin and the linear mode connectivity. arXiv preprint 2:Xix:2402005960 2024.
[57] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.
[58] Aviv Shamsian, Aviv Navon, David W Zhang, Yan Zhang, Ethan Fetaya, Gal Chechik, and Haggai Maron. Improved generalization of weight space networks via augmentations. arXiv preprint arXiv:2402. 04081, 2024.
[59] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
13

---

[60] George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoftmmnn. Zipit! merging models from different tasks without training. In The Twelfth International Conference on Learning Representations, 2024. URL httlttt:////peereeiiee http/forummid-LEEDvvddaa
[61] Héctor J Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural networks, 5(4):589-593) 1992.
[62] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 2826, 2016.
[63] Norman Tatro, Pin- Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Op- timizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 3ystems, 33:15300-111311111 15311,, 2020.
[64] Marcin Tomczak, Siddharth Swaroop, and Richard Turner. Efficient low rank gaussian varia- tional inference for neural networks. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4610- 4622. Curran Associates, Inc., 2020. URL
pp f http://pw..bopeoeeo///o//cceeff.
pdf [65] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and 1lya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002. 1148, 2020.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in nearal information processing systems, 30, 2017.
[67] Neha Verma and Maha Elbayad. Merging text transformer models from different initializations. arXiv preprint arXiv:2403.cm 00986, 2024.
[68] Tiffany J. Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? ICML, 2022.
[69] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.0644 2020.
[70] Xiang Wang, Annie N. Wang, Mo Zhou, and Rong Ge. Plateau in monotonic linear interpo- lation - a "biased" view of loss landscape for deep networks. In The Eleventh International Conference on Learning Representations, 2023. URL https://oppnrrviee...tttt nww.:orunfffff id=228951Q0Qna
[71] Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, and David Rügamer. Towards efficient meme sampling bayesian neural networks by exploiting symmetry. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 459 474. Springer, 2023.
[72] Tim Z Xiao, Weiyang Liu, and Robert Bamler. A compact representation for bayesian neural networks by removing permutation symmetry. arXiv preprint arXiv:2401.000 2023.
[73] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https
//openreview wwt.r/viweeeeefffrun het//forumiie-yyGG61.....
[74] Adam X Yang, Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, and Laurence Aitchison. A theory of representation learning gives deep generalisation of kernel methods. In International Conference on Machine Learning, pages 39380-39415. PMLR, 2023.
[75] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020. URL btsttp:///peerrevee..eest
id=ByxRMONtvi
[76] David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
14

---

[77] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[78] Bo Zhao, Tordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy. Symmetries, flat minima, and the conserved quantities of gradient flow. arXiv preprini urXiv:2210.1121) 2022.
[79] Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Understanding mode connectivity via parameter space symmetry. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023.
[80] Bo Zhao, Robert M. Gower, Robin Walters, and Rose Yu. Improving convergence and gener- alization using parameter symmetries. In The Twelfth International Conference on Learning Representations, 2024. URL. http:ttts:://peeeeeeiei
net/forrmm2dd-OOrOo
[81] Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in Neural Information Processing Systems, 36, 2023.
[82] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in Neural Information Processing Systems, 36, 2023.
[83] Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. arXiv preprint 2rXiv:2402.05232, 2024.
[84] Liu Ziyin. Symmetry leads to structured constraint of learning, arXiv preprint arXiv:2309.16932 2023.
15

---

Number of Epochs until 70%
70
0.0
11
12
60
Standard Deviation
0.75
11
12
15
19
22
50
11
2:25
11
24
36
44
53
20
3.0
-10000
-20000
-30000
-40000
Num Fixed
Figure 6: Epochs until reaching 70% training accuracy on CIFAR-10 when varying the hyperparame- ters of W-Asymmetric ResNets; we vary number of fixed entries flfix and standard deviation x of the fixed entries F. Entries further the bottom and right are more asymmetric, while the entries further to the top and left are more like standard networks (the leftmost column are all standard networks). We see that more more-asymmnn asymmmrcymttiic networks need more time to train.
A Additional Observations on Asymmetric Networks
There are several other interesting differences in the optimization and loss landscape properties of Asymmetric and standard neural networks. For one, even though Asymmetric networks generally interpolate significantly better than standard networks, this cannot be seen by measuring distances in parameter space. For instance, in GNN experiments following the setup of Section [..] pairs of standard GNNs have a distance per parameter of .000174 on average, whereas W-Asymmetric
GNNs have 5000159, which is only slightly lower. However, the average test loss barrier is 1.448 for standard GNNs while only 0.069 for W-Asymmetric GNNs. Likewise, our datasets of 10,000 standard and W W-Asymmetric ResNets, the average distance per parameter between the weights of trained standard classifiers is .0034, which is actually lower than the distance per parameter of .0051 for W-Asymmetric ResNets (estimated 20,000 pairs of networks). Thus, although sometimes imagine well-ineerpolating pairs of networks to lie in the same local basin of parameter space, W -s-ymmmettii networks actually rather far apart in parameter space, but nonetheless have linear segments low loss between them.
We also find that Asymmetric networks often do not overfit much as standard networks. For instance, the GNN setup of Section 5.1 standard GNNs have max training accuracy of 84.6% on average, with validation accuracy of 71.6%. On the other hand, a-Asym GNNs have 70.8%/70.19 train/validation accuracy, while W-Asym GNNs have 70.7% (1.7%//77.067 train/validation accuracy. This difference does not show as much in datasets of 10,000 standard ResNets and W-Asym ResNets, possibly because of the substantial regularization (data augmentation, weight decay, and label smoothing) used for training (standard gets 74.8% 73.8% train/test accuracy while W W-Asymmetric gets (..0).........................o)
Further, in Figure 6 we see that training speed is slower for W --yymmettiic ResNets when we increase the amount of asymmetry (by increasing the number of fixed entries and the standard deviation of the fixed entries). While standard ResNets take on average 11 epochs to reach 70% training accuracy on CIFAR-10, W-A --ysmmmerric ResNets with the most extreme hyperparameters take up to 72 epochs.
16

---

B Proofs of Theoretical Results
B.1 Graph-based approach
Here, we prove that as long as each mask matrix M in our W- W-Asymmetric MLPs with fixed entries set has unique nonzero rows, then architecture has nontrivial neural DAG automorphisms. In practice, we find that setting the standard deviation x the fixed entries F to be positive (and in fact orders of magnitude larger than the standard deviation that typically initialize trainable weights with) is important to achieve properties such as linear mode connectivity that Asymmetric networks have but standard networks do have. When zero), we can directly work in the framework of Lim et al.
(i.e. when fixed entries set to [39] that connects parameter symmetries computation graph automorphisms. To work towards generalizing our result
we would have to modify the definitions and results of Lim et al. [39]; for instance, we would need to add edges associated to untrainable parameters in the computation graph, and redefine the concept of neural DAG automorphisms. We leave such exploration to future work.
Theorem If each mask matrix M has unique nonzero rows, then W --yymmettiic MLPs with have no nontrivial neural DAG automorphisms.
Proof. Consider an L-layer W --ssymmetric MLP with fixed entries set to zero. Denote its weights as W..... W1 and the corresponding binary masks as ML,... M1. The forward pass of such network on an input x is then
W Myla(....aa
M1]z).... .....
(5)
for some elementwise nonlinearity a. The dimension of W, is d, d;- In the framework of Lim et al. [39] this is feedforward neural network with a computation graph defined as follows.
The node set is Vo x V1 .... x VL, where V; has d; nodes, and nodes shared between different V. If a node is in Ve, then we say that layer(u) = i. Vo contains the input nodes and Vz contains the output nodes. The adjacency matrix can be written
0
M1
0
M2
A
(6)
0
Mr
0
Every block besides the ones containing masks zero. There L L+ blocks, and the (i, block of size d, x dj-
Recall that neural DAG automorphism is relabelling of nodes V V such that is bijective, (i.5) E E if and only (T(I), T()) E E, and every input node and output node is a fixed point of т.
Now, let V + V be neural DAG automorphism. Further, let P be the corresponding permutation matrix. We will show that is the identity, i.e. that P I. By Lemmal layer number of nodes, meaning layer(r(1)) = layer(i) Thus, P is block diagonal permutation matrix:
know that preserves
Po
P
p
PL
where P. is d, x d, Morever, Po = and Pt = / because input nodes and output nodes are fixed points. Applying this to the adjacency matrix, we see that
0
P.MPP 0
(A) PAPT
(8)
PLM 0
Since is neural DAG automorphism, have that 7(A) A. Equating blocks, this means that
P Mj. As Po = L, we have M,M, M1. But M, has unique rows, so P = I as well.
17

---

For the inductive step, assume P: I for some i. Then Po+1M = P1+1M1+11 = M1+1. so since M,+1 has unique rows, we have that P.+1 As this holds for any by induction, this means that P = I, so is trivial neural DAG automorphism and we are done.
Lemma 1. Neural DAG automorphisms preserve layer number in W W-Asymmetric MLPs that have masks with nonzero rows.
Proof. Let be neural DAG automorphism. This means that PAPT A, where P is the permutation matrix associated to Then, using PAP A for the first equality and the definition of P the second, we have that
(0),) (PAPT) (1())) (PA))))( An,-
We proceed by induction on layer number 1. For any input node we know that r(1) 1, so of course layer(r(i) = layer(i).
Now, suppose that layer(r(i)) layer(i) for any i in layer 2 If node j is in layer / 1, then there is some in layer such that (i.j) E E because M++1 has no nonzero rows. We have that (7()).()) = Ai , (i) is connected to T(j). As we know that r(i) is in layer we have that 7 is in layer ++1. 1.
B.2 Symmetry Breaking via Nonlinearities
Proposition 3. Let the parameter space O be all pairs of square invertible matrices 0 (W2, W1) for 22, W1 E GL(d), and let fo(x) = ( (1æ)). If a has no linear equivariances, then for = foz if and only 0 = B2. In other words, there are no nontrivial parameter space symmetries.
Proof. If 01 02, then elearly foz= For the other direction, suppose fo, Sng. and denote 0, = (W2, W) and 02 = (W2, Wi). Then for any input IR", we have
W20(W 12) 20 W1)
(10)
W₂ Woo (W12) = (Wi2)
(11)
Now, choose an arbitrary R.. We let 2 in the above equation III be W. x, so we have
W W20(1) = o WW.,
(12)
This holds for any so W W2 000- do W1 WT i.c. we have found linear equivariance of a. Since a has no linear equivariances,
W. T= W.
(13)
meaning that W₂ W2 and W W, i.e 01 02 so we are done.
B.2.1 FIGLU nonlinearity proofs (Proposition) 2
Now, we study the properties of our FiGLU nonlinearity (x) (Fz)) @ x, where rj is the sigmoid function 7/x)) T+c-a. For proving Proposition we want to prove that with probability I over samples of F, has permutation diagonal equivariances.
We say that a has no permutation equivariances whenever P200 = a Pi for permutation matrices P1 and Pg. then P1 P2 I. Likewise, say that has diagonal equivariances if whenever B o = a o for invertible diagonal matrices A and B, then A B
We will show that these two properties hold for any F that has permutation symmetries and no zero entries. We say that F has no permutation symmetries if P,FP:
for permutation matrices Pi and P2 implies that Pi P3 = I. Note that if F has distinct entries, then it has no permutation symmetries. Thus, F' satisfies both of these conditions with probability 1, since the set of matrices with nondistinct entries or with at least one zero entry are of Lebesgue measure zero, so they have zero probability under the Gaussian distribution. We now proceed to show that a has no permutation or diagonal equivariances under these conditions on F.
18

---

Proposition 4. F is square matrix with no permutation symmetries. then σ(x) = ,n(x)) o z has no permutation equivariances.
Proof. Suppose a o P1 = P2 a for permutation matrices P1, P2 We will show that P1 P2 1.
For any input x, we have
(FP @ Pa = P2 (zx) ox]
(14)
P (FP(a) P (Fx) cx
(15)
PP3 @ P, P (Fx) C..
(16)
where used permutation equivariance of 7), which acts elementwise. Let e,, the standard basis vector that is in the ith coordinate and 0 elsewhere. If is not a fixed point of the permutation P. Pt, then let, j be the index that it is mapped to. Then equation (16) gives that
n(P Pie) @ Ps Pie = (Fe;)) O
(17)
Oe = Fye,,, Q e,
(18)
In the ith coordinate of this equality of vectors, see that n)Fe;) = O, which is impossible, since 1 is the sigmoid function. Thus, i cannot be fixed point of P2 PI. PT Pı I is the identity permutation. Now, let æ be an arbitrary vector with no zero entries. Equation [6 gives that
() F. 2) @ x = (Fx) @ x.
(19)
Since is nonzero, we can divide by x, in the ith coordinate of this vector equality for each get that
FP2)) (Fx).
(20)
As is bijective,
F2 Fr.
(21)
Because this holds for all with no zero entries (and in particular for basis of the input space), we know that
ps FP =
(22)
as matrices. But since F has no permutation symmetries, we have that P1 = P2 = I, so we are done.
Proposition 5. F is a square matrix with no zero entries, then oxx) = 17( FFx) O x has no diagonal equivariances.
Proof. Let A = Diag(a) and B = Diag( be invertible diagonal matrices, and suppose that a Boo. We will show that A = = For any input æ, we have
(FF(a @ x)) @ (a o x) 8 @ [n(Fxr @x]
(23)
Let x = cE,, where e; is the ith standard basis vector and + 0 is any nonzero number. Then
((Faaee; O co;e; 8 o [n(cFes) ce:]
(24)
At the ith coordinate of this equality, we have (FFones) = Brn(cFes)ic
(25)
0; n(cFes)s
(26)
3. n(a,cFee);
Thus, the right hand side is constant in c. We must have that 0. because if not, then increasing would increase either the numerator or denominator and decrease the other, hence contradicting the equality (here ai c + 00, we see that
that F has zero entries, cFe, is nonzero in every entry). Thus, letting
R. 1, so a4 = 3r. Plugging this back into Equation (66), we have
1 n(cFes)s
(27)
n(a,cFes);
n(a,cFe;): = n(cFess
(28)
(Fc(Fe;): = c(Fei)i
(29)
a; = 1,
(30)
where in the third line we used the fact that r) invertible. We have shown that o; 33 1 for each
i, so A B =I and we are done.
19

---

We note that the proofs of these two results about FiGLU are reminiscent of some proof techniques from Godfrey et al. 220] such as those used in their analysis of GELU nonlinearities.
B.3 Proofs for Universal Approximation
Here, we prove the universal approximation result for our W-Asymmetric MLPs.
Theorem 4. Let n be any nonpolynomial elementwise nonlinearity with n7x) 17( I (e.g. ReLU, GELU, swish), let Ω C RR be compact domain, and let ftarget Q + R be continuous target function. Fix ε and O.
There exists a width n' such that for all n >> n!, with probability 1-6, f.r. randomly sampled 4-layer W-A W-Asymmetric MLP f with y nonlinearity, hidden dimensions 24n -> + 24n, and nfix 1/1) hardwired entries per neuron, there will exist E θ such that the W-Asymmetric MLP f: R - I
approximates ftarget to
[o 0)) - Starget (2) for all E Q.
(31)
Importantly, we require that the input to fo be padded with n zeroes, so (x; 0] R..
B.3.1 Proof sketch
To approximate Starget to 8 > we will leverage the universal approximation for standard MLPs with nonlinearity 7) to first obtain a standard 2-layer MLP that approximates ftarget to within meaning [targett (x) [MLL(x)] || < for all x E Ω. Then we will exactly represent fMLP using an Asymmetric Network f-
This will be done by approximating each linear map W of fMLP by two layers of an Asymmetric network: W1 i) o W = W for Asymmetric linear maps W and W1. For the sake of exposition, we will show how to do this first when both W32 and W/ have no Asymmetric mask (i.e. fitting linear map W using standard two-layer 17-MPP), then when only W2 has an Asymmetric mask, and finally when both W and W have an Asymmetric mask.
B.3.2 Fitting Linear Map with a Two-Layer Standard MLP
Let W E RRx= be the target linear map, and let B E R and A E Rnx2nn be parameters of two-layer MLP, defined by fa.((z) Arr) will choose A and such that fa,B(*) = W for all x E I
Denote the ith row of W by W. that
Wo :
Wo x
W
Wx
(32)
Wn-1
W
We set A and B as follows, where Ln is the n identity matrix:
Wo
-Wo
Wi
A In 8 -1]
==
-Wi
(33)
W
W
Then that SA,B exactly computes the linear transformation Wa.
nWW z) WW xx)
Wo x
An(Br)=
=
W..
(34)
WWWn- 2) 1 W n- Wn x)
Wn-13 σ
20

---

B.3.3 Fitting Linear Map with One Asymmetric and One Standard Linear Map
Let nix and let each row N {0, 1 J have fäx entries equal to 0, selected at random. Let B E RR and A E R Define A' to be an Asymmetric linear map: A' = A where N a randomly sampled binary mask, and P a randomly sampled Gaussian matrix. We consider two-layer network with one Asymmetric and one standard linear map: fa.((z) Az). We want (z) = Wz for all x. For the remainder of this proof, we will assume that N never has three consecutive entries in row set to zero; will later show that this holds with high probability over the sampling of N.
First, we define B in similar way to the purely linear setting, but with additional copies of entries to allow for error correction of the random noisy entries fixed in A'.
Wo
Wo
W
Wo
Wo
Wo
B
(35)
W
W n
W
W
Wn-1
W Wn-1
Recall that each A of A' has fixx entries that randomly hardwired constants. Ideally, would want A₁ to pick out 17W, -x) 77-WW W. but because of the hardwired constants, A might randomly add c* ( x). However, since there are three copies (W, w) 17( Bxx). (...aa as long not all three corresponding entries in A; are fixed, one of the un-fixed copies can be changed such that the coefficients sum O. Since by our assumption N never has three consecutive entries all set to O, the coefficients of A can be picked such that A' Bz) = Wa. For example, a possible A' matrix would be
0
.37
1.47
A' =
89
89
0
0
.37
63
0
Thus we have shown that under the assumption that N never has three consecutive en- tries equal to 0, can be picked such that A' Aar) Wx. We will now show that P(N never has three consecutive entries equal to 0) can be made arbitrarily small by increasing the width while keeping to be o/3
The probability there are three consecutive entries given row of N that are zero is O By the union bound, the probability that any row has 3 consecutive hardwired entries is O Pis nix E o(n 1/3), this tends towards 0. Thus, with probability 2 O "ixx can be picked such that ABn( = Wx.
n²
For any
B.3.4 Fitting Linear Map with Two Asymmetric Linear Maps
Once again let Rix > 0, and let each row of M have nix entries equal to 0, selected at random. Let B E 2inxx and E Rnx22nn Further, define Asymmetric maps
A M -
B' Bo
Q
(36)
where N, M are randomly sampled masks, and P, Q are normal matrices. Then we let fA,n(x)
=
A'n(B' (x)), and once again desire choices of parameters A and B such that fa. WI.
21

---

Constructing B
Consider the randomly drawn mask M E {0, 1) 122 and denote the ith row by M,.
Mo
M1
M
(37)
M24n-1
where M, E 10,1)". We partition M's rows into blocks of 24 rows. $1 (Mo.... M2s)..... ..... {M2ai M211+23). Now, consider $1, the first 24 rows of M.
Definition B.1. We say two rows Mj. My are intersecting if there is some column index a such that Mjn = Ma 0. That is, two rows of are intersecting if they share 0 the same index.
Note that for any two given rows of M, the probability that they share 0 in the same location is nixx
We assume that 3, contains no more than one pair of intersecting rows; later, we show this to hold with high probability. Then, every 3; can be broken into two disjoint sets of 12 rows, 8,,0 and B., such that neither set of 12 contains a single pair of intersecting rows. Intuitively, this means that each row in B corresponding
3i, will have unique fixed indices.
Our goal will be for the rows in Bg to mimic the row W;. We will show how to do this for each i. Fix an arbitrary index E nO. 1}.
Without loss of generality, assume 9.,0 and B are continguous, so 3 = M240241+11 and B M24++11:245+23 By our assumption, for j. k E $.,0 (i.e. j. E (0.....11 the mask rows M24i+j and M2114k are never in the same two column indices. Similarly, for k 8.1 (i.e. j.k E {12.... 23),, the mask rows M242+5 and M24i44 are never 0 in the same two column indices.
Next, we define Cijj as the difference between B' and in the (24i + j)th row:
Cij = B2 B241+5-
(38)
In particular, we have that
C.j = B24i+s ① (1 M24i+z) + (1 -Mziit)) O Q24it5-
(39)
Lemma 2. For any indices j + k such that j, k E Br,0 or j,k k Bi, we have that
Cij 2 M24148 Ci.j-
(40)
Proof. By the definition of Ci we know that C., only nonzero at indices where M241+ equal to zero. Since j, k are either both in 8L, B. we know that M2ti+ cannot also be zero at indices where M241+; is zero. Thus, M241+* is equal to at every index where Cr is nonzero, so Cij 3 M241+k Cij as desired.
Next, we construct B, by constructing this block of 24 rows. Let [cr,o, C,,1, C222, C,a3] be continguous sums of length-3 segments of c;.:
C = C; + 4,1 + 0,2
(41)
C, 4,3 9,4 + 4,5
(42)
C222 = 6,6 4.7 + 0
(43)
C1 = 6,9 4,10 + 11
(44)
We assign the first 12 rows of B as follows.
(0 S < 3) + B24145 = W; + C₁ - C₁ + C122 - C,,3 - Cij
(45)
(3 6) + B2ti+j = -W, C,, + C; 022 C4,3 Gjj
(46)
(65j<9 9) + B24i+5 = +C;,0 - C, + C.2 C133 Cjj
(47)
(9$j<1) 12) + B24i45 = Cc,0 + C;,1 C.2 C1,3 Cij
(48)
22

---

Defining C;,0 C1,1 + C122 Cr,,, we have the nice property:
(0$i<3) - Bruts = W; +
(49)
(3$j<6) + B = - W,
(50)
(6$)<9) + B2i1+s = +C
(51)
(9 < 12) B24143
(52)
By the construction above, Bl B244443 and Bl4i46 B244449- This means that
7( (42 2) m (B244433 2) = B24 x == (W; + c)
(53)
and likewise that
n BB2444 x) n (B22449 x) c
(54)
So that a simple linear map gives our desired output:
(B2as 2) n(B22sita =) [n)(Buute 2) B2244 z)] (W; c cc
(55)
W
(56)
In the next part, will construct A' to compute this linear map, which will follow the method of Appendix B.3.3 (because A' has certain fixed entries).
What remains is to define the rows of B corresponding to Br, in an enror correctible manner. This can be done easily by defining
23 d
Σ
(57)
j 12
and then defining
(12 < < 24) - BBus+ = Cij
(58)
By similar reasoning to before, this means that
1 24) + Bhaits d.
(59)
Recall that constructed B' under the assumption that no 3y has at most pair of intersecting rows. We now show that the 3g each have at most one pair of intersecting rows with high probability. Within the 24 rows of any given 3, the probability that more than one pair of rows are intersecting is < Si for some constant C. So, by the union bound, the probability over M that any of the 3 have T
more than one pair of intersecting rows is < C nix Thus, we can construct B in this manner with
probability - - For sufficiently large and nax o(n)/1),, this probability approaches 1.
23

---

Construction of A
With our above construction, each block of the 24 rows in 3 of n(Bxx) is of the form
7 +c) .2)
n(WW, (00
7(WW, +(W
m(-W; -
( (WW,, (c) -x)
(((-W) (((
W -c)
7/c xx)
C
n -
7 -C.0
M C.--0
(60)
m(d [d.
n(d
7 (
77
m(d x
n(d
7(d
n(d-
n(d
n(d
Importantly, each row here is wide. Recall that A' E Rax2in Denote the ith row of A' by A/ E 224 with A E 24n A' had hardwired entries, then setting A, = 121 121133
(121166 (21+9) would give Am(BB' =) W by the same argument as in Appendix B.3.2
Unfortunately this A' has nix fixed entries in each row. This means that N; has Rix entries equal to 0. Since every entry of B'x has three copies, as long as N, does not have three elements set to 0 in row, A₁ can be made equivalent to A; 1244 - 1244+3 - (124i+6 - 124449). This is because, as in Appendix B.3.3 if at most elements out any three copies are hardwired, then the third can changed arbitrarlly to offset the hardwiring.
the case, so we have to use the construction in Appendix B.3.3 Recall, that
Further, just as in Appendix B.3.3 the probability that given row of A' has three items hardwired Thus, by the union bound, the probability that some row of A' has three
in row is O nag n
items hardwired in row is O pis So, with large enough width A can be chosen such that (B'( Wx.
Similarly, any linear map in Rax for ñ also be fit using this method.
Conclusion
We have shown that --ymmmettii MLP with hidden dimension 24n exactly fit an linear map with high probability over the choice of Asymmetric masks M. It known by ( that for any continuous function Starget Ω RD IR and any 0, there exists width 1 such that 2-layer MLPs of width k can approximate f to within e.
Let & sufficiently big so that the probability that the masks do not satisfy the conditions of Appendix is less than Such a exists as long as näx E o(k²). Let m 2 max(k, k' D). Importantly, if a 2-layer MIP of width A'' can approximate Starget to within s, 2-layer MLP of width m with m k can also approximate f to within e. SMLP be a width m MI.P that approximates frarget to within s.
24

---

We now pad the input I to frarget, with m D zeros. This allows us to define a new function ftaggtt TR + R by Searged (10; 0]) = (x) Clearly fauget can also be approximated by a width m MLP.
Let MMIP denote the width m MLP that approximates fo to within E. Now, MMLP has dimensions m - 1, with corresponding linear maps W1 E R W2 E R Each of these maps can be exactly fit using a 2-layer W-Asymmetrie MLP, since their corresponding matrices have at least many columns rows. Concatenating these two exact fits yields an asymmetric MLP whose output exactly matches SML and thus approximates fo to within E.
Thus, setting n' m, there exists width n' such that for all > n', with probability 1 8, for a randomly sampled 4-layer W-Asymmetrie MLP with y nonlinearity, hidden dimensions 24n - - 24n, and náx o(n!/4) hardwired entries per neuron, there will exist 0 such that the W-Asymmetric MLP f R" + E approximates ftarget to
On parameter symmetries of the 4-layer W-Asymmetric Network
As an aside, this procedure of mapping 2- 2-layer standard MLPs to 4-layer W-Asymmetric MLPs implies that these 4-layer W-Asymmetric MLPs have least as many symmetries as 2 layer standard MLPs. To fix this, we may want to consider nonlinearity 7 such that n7()) (-=x) I.
C Limitations
Although our W-Asymmetric and o-Asmmeetric networks are motivated by removing parameter space symmetries, their distinct empirical behavior may be caused by other factors besides just parameter space symmetries. For instance, the fixed entries F for the W-Asymmetric approach are taken to be much larger than the standard initialization of linear maps, which could cause several changes to optimization and loss landscapes besides just parameter symmetry breaking.
Also, our theoretical results could be strengthened by future work in several ways. For instance, for the --Asmmmetric approach, Proposition (only gives guarantee of no parameter symmetries in the two-layer network case with square invertible weights. Future work could also give tighter analysis of the required width and depth for universal approximation using W-Asymmetric architecture.
D Broader Impacts
This work does not focus any particular application area. Instead, study fundamental phenom- ena and theory of deep learning in general. Our work has potential to improve known deficits of neural networks: by making neural network loss landscapes more similar to convex landscapes, can improve our understanding of them, and by improving Bayesian neural networks we advance one paradigm for bettering uncertainty quantification in neural networks. However, unlike standard neural networks, which have millions of papers studying them, we have only scratched the surface of Asymmetric networks. Important properties such as generalization, robustness to distribution shifts, and adversarial robustness have not been extensively studied for Asymmetric networks, and the interaction of parameter symmetries with these properties is not clear. Future research should further explore these important properties.
E Experimental Ablations
Here, present experiments with various ablations. These were in-part suggested by anonymous reviewers for the NeurlPS 2024 conference (thanks!).
E.1 Matching Learnable Parameters
For the experiments in the main text, our W-Asym networks often had less learnable parameters than the standard networks that they were compared against (since we take them to have the same architecture, but the W W-Asym approach fixes certain learnable parameters to constants). See Table for the learnable parameter counts.
25

---

Table 5: Number of learnable parameters of Standard o-Asym and W-Asym nets for our experi- ments.
Experiment / Architecture
Standard / Ø-Asym
W-Asym
5.1
MLP
935,434
834,570
ResNet
272,474
230,024
5.1
ResNet 8x
17,289,80
16,273,946
GNN
176,424
171,576
5.2
MLP-8
3,242,146
3,324,466
5.2
MLP-16
2,960,242
5,796,002
5.2
ResNet-20 Ix
1,356,098
1,143,858
5.2
ResNet-20 2x
5,410,386
5,044,756
5.2
ResNet-110 1x
8,620.418
7,37
5.2
ResNet- 2x
34,512,276
32,014,996
5.2
ResNet-22 2x
5,410,386
5,044,756
5.3
ResNet
78,042
60,634
3.4
MLI ResNet
78,042
60,634
Table 6: Metanetwork performance, including results with smaller standard networks (Smaller
ResNet) at same number of parameters at W-Asym. There is substantial difference W-Asym ResNets are still substantially easier to predict performance
ResNet
Smaller ResNet
W -sysym ResNet
R²
R²
R²
MLP
.330
.389 .03
.400
.594
.864 1
DMC
.950 = 001
.787 02
.943 + .01
.779
01
.967 .01
.91 00
DeepSets
.855 = .01
.617 + 003
.849 001
.627 + 001
.936 - -00
.858 + .00
StatNN
.966 .00
.866 + .00
.976 .00
.869 + .00
.978 00
.935 + .01
In this section, control for this, by decreasing the width of the standard networks that they have same number parameters as the W-Asymeetric networks they are compared against. Results are shown in Tables [nd 7 We see that there is little to change in metanetwork performance or Bayesian neural network performance for these smaller standard networks, so our results from the main text all still hold.
E.2 Changing Number of Warmup Steps
Altintas al. 3 showed that amount of learning rate warmup affect the extent of linear mode connectivity between training runs. In Table we see that varying the number of warmup epochs between and 20 does not change the qualitative results much test loss barrier for linear mode connectivity experiments.
E.3 Failure to Break Symmetries by Fixing Biases
Intuitively, one way to break permutation symmetries of an MLP is to order the hidden neurons in some way. One possible way to do this is to fix the biases (so that they are untrained). As shown in Table 9 basic attempt at this fails, and has much larger test barrier than our W-Asym and A-Asym networks.
F
Experimental Details
F1
Linear Mode Connectivity Experimental Details
F.111 Image Classifier Interpolation
For the image classification experiments, use two types of models.
26

---

Table 7: Bayesian NN test accuracy after 25 epochs. Decreasing standard ResNet20 parameters to match that of W-Asym ResNet20 does not substantially change performance.
Base Network
Test Accuracy
W-Asym ResNet20
49.3 + 0.4
Standard ResNer20
46.8 + 0.9
Smaller Standard ResNet20
46.5 = 1.1
Table 8: Test loss barrier when changing warmup steps. Results are very similar when lowering
number of warmup epochs (W-Asym interpolates significantly better than Git-ReBasin). Adam optimizer with learnigg rate le-2 (ResNer20) and le-3 (GNN) is used.
ResNer20 (ReBasin)
W-Asym ResNet20
GNN (ReBasin)
W-Asym GNN
Epoch Warmup
.80 2.0 + .21
.673 1: .29 .934 + .72
.249
.075 + .04
20 Epoch Warmup
.292 + .04
.074 + .02
1. ResNet We train ResNet20s with LayerNorm of width 64 and 8 64. We batch size of 128 and a learning rate that warms up from .0001 to .01 over 20 epochs. In the width 8x multiplier case train for 50 epochs, and in the width 1 x multiplier case train for 100. For o-Asymmetric ResNets, we warm up to a learning rate of .001 instead of .01 due to training instability.
2. MLP We train MLPs with 4 layers, LayerNorm, and width 512. For MNIST we tuned the hyperparameters (epochs, learning rate, weight decay) of both the Asymmetric and Standard models minimize loss barrier. We use batch size of 64.
For MNIST we data augmentation, and for CIFAR-IO we use random cropping and horizontal flipping. For the Git-ReBasin tests, we use the weight matching algorithm from [T. For MLPs on MNIST, we used the Asymmetry hyperparameters in Table I0 Table giives the Asymmetric hyperparameters for ResNet20 on CIFAR-10, 10, and Table 12 width.
lists the same for ResNet20 with 8x larger
F.1.2 Graph Neural Network Interpolation
For the GNN experiments, we use GNN architecture similar to GIN [73 with mean aggregation. The base GNN has three message passing layers and a hidden dimension of 256, which gives 176,424 trainable parameters. The dataset is ogbn-arXiv (27), which is citation network of computer science arXiv papers with 169,343 nodes and 1,166,243 edges. The task transductive node classification, where the label of each paper node is the primary subject area of the paper.
As is common in transductive node classification on modestly sized graphs, train each network with full-batcd gradient on the whole graph. Thus, the randomness in training is purely from the initialization there no noise from minibatch selection in SGD. We use the Adam optimizer [31] with peak learning rate of ,001. The learning rate is linearly warmed up for 25 epochs to the peak, and then is held constant. Each network trained for 500 epochs.
For the Git-ReBasin alignment, we implement the activation matching approach. For the a- Asymmetric GNN, take to be FiGLU, in which as standard normal matrix with standard deviation .01 01/vd where d is the number of hidden channels; we found that having small standard deviation helped with training and interpolation. For the --ssmmmetric GNN, we fix 6 constants in each row of each linear map, and randomly initialize these constants from normal distribution with standard deviation .5.
randomly initialize each fixed matrix
F22 Bayesian Neural Network Experimental Details
For training Bayesian neural networks, we use the variational inference approach of Tomezak et al. 644. which fits an approximate posterior that is Gaussian with diagonal plus rank-4 covariance matrix structure. For the W-Asym ResNet tests, we train ResNer20s with the same Asymmetric heprraarameters in Table II though with 55. For the CIFAR- -00 experiments, use
27

---

Table 9: Loss barrier when fixing biases to attempt to break symmetries for standard ResNet20 on CIFAR-10. Biases are between |-k, k.. Barriers are much larger than for W-Asymmetric Nets (.934 + 72) and a-Asym Nets (2.521 + 46).
k
Loss Barrier
1
5.81 + 3.67
3
3.76 = 0.38
9
4.62 1.03
27
10.6 4.29
Table 10: W-Asymmetric network hyperparameters for depth 4 MLPs. Dix refers to the number of weights randomly fix per neuron. к refers to the standard deviation of the normal distribution that the fixed entries F are drawn from.
Layer
Räx K
Lincar-1
Lincar-2
64
Linear-3
64
Lincar-4
256
Table 11: W-Asymmetric network hyperparameters for ResNet20s with width multiplier 1. nix refers to the number of weights we randomly fix per output channel (for convolutional layers) or neuron (for linear layers). refers to the standard deviation normal distribution that the fixed entries F are drawn from.
Block
f1x
First Conv
12
Block Conv
36
Block Skip
Block 2 Conv
54
Block 2 Skip
Block 3 Conv
72
Block 3- Skip
Linear
Table 12: W-Asymmetric network hyperparameters for ResNet20s with width multiplier 8 CIFAR-10. 10. We use 3 times more fixed entries per output channel or neuron than for Table TT
Block
Fix
Conv
Block - Conv
108
Block 1 Skip
12
Block 2 Conv
162
Block Skip
Block 3 Conv
216
Block 3 Skip
24
Linear
24
standard linear layer instead of hardwiring weights for the last fully-connected linear layer. On CIFAR-100 100 we also use width multiplier of 2 for our ResNets. For the ResNet experiments, we use a learning rate of .001. We train with batch size of 250 for 50 epochs.
For the MLP experiments, we use i = .5, 8 hardwired entries per neuron, and a learning rate of .0005. A batch size of 250 is used for 50 epochs again.
We use standard data augmentation (horizontal flips and random crops) on CIFAR-10 and CIFAR-100, and no data augmentations for MNIST. All training is done with the Adam optimizer [3I].
28

---

F.3
Metanetwork Experimental Details
F33.1 Dataset Details
We trained two datasets of image classifiers on CIFAR-10:
one consisting of 10,000 small ResNet-likk convolutional neural networks, and one consisting of 10,000 networks with a similar architecture, that use our graph-based approach to removing parameter symmetries. For fast training of many image classifiers, we use the FFCV package [37]. In particular, we use their CIFAR-I0 sample script https://ga twg..htt....... wttp:////wccw///////////////a//a http://tteeeeeeee///e//eeeaaeaae http///aaapplleeeeeeeeeeeeeeeeee /air which includes data aug- mentation (random horizontal flips, random translations, and Cutout [IQ),, label smoothing [2], and a linear learning rate warmup and decay. In total, training all 20,000 classifiers takes just under 400 GPU hours (about GPU-wecks) on NVIDIA RTX 2080 Ti GPUs.
See Table 13 for the hyperparameters and ranges that we varied across the networks in our datasets. In each dataset, the trained networks all have the same architecture.
Each ResNet has 78,042 trainable parameters, and each W-Asym ResNet has 60,634 trainable parameters. Both have the same architecture, except the W-Asym ResNet has certain filters that are fixed to constants to break the parameter symmetries. The ResNets each have convolution layers, LayerNorm (5), and a final fully-connected linear classification layer after average pooling across spatial dimensions.
Table 13: Hyperparameters and distributions we sampled from for the datasets of image classifiers that we trained on CIFAR-10. Unif(a, b) the uniform distribution over (a, b). and RandInt (a, b) the uniform distribution over integers in (a, bj (inclusive of endpoints).
Hyperparameter
Distribution
Learning rate Weight decay
.5 10 Unif(0.2)
10 Unif(1,5)
Label smoothing Epochs
Unif 2) RandInt (10, 40)
F.3.2
Metanetwork Details
Table 14: Learning rate and number of parameters for each type of metanetwork trained in Table
ResNet
W-Asym ResNet
LR
# Params
LR
# Params
MI.P
10-1
4,994,945
10-4
3,880,833
DMC
12
10-3
105,357
5 10-3
105,357
DeepSets 77
10-2
8,897
5 10-3
8,897
StatNN 65
10-3
119,297
10-2
119,297
We trained several types of metanetworks for our experiments. All of these metanetworks are trained for 50 epochs using the Adam W optimizer (40). For each metanetwork, each dataset, choose the learning rate in {10-5, 10-4,5 10 1003,13 5 10-X 10-2} that gives the best validation lt2 performance one training run. Then we run train each type of metanetwork 5 times on each dataset, and report the mean and standard deviation for each metric in Table 3
F.4 Monotonic Linear Interpolation Experimental Details
For the monotonic linear interpolation experiments, used the same setup as in the training of the datasets of CIFAR-10 image classifiers in Section [.3 For each architecture, we sample 300 sets of hyperparameters from the distributions in Table 13 and train network for each set of these sampled hyperparameters. When evaluating training loss, we include the labeling smoothing term.
For the --Asmmmetric networks, we initialize the FiGLU with standard deviation of 1/Vd, where d is the number of channels in the layer. Note that this is considerably larger than the standard
29

---

deviation of .01//ád used in the GNN experiments of Section 5.1 (note that this initialization for the o-Asymmetric networks, 24 out of the 300 networks diverged during training (giving NaNs), so we exclude them from the computation of statistics in Table 4 From manual inspection, this divergence seems to happen when the learning rate is high (greater than .1). In contrast, none of the standard or W-Asymmetric networks diverged.
we found this setting to train better line with standard initializations of trainable parameters). Further,
F.5 Miscellaneous Experimental Details
The datasets we use are MNIST [38, CIFAR-I0 [3), CIFAR-100 [33, and ogbn-arXiv [27]. which all widely used in machine learning research. The first three appear to not have licenses and open to use, while the last dataset is from the Open Graph Benchmark, which has an MIT License in the Github repository.
We use software packages including Py [52] (for all neural network experiments), FFCV (37 (for building our dataset Section 5.3). and PyTorch Geometric [15] (for GNN experiments).
ran our experiments on several types of NVIDIA GPUs and compute systems, including 2080 Ti, 3090 Ti, 4090 Ti, and V100 GPUs. Every training run conducted on at most one GPU.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Sections 4 and 5, include theoretical proofs, description of the methods and experimental setup, and experimental results to support the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix C we detail empirical and theoretical limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
We give the full set of assumptions and proofs for each result in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give high-level experimental setup information in the main paper (Sec- 5i and more detailed information about experiments in the Appendix F
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: We have open sourced the code to reproduce our experiments
https
//github.com www.b/..ccccceeooeooeeeeeeeceeee cptq/
asymmetric-netxorrk http://////pppp/aaaaaettttteeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https wwt.:Suwwisssispoooooooooooooooe
/nipps.cc/pulli///uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe high-level experimental setup in Section 5 and fine-grained experimental details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We generally do several runs of experiments, and report error bars that measure the variance in the results. We also conduct statistical significance tests Table 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
33

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative enrrr rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: While we are not fully comprehensive, we include some details compute Appendix F5 and estimate the compute required for generating our datasets of image classifiers in Appendix F3 All other experiments follow popular training paradigms and are small in scale.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::/nnuriippp wwwpeeppp..ppppppii///ppp..p http:///wbc..ct.tccc.....eeeeae.
Answer: [Yes]
Justification:
We have read and complied with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss both potential positive and negative societal impacts in
Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not train models data that have high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit code and data used in Appendix R5 For other types of models, cite papers in which they were introduced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have open-sourced new assets here: https://githut: hwt.ggtthh..ccocppp.ooeeee com/cptq/
asyimetrric-netwokks
The documentation can be found at the link.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
crowdsourcing or human subjects were used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or human subjects were used.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "NJFaE4u6pE", "reviewer": "Reviewer_hbBt", "summary": "This paper studies how removing parameter symmetry of neural networks affects the loss landscape, Bayesian Neural Networks and meta-networks. The authors proposed two ways to remove the parameter symmetry: one is similar to pruning that making some parameters untrainable, the other is to adopt non-elementwise activations. Both ways can remove the permutation symmetry in parameter space. After that, the authors empirically demonstrate that removing permutation symmetry substantially 1) make LMC easier to satisfy 2) make BNN's training more efficient 3) improve the performance of meta-network (I am not quite familiar with meta-network or neural functionals.) 4) make training loss along the line segment between initialization and trained parameters more monotonic.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The writing is clear and easy to follow. Especially, the structure of this paper is clear: front part is about the two ways to remove parameter symmetry, back part is to demonstrate the effect of removing parameter symmetry on loss landscape, BNN, meta-network and MLI.\n2. This topic is pretty interesting. Permutation symmetry persists in most neural network architectures and impose structure beyond Euclidean structure to parameter space, however, the community has no deep understanding of how permutation symmetry relates to the success of Deep Learning.\n3. The experiments on BNN, meta-networks and MLI are interesting.", "weaknesses": "1. A major issue about this paper is that some findings are not new. Especially, some studies have already demonstrated that asymmetric networks are more easily to satisfy LMC. [Cite 1] showed that pruning solutions are within the same basin and pruned networks can be viewed as a special case of W-asymmetric networks.\n\n[Cite 1] Evci, Utku, Yani Ioannou, Cem Keskin, and Yann Dauphin. \"Gradient flow in sparse neural networks and how lottery tickets win.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 6, pp. 6577-6586. 2022.\n\n2. Another issue is that in experimental part, the four hypotheses seemingly do not relate to each other. There is not a unified conclusion drawn from the experiments.\n3. The most important question about the permutation symmetry remains unresolved. As permutation symmetry holds in most neural network architectures, should we remove permutation symmetry or not? Is the success of deep learning related to the permutation symmetry? This paper seemingly cannot give an insightful answer.\n3. In Sec. 5, despite each experiment are motivated from some simple intuition, there is no rigorous theoretical foundation for each hypothesis, which could potentially lower the value of this study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer appreciates the writing, topic, and experiments of the paper, and we appreciate the reviewer’s comments. Below, we address them one-by-one.\n\n> “A major issue about this paper is that some findings are not new. Especially, some studies have already demonstrated that asymmetric networks are more easily to satisfy LMC. [Cite 1] showed that pruning solutions are within the same basin and pruned networks can be viewed as a special case of W-asymmetric networks …“\n\nWe respectfully disagree. The citation refers to methods for pruning a standard neural net, which are very different from our W-Asymmetric nets. The pruning methods for standard neural networks require specialized training algorithms that differ significantly from standard training (e.g. lottery tickets require repeated training and resetting, dynamic sparse training requires updating connectivity during training).\n\nAs noted also by reviewers ceUs and tX9W, our methods are novel. Unliked pruned networks, our W-Asymmetric networks have a fixed sparsity. Thus, they can be trained by standard training algorithms like Adam, just like standard neural networks. This is crucial, since we want our Asymmetric networks to be as similar as standard networks as possible, so that we can gain insights into standard networks. See more on this important point (importance of using standard optimization algorithms) in the general comment.\n\n> “... in experimental part, the four hypotheses seemingly do not relate to each other. There is not a unified conclusion drawn from the experiments.”\n\nThis is true, but we do not see this as a downside. In fact, the point of our work is to provide a tool (analysis of asymmetric networks) for us and future works to study the effects of parameter symmetries in many different phenomena at once.\n\nAlso, there are some higher-level conclusions that we will emphasize more in the revision. For instance, asymmetric network loss landscapes are somewhat more well-behaved and similar to convex loss landscapes than standard networks, and we show that understanding aspects of neural network optimization requires consideration of parameter symmetries.\n\n> “The most important question about the permutation symmetry remains unresolved. As permutation symmetry holds in most neural network architectures, should we remove permutation symmetry or not? Is the success of deep learning related to the permutation symmetry? This paper seemingly cannot give an insightful answer.”\n\nThis is not the point of our work. We probe the effect of parameter symmetries (not just permutation symmetries) in already many different phenomena in deep learning. For instance, such symmetries should be accounted for when merging models, processing models with metanetworks, or training Bayesian neural networks.\n\n> “In Sec. 5, despite each experiment are motivated from some simple intuition, there is no rigorous theoretical foundation for each hypothesis, which could potentially lower the value of this study.“\n\nAs mentioned in our abstract, “theoretical analysis of the relationship between parameter space symmetries and these phenomena is difficult.” As in most of the study of deep learning theory, it is very difficult and sometimes intractable to theoretically analyze the effects of symmetries in all of these domains. Even when some theory can be done, it is usually done in very restricted settings (e.g. infinite width, one or two-layer networks, optimization assumed to reach a global optima), which are very unrealistic.\n\nWe can consider examples from the literature of theoretical analysis of these hypotheses. In [Ferbach et al. 2024], the authors can only theoretically prove linear mode connectivity up to permutations for the (unrealistic) cases of two-layer mean-field MLPs or untrained MLPs. In [Kurle et al. 2022], parameter symmetries in Bayesian learning are theoretically analyzed for a linear model trained on one datapoint.\n\nThe point of our paper is to analyze the effects of symmetries in many domains at once (including some that are not covered in our paper) via empirical studies, which have arguably been more impactful in the study of deep learning. Many other works study deep learning phenomena in a purely empirical way, and they give inspiration for future theory; these include seminal works like those on the Lottery Ticket Hypothesis [Frankle & Carbin 2018], permutation matching for merging [Ainsworth et al. 2022], and early empirical investigations into neural networks [Goodfellow et al. 2014].\n\n**References**  \n[Ferbach et al. 2024] Proving linear mode connectivity of neural networks via optimal transport.  \n[Kurle et al. 2022] On the detrimental effect of invariances in the likelihood for variational inference.    \n[Frankle & Carbin 2018] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.  \n[Ainsworth et al. 2022] Git Re-Basin: Merging Models modulo Permutation Symmetries  \n[Goodfellow et al. 2014] Qualitatively characterizing neural network optimization problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zStQwP6iHE", "reviewer": "Reviewer_ajZW", "summary": "This paper proposes to study the effect of parameter symmetries on the neural networks' training and final properties by analyzing the behavior of networks without such symmetries (or with fewer of them). To do so, the authors develop two methods of parameterizing neural network architectures without parameter symmetries: W-asymmetric parametrization fixes the different subsets of weights in each row of weight matrices to be constant and untrainable, and a $\\sigma$-asymmetric network uses a new FiGLU nonlinearity. The paper analyzes the asymmetric properties of both proposed methods theoretically and demonstrates empirically that asymmetry improves the behavior of neural networks in several setups. Specifically, asymmetric neural networks have better linear mode connectivity after training and more stable monotonic linear interpolation between initialization and trained model, are more effective for Bayesian deep learning, and are easier to model with metanetworks.", "questions": "I would kindly ask the authors to address the concerns from the Weaknesses section and focus on the following questions:\n1. Could you please elaborate on how the asymmetric network perspective differs from the constrained optimization one and in which cases it shows new insights, in your opinion? \n2. Could you please clarify whether the W-asymmetric and $\\sigma$-asymmetric networks are fully asymmetric in the experiments or not? Which of the known symmetries (neuron permutations, ReLU scaling, pre-normalization parameters scaling, etc.) does each method remove in practice?\n3. Could you please comment on the poor results of $\\sigma$-asymmetric networks in linear mode connectivity and linear interpolation experiments? Is there any specific reason why the Bayesian neural networks and metanetworks experiments are not conducted for $\\sigma$-asymmetric networks?\n4. Could you please comment on the technical differences between standard and W-asymmetric networks (optimal hyperparameters and parameter count) in Bayesian neural networks and metanetworks experiments and whether they can affect the conclusions? \n\nMinor questions:\n1. Do I understand correctly that the standard deviation $\\kappa$ used in W-asymmetric and $\\sigma$-asymmetric networks is a hyperparameter tuned separately from the trainable weights? (lines 134 and 163).\n2. Do the non-trainable weights of all W-asymmetric networks have the same values in the metanetwork experiment? Does a metanetwork take only the trainable parameters of W-asymmetric networks as input?", "limitations": "The authors adequately discuss the limitations of the paper in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper proposes a new idea of analyzing the effect of parameter symmetries on neural networks through comparison with asymmetric networks in practical settings. I think this perspective is interesting and has potential for future research.\n2. The authors develop easy and effective asymmetric parametrizations and analyze them theoretically. \n3. The applicability of the proposed asymmetric perspective is demonstrated on a wide range of related problems.\n4. The paper is clearly written and easy to follow.", "weaknesses": "My main concerns are related to the limited discussion on the benefits of the proposed perspective in comparison to previous works, asymmetric properties of the proposed methods in practical setups, and poor results of $\\sigma$-asymmetric networks:\n1. As mentioned in the Related Work section, previous works examined the influence of parameter symmetries on neural network training by changing the optimization schemes instead of the network parametrization. Even though I find the idea of studying the behavior of asymmetric neural networks interesting, it is not clear to me how this perspective is beneficial compared to the constrained optimization one. Adding a more thorough discussion on that in the Related Work section would improve the paper. \n2. It is not clear from the paper if W-asymmetric and $\\sigma$-asymmetric networks are fully asymmetric or just have fewer symmetries in practical setups. A more accurate discussion on which symmetries are removed by the proposed methods in the experimental setups should be added to the paper. For example, the paper does not cover the normalization layer symmetries, even though layer and batch normalization are used in the experiments. It seems W-asymmetric networks remove normalization symmetries, while $\\sigma$-asymmetric ones do not. It may be one of the reasons why $\\sigma$-asymmetric networks show weaker results in linear mode connectivity and linear interpolation sections. The ReLU scale symmetry is also not adequately discussed.\n3. The analysis of the $\\sigma$-asymmetric networks is limited. There is no proof or discussion of the universal approximation for this method. The paper does not explain poor results on linear mode connectivity and linear interpolation and does not include the results on Bayesian neural networks and metanetworks. A more detailed analysis of the $\\sigma$-asymmetric networks, or at least a discussion of its suboptimal results, would benefit the paper.\n\nAdditionally, I have some minor concerns regarding the experiments:\n1. In the Bayesian neural network experiment, the W-asymmetric networks have fewer parameters than standard ones, which may influence the optimal training hyperparameters. Hence, it may be the case that the difference in the results is due to, e.g., different effective learning rates and not the asymmetric network structure.\n2. In the metanetwork experiments, standard and W-asymmetric networks have different numbers of parameters. Hence, it may be the case that the difference in the results is due to the different dimensionality of the input space and different optimal training hyperparameters for metanetworks and not the asymmetric network structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the novelty of our ideas, the effectiveness of our asymmetric parameterizations, the wide range of problems that we consider, and our writing. We think that we have improved our work through your suggested clarifications and ablations.\n\n> “Even though I find the idea of studying the behavior of asymmetric neural networks interesting, it is not clear to me how this perspective is beneficial compared to the constrained optimization one…”\n\nThis is indeed important, see general comment for our elaboration on this.\n\n> “It is not clear from the paper if W-asymmetric and $\\sigma$-asymmetric networks are fully asymmetric or just have fewer symmetries in practical setups. A more accurate discussion on which symmetries are removed by the proposed methods in the experimental setups should be added to the paper.”\n\nGood point. We will more clearly explain this. Our theoretical results are summarized in this table:\n\n| --- | W-Asym | $\\sigma$-Asym |\n| ---| --- | --- |\n| Permutation | removed | removed |\n| Scale | unclear | removed |\n\nScale symmetries are removed by FiGLU, as shown in Proposition 2. Although it is not formally proven that W-Asym networks remove scale symmetries, we believe that they do (intuitively, the fixed weights also fix a scale).\n\n> “For example, the paper does not cover the normalization layer symmetries, even though layer and batch normalization are used in the experiments. It seems W-asymmetric networks remove normalization symmetries, while $\\sigma$-asymmetric ones do not. It may be one of the reasons why $\\sigma$-asymmetric networks show weaker results in linear mode connectivity and linear interpolation sections.  The ReLU scale symmetry is also not adequately discussed.“\n\nThis is a good point. We agree that the W-Asymmetric networks appear to remove normalization symmetries, whereas the $\\sigma$-Asymmetric ones do not in general.\n\nThe ReLU scale symmetry can be handled by changing the nonlinearity to e.g. GELU (Godfrey et al. [19] prove that this does not have scale symmetries). Also, our ResNet experiments still use ReLU nonlinearity, yet they achieve good symmetry breaking.\n\n> “The analysis of the $\\sigma$-asymmetric networks is limited. There is no proof or discussion of the universal approximation for this method…”\n\nWe did try to prove universal approximation for $\\sigma$-Asymmetric networks before submission, but we could not do it for a few reasons; we will add discussion of this to the paper. Classical universal approximation results with MLPs do not apply, because those generally assume elementwise nonlinearities. We do think there is a potential proof via related constructions to the proof of the W-Asym universality and interesting symmetries of SiLU-type nonlinearities [Martinelli et al. 2023], but we leave this to future work.\n\n[Martinelli et al. 2023] Expand-and-Cluster: Parameter Recovery of Neural Networks. https://arxiv.org/abs/2304.12794\n\n> “The analysis of the $\\sigma$-asymmetric networks is limited… The paper does not explain poor results on linear mode connectivity and linear interpolation and does not include the results on Bayesian neural networks and metanetworks. A more detailed analysis of the $\\sigma$-asymmetric networks, or at least a discussion of its suboptimal results, would benefit the paper.”\n\nIndeed. See our general comment for more on this. We will add more discussion on this interesting point to our paper.\n\n> “In the Bayesian neural network experiment, the W-asymmetric networks have fewer parameters than standard ones, which may influence the optimal training hyperparameters. Hence, it may be the case that the difference in the results is due to, e.g., different effective learning rates and not the asymmetric network structure.”\n\nGood point. We have rerun the experiments, with 90% shallower standard ResNets to match the number of parameters of the W-Asymmetric networks, and we find essentially the same performance: W-Asymmetric test accuracy ($49.3\\pm .4$) is still substantially better ($46.8 \\pm.9$ for standard and $46.5\\pm1.1$ for the shallower one). See our 1-page results PDF for more.\n\n> “In the metanetwork experiments, standard and W-asymmetric networks have different numbers of parameters. Hence, it may be the case that the difference in the results is due to the different dimensionality of the input space and different optimal training hyperparameters for metanetworks and not the asymmetric network structure.”\n\nAgain, good point. We have trained a whole new dataset of smaller standard networks, of the same number of parameters as the W-Asymmetric networks (~ 60,000). There is little to no change in the results of these networks and our previous dataset of standard networks, so the difference in results seems to be from the asymmetric network structure. We will include these results in the revision; see our 1-page results PDF for full results.\n\n\n> “Do I understand correctly that the standard deviation $\\kappa$ … is a hyperparameter tuned separately…”\n\nYes, you are correct. We previously said “standard deviation $\\kappa$ > 0 that we tune”, but\n we will note this more clearly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jyCyvCEamO", "reviewer": "Reviewer_ceUs", "summary": "The paper suggested asymmetric neural networks in terms of an asymmetric nonlinearity and weight. It demonstrated some tasks including linear mode connectivity without permutation alignment, Bayesian neural networks, training metanetworks, and monotonic linear interpolation to show the role of permutation symmetry inherited in the standard neural networks.", "questions": "What do you mean by “loss” in BNN. Is it an ELBO? Why don’t you report the negative log-likelihood (NLL)?", "limitations": "1. Fixing some weights or adding anisotropic activation function implies constraining not only symmetry but also distorted parameter manifold (like equivariant NNs) that also leads to increasing correlation between solutions. The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation.\n1. The BMA and MLI tasks are based on the strong assumption, where its evidence is absent, that no symmetry induces the reduced number of modes.\n1. Although the paper partially gives insights on the loss surface, it does not suggest a new practical method utilizing the insights.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Various evaluations to show the role of permutation symmetry.\n1. Novel approaches to build neural networks that have no permutation symmetry.", "weaknesses": "1. The number of learnable parameters of the standard NN and the asymmetric NN is reported only for the metanetwork task. It should also be reported for the LMC, BMA, and MLI tasks.\n1. Comprehensible reason is required for the proposition “the posterior will have less modes” in the hypothesis of the BMA task in order to understand the purpose of the BMA task. No symmetry does not imply less modes. Thus, the improved performance of BMA may be induced from another reason other than asymmetry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the novelty and evaluations behind our work, and for their comments, which we now address one at at time:\n\n> “The number of learnable parameters of the standard NN and the asymmetric NN is reported only for the metanetwork task. It should also be reported for the LMC, BMA, and MLI tasks.”\n\nGood point, we have now added the number of learnable parameters for the other experiments to our paper. They are in this table (note that sigma-Asym networks always have the same number of learnable parameters as the corresponding standard network).\n\n| Experiment / Architecture  | Standard / sigma-Asym | W-Asym|\n| --- | --- | --- |\n| Sec 5.1 MLP |  935434 | 834570  |\n| Sec 5.1 Resnet 1x  | 272,474 | 230,024 |\n| Sec 5.1 Resnet 8x  | 17,289,866 | 16,273,946 |\n| Sec 5.1 GNN | 176,424 | 171,576 |\n| Sec 5.2 MLP-8 | 3,242,146 | 3,324,466 |\n| Sec 5.2 MLP-16 | 5,796,002 | 5,960,242 |\n| Sec 5.2 Resnet-20 1x | 1,143,858 | 1,356,098 |\n| Sec 5.2 Resnet-20 2x | 5,044,756 | 5,410,386 |\n| Sec 5.2 Resnet-110 1x | 7,371,378 | 8,620,418 |\n| Sec 5.2 Resnet-110 2x | 32,014,996 | 34,512,276 |\n| Sec 5.2 Resnet-20 2x | 5,044,756 | 5,410,386 |\n| Sec 5.4 MLI ResNet | 78,042 | 60,634 |\n\n\nWe also now report the number of learned parameters for metanetworks in Section 5.3 (not the input data) as follows:\n\n| Metanet | Num Params |\n| --- | --- |\n| MLP | 4,994,945 (ResNet) / 3,836,673 (Smaller ResNet) / 3,880,833 (W-Asym ResNet) |\n| DMC | 105,357 |\n| DeepSets | 8,897 |\n| StatNN | 119,297 |\n\n> “Comprehensible reason is required for the proposition “the posterior will have less modes” in the hypothesis of the BMA task in order to understand the purpose of the BMA task. No symmetry does not imply less modes… “\n\n> “The BMA and MLI tasks are based on the strong assumption, where its evidence is absent, that no symmetry induces the reduced number of modes.”\n\nWe gave several citations in that section (5.2) on previous works noting that symmetries induce problematic modes in Bayesian NNs (e.g. [2, 33, 71, 49, 70, 35]).\n\nThe argument is simple: if $\\tau$ is a parameter symmetry, then $L(\\theta) = L(\\tau(\\theta))$ for loss functions $L$ such as NLL and parameters $\\theta$ (because the neural network function is left unchanged by $\\tau$, and the loss only depends on the neural network function). Thus, for any mode $\\theta^* \\in \\mathrm{argmin}_{\\theta} L(\\theta)$, we have that $\\tau(\\theta^*)$ is also a mode, because it has the same loss value. Removing parameter symmetries $\\tau$ also removes these additional modes. We will spell out this argument in the revision.\n\nAlso, the empirical results do not rely on these “assumptions”. But rather, previous work shows that we expect these assumptions to hold, so we empirically investigate these hypotheses via Asymmetric Networks.\n\n> “What do you mean by “loss” in BNN. Is it an ELBO? Why don’t you report the negative log-likelihood (NLL)?”\n\nWe do mean NLL loss. We will make this clearer in the revision.\n\n> “Fixing some weights or adding anisotropic activation function implies constraining not only symmetry but also distorted parameter manifold (like equivariant NNs) that also leads to increasing correlation between solutions. The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation.”\n\nRegarding the comparison to equivariant NNs, coefficients for a steerable basis in an equivariant network is probably more distorting of the parameter manifold. Yes, we do distort the parameter manifold, but we do so in a way that maintains the vector space structure and calculus, so we can optimize Asymmetric Networks with standard gradient-based methods like Adam. In contrast, some previous works restrict parameters to nonlinear manifolds, and require different optimization algorithms such as projected gradient descent or Riemannian optimization methods. See the general comment for more information, as well as our related work section. \n\n>  “The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation.”\nWe are not sure what you mean here, but this is an important positive property of asymmetric networks. Since they are like standard networks with a fixed permutation, we do not need to account for parameter symmetries when doing interpolation, metanetwork processing, or Bayesian NN training for asymmetric networks.\n\n> “Although the paper partially gives insights on the loss surface, it does not suggest a new practical method utilizing the insights.”\n\nThis is not quite the purpose of our paper; we moreso seek to understand various phenomena in deep learning. Some phenomena like Monotonic Linear Interpolation (MLI) have no real applications, and some others like Linear Mode Connectivity (LMC) have related downstream applications in things like model merging and federated model averaging.\n\nMoreover, the BNN results are in some sense a practical method for improving training of BNNs.\n\nThat being said, we do believe there are some other potential practical methods inspired by our work, which could be explored in future work. For instance, model merging is extremely powerful for open-weight large language models (see top methods on [open LLM leaderboard](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard)), and symmetries have not been explored as much there. Our Asymmetry methods cause little overhead (only modifying weights and/or nonlinearities a little), so they could be used for very large models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDAyfHJfc4", "reviewer": "Reviewer_tX9W", "summary": "This paper develops two types of modifications to neural networks to remove permutation symmetries: fixing certain weights, or using a non-elementwise activation function. The resulting \"asymmetric\" neural networks are found to improve on certain metrics that are observed in networks with permutation symmetries, and thought to be caused by such symmetries: lack of linear mode connectivity between randomly initialized/trained networks, reduced Bayesian network performance, and reduced metanetwork performance. The asymmetric networks also are more likely to have monotonic linear interpolation between their initial and trained weights.", "questions": "Are the results in table 1 for networks with different random initializations, or the same identical initialization? If the former, are the masks and fixed weights different or the same between the two networks? I find it highly unlikely that networks with different fixed weights or different initializations would have low barrier after training without post-processing.\n\nThe fixed weights in $W$-asymmetry are distributed very differently to the non-fixed weights. If the fixed linear transform $F$ in $\\sigma$-asymmetry is similarly shifted in distribution, would that make its results closer to $W$-asymmetry?\n\nWhy not fix biases to get asymmetry instead?", "limitations": "The reproducibility issues discussed previously, combined with the experimentally-focused framing, detract from an otherwise very strong and impactful paper.\n\nThe results for $\\sigma$-asymmetry are considerably weaker than $W$-asymmetry (and rely on even smaller learning rates). Another work has found that networks mainly use linear transformations [1]. Since $\\sigma$-asymmetry is effectively a non-elementwise sigmoid activation function, I wonder if the network's activations mainly stay in the approximately linear region of the sigmoid, allowing many permutations to be approximately equivalent. Some other comments or experiments on why $\\sigma$-asymmetry is less effective in practice would be helpful, especially since theoretically speaking, both methods are clearly asymmetric.\n\n[1] Mehmeti-Göpel & Disselhoff. Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think. ICML 2023. https://proceedings.mlr.press/v202/ali-mehmeti-gopel23a.html", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper is effectively an ablation study on permutation symmetries, which provides a much-needed tool and counterfactual perspective for investigations of permutation symmetries. The methods appear both novel and logical. A thorough survey of prior literature is given, which is extremely welcome. The paper is well organized, with well-motivated methodology and easy-to-follow experiments and hypotheses are easy to follow. The proofs are intuitive and comprehensive.", "weaknesses": "The main issue is that the paper relies heavily on experimental rather than theoretical results, but the strongest results are for a single method ($W$-asymmetry) with settings much more different than \"standard\" training than seems at first glance - specifically, lower learning rates, longer warmup, and larger fixed weights that all differ from standard training by order(s) of magnitude. These settings, plus a lack of available code, seriously impact the relevance/reproducibility of the methods/results.\n\nThere should be a comparison with standard models initialized to the same weights as the $W$-asymmetric networks. This would make certain metrics like $L^2$ weight distance comparable (currently the $W$-asymmetric networks are more likely to have extremely large magnitude weights which could increase the $L^2$ distance between independently initialized/trained networks). This would also eliminate some other variables such as reduced performance due to suboptimal weight initialization.\n\nLow learning rates and long warmup are likely to improve similarity and reduce barriers, particularly between identically initialized neural networks. An analysis of how learning rate/warmup affects the experimental results (particularly barriers after training) is needed to determine if the observed results are mainly due to asymmetry.\n\nSome additional details are needed to properly replicate the asymmetry methods, specifically in tables 5-7.\n- how are $n_{\\text{fix}}$ and $\\kappa$ determined for each architecture?\n- what do the names of the blocks refer to? There are not enough for a 1-to-1 mapping onto ResNet20 layers or permutations, so more explanation is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of the novelty of our methods, thoroughness of our discussion of previous work, organization of the paper and methodology, and comprehensiveness of our proofs. Also, we thank the reviewer for putting effort into understanding the empirical setup of our work, and making suggestions to improve the $\\sigma$-Asymmetric networks.\n\n**On Reproducibility:** As mentioned in the paper checklist, we plan to open source the data and code to the public, and we are sharing the code with the reviewers now. We have sent the AC our data and code to reproduce the experiments, which they will share with you.\n\nWe have also run useful ablations that the reviewer suggested, and found that the results do not significantly change! In fact, some of the reviewer’s suggestions actually improved the results (e.g. shorter warmup period improved W-Asymmetry interpolation while making Git-ReBasin interpolation worse).\n\n> “The main issue is that the paper relies heavily on experimental rather than theoretical results, but the strongest results are for a single method (W-asymmetry) with settings much more different than \"standard\" training … lower learning rates, longer warmup, and larger fixed weights that all differ from standard training by order(s) of magnitude.”\n\nWe have rerun our ResNet20 experiment on CIFAR10, and find that using a more standard warmup schedule (1 epoch) yields similar results for linear interpolation. In fact, using a 1 epoch warmup has a lower loss barrier (.691) than the 20 epoch warmup (.931). Performance is also similar on GNNs, see our 1-page results PDF for more.\n\nOur learning rates used for our ResNet training are pretty standard for the Adam optimizer (1e-2 peak). When using 1e-3 learning rate for W-Asymmetric nets, we actually get a lower barrier of $.285 \\pm .065$, but test loss that is significantly worse. Perhaps the reviewer is referring to the higher learning rates that people use with SGD (such as 1e-1 in Git-ReBasin), but these learning rates are normal for Adam (e.g. Git-ReBasin uses 1e-3 for Adam on their MLP, and the folklore [3e-4 of Karpathy](https://karpathy.github.io/2019/04/25/recipe/) is common).\n\nIndeed the magnitude of the fixed weights we use are very large but that is by design. We note that we require high magnitude fixed weights in order to effectively break both approximate and exact parameter symmetries. Nevertheless, we agree that the large fixed weights are not ideal for W-Asymmetry. Our work is fairly novel, in that others have not really worked on making Asymmetric networks like this. We envision that future works will develop new types of Asymmetric networks that get around some of these issues.\n\n> “There should be a comparison with standard models initialized to the same weights as the W-asymmetric networks… \n\nStandard networks initialized to the same weights have similar performance ($.79 \\pm .2$ loss barrier with 1 epoch warmup, versus $.67 \\pm .3$ for W-Asym, for ResNet20 on CIFAR-10). We also find that these networks are comparably asymmetric though have slightly worse linear interpolation. We find this not too surprising since the symmetry-breaking weights have such high magnitudes and thus are similar before and after training. \n\n> “how are $n_{\\mathrm{fix}}$  and $\\kappa$ determined for each architecture? … what do the names of the blocks refer to? There are not enough for a 1-to-1 mapping onto ResNet20 layers or permutations, so more explanation is needed.”\n\nOur theory suggests that $n_{\\mathrm{fix}}$ should be at least $\\log_2(dim)$ so that the conditions in Theorem 3 hold. The exact values of $n_{\\mathrm{fix}}$ and $\\kappa$ are chosen manually (these are hyperparameters).\n\nThe ResNet20 code we use is in `lmc/models/models_resnet.py` in our attached code. It consists of an initial convolution followed by 3 blocks. Each block contains 6 convolutions (and 36 convolutions for ResNet110). The parameters we give in Tables 6 and 7 apply to each convolution of the described block.\n\n> “Are the results in table 1 for networks with different random initializations, or the same identical initialization? … are the masks and fixed weights different or the same between the two networks? ...”\n\nThe learnable parameters are from different random initializations, but the masks and fixed weights are the same between the two networks. See also the general comment for more on this.\n\n> “... If the fixed linear transform F in $\\sigma$-asymmetry is similarly shifted in distribution, would that make its results closer to W-asymmetry?”\n\nGood question. We have tried this as well (as noted in the paper, we tune the standard deviation that $\\mathbf{F}$ is drawn from for $\\sigma$-Asymmetry as well), but this was not sufficient.\n\n> “Why not fix biases to get asymmetry instead?”\nWe tried this and it didn’t work. See the results in our results PDF. Intuitively, there is only one dimension for the biases to vary, so for instance if you sort the biases, then two neighboring biases may be very close to each other. This causes two neurons to be approximately automorphic, so there are still approximate permutation symmetries.\n\n> “for $\\sigma$-asymmetry … I wonder if the network's activations mainly stay in the approximately linear region of the sigmoid … Some other comments or experiments on why $\\sigma$-asymmetry is less effective in practice would be helpful …”\n\nWe thank the reviewer for this suggestion. This is a plausible cause, but we investigated it, and it does not seem like this is the main issue. For instance, to make the nonlinearity more nonlinear, we switched to cosine nonlinearity instead of sigmoid, but this did not work well either. Also, we have already tried changing the magnitude of the fixed weights (via tuning $\\kappa$), to move the activations into different parts of the sigmoid. See general comment for more.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their comments and suggestions. We have added new experiments and readied changes for the manuscript, which we think will improve the paper significantly. See our 1-page results PDF for more. Also, we have sent our code for reproducing experiments to the area chair, which should be shared with you too.

Here are our responses to some selected comments by reviewers:

**Why do we not change the optimization algorithm to break symmetries?** (Reviewers hbBt, ajZW)  This is already somewhat touched-upon in our related work section, but we will add more discussion about this:

> “Our models are optimized using standard unconstrained gradient-descent based methods like Adam. Hence, our networks do not require any non-standard optimization algorithms such as manifold optimization or projected gradient descent [5, 54], nor do they require post-training-processing to remove symmetries or special care during analysis of parameters (such as geodesic interpolation in a Riemannian weight space [53]).“

This is a very important point that we will elaborate on further in the revision. We purposefully parameterize Asymmetric networks so that we can use standard optimization algorithms like Adam. This is because the main goal of Asymmetric networks is to provide a counterfactual system that is as similar to standard networks as possible, but with removed parameter symmetries. These other methods that require e.g. optimizing over manifolds like spheres, or iterative retraining for pruning, have significantly different optimization and loss landscape behaviors (e.g. linear interpolation is not even well-defined on general nonlinear parameter manifolds), so they are not suitable for gathering insights into standard networks.

**Masks and constants $\mathbf{F}$ are fixed between runs** (Reviewers ajZW, tX9W). When interpolating between two W-Asymmetric or two $\sigma$-Asymmetric nets, the two networks have the same exact fixed constants $\mathbf{F}$ and masks $M$.

One way to think about this is: when defining a standard network architecture (i.e. a mapping from parameters $\theta$ to functions $f_\theta$), we must specify things like hidden dimension, number of layers, and architecture class (MLP, CNN, etc). When specifying an Asymmetric network of the same architecture class (e.g. a W-Asymmetric MLP), we additionally need to choose masks $M$ and fixed constants $\mathbf{F}$. Thus, since we only ever do linear interpolation between standard networks of the same architecture, we also only ever do linear interpolation between W-Asymmetric networks of the same architecture (so the two networks will have the same masks and constants).

We will include this clarification and way of thinking about the architecture in the rebuttal.

**$\sigma$-Asymmetric network performance** (Reviewers ajZW, tX9W). The $\sigma$-Asymmetric networks do not appear to break symmetries as well as the W-Asymmetric networks, even though both have some theoretical results for symmetry removal.

We have run preliminary empirical tests on many variations of $\sigma$-Asym networks, such as: $sigma(\mathbf{F} sigma(x))$, orthogonal $\mathbf{F}$, sparse $\mathbf{F}$, putting these nonlinearities before the first and after the last layers, adding instead of multiplying the gate, using cosine instead of sigmoid, using square instead of sigmoid, and adding layernorm in the nonlinearity. None of these worked well.

Thus, we feel that there are interesting fundamental questions arising from this relative failure, which could lead to very interesting future work. These are approaches that act on activations, rather than weights, of a neural network. W-Asym Networks act on weights, and do much better in terms of breaking symmetries. We don’t think anything like this has been noted before in the literature (differences between symmetry breaking in weights versus activations).

We will add discussion on these important points in the revised form of our paper.

Meta-review: This paper studies (1) how to construct neural network architectures without "trivial" parameter space symmetries (e.g. permutation or scale invariance) and (2) the downstream, empirical affect of having fewer symmetries on applications ranging from posterior inference to Bayesian NNs to studying loss landscapes.   All reviewers were in favor of acceptance to some degree ( 2 x accept, 1 x weak accept, 1 x borderline accept), applauding the paper's clarity, importance of topic, connection to downstream applications.  The reviewers' most critical issues were about connections to / comparison with optimization-based approached and how the training conditions for the reduced-symmetry networks are substantially different than typical training (jeopardizing the generalizability of the results).  Yet both I and reviewers believe the authors have adequately addressed these concerns in their rebuttal, as the authors' general rebuttal has promised to add more discussion of constrained optimization to the camera-ready version.
Decision: Accept (poster)