Title: The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof
Venue: NeurIPS 2024
Accepted: yes
Keywords: Parameter symmetry

loss landscapes

identifiability

mode connectivity

Bayesian neural networks
Abstract: Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries --- transformations of neural network parameters that do not change the underlying neural network function. These include linear mode connectivity, model merging, Bayesian neural network inference, metanetworks, and several other characteristics of optimization or loss-landscapes. However, theoretical analysis of the relationship between parameter space symmetries and these phenonmena is difficult. In this work, we empirically investigate the impact of neural parameter symmetries by introducing new neural network architectures that have reduced parameter space symmetries. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries.  With these new methods, we conduct a comprehensive experimental study consisting of multiple tasks aimed at assessing the effect of removing parameter symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries; for instance, we observe linear mode connectivity between our networks without alignment of weight spaces, and we find that our networks allow for faster and more effective Bayesian neural network training.

The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof

Abstract

Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries - transformations of neural network parameters that do not change the underlying neural network function. These include linear mode connectivity, model merging, Bayesian neural network inference, metanetworks, and several other characteristics of optimization or loss-landscapes. However, theoretical analysis of the relationship between parameter space symmetries and these phenomena is difficult. In this work, we empirically investigate the impact of neural parameter symmetries by introducing new neural network architectures that have reduced parameter space symmetries. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries. With these new methods, we conduct a comprehensive experimental study consisting of multiple tasks aimed at assessing the effect of removing parameter symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries; for instance, we observe linear mode connectivity between our networks without alignment of weight spaces, and we find that our networks allow for faster and more effective Bayesian neural network training. Our code is available at https://github.com/cptq/asymmetric-networks .

1 Introduction

Neural networks have found profound empirical success, but have many associated behaviors and phenomena that are difficult to understand. One important property of neural networks is that they generally have many parameter space symmetries -for any set of parameters, there are typically many other choices of parameters that correspond to the same exact neural network function [24]. For instance, permutations of hidden neurons in a multi-layer perceptron (MLP) induce permutations of weights that leave the overall input-output relationship unchanged. These parameter symmetries are a type of (not-necessarily detrimental) redundancy in the parameterization of neural networks, that adds much non-Euclidean structure to parameter space.

Parameter space symmetries appear to influence several phenomena observed in neural networks. For example, when linearly interpolating between the parameters of two independently trained networks with the same architecture, the intermediate networks typically perform poorly [59, 13]. However, if we first align the two networks via a permutation of parameters that does not affect the network function, then the intermediate networks can perform just as well as the unmerged networks [59, 1]. In some sense, this suggests that neural network loss landscapes are more convex

TU Munich, MIT

ΓıΓ$ΓV 'ΓPΓPΓHΓWΓUΓLΓF

Γ6ΓWΓDΓQΓGΓDΓUΓG

ΓıΓ$ΓVΓ'ΓPΓPΓHΓWΓUΓLΓF

Γ7 ΓUΓDΓLΓQΓHΓG

Γ)ΓLΓ[ΓHΓG

Γ)ΓLΓ*Γ/Γ8

or well-behaved after removing permutation symmetries. Other areas that parameter symmetries play a role in include interpretability of neurons [20], optimization [48, 84, 80], model merging [60], learned equivariance [7], Bayesian deep learning [34], loss landscape geometry [54], processing neural network weights as input data using metanetworks [39], and generalization measures [49, 11].

To rigorously study the effect of parameter symmetries, we study the effect of removing them. In particular, we introduce two ways of modifying neural network architectures to remove parameter space symmetries (see Figure 1):

W -Asymmetric networks fix certain elements of each linear map to break symmetries in the computation graph.

σ -Asymmetric networks use a new nonlinearity (FiGLU) that does not act elementwise, and hence does not induce symmetries such as permutations.

These two approaches are inspired by previous work, which shows that both symmetries of computation graphs [39] and equivariances of nonlinearities [20] induce parameter symmetries in standard neural networks. We theoretically prove that both of our approaches remove parameter symmetries under certain conditions. Our Asymmetric networks are similar structurally to standard networks and can be trained with standard backpropagation and first-order optimization algorithms like Adam. Thus, they are a reasonable 'counterfactual' system for studying neural networks that are similar to standard neural networks, but that do not have as many parameter symmetries.

With our Asymmetric networks, we run a suite of experiments to study the effects of removing parameter symmetries on several base architectures, including MLPs, ResNets, and graph neural networks. We investigate linear mode connectivity, Bayesian deep learning, metanetworks, and monotonic linear interpolation. Through the lenses of linear mode connectivity and monotonic linear interpolation, we see that the loss landscapes of our Asymmetric networks are remarkably more well-behaved and closer to convex than the loss landscapes of standard neural networks. When using our Asymmetric networks as the base model in a Bayesian neural network, we find faster training and better performance than using standard neural networks that have many parameter symmetries. When using metanetworks to predict properties such as test accuracy of an input neural network, we see that all tested metanetworks more accurately predict the accuracy of Asymmetric networks than standard networks. Overall, our Asymmetric networks provide valuable insights for empirical study and hold promise for advancing our understanding of the impact of neural parameter symmetries.

2 Background and Definitions

Let Θ be the space of parameters of a fixed neural network architecture. For any choice of parameters θ ∈ Θ , we have a neural network function f θ : X → Y from an input space X to an output space Y . We call a function ϕ : Θ → Θ a parameter space symmetry if f θ ( x ) = f ϕ ( θ ) ( x ) for all inputs x and parameters θ ∈ Θ (i.e. if f θ and f ϕ ( θ ) are always the same function).

For instance, consider a two-layer MLP with no biases, parameterized by matrices θ = ( W 2 , W 1 ) with an elementwise nonlinearity σ . Then f θ ( x ) = W 2 σ ( W 1 x ) . Let P be a permutation matrix,

and let ϕ ( θ ) = ( W 2 P ⊤ , P W 1 ) . Then for any input x ,

so ϕ is a parameter space symmetry. A key step is the second equality, which holds because Pσ ( x ) = σ ( Px ) : any elementwise nonlinearity σ is permutation equivariant. Any other equivariance of σ also induces a parameter symmetry; for instance, if σ ( x ) = max(0 , x ) is the ReLU function, then ασ ( x ) = σ ( αx ) for any α > 0 , so there is a positive-scaling-based parameter symmetry [49, 11, 20].

3 Related Work

Characterizing parameter space symmetries. While many works spanning several decades have noted specific parameter space symmetries in neural networks [24, 61], some works take a more systematic approach to deriving parameter space symmetries. Godfrey et al. [20] characterize all global linear symmetries induced by the nonlinearity for two-layer multi-layer perceptrons with pointwise nonlinearities. Zhao et al. [78] study several types of symmetries, and derive nonlinear, data-dependent parameter space symmetries. Lim et al. [39] show that graph automorphisms of the computation graph of a neural network induce permutation parameter symmetries, which captures hidden neuron permutations in MLPs and hidden channel permutations in CNNs.

Constraints and post-processing to break parameter space symmetries. Several works develop methods for constraining or post-processing the weights of a single neural network to remove ambiguities from parameter space symmetries. This includes methods to remove scaling symmetries induced by normalization layers or positively-homogeneous nonlinearities such as ReLU [6, 55, 54, 36], methods to remove permutation symmetries [55, 54, 71, 36], and methods to remove sign symmetries induced by odd activation functions [71].

Unlike these previous works, we develop neural network architectures that have reduced parameter space symmetries. Our models are optimized using standard unconstrained gradient-descent based methods like Adam. Hence, our networks do not require any non-standard optimization algorithms such as manifold optimization or projected gradient descent [6, 55], nor do they require post-trainingprocessing to remove symmetries or special care during analysis of parameters (such as geodesic interpolation in a Riemannian weight space [54]). These methods based on constraining weights or post-processing have significantly different optimization and loss landscape properties (for instance, linear interpolation is not even well-defined on general nonlinear parameter manifolds), so they are less suitable than our Asymmetric networks for studying phenomena that may generalize to standard neural networks.

Aligning multiple networks for relative invariance to parameter symmetries. One way to reduce the impact of parameter symmetries in certain settings, especially for model merging, is to align the parameters of one network to another. Several methods have been proposed for choosing permutations that align the parameters of two neural networks of the same architecture, via efficient heuristics or learned methods [4, 69, 63, 13, 1, 53, 47, 67]. Other approaches relax the exact permutation-parameter-symmetry constraint or do additional postprocessing to achieve effective merging of models in parameter space [59, 28, 29, 60, 56]. As our Asymmetric networks have removed parameter symmetries, they can often be successfully merged and linearly interpolated between without any alignment.

4 Asymmetric Networks

We develop two methods of parameterizing neural network architectures without parameter symmetries, both of which are justified by theoretical results. We can prove that our σ -Asym networks have permutation and scale symmetries removed, and that our W -Asym networks have permutation symmetries removed. Although we have not formally proven that W -Asym networks have scale symmetries removed, we believe that they do (intuitively, the fixed weights fix a scale).

We first focus on the case of fully-connected MLPs with no biases, which take the form f θ ( x ) = W L σ ( W L -1 · · · σ ( W 1 x )) for weights θ = ( W L , . . . , W 1 ) and nonlinearity σ . Then in Section 4.3, we discuss how we use these approaches to remove parameter symmetries in other architectures (e.g. CNNs, GNNs) as well.

Standard

Standard

-Asymmetric

(Fix entries)

-Asymmetric

(Fix filters)

-Asymmetric

Linear

Convolution

4.1 Computation Graph Approach ( W -Asymmetric Networks)

Our first approach to developing neural networks with greatly reduced parameter space symmetries relies on their computation graph. In particular, we can write a feedforward neural network architecture as a DAG G = ( V, E ) with neurons as nodes V and connections between them as edges E . For a choice of parameters θ ∈ R | E | , we get a function f θ from input neuron space to output neuron space [19, 49, 39]

Lim et al. [39] showed that neural DAG automorphisms ϕ , which are graph automorphisms of the DAG G that preserve types of nodes and weight-sharing constraints, induce permutation parameter symmetries ϕ that leave the function unchanged: f θ = f ϕ ( θ ) . Thus, any feedforward neural network architecture that has no permutation parameter symmetries must necessarily have a computation graph with no nontrivial neural DAG automorphisms.

To modify MLPs so they have no nontrivial neural DAG automorphisms, we mask edges in the computation graph, by setting certain edge weights to constant values that are not updated during training. For an MLP, we can do this by enforcing that every linear layer T : R d 1 → R d 2 takes the form of a matrix W ∈ R d 2 × d 1 , where each row has a unique pattern of untrained weights. To achieve this, define a mask M ∈ { 0 , 1 } d 2 × d 1 such that W ij is a trainable parameter if and only if M ij = 1 , and the rows of M are pairwise distinct binary vectors in { 0 , 1 } d 1 . We call any neural network with linear maps masked as such a W -Asymmetric neural network. In Appendix B.1, we show that masking these entries so that they are not trained is sufficient to remove all nontrivial neural DAG automorphisms.

Theorem 1. If each mask matrix M has unique nonzero rows, then W -Asymmetric MLPs with fixed entries set to zero have no nontrivial neural DAG automorphisms.

In practice, we generate a binary mask M by randomly selecting a subset of n fix fixed elements for each row. For the fixed entries, we sample them from a normal distribution N (0 , κI ) with standard deviation κ > 0 that is a hyperparameter that we tune. Our asymmetric linear layer can be written as

where W ∈ R d 2 × d 1 is a matrix of trainable parameters, and F ∈ R d 2 × d 1 is a matrix of fixed elements, sampled from N (0 , κI ) . The only trainable parameters are the unmasked entries of M ⊙ W , of which there are d 2 · ( d 1 -n fix ) . We empirically find that having κ be significantly larger than the standard deviation of typical initializations for weight matrices (e.g. κ = 1 while the trained coefficients have standard deviation about 1 / √ 1000 ) is important for breaking parameter symmetries.

4.2 Nonlinearity Approach ( σ -Asymmetric Networks)

Another approach for removing parameter symmetries is to change the nonlinearity. As studied by Godfrey et al. [20], equivariances of the nonlinearity induce parameter symmetries in MLPs with elementwise nonlinearities. Recall that an elementwise nonlinearity acts by using the same function on each coordinate of the input; σ : R d → R d is elementwise if it takes the form σ ( x ) = ( σ 1 ( x 1 ) , . . . , σ 1 ( x d )) for some real function σ 1 : R → R . Any elementwise nonlinearity is permutation equivariant, and hence induces a permutation parameter symmetry.

Thus, in contrast to most neural network architectures, for Asymmetric networks we must use a nonlinearity that does not act elementwise. Likewise, the nonlinearity cannot have any linear symmetry itself, since if σ ◦ A = B ◦ σ for A,B ∈ GL ( d ) , then for a two-layer network:

So ( W 2 , W 1 ) and ( W 2 B -1 , A W 1 ) give the same neural network function. Thus, in order to define a model class without parameter symmetries, it is necessary for σ to have no linear equivariances , i.e. we desire that if σ ◦ A = B ◦ σ for A,B ∈ GL ( d ) , then A = B = I . For two-layer MLPs with square invertible weights, this is in fact sufficient to remove all parameter symmetries: we prove this in Appendix B.2.

Proposition 1. Let the parameter space Θ be all pairs of square invertible matrices θ = ( W 2 , W 1 ) for W 2 , W 1 ∈ GL ( d ) , and let f θ ( x ) = W 2 σ ( W 1 x ) . If σ has no linear equivariances, then f θ 1 = f θ 2 if and only if θ 1 = θ 2 . In other words, there are no nontrivial parameter space symmetries.

4.2.1 FiGLU: the Fixed Gated Linear Unit Nonlinearity

Motivated by Proposition 1, we define a non-elementwise nonlinearity that does not have the equivariances of standard nonlinearities. Letting η be the sigmoid function η ( x ) = 1 1+ e -x , we define our nonlinearity as

for a randomly sampled, untrained matrix F . Similarly to W -Asym nets, we sample F as an i.i.d Gaussian matrix with variance that we tune. This nonlinearity is similar to Swish / SiLU [57, 26] with an additional matrix F to mix feature dimensions (to break permutation equivariance), and it is also similar to a gated linear unit (GLU) with no trainable parameters [9]. Thus, we call our nonlinearity FiGLU: the Fixed Gated Linear Unit.

In Appendix B.2.1, we prove that FiGLU does not have permutation equivariances or diagonal equivariances, which are the only equivariances for most elementwise nonlinearities [20].

Proposition 2. With probability 1 over the sampling of F , FiGLU has no permutation equivariances or diagonal equivariances.

We call any network with our symmetry-breaking FiGLU nonlinearity a σ -Asymmetric Network.

4.3 Extension to Other Architectures

The graph-based approach ( W -Asymmetric Networks) works naturally for neural network architectures with 'channel' dimensions, such as convolutional neural networks (CNNs), graph neural networks (GNNs) [22], Transformers [66], and equivariant neural networks based on equivariant linear maps [16]. In these types of networks, permutations of entire channels induce permutation parameter symmetries [39]. For such networks, we thus mask entire connections between channels, e.g. entire filters in CNNs. For CNNs, we also experiment with randomly masking some number of entries in each filter (instead of masking entire filters), and find that this also works well in removing parameter symmetries. For neural networks with linear layers that include bias terms, we do not modify the biases in any way, as they do not introduce new computation graph automorphisms [39].

The nonlinearity-based approach ( σ -Asymmetric Networks) can be straightforwardly applied to many general architectures as well. Though, the fixed matrix F may have to be changed to a structured linear map; for instance, in CNNs we take F to be a 1D convolution.

0.0

0.2

0.4

0.6

0.8

1.0

Interpolation

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Test Loss

MLP MNIST

Standard

Rebasin

Asym

Asym

W

0.0

0.2

0.4

0.6

0.8

1.0

Interpolation

0.5

1.0

1.5

2.0

2.5

3.0

Test Loss

ResNet 8x width CIFAR-10

Standard

Rebasin

Asym

Asym

W

0.0

0.2

0.4

0.6

0.8

1.0

Interpolation

1.00

1.25

1.50

1.75

2.00

2.25

2.50

2.75

Test Loss

GNN ogbn-arXiv

Standard

Rebasin

Asym

Asym

W

4.4 Universal Approximation

Our two approaches remove parameter symmetries from standard neural networks, but still intuitively retain much of the structure of standard networks. One important property of widely-used neural network architectures is universal approximation - for any target function of a certain type, there exists a neural network of the given architecture that approximates the target to an arbitrary accuracy [8, 25, 42, 75]. In Appendix B.3, we show that W -Asymmetric MLPs retain this property:

Theorem 2 (Informal) . For n fix ∈ o ( n 1 4 ) , where n is the hidden dimension, W -Asymmetric MLPs are Universal Approximators with probability 1 over the choice of hardwired entries.

We have not been able to prove a similar result for σ -Asymmetric networks. Classical universal approximation results for standard neural networks do not apply to σ -Asym nets, as they tend to assume elementwise nonlinearities.

5 Experiments

5.1 Linear Mode Connectivity without Permutation Alignment

Background. Under certain conditions, neural networks have been found to exhibit linear mode connectivity, which is when all networks on the line segment in parameter space between two well-performing trained networks are also well-performing. Starting with Frankle et al. [18], who coined the term and provided the first in-depth analysis, many works have studied this phenomenon [44, 13, 76, 14]. When the two networks are randomly initialized and trained independently, linear mode connectivity generally does not hold [13, 1]. However, if one of the two networks is permuted with a parameter symmetry that does not change its function, but that aligns its parameters with the other network, then linear mode connectivity empirically and theoretically holds for many more model / task combinations [13, 1, 79, 14]. In fact, Entezari et al. [13] conjectures that if all permutation symmetries are accounted for, then linear mode connectivity generally holds. Since our Asymmetric networks remove parameter space symmetries, we may expect linear mode connectivity to hold, without any post-processing or alignment step.

Hypothesis. Asymmetric networks are more linearly mode connected than standard networks, and do not require post-processing or alignment of pairs of networks before merging.

Experimental Setup. We consider several networks and tasks: MLPs on MNIST, ResNets [23] on CIFAR-10, and Graph Neural Networks [73] on ogbn-arXiv [27]. For each architecture and task, we compute the midpoint test loss barrier: L ( 1 2 θ 1 + 1 2 θ 2 ) -1 2 ( L ( θ 1 ) + L ( θ 2 )) . This measures how much worse the interpolated network with parameters 1 2 θ 1 + 1 2 θ 2 is than the original networks with parameters θ 1 and θ 2 . We measure this barrier for standard networks, pairs of networks aligned with Git-Rebasin [1], and networks with our two approaches ( σ -Asym and W -Asym) applied. To be clear, whenever we interpolate between the weights of two Asymmetric networks, they have the same exact fixed weights F (for both W -Asym and σ -Asym) and the same exact mask M (for W -Asym).

Results. Figure 3 plots interpolation curves and Table 1 displays midpoint test loss barriers of various methods. Our σ -Asymmetric approach lowers the test loss barrier compared to standard networks, but falls short of the alignment approach of Git-Rebasin. On the other hand, our W -Asymmetric

approach achieves strong (and sometimes perfect) interpolation, and interpolates better than standard networks aligned via Git-ReBasin. This may be caused by failure of the Git-ReBasin approaches to find the optimal permutations, importance of other parameter symmetries besides layer-wise permutations, or other properties of W -Asymmetric networks.

5.2 Bayesian Neural Networks

Background. Bayesian deep learning is a promising approach to improve several deficits of mainstream deep learning methods, such as uncertainty quantification and integration of priors [30, 51]. However, parameter symmetries are problematic in Bayesian neural networks, as they are a major source of statistical nonidentifiability [30]. Parameter symmetries introduce modes in the posterior p ( θ |D ) that make the posterior harder to approximate [2, 34, 72], sample from [50, 71], and otherwise analyze [36]. For instance, one common technique for training Bayesian neural networks is variational inference via fitting a Gaussian distribution to the true posterior p ( θ |D ) . This approach suffers because the Gaussian distribution has only one mode, whereas the true posterior has at least one mode for every parameter symmetry. As such, some approaches treat kernel matrices are random variables, which has less symmetries than treating features or weights as random variables, and allows better approximation by unimodal posteriors [74, 43]. Instead, we consider traditional, commonly-used Bayesian deep learning techniques applied on the features or weights of our Asymmetric networks.

Experimental setup. We train Standard Bayesian and Asymmetric Bayesian Networks for image classification using variational inference. We use the method of [64] for variational inference, which fits a Gaussian approximate posterior with a diagonal plus low-rank covariance. We train 10 instances of each model and then report train loss, test loss, test accuracy, and Expected Calibration Error (ECE) [45], which is a measure of calibration.

Results. See training curves in Figure 4, and quantiative results in Table 2. Using W -Asymmetric networks as a base for Bayesian deep learning improves training speed and convergence. Most

strikingly, Bayesian MLPs of depth 16 cannot train at all, while W -Asymmetric Bayesian MLPs train well. In general, the W -Asymmetric approach improves training and test accuracy across the several models (MLPs, ResNets of varying sizes, and ResNets with either batch norm or layer norm).

0

10

20

30

40

50

Epoch

10

1

10 0

Train Loss

MLP MNIST

Standard

W

-Asym

0

10

20

30

40

50

Epoch

10 0

2×10 0

3×10 0

Train Loss

ResNet110 CIFAR-10

Standard

W

-Asym

0

10

20

30

40

50

Epoch

2×10 0

3×10 0

4×10 0

Train Loss

ResNet20 BN CIFAR-100

Standard

W

-Asym

5.3 Metanetworks

Background. Metanetworks [39] - also referred to as deep weight-space networks [46, 58], meta-models [35], or neural functionals [81, 82, 83] - are neural networks that take as inputs the parameters of other neural networks. Recent work has found that making metanetworks invariant or equivariant to parameter-space symmetries of the input neural networks can substantially improve metanetwork performance [46, 81, 39, 32].

Hypothesis. Asymmetric networks are easier to train metanetworks on because they do not have to explicitly account for symmetries.

Experimental setup. We experiment with metanetworks on the task of predicting the CIFAR-10 test accuracy of an input image classifier, which many metanetworks have been tested on [65, 12, 81, 39]. We use metanetworks based on simple MLPs, 1D-CNN metanetworks [12], and metanetworks that are exactly invariant to permutation parameter symmetries: DeepSets [77] and StatNN [65]. We train two separate datasets of 10,000 image classifiers: one dataset of small ResNet models, and one dataset of W -Asymmetric ResNet models. More information on the data, metanetworks, and training details are in Appendix F.3.

Results. In Table 3, we see that metanetworks are signficantly better at predicting the performance of our W -Asymmetric ResNets than standard ResNets. Interestingly, simple MLP metanetworks, which view the input parameters as a flattened vector, can predict the test accuracy of Asymmetric Networks quite well, but fail on standard networks. Also, the permutation equivariant metanetworks (DeepSets and StatNN) both improve on W -Asym ResNets compared to on ResNets, even though the permutation symmetries of standard ResNets do not affect these metanetworks; thus, it may be possible that other symmetries in standard ResNets (but not Asym-ResNets) harm metanetwork performance, or they may be other factors besides symmetries that improve metanetwork performance for Asym-ResNets.

0.0

0.2

0.4

0.6

0.8

1.0

Interpolation

1

2

3

4

5

Train Loss

Standard ResNet

0.0

0.2

0.4

0.6

0.8

1.0

Interpolation

0.5

1.0

1.5

2.0

2.5

Train Loss

-Asymmetric ResNet

0.0

0.2

0.4

0.6

0.8

1.0

Interpolation

2

4

6

8

Train Loss

W

-Asymmetric ResNet

5.4 Monotonic Linear Interpolation

Background. One common method of studying the loss landscapes of neural networks is by studying the one-dimensional line segment of parameters attained by linear interpolation between parameters at initialization and parameters after training. Many works have observed monotonic linear interpolation (MLI), which is when the training loss monotonically decreases along this line segment [21, 17, 41, 68]. Loss landscapes of convex problems have this property as well, so presence of the monotonic linear interpolation property has been used as a rough measure of how well-behaved the loss landscape is. However, with many types of models, tasks, or hyperparameter settings, monotonic linear interpolation does not hold [41, 68], or there is a large plateau where the loss barely changes for much of the line segment [17, 70]; neither of these properties can happen for convex objectives trained to completion. To the best of our knowledge, there has been little work on the role of parameter symmetries - or lack thereof - in monotonic linear interpolation (besides one minor experiment in [41] Appendix C.9). Nonetheless, since removing parameter symmetries substantially improves linear interpolation between trained networks (Section 5.1), one may expect removing parameter symmetries to improve monotonic linear interpolation.

Hypothesis. The training loss along the line segment between initialization and trained parameters is more monotonic and convex for Asymmetric networks.

Experimental setup. For the learning task, we follow the setup used for creating the dataset of image classifiers in Section 5.3. In particular, we train 300 standard ResNets and 300 W -Asymmetric ResNets with varying hyperparameters sampled from the same distributions as used for the dataset of image classifiers (see Appendix Table 13). For each of these networks, we linearly interpolate between its initial parameters θ 0 and its final trained parameters θ T : (1 -α ) θ 0 + αθ T for 25 uniformly spaced values 0 = α 1 < α 2 < . . . < α 25 = 1 . To measure monotonicity, we record the maximum increase between adjacent networks ∆ = max( L ( α i +1 ) -L ( α i )) , and the percentage of networks that have ∆ ≤ 0 i.e. the percentage of networks that satisfy monotonic linear interpolation. To measure convexity, we consider a local convexity measure (the proportion of α i where the centered difference second derivative approximation is nonnegative) and a global convexity measure (the proportion of α i such that L ( α i ) lies below the line segment between the endpoints, i.e. L ( α i ) ≤ (1 -α i ) L (0) + α i L (1) ).

Results. Table 4 shows the measures of monotonicity and convexity for standard, σ -Asymmetric, and W -Asymmetric ResNets. Remarkably, every single one of the 300 W -Asymmetric ResNets satisfies

monotonic linear interpolation and has a trajectory that lies underneath the line segment between the endpoints. Qualitatively, we can see in Figure 5 that W -Asymmetric ResNets do not have any clear loss barriers from initialization, nor any loss plateaus that indicate nonconvexity. In contrast, the majority of standard ResNets have non-monotonic trajectories, and the monotonic trajectories seem to be more nonconvex. σ -Asymmetric network trajectories are signficantly more convex and monotonic than standard network trajectories, but there are some non-monotonic or nonconvex trajectories still.

5.5 Other Optimization and Loss Landscape Properties

In Appendix A, we note other interesting differences in optimization and loss landscape properties of Asymmetric and standard neural networks. These can be summarized as:

Even though Asymmetric networks interpolate much better than standard networks, the parameters of trained Asymmetric networks are often basically the same distance away from each other in weight space as standard networks.

Asymmetric networks do not tend to overfit as much: the difference in train performance and test performance can be substantially lower than that of standard networks.

Asymmetric networks can take longer to train, especially when choosing hyperparameters that make them more dissimilar to standard networks.

6 Discussion

While many properties of Asymmetric networks are in line with our hypotheses and intuition about the impact of removing parameter symmetries, there are many unexpected effects and unanswered questions that are promising to further investigate. For instance, we did not extensively explore Asymmetric networks in the context of model interpretability, generalization measures in weight spaces, or optimization improvements, all of which are known to be influenced to some extent by parameter symmetries. Further studying the properties in Section 5.5, the dependence of behavior on the choices of Asymmetry-inducing hyperparameters, and other design choices in making networks asymmetric could also bring more insights into parameter space symmetries.

Also, it is interesting that our σ -Asymmetric networks do not appear to break parameter symmetries as well as our W -Asymmetric networks. We have run preliminary empirical tests on several variants of σ -Asym networks, such as: σ ◦ F ◦ σ as the nonlinearity, sparsifying F , adding instead of multiplying the gate, using cosine instead of sigmoid, squaring instead of using sigmoid, and putting a LayerNorm in the nonlinearity. However, none of these approaches worked well. We believe that such failures may be because σ -Asymmetry breaks symmetries at the activation / neuron level, whereas W -Asymmetry breaks symmetries in the larger space of weights (for more evidence, see Appendix E, where we show that fixing biases at neurons also fails to effectively remove symmetries). These curiosities provide interesting directions for future work.

All in all, we believe that future theoretical and empirical study of Asymmetric networks could garner many insights into the role of parameter symmetries in deep learning.

References

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T .

Laurence Aitchison, Adam Yang, and Sebastian W Ober. Deep kernel processes. In International Conference on Machine Learning , pages 130-140. PMLR, 2021.

Gul Sena Altintas, Gregor Bachmann, Lorenzo Noci, and Thomas Hofmann. Disentangling linear mode-connectivity. arXiv preprint arXiv:2312.09832 , 2023.

Stephen Ashmore and Michael Gashler. A method for finding similarity between multi-layer perceptrons by forward bipartite alignment. In 2015 International Joint Conference on Neural Networks (IJCNN) , pages 1-7. IEEE, 2015.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. arXiv preprint arXiv:1511.01029 , 2015.

Georg Bökman and Fredrik Kahl. Investigating how reLU-networks encode symmetries. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https: //openreview.net/forum?id=8lbFwpebeu .

George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314, 1989.

Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning , pages 933-941. PMLR, 2017.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning , pages 1019-1028. PMLR, 2017.

Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: dissecting the weight space of neural networks. arXiv preprint arXiv:2002.05688 , 2020.

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=dNigytemkL .

Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving linear mode connectivity of neural networks via optimal transport. In International Conference on Artificial Intelligence and Statistics , pages 3853-3861. PMLR, 2024.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 , 2019.

Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International conference on machine learning , pages 3318-3328. PMLR, 2021.

Jonathan Frankle. Revisiting "qualitatively characterizing neural network optimization problems", 2020.

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

Cedric Gegout, Bernard Girau, and Fabrice Rossi. A mathematical model for feed-forward neural networks: theoretical description and parallel applications. PhD thesis, Laboratoire de l'informatique du parallélisme, 1995.

Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems , 35:11893-11905, 2022.

Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. ICLR , 2015.

William L Hamilton. Graph representation learning . Morgan & Claypool Publishers, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers , pages 129-135. Elsevier, 1990.

Robert Hecht-Nielsen. Theory of the backpropagation neural network. In Neural networks for perception , pages 65-93. Elsevier, 1992.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719 , 2023.

Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=gU5sJ6ZggcX .

Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands-on bayesian neural networks-a tutorial for deep learning users. IEEE Computational Intelligence Magazine , 17(2):29-48, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efstratios Gavves, Cees GM Snoek, and David W Zhang. Graph neural networks for learning equivariant representations of neural networks. arXiv preprint arXiv:2403.12143 , 2024.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.

Richard Kurle, Ralf Herbrich, Tim Januschowski, Yuyang Bernie Wang, and Jan Gasthaus. On the detrimental effect of invariances in the likelihood for variational inference. Advances in Neural Information Processing Systems , 35:4531-4542, 2022.

Lauro Langosco, Neel Alex, William Baker, David John Quarel, Herbie Bradley, and David Krueger. Towards meta-models for automated interpretability, 2024. URL https: //openreview.net/forum?id=fM1ETm3ssl .

Olivier Laurent, Emanuel Aldea, and Gianni Franchi. A symmetry-aware exploration of bayesian neural network posteriors. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=FOSBQuXgAq .

Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander M ˛ adry. Ffcv: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12011-12020, 2023.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Derek Lim, Haggai Maron, Marc T. Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=ijK5hyxs0n .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

James Lucas, Juhan Bae, Michael R Zhang, Stanislav Fort, Richard Zemel, and Roger Grosse. Analyzing monotonic linear interpolation in neural network loss landscapes. arXiv preprint arXiv:2104.11044 , 2021.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International conference on machine learning , pages 4363-4371. PMLR, 2019.

Edward Milsom, Ben Anson, and Laurence Aitchison. Convolutional deep kernel machines. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=1oqedRt6Z7 .

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations , 2021. URL https://openreview.net/ forum?id=Fmg_fQYUejf .

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence , volume 29, 2015.

Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning , pages 25790-25816. PMLR, 2023.

Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment. arXiv preprint arXiv:2310.13397 , 2023.

Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems , 28, 2015.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on learning theory , pages 1376-1401. PMLR, 2015.

Theodore Papamarkou, Jacob Hinkle, M Todd Young, and David Womble. Challenges in markov chain monte carlo for bayesian neural networks. Statistical Science , 37(3):425-442, 2022.

Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, Aliaksandr Hubin, et al. Position paper: Bayesian deep learning in the age of large-scale ai. arXiv preprint arXiv:2402.00809 , 2024.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20237-20246, 2023.

Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry. In International Conference on Machine Learning , pages 17759-17781. PMLR, 2022.

Arya A Pourzanjani, Richard M Jiang, and Linda R Petzold. Improving the identifiability of neural networks for bayesian inference. In NIPS workshop on bayesian deep learning , volume 4, page 31, 2017.

Xingyu Qu and Samuel Horvath. Rethink model re-basin and the linear mode connectivity. arXiv preprint arXiv:2402.05966 , 2024.

Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.

Aviv Shamsian, Aviv Navon, David W Zhang, Yan Zhang, Ethan Fetaya, Gal Chechik, and Haggai Maron. Improved generalization of weight space networks via augmentations. arXiv preprint arXiv:2402.04081 , 2024.

Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems , 33:22045-22055, 2020.

George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview. net/forum?id=LEYUkvdUhq .

Héctor J Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural networks , 5(4):589-593, 1992.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems , 33:15300-15311, 2020.

Marcin Tomczak, Siddharth Swaroop, and Richard Turner. Efficient low rank gaussian variational inference for neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 46104622. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/310cc7ca5a76a446f85c1a0d641ba96d-Paper.pdf .

Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.11448 , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Neha Verma and Maha Elbayad. Merging text transformer models from different initializations. arXiv preprint arXiv:2403.00986 , 2024.

Tiffany J. Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? ICML , 2022.

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440 , 2020.

Xiang Wang, Annie N. Wang, Mo Zhou, and Rong Ge. Plateau in monotonic linear interpolation - a 'biased' view of loss landscape for deep networks. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=z289SIQOQna .

Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, and David Rügamer. Towards efficient mcmc sampling in bayesian neural networks by exploiting symmetry. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 459-474. Springer, 2023.

Tim Z Xiao, Weiyang Liu, and Robert Bamler. A compact representation for bayesian neural networks by removing permutation symmetry. arXiv preprint arXiv:2401.00611 , 2023.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019. URL https: //openreview.net/forum?id=ryGs6iA5Km .

Adam X Yang, Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, and Laurence Aitchison. A theory of representation learning gives a deep generalisation of kernel methods. In International Conference on Machine Learning , pages 39380-39415. PMLR, 2023.

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations , 2020. URL https://openreview.net/forum? id=ByxRM0Ntvr .

David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) , 2022.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems , 30, 2017.

Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy. Symmetries, flat minima, and the conserved quantities of gradient flow. arXiv preprint arXiv:2210.17216 , 2022.

Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Understanding mode connectivity via parameter space symmetry. In UniReps: the First Workshop on Unifying Representations in Neural Models , 2023.

Bo Zhao, Robert M. Gower, Robin Walters, and Rose Yu. Improving convergence and generalization using parameter symmetries. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=L0r0GphlIL .

Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in Neural Information Processing Systems , 36, 2023.

Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in Neural Information Processing Systems , 36, 2023.

Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. arXiv preprint arXiv:2402.05232 , 2024.

Liu Ziyin. Symmetry leads to structured constraint of learning. arXiv preprint arXiv:2309.16932 , 2023.

0

~10000

~20000

~30000

~40000

0.0

0.75

1.5

2.25

3.0

Standard Deviation

11

10

11

12

11

11

12

15

19

22

11

19

25

31

38

11

24

36

44

53

11

30

46

60

72

Number of Epochs until 70%

20

30

40

50

60

70

A Additional Observations on Asymmetric Networks

There are several other interesting differences in the optimization and loss landscape properties of Asymmetric and standard neural networks. For one, even though Asymmetric networks generally interpolate significantly better than standard networks, this cannot be seen by measuring distances in parameter space. For instance, in GNN experiments following the setup of Section 5.1, pairs of standard GNNs have a distance per parameter of .000174 on average, whereas W -Asymmetric GNNs have .000159, which is only slightly lower. However, the average test loss barrier is 1.448 for standard GNNs while it is only 0.069 for W -Asymmetric GNNs. Likewise, in our datasets of 10,000 standard and W -Asymmetric ResNets, the average distance per parameter between the weights of trained standard classifiers is .0034, which is actually lower than the distance per parameter of .0051 for W -Asymmetric ResNets (estimated on 20,000 pairs of networks). Thus, although we sometimes imagine well-interpolating pairs of networks to lie in the same local basin of parameter space, W -Asymmetric networks are actually rather far apart in parameter space, but nonetheless have linear segments of low loss between them.

We also find that Asymmetric networks often do not overfit as much as standard networks. For instance, in the GNN setup of Section 5.1, standard GNNs have a max training accuracy of 84 . 6% on average, with a validation accuracy of 71 . 6% . On the other hand, σ -Asym GNNs have 70 . 8% / 70 . 1% train/validation accuracy, while W -Asym GNNs have 70 . 7% / 70 . 06% train/validation accuracy. This difference does not show as much in our datasets of 10,000 standard ResNets and W -Asym ResNets, possibly because of the substantial regularization (data augmentation, weight decay, and label smoothing) used for training (standard gets 74 . 8% / 73 . 8% train/test accuracy while W -Asymmetric gets 64 . 0% / 64 . 0% ).

Further, in Figure 6, we see that training speed is slower for W -Asymmetric ResNets when we increase the amount of asymmetry (by increasing the number of fixed entries and the standard deviation of the fixed entries). While standard ResNets take on average 11 epochs to reach 70% training accuracy on CIFAR-10, W -Asymmetric ResNets with the most extreme hyperparameters take up to 72 epochs.

B Proofs of Theoretical Results

B.1 Graph-based approach

Here, we prove that as long as each mask matrix M in our W -Asymmetric MLPs with fixed entries set to zero has unique nonzero rows, then our architecture has no nontrivial neural DAG automorphisms. In practice, we find that setting the standard deviation κ of the fixed entries F to be positive (and in fact orders of magnitude larger than the standard deviation that we typically initialize trainable weights with) is important to achieve properties such as linear mode connectivity that Asymmetric networks have but standard networks do not have. When κ = 0 (i.e. when fixed entries are set to zero), we can directly work in the framework of Lim et al. [39] that connects parameter symmetries to computation graph automorphisms. To work towards generalizing our result to κ > 0 , we would have to modify the definitions and results of Lim et al. [39]; for instance, we would need to add edges associated to untrainable parameters in the computation graph, and redefine the concept of neural DAG automorphisms. We leave such exploration to future work.

Theorem 3. If each mask matrix M has unique nonzero rows, then W -Asymmetric MLPs with κ = 0 have no nontrivial neural DAG automorphisms.

Proof. Consider an L -layer W -Asymmetric MLP with fixed entries set to zero. Denote its weights as W L , . . . , W 1 and the corresponding binary masks as M L , . . . , M 1 . The forward pass of such a network on an input x is then

for some elementwise nonlinearity σ . The dimension of W i is d i × d i -1 . In the framework of Lim et al. [39], this is a feedforward neural network with a computation graph defined as follows.

The node set is V 0 × V 1 × . . . × V L , where V i has d i nodes, and no nodes are shared between different V i . If a node v is in V i , then we say that layer( v ) = i . V 0 contains the input nodes and V L contains the output nodes. The adjacency matrix can be written as

Every block besides the ones containing masks is zero. There are L +1 × L +1 blocks, and the ( i, j ) block is of size d i × d j .

Recall that a neural DAG automorphism is a relabelling of nodes τ : V → V such that τ is bijective, ( i, j ) ∈ E if and only if ( τ ( i ) , τ ( j )) ∈ E , and every input node and output node is a fixed point of τ .

Now, let τ : V → V be a neural DAG automorphism. Further, let P be the corresponding permutation matrix. We will show that τ is the identity, i.e. that P = I . By Lemma 1, we know that τ preserves layer number of nodes, meaning layer( τ ( i )) = layer( i ) . Thus, P is a block diagonal permutation matrix:

where P i is d i × d i . Morever, P 0 = I and P L = I because input nodes and output nodes are fixed points. Applying this to the adjacency matrix, we see that

Since τ is a neural DAG automorphism, we have that τ ( A ) = A . Equating blocks, this means that P 1 M 1 P ⊤ 0 = M 1 . As P 0 = I , we have P 1 M 1 = M 1 . But M 1 has unique rows, so P 1 = I as well.

For the inductive step, assume P i = I for some i . Then P i +1 M i +1 P ⊤ i = P i +1 M i +1 = M i +1 , so since M i +1 has unique rows, we have that P i +1 = I . As this holds for any i by induction, this means that P = I , so τ is a trivial neural DAG automorphism and we are done.

Lemma 1. Neural DAG automorphisms preserve layer number in W -Asymmetric MLPs that have masks with nonzero rows.

Proof. Let τ be a neural DAG automorphism. This means that PAP ⊤ = A , where P is the permutation matrix associated to τ . Then, using PAP ⊤ = A for the first equality and the definition of P in the second, we have that

We proceed by induction on layer number l . For any input node i we know that τ ( i ) = i , so of course layer( τ ( i )) = layer( i ) .

Now, suppose that layer( τ ( i )) = layer( i ) for any i in layer l ≥ 1 . If node j is in layer l +1 , then there is some i in layer l such that ( i, j ) ∈ E because M l +1 has no nonzero rows. We have that A τ ( i ) ,τ ( j ) = A i,j , so τ ( i ) is connected to τ ( j ) . As we know that τ ( i ) is in layer l , we have that τ ( j ) is in layer l +1 .

B.2 Symmetry Breaking via Nonlinearities

Proposition 3. Let the parameter space Θ be all pairs of square invertible matrices θ = ( W 2 , W 1 ) for W 2 , W 1 ∈ GL ( d ) , and let f θ ( x ) = W 2 σ ( W 1 x ) . If σ has no linear equivariances, then f θ 1 = f θ 2 if and only if θ 1 = θ 2 . In other words, there are no nontrivial parameter space symmetries.

Proof. If θ 1 = θ 2 , then clearly f θ 1 = f θ 2 . For the other direction, suppose f θ 1 = f θ 2 , and denote θ 1 = ( W 2 , W 1 ) and θ 2 = ( ˜ W 2 , ˜ W 1 ) . Then for any input z ∈ R n , we have

Now, choose an arbitrary x ∈ R n . We let z in the above equation (11) be W -1 1 x , so we have

This holds for any x , so ˜ W -1 2 W 2 ◦ σ = σ ◦ ˜ W 1 W -1 1 , i.e. we have found a linear equivariance of σ . Since σ has no linear equivariances,

meaning that ˜ W 2 = W 2 and ˜ W 1 = W 1 , i.e. θ 1 = θ 2 , so we are done.

B.2.1 FiGLU nonlinearity proofs (Proposition 2)

Now, we study the properties of our FiGLU nonlinearity σ ( x ) = η ( F x ) ⊙ x , where η is the sigmoid function η ( x ) = 1 1+ e -x . For proving Proposition 2, we want to prove that with probability 1 over samples of F , σ has no permutation or diagonal equivariances.

We say that σ has no permutation equivariances if whenever P 2 ◦ σ = σ ◦ P 1 for permutation matrices P 1 and P 2 , then P 1 = P 2 = I . Likewise, we say that σ has no diagonal equivariances if whenever B ◦ σ = σ ◦ A for invertible diagonal matrices A and B , then A = B = I .

We will show that these two properties hold for any F that has no permutation symmetries and no zero entries. We say that F has no permutation symmetries if P 2 F P 1 = F for permutation matrices P 1 and P 2 implies that P 1 = P 2 = I . Note that if F has distinct entries, then it has no permutation symmetries. Thus, F satisfies both of these conditions with probability 1, since the set of matrices with nondistinct entries or with at least one zero entry are of Lebesgue measure zero, so they have zero probability under the Gaussian distribution. We now proceed to show that σ has no permutation or diagonal equivariances under these conditions on F .

Proposition 4. If F is a square matrix with no permutation symmetries, then σ ( x ) = η ( F x ) ⊙ x has no permutation equivariances.

Proof. Suppose σ ◦ P 1 = P 2 ◦ σ for permutation matrices P 1 , P 2 . We will show that P 1 = P 2 = I . For any input x , we have

where we used permutation equivariance of η , which acts elementwise. Let x = e i , the standard basis vector that is 1 in the i th coordinate and 0 elsewhere. If i is not a fixed point of the permutation P ⊤ 2 P 1 , then let j be the index that it is mapped to. Then equation (16) gives that

In the i th coordinate of this equality of vectors, we see that η ( F e i ) = 0 , which is impossible, since η is the sigmoid function. Thus, i cannot be a fixed point of P ⊤ 2 P 1 , so P ⊤ 2 P 1 = I is the identity permutation. Now, let x be an arbitrary vector with no zero entries. Equation (16) gives that

Since x is nonzero, we can divide by x i in the i th coordinate of this vector equality for each i to get that

As η is bijective,

Because this holds for all x with no zero entries (and in particular for a basis of the input space), we know that

as matrices. But since F has no permutation symmetries, we have that P 1 = P 2 = I , so we are done.

Proposition 5. If F is a square matrix with no zero entries, then σ ( x ) = η ( F x ) ⊙ x has no diagonal equivariances.

Proof. Let A = Diag( α ) and B = Diag( β ) be invertible diagonal matrices, and suppose that σ ◦ A = B ◦ σ . We will show that A = B = I . For any input x , we have

̸

Let x = ce i , where e i is the i th standard basis vector and c = 0 is any nonzero number. Then

At the i th coordinate of this equality, we have

Thus, the right hand side is constant in c . We must have that α i > 0 , because if not, then increasing c would increase either the numerator or denominator and decrease the other, hence contradicting the equality (here we use that F has no zero entries, so c F e i is nonzero in every entry). Thus, letting c →∞ , we see that α i β i = 1 , so α i = β i . Plugging this back into Equation (26), we have

where in the third line we used the fact that η is invertible. We have shown that α i = β i = 1 for each i , so A = B = I and we are done.

We note that the proofs of these two results about FiGLU are reminiscent of some proof techniques from Godfrey et al. [20], such as those used in their analysis of GELU nonlinearities.

B.3 Proofs for Universal Approximation

Here, we prove the universal approximation result for our W -Asymmetric MLPs.

Theorem 4. Let η be any nonpolynomial elementwise nonlinearity with η ( x ) -η ( -x ) = x (e.g. ReLU , GELU , swish ), let Ω ⊆ R D be a compact domain, and let f target : Ω → R be a continuous target function. Fix ε > 0 and δ > 0 .

There exists a width n ′ such that for all n > n ′ , with probability 1 -δ , for a randomly sampled 4-layer W -Asymmetric MLP f with η nonlinearity, hidden dimensions 24 n → n → 24 n , and n fix ∈ o ( n 1 / 4 ) hardwired entries per neuron, there will exist θ ∈ Θ such that the W -Asymmetric MLP f : R n → R approximates f target to ε :

Importantly, we require that the input to f θ be padded with n -D zeroes, so [ x ; 0] ∈ R n .

B.3.1 Proof sketch

To approximate f target to ε > 0 , we will leverage the universal approximation for standard MLPs with nonlinearity η to first obtain a standard 2-layer MLP that approximates f target to within ε , meaning ∥ ∥ f target ( x ) -f MLP ( x ) ∥ ∥ < ε for all x ∈ Ω . Then we will exactly represent f MLP using an Asymmetric Network f .

This will be done by approximating each linear map W of f MLP by two layers of an Asymmetric network: W ′ 2 ◦ η ◦ W ′ 1 = W for Asymmetric linear maps W ′ 2 and W ′ 1 . For the sake of exposition, we will show how to do this first when both W ′ 2 and W ′ 1 have no Asymmetric mask (i.e. fitting a linear map W using a standard two-layer η -MLP), then when only W ′ 2 has an Asymmetric mask, and finally when both W ′ 2 and W ′ 1 have an Asymmetric mask.

B.3.2 Fitting a Linear Map with a Two-Layer Standard MLP

Let W ∈ R n × n be the target linear map, and let B ∈ R 2 n × n and A ∈ R n × 2 n be parameters of a two-layer MLP, defined by f A,B ( x ) = Aη ( Bx ) . We will choose A and B such that f A,B ( x ) = Wx for all x ∈ R n .

Denote the i th row of W by W i , so that

We set A and B as follows, where I n is the n × n identity matrix:

Then we can see that f A,B exactly computes the linear transformation Wx .

B.3.3 Fitting a Linear Map with One Asymmetric and One Standard Linear Map

Let n fix > 0 and let each row of N ∈ { 0 , 1 } n × 6 n have n fix entries equal to 0, selected at random. Let B ∈ R 6 n × n and A ∈ R n × 6 n . Define A ′ to be an Asymmetric linear map: A ′ = A ⊙ N +(1 -N ) ⊙ P , where N is a randomly sampled binary mask, and P a randomly sampled Gaussian matrix. We consider a two-layer network with one Asymmetric and one standard linear map: f A,B ( x ) = A ′ η ( Bx ) . We want f A,B ( x ) = Wx for all x. For the remainder of this proof, we will assume that N never has three consecutive entries in a row set to zero; we will later show that this holds with high probability over the sampling of N .

First, we define B in a similar way to the purely linear setting, but with additional copies of entries to allow for error correction of the random noisy entries fixed in A ′ .

Recall that each row A ′ i of A ′ has n fix entries that are randomly hardwired to constants. Ideally, we would want A ′ i to pick out η ( W i · x ) -η ( -W i · x ) = W i · x , but because of the hardwired constants, A ′ i might randomly add c ∗ η ( W j · x ) . However, since there are three copies of η ( W j · x ) in η ( Bx ) , as long as not all three corresponding entries in A ′ i are fixed, one of the un-fixed copies can be changed such that the coefficients sum to 0 . Since by our assumption N never has three consecutive entries all set to 0, the coefficients of A can be picked such that A ′ η ( Bx ) = Wx . For example, a possible A ′ matrix would be

Thus we have shown that under the assumption that N never has three consecutive entries equal to 0, A can be picked such that A ′ η ( Bx ) = Wx . We will now show that P ( N never has three consecutive entries equal to 0 ) can be made arbitrarily small by increasing the width n while keeping n fix to be o ( n 1 / 3 ) .

The probability there are three consecutive entries in a given row of N that are zero is O ( n 3 fix n 2 ) . By the union bound, the probability that any row has 3 consecutive hardwired entries is O ( n 3 fix n ) . For any n fix ∈ o ( n 1 / 3 ) , this tends towards 0 . Thus, with probability ≥ 1 -O ( n 3 fix n ) , A can be picked such that A ′ η ( Bx ) = Wx .

B.3.4 Fitting a Linear Map with Two Asymmetric Linear Maps

Once again let n fix > 0 , and let each row of M have n fix entries equal to 0, selected at random. Let B ∈ R 24 n × n and A ∈ R n × 24 n . Further, define Asymmetric maps

where N , M are randomly sampled masks, and P , Q are normal matrices. Then we let f A,B ( x ) = A ′ η ( B ′ ( x )) , and we once again desire choices of parameters A and B such that f A,B ( x ) = Wx .

Constructing B

Consider the randomly drawn mask M ∈ { 0 , 1 } 24 n × n , and denote the i th row by M i .

where M i ∈ { 0 , 1 } n . We partition M 's rows into n blocks of 24 rows. β 1 = { M 0 . . . M 23 } , . . . β i = { M 24 i . . . M 24 i +23 } . Now, consider β 1 , the first 24 rows of M .

Definition B.1. We say two rows M j , M k are intersecting if there is some column index α such that M j,α = M k,α = 0 . That is, two rows of are intersecting if they share a 0 at the same index.

Note that for any two given rows of M , the probability that they share a 0 in the same location is ≤ n 2 fix n .

We assume that β i contains no more than one pair of intersecting rows; later, we show this to hold with high probability. Then, every β i can be broken into two disjoint sets of 12 rows, β i, 0 and β i, 1 , such that neither set of 12 contains a single pair of intersecting rows. Intuitively, this means that each row in B corresponding to β i, 0 will have unique fixed indices.

Our goal will be for the rows in β i to mimic the row W i . We will show how to do this for each i . Fix an arbitrary index i ∈ { 0 , . . . , n -1 } .

Without loss of generality, assume β i, 0 and β i, 1 are continguous, so β i, 0 = M 24 i :24 i +11 and β i, 1 = M 24 i +11:24 i +23 . By our assumption, for j, k ∈ β i, 0 (i.e. j, k ∈ { 0 , . . . , 11 } ), the mask rows M 24 i + j and M 24 i + k are never 0 in the same two column indices. Similarly, for j, k ∈ β i, 1 (i.e. j, k ∈ { 12 , . . . , 23 } ), the mask rows M 24 i + j and M 24 i + k are never 0 in the same two column indices.

Next, we define c i,j as the difference between B ′ and B in the (24 i + j ) th row:

In particular, we have that

̸

Lemma 2. For any indices j = k such that j, k ∈ β i, 0 or j, k ∈ β i, 1 , we have that

Proof. By the definition of c i,j , we know that c i,j is only nonzero at indices where M 24 i + j is equal to zero. Since j, k are either both in β i, 0 or β i, 1 , we know that M 24 i + k cannot also be zero at indices where M 24 i + j is zero. Thus, M 24 i + k is equal to 1 at every index where c i,j is nonzero, so c i,j ⊙ M 24 i + k = c i,j as desired.

Next, we construct B , by constructing this block of 24 rows. Let [ c i, 0 , c i, 1 , c i, 2 , c i, 3 ] be continguous sums of length-3 segments of c i, : :

We assign the first 12 rows of B as follows.

Defining c = c i, 0 -c i, 1 + c i, 2 -c i, 3 , we have the nice property:

By the construction above, B ′ 24 i = -B ′ 24 i +3 , and B ′ 24 i +6 = -B ′ 24 i +9 . This means that

and likewise that

So that a simple linear map gives our desired output:

In the next part, we will construct A ′ to compute this linear map, which will follow the method of Appendix B.3.3 (because A ′ has certain fixed entries).

What remains is to define the rows of B corresponding to β i, 1 in an error correctible manner. This can be done easily by defining

By similar reasoning to before, this means that

Recall that we constructed B ′ under the assumption that no β i has at most one pair of intersecting rows. We now show that the β i each have at most one pair of intersecting rows with high probability. Within the 24 rows of any given β i , the probability that more than one pair of rows are intersecting is ≤ C n 4 fix n 2 for some constant C . So, by the union bound, the probability over M that any of the β i have more than one pair of intersecting rows is ≤ C n 4 fix n . Thus, we can construct B in this manner with probability ≥ 1 -C n 4 fix n . For sufficiently large n and n fix ∈ o ( n 1 / 4 ) , this probability approaches 1 .

and then defining

Construction of A

With our above construction, each block of the 24 rows in β i of η ( B ′ x ) is of the form

(60)

Importantly, each row here is n wide. Recall that A ′ ∈ R n × 24 n . Denote the i th row of A ′ by A ′ i ∈ R 24 n with A ′ i ∈ R 24 n . If A ′ had 0 hardwired entries, then setting A i = ✶ 24 i -✶ 24 i +3 -( 24 i +6 -24 i +9 ) would give A i η ( B ′ · x ) = W i · x , by the same argument as in Appendix B.3.2.

Further, just as in Appendix B.3.3, the probability that a given row of A ′ has three items hardwired in a row is O ( n 3 fix n 2 ) . Thus, by the union bound, the probability that some row of A ′ has three items hardwired in a row is O ( n 3 fix n ) . So, with large enough width n , A can be chosen such that A ′ η ( B ′ x ) = Wx .

✶ ✶ Unfortunately this is not the case, so we have to use the construction in Appendix B.3.3. Recall, that A ′ has n fix fixed entries in each row. This means that N i has n fix entries equal to 0 . Since every entry of B ′ x has three copies, as long as N i does not have three elements set to 0 in a row, A ′ i can be made equivalent to A i = ✶ 24 i -✶ 24 i +3 -( ✶ 24 i +6 -✶ 24 i +9 ) . This is because, as in Appendix B.3.3, if at most 2 elements out of any 3 three copies are hardwired, then the third can be changed arbitrarily to offset the hardwiring.

Similarly, any linear map in R ˜ n × n for ˜ n < n can also be fit using this method.

Conclusion

We have shown that a W -Asymmetric MLP with hidden dimension 24 n can exactly fit an n × n linear map with high probability over the choice of Asymmetric masks M . It is known by [8] that for any continuous function f target : Ω ⊆ R D → R and any ε > 0 , there exists a width k ′ such that 2-layer MLPs of width k ′ can approximate f to within ε .

Let k be sufficiently big so that the probability that the masks do not satisfy the conditions of Appendix B.3.4 is less than δ . Such a k exists as long as n fix ∈ o ( k 1 4 ) . Let m ≥ max( k, k ′ , D ) .

Importantly, if a 2-layer MLP of width k ′ can approximate f target to within ε , a 2-layer MLP of width m with m ≥ k ′ can also approximate f to within ε . Let f MLP be a width m MLP that approximates f target to within ε .

We now pad the input x to f target , with m -D zeros. This allows us to define a new function f 0 target : R m → R by f 0 target ([ x ; 0]) = f ( x ) . Clearly f 0 target can also be approximated by a width m MLP.

Let f 0 MLP denote the width m MLP that approximates f 0 to within ε . Now, f 0 MLP has dimensions m → m → 1 , with corresponding linear maps W 1 ∈ R m × m , W 2 ∈ R 1 × m . Each of these maps can be exactly fit using a 2-layer W -Asymmetric MLP, since their corresponding matrices have at least as many columns as rows. Concatenating these two exact fits yields an asymmetric MLP whose output exactly matches f 0 MLP and thus approximates f 0 to within ε .

Thus, setting n ′ = m , there exists a width n ′ such that for all n > n ′ , with probability 1 -δ , for a randomly sampled 4-layer W -Asymmetric MLP f with η nonlinearity, hidden dimensions 24 n → n → 24 n , and n fix ∈ o ( n 1 / 4 ) hardwired entries per neuron, there will exist θ ∈ Θ such that the W -Asymmetric MLP f : R n → R approximates f target to ε .

On parameter symmetries of the 4-layer W -Asymmetric Network

As an aside, this procedure of mapping 2-layer standard MLPs to 4-layer W -Asymmetric MLPs implies that these 4-layer W -Asymmetric MLPs have at least as many symmetries as 2 layer standard MLPs. To fix this, we may want to consider a nonlinearity η such that η ( x ) -η ( -x ) = x .

C Limitations

Although our W -Asymmetric and σ -Asymmetric networks are motivated by removing parameter space symmetries, their distinct empirical behavior may be caused by other factors besides just parameter space symmetries. For instance, the fixed entries F for the W -Asymmetric approach are taken to be much larger than the standard initialization of linear maps, which could cause several changes to optimization and loss landscapes besides just parameter symmetry breaking.

Also, our theoretical results could be strengthened by future work in several ways. For instance, for the σ -Asymmetric approach, Proposition 1 only gives a guarantee of no parameter symmetries in the two-layer network case with square invertible weights. Future work could also give tighter analysis of the required width and depth for universal approximation using our W -Asymmetric architecture.

D Broader Impacts

This work does not focus on any particular application area. Instead, we study fundamental phenomena and theory of deep learning in general. Our work has potential to improve known deficits of neural networks: by making neural network loss landscapes more similar to convex landscapes, we can improve our understanding of them, and by improving Bayesian neural networks we advance one paradigm for bettering uncertainty quantification in neural networks. However, unlike standard neural networks, which have millions of papers studying them, we have only scratched the surface of Asymmetric networks. Important properties such as generalization, robustness to distribution shifts, and adversarial robustness have not been extensively studied for Asymmetric networks, and the interaction of parameter symmetries with these properties is not clear. Future research should further explore these important properties.

E Experimental Ablations

Here, we present experiments with various ablations. These were in-part suggested by anonymous reviewers for the NeurIPS 2024 conference (thanks!).

E.1 Matching Learnable Parameters

For the experiments in the main text, our W -Asym networks often had less learnable parameters than the standard networks that they were compared against (since we take them to have the same architecture, but the W -Asym approach fixes certain learnable parameters to constants). See Table 5 for the learnable parameter counts.

̸

In this section, we control for this, by decreasing the width of the standard networks so that they have the same number of parameters as the W -Asymmetric networks they are compared against. Results are shown in Tables 6 and 7. We see that there is little to no change in metanetwork performance or Bayesian neural network performance for these smaller standard networks, so our results from the main text all still hold.

E.2 Changing Number of Warmup Steps

Altintas et al. [3] showed that amount of learning rate warmup can affect the extent of linear mode connectivity between training runs. In Table 8, we see that varying the number of warmup epochs between 1 and 20 does not change the qualitative results much on test loss barrier for our linear mode connectivity experiments.

E.3 Failure to Break Symmetries by Fixing Biases

Intuitively, one way to break permutation symmetries of an MLP is to order the hidden neurons in some way. One possible way to do this is to fix the biases (so that they are untrained). As shown in Table 9, a basic attempt at this fails, and has a much larger test barrier than our W -Asym and σ -Asym networks.

F Experimental Details

F.1 Linear Mode Connectivity Experimental Details

F.1.1 Image Classifier Interpolation

For the image classification experiments, we use two types of models.

ResNet We train ResNet20s with LayerNorm of width 64 and 8 · 64 . We use a batch size of 128 and a learning rate that warms up from . 0001 to . 01 over 20 epochs. In the width 8 × multiplier case we train for 50 epochs, and in the width 1 × multiplier case we train for 100. For σ -Asymmetric ResNets, we warm up to a learning rate of . 001 instead of . 01 due to training instability.

MLP We train MLPs with 4 layers, LayerNorm, and width 512. For MNIST we tuned the hyperparameters (epochs, learning rate, weight decay) of both the Asymmetric and Standard models to minimize loss barrier. We use a batch size of 64.

For MNIST we use no data augmentation, and for CIFAR-10 we use random cropping and horizontal flipping. For the Git-ReBasin tests, we use the weight matching algorithm from [1]. For MLPs on MNIST, we used the Asymmetry hyperparameters in Table 10. Table 11 gives the Asymmetric hyperparameters for ResNet20 on CIFAR-10, and Table 12 lists the same for ResNet20 with 8 x larger width.

F.1.2 Graph Neural Network Interpolation

For the GNN experiments, we use a GNN architecture similar to GIN [73] with mean aggregation. The base GNN has three message passing layers and a hidden dimension of 256, which gives 176,424 trainable parameters. The dataset is ogbn-arXiv [27], which is a citation network of computer science arXiv papers with 169,343 nodes and 1,166,243 edges. The task is transductive node classification, where the label of each paper node is the primary subject area of the paper.

As is common in transductive node classification on modestly sized graphs, we train each network with full-batch gradient on the whole graph. Thus, the randomness in training is purely from the initialization - there is no noise from minibatch selection in SGD. We use the Adam optimizer [31] with a peak learning rate of .001. The learning rate is linearly warmed up for 25 epochs to the peak, and then is held constant. Each network is trained for 500 epochs.

For the Git-ReBasin alignment, we implement the activation matching approach. For the σ -Asymmetric GNN, we take σ to be FiGLU, in which we randomly initialize each fixed matrix F as a standard normal matrix with standard deviation . 01 / √ d where d is the number of hidden channels; we found that having small standard deviation helped with training and interpolation. For the W -Asymmetric GNN, we fix 6 constants in each row of each linear map, and randomly initialize these constants from a normal distribution with standard deviation . 5 .

F.2 Bayesian Neural Network Experimental Details

For training Bayesian neural networks, we use the variational inference approach of Tomczak et al. [64], which fits an approximate posterior that is Gaussian with a diagonal plus rank-4 covariance matrix structure. For the W -Asym ResNet tests, we train ResNet20s with the same Asymmetric hyperparameters as in Table 11, though with κ = . 5 . For the CIFAR-100 experiments, we use a

standard linear layer instead of hardwiring weights for the last fully-connected linear layer. On CIFAR-100 we also use a width multiplier of 2 for our ResNets. For the ResNet experiments, we use a learning rate of . 001 . We train with a batch size of 250 for 50 epochs.

For the MLP experiments, we use κ = . 5 , 8 hardwired entries per neuron, and a learning rate of .0005. A batch size of 250 is used for 50 epochs again.

We use standard data augmentation (horizontal flips and random crops) on CIFAR-10 and CIFAR-100, and no data augmentations for MNIST. All training is done with the Adam optimizer [31].

F.3 Metanetwork Experimental Details

F.3.1 Dataset Details

We trained two datasets of image classifiers on CIFAR-10: one consisting of 10,000 small ResNet-like convolutional neural networks, and one consisting of 10,000 networks with a similar architecture, that use our graph-based approach to removing parameter symmetries. For fast training of many image classifiers, we use the FFCV package [37]. In particular, we use their CIFAR-10 sample script https://github.com/libffcv/ffcv/tree/main/examples/cifar , which includes data augmentation (random horizontal flips, random translations, and Cutout [10]), label smoothing [62], and a linear learning rate warmup and decay. In total, training all 20,000 classifiers takes just under 400 GPU hours (about 2 GPU-weeks) on NVIDIA RTX 2080 Ti GPUs.

See Table 13 for the hyperparameters and ranges that we varied across the networks in our datasets. In each dataset, the trained networks all have the same architecture.

Each ResNet has 78,042 trainable parameters, and each W -Asym ResNet has 60,634 trainable parameters. Both have the same architecture, except the W -Asym ResNet has certain filters that are fixed to constants to break the parameter symmetries. The ResNets each have 8 convolution layers, LayerNorm [5], and a final fully-connected linear classification layer after average pooling across spatial dimensions.

F.3.2 Metanetwork Details

We trained several types of metanetworks for our experiments. All of these metanetworks are trained for 50 epochs using the AdamW optimizer [40]. For each metanetwork, on each dataset, we choose the learning rate in { 10 -5 , 10 -4 , 5 · 10 -4 , 10 -3 , 5 · 10 -3 , 10 -2 } that gives the best validation R 2 performance on one training run. Then we run train each type of metanetwork 5 times on each dataset, and report the mean and standard deviation for each metric in Table 3.

F.4 Monotonic Linear Interpolation Experimental Details

For the monotonic linear interpolation experiments, we used the same setup as in the training of the datasets of CIFAR-10 image classifiers in Section 5.3. For each architecture, we sample 300 sets of hyperparameters from the distributions in Table 13, and train one network for each set of these sampled hyperparameters. When evaluating training loss, we include the labeling smoothing term.

√

For the σ -Asymmetric networks, we initialize the FiGLU F with a standard deviation of 1 / d , where d is the number of channels in the layer. Note that this is considerably larger than the standard

√

deviation of . 01 / d used in the GNN experiments of Section 5.1; we found this setting to train better (note that this initialization is in line with standard initializations of trainable parameters). Further, for the σ -Asymmetric networks, 24 out of the 300 networks diverged during training (giving NaNs ), so we exclude them from the computation of statistics in Table 4. From manual inspection, this divergence seems to happen when the learning rate is high (greater than . 1 ). In contrast, none of the standard or W -Asymmetric networks diverged.

F.5 Miscellaneous Experimental Details

The datasets we use are MNIST [38], CIFAR-10 [33], CIFAR-100 [33], and ogbn-arXiv [27], which are all widely used in machine learning research. The first three appear to not have licenses and are open to use, while the last dataset is from the Open Graph Benchmark, which has an MIT License in the Github repository.

We use software packages including PyTorch [52] (for all neural network experiments), FFCV [37] (for building our dataset in Section 5.3), and PyTorch Geometric [15] (for GNN experiments).

We ran our experiments on several types of NVIDIA GPUs and compute systems, including 2080 Ti, 3090 Ti, 4090 Ti, and V100 GPUs. Every training run was conducted on at most one GPU.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In Sections 4 and 5, we include theoretical proofs, a description of the methods and experimental setup, and experimental results to support the claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Appendix C, we detail empirical and theoretical limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We give the full set of assumptions and proofs for each result in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We give high-level experimental setup information in the main paper (Section 5), and more detailed information about experiments in the Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have open sourced the code to reproduce our experiments in https: //github.com/cptq/asymmetric-networks .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe high-level experimental setup in Section 5, and fine-grained experimental details in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We generally do several runs of experiments, and report error bars that measure the variance in the results. We also conduct statistical significance tests in Table 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: While we are not fully comprehensive, we include some details on compute in Appendix F.5, and estimate the compute required for generating our datasets of image classifiers in Appendix F.3. All other experiments follow popular training paradigms and are small in scale.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and complied with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, we discuss both potential positive and negative societal impacts in Appendix D.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not train models or use data that have a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We credit code and data used in Appendix F.5. For other types of models, we cite papers in which they were introduced.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or human subjects were used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or human subjects were used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops two types of modifications to neural networks to remove permutation symmetries: fixing certain weights, or using a non-elementwise activation function. The resulting \"asymmetric\" neural networks are found to improve on certain metrics that are observed in networks with permutation symmetries, and thought to be caused by such symmetries: lack of linear mode connectivity between randomly initialized/trained networks, reduced Bayesian network performance, and reduced metanetwork performance. The asymmetric networks also are more likely to have monotonic linear interpolation between their initial and trained weights.\n\nSTRENGTHS:\nThis paper is effectively an ablation study on permutation symmetries, which provides a much-needed tool and counterfactual perspective for investigations of permutation symmetries. The methods appear both novel and logical. A thorough survey of prior literature is given, which is extremely welcome. The paper is well organized, with well-motivated methodology and easy-to-follow experiments and hypotheses are easy to follow. The proofs are intuitive and comprehensive.\n\nWEAKNESSES:\nThe main issue is that the paper relies heavily on experimental rather than theoretical results, but the strongest results are for a single method ($W$-asymmetry) with settings much more different than \"standard\" training than seems at first glance - specifically, lower learning rates, longer warmup, and larger fixed weights that all differ from standard training by order(s) of magnitude. These settings, plus a lack of available code, seriously impact the relevance/reproducibility of the methods/results.\n\nThere should be a comparison with standard models initialized to the same weights as the $W$-asymmetric networks. This would make certain metrics like $L^2$ weight distance comparable (currently the $W$-asymmetric networks are more likely to have extremely large magnitude weights which could increase the $L^2$ distance between independently initialized/trained networks). This would also eliminate some other variables such as reduced performance due to suboptimal weight initialization.\n\nLow learning rates and long warmup are likely to improve similarity and reduce barriers, particularly between identically initialized neural networks. An analysis of how learning rate/warmup affects the experimental results (particularly barriers after training) is needed to determine if the observed results are mainly due to asymmetry.\n\nSome additional details are needed to properly replicate the asymmetry methods, specifically in tables 5-7.\n- how are $n_{\\text{fix}}$ and $\\kappa$ determined for each architecture?\n- what do the names of the blocks refer to? There are not enough for a 1-to-1 mapping onto ResNet20 layers or permutations, so more explanation is needed.\n\nQUESTIONS:\nAre the results in table 1 for networks with different random initializations, or the same identical initialization? If the former, are the masks and fixed weights different or the same between the two networks? I find it highly unlikely that networks with different fixed weights or different initializations would have low barrier after training without post-processing.\n\nThe fixed weights in $W$-asymmetry are distributed very differently to the non-fixed weights. If the fixed linear transform $F$ in $\\sigma$-asymmetry is similarly shifted in distribution, would that make its results closer to $W$-asymmetry?\n\nWhy not fix biases to get asymmetry instead?\n\nLIMITATIONS:\nThe reproducibility issues discussed previously, combined with the experimentally-focused framing, detract from an otherwise very strong and impactful paper.\n\nThe results for $\\sigma$-asymmetry are considerably weaker than $W$-asymmetry (and rely on even smaller learning rates). Another work has found that networks mainly use linear transformations [1]. Since $\\sigma$-asymmetry is effectively a non-elementwise sigmoid activation function, I wonder if the network's activations mainly stay in the approximately linear region of the sigmoid, allowing many permutations to be approximately equivalent. Some other comments or experiments on why $\\sigma$-asymmetry is less effective in practice would be helpful, especially since theoretically speaking, both methods are clearly asymmetric.\n\n[1] Mehmeti-Göpel & Disselhoff. Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think. ICML 2023. https://proceedings.mlr.press/v202/ali-mehmeti-gopel23a.html", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper suggested asymmetric neural networks in terms of an asymmetric nonlinearity and weight. It demonstrated some tasks including linear mode connectivity without permutation alignment, Bayesian neural networks, training metanetworks, and monotonic linear interpolation to show the role of permutation symmetry inherited in the standard neural networks.

STRENGTHS:
1. Various evaluations to show the role of permutation symmetry.
1. Novel approaches to build neural networks that have no permutation symmetry.

WEAKNESSES:
1. The number of learnable parameters of the standard NN and the asymmetric NN is reported only for the metanetwork task. It should also be reported for the LMC, BMA, and MLI tasks.
1. Comprehensible reason is required for the proposition “the posterior will have less modes” in the hypothesis of the BMA task in order to understand the purpose of the BMA task. No symmetry does not imply less modes. Thus, the improved performance of BMA may be induced from another reason other than asymmetry.

QUESTIONS:
What do you mean by “loss” in BNN. Is it an ELBO? Why don’t you report the negative log-likelihood (NLL)?

LIMITATIONS:
1. Fixing some weights or adding anisotropic activation function implies constraining not only symmetry but also distorted parameter manifold (like equivariant NNs) that also leads to increasing correlation between solutions. The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation.
1. The BMA and MLI tasks are based on the strong assumption, where its evidence is absent, that no symmetry induces the reduced number of modes.
1. Although the paper partially gives insights on the loss surface, it does not suggest a new practical method utilizing the insights.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to study the effect of parameter symmetries on the neural networks' training and final properties by analyzing the behavior of networks without such symmetries (or with fewer of them). To do so, the authors develop two methods of parameterizing neural network architectures without parameter symmetries: W-asymmetric parametrization fixes the different subsets of weights in each row of weight matrices to be constant and untrainable, and a $\\sigma$-asymmetric network uses a new FiGLU nonlinearity. The paper analyzes the asymmetric properties of both proposed methods theoretically and demonstrates empirically that asymmetry improves the behavior of neural networks in several setups. Specifically, asymmetric neural networks have better linear mode connectivity after training and more stable monotonic linear interpolation between initialization and trained model, are more effective for Bayesian deep learning, and are easier to model with metanetworks.\n\nSTRENGTHS:\n1. The paper proposes a new idea of analyzing the effect of parameter symmetries on neural networks through comparison with asymmetric networks in practical settings. I think this perspective is interesting and has potential for future research.\n2. The authors develop easy and effective asymmetric parametrizations and analyze them theoretically. \n3. The applicability of the proposed asymmetric perspective is demonstrated on a wide range of related problems.\n4. The paper is clearly written and easy to follow.\n\nWEAKNESSES:\nMy main concerns are related to the limited discussion on the benefits of the proposed perspective in comparison to previous works, asymmetric properties of the proposed methods in practical setups, and poor results of $\\sigma$-asymmetric networks:\n1. As mentioned in the Related Work section, previous works examined the influence of parameter symmetries on neural network training by changing the optimization schemes instead of the network parametrization. Even though I find the idea of studying the behavior of asymmetric neural networks interesting, it is not clear to me how this perspective is beneficial compared to the constrained optimization one. Adding a more thorough discussion on that in the Related Work section would improve the paper. \n2. It is not clear from the paper if W-asymmetric and $\\sigma$-asymmetric networks are fully asymmetric or just have fewer symmetries in practical setups. A more accurate discussion on which symmetries are removed by the proposed methods in the experimental setups should be added to the paper. For example, the paper does not cover the normalization layer symmetries, even though layer and batch normalization are used in the experiments. It seems W-asymmetric networks remove normalization symmetries, while $\\sigma$-asymmetric ones do not. It may be one of the reasons why $\\sigma$-asymmetric networks show weaker results in linear mode connectivity and linear interpolation sections. The ReLU scale symmetry is also not adequately discussed.\n3. The analysis of the $\\sigma$-asymmetric networks is limited. There is no proof or discussion of the universal approximation for this method. The paper does not explain poor results on linear mode connectivity and linear interpolation and does not include the results on Bayesian neural networks and metanetworks. A more detailed analysis of the $\\sigma$-asymmetric networks, or at least a discussion of its suboptimal results, would benefit the paper.\n\nAdditionally, I have some minor concerns regarding the experiments:\n1. In the Bayesian neural network experiment, the W-asymmetric networks have fewer parameters than standard ones, which may influence the optimal training hyperparameters. Hence, it may be the case that the difference in the results is due to, e.g., different effective learning rates and not the asymmetric network structure.\n2. In the metanetwork experiments, standard and W-asymmetric networks have different numbers of parameters. Hence, it may be the case that the difference in the results is due to the different dimensionality of the input space and different optimal training hyperparameters for metanetworks and not the asymmetric network structure.\n\nQUESTIONS:\nI would kindly ask the authors to address the concerns from the Weaknesses section and focus on the following questions:\n1. Could you please elaborate on how the asymmetric network perspective differs from the constrained optimization one and in which cases it shows new insights, in your opinion? \n2. Could you please clarify whether the W-asymmetric and $\\sigma$-asymmetric networks are fully asymmetric in the experiments or not? Which of the known symmetries (neuron permutations, ReLU scaling, pre-normalization parameters scaling, etc.) does each method remove in practice?\n3. Could you please comment on the poor results of $\\sigma$-asymmetric networks in linear mode connectivity and linear interpolation experiments? Is there any specific reason why the Bayesian neural networks and metanetworks experiments are not conducted for $\\sigma$-asymmetric networks?\n4. Could you please comment on the technical differences between standard and W-asymmetric networks (optimal hyperparameters and parameter count) in Bayesian neural networks and metanetworks experiments and whether they can affect the conclusions? \n\nMinor questions:\n1. Do I understand correctly that the standard deviation $\\kappa$ used in W-asymmetric and $\\sigma$-asymmetric networks is a hyperparameter tuned separately from the trainable weights? (lines 134 and 163).\n2. Do the non-trainable weights of all W-asymmetric networks have the same values in the metanetwork experiment? Does a metanetwork take only the trainable parameters of W-asymmetric networks as input?\n\nLIMITATIONS:\nThe authors adequately discuss the limitations of the paper in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how removing parameter symmetry of neural networks affects the loss landscape, Bayesian Neural Networks and meta-networks. The authors proposed two ways to remove the parameter symmetry: one is similar to pruning that making some parameters untrainable, the other is to adopt non-elementwise activations. Both ways can remove the permutation symmetry in parameter space. After that, the authors empirically demonstrate that removing permutation symmetry substantially 1) make LMC easier to satisfy 2) make BNN's training more efficient 3) improve the performance of meta-network (I am not quite familiar with meta-network or neural functionals.) 4) make training loss along the line segment between initialization and trained parameters more monotonic.\n\nSTRENGTHS:\n1. The writing is clear and easy to follow. Especially, the structure of this paper is clear: front part is about the two ways to remove parameter symmetry, back part is to demonstrate the effect of removing parameter symmetry on loss landscape, BNN, meta-network and MLI.\n2. This topic is pretty interesting. Permutation symmetry persists in most neural network architectures and impose structure beyond Euclidean structure to parameter space, however, the community has no deep understanding of how permutation symmetry relates to the success of Deep Learning.\n3. The experiments on BNN, meta-networks and MLI are interesting.\n\nWEAKNESSES:\n1. A major issue about this paper is that some findings are not new. Especially, some studies have already demonstrated that asymmetric networks are more easily to satisfy LMC. [Cite 1] showed that pruning solutions are within the same basin and pruned networks can be viewed as a special case of W-asymmetric networks.\n\n[Cite 1] Evci, Utku, Yani Ioannou, Cem Keskin, and Yann Dauphin. \"Gradient flow in sparse neural networks and how lottery tickets win.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 6, pp. 6577-6586. 2022.\n\n2. Another issue is that in experimental part, the four hypotheses seemingly do not relate to each other. There is not a unified conclusion drawn from the experiments.\n3. The most important question about the permutation symmetry remains unresolved. As permutation symmetry holds in most neural network architectures, should we remove permutation symmetry or not? Is the success of deep learning related to the permutation symmetry? This paper seemingly cannot give an insightful answer.\n3. In Sec. 5, despite each experiment are motivated from some simple intuition, there is no rigorous theoretical foundation for each hypothesis, which could potentially lower the value of this study.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}