The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof

Authors: derek  lim
mit  csail
moe  putterman
robin  walters
stefanie  jegelka
t u munich
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:24:12

Abstract: Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries --- transformations of neural network parameters that do not change the underlying neural network function. These include linear mode connectivity, model merging, Bayesian neural network inference, metanetworks, and several other characteristics of optimization or loss-landscapes. However, theoretical analysis of the relationship between parameter space symmetries and these phenonmena is difficult. In this work, we empirically investigate the impact of neural parameter symmetries by introducing new neural network architectures that have reduced parameter space symmetries. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries.  With these new methods, we conduct a comprehensive experimental study consisting of multiple tasks aimed at assessing the effect of removing parameter symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries; for instance, we observe linear mode connectivity between our networks without alignment of weight spaces, and we find that our networks allow for faster and more effective Bayesian neural network training.

Main paper: 
 introduction: 
 Neural networks have found profound empirical success, but have many associated behaviors and phenomena that are difficult to understand. One important property of neural networks is that they generally have many parameter space symmetries -for any set of parameters, there are typically many other choices of parameters that correspond to the same exact neural network function [24] . For instance, permutations of hidden neurons in a multi-layer perceptron (MLP) induce permutations of weights that leave the overall input-output relationship unchanged. These parameter symmetries are a type of (not-necessarily detrimental) redundancy in the parameterization of neural networks, that adds much non-Euclidean structure to parameter space.Parameter space symmetries appear to influence several phenomena observed in neural networks. For example, when linearly interpolating between the parameters of two independently trained networks with the same architecture, the intermediate networks typically perform poorly [59, 13] . However, if we first align the two networks via a permutation of parameters that does not affect the network function, then the intermediate networks can perform just as well as the unmerged networks [59, 1] . In some sense, this suggests that neural network loss landscapes are more convex or well-behaved after removing permutation symmetries. Other areas that parameter symmetries play a role in include interpretability of neurons [20] , optimization [48, 84, 80] , model merging [60] , learned equivariance [7] , Bayesian deep learning [34] , loss landscape geometry [54] , processing neural network weights as input data using metanetworks [39] , and generalization measures [49, 11] .Trained Fixed FiGLUTo rigorously study the effect of parameter symmetries, we study the effect of removing them. In particular, we introduce two ways of modifying neural network architectures to remove parameter space symmetries (see Figure 1 ):(1) W-Asymmetric networks fix certain elements of each linear map to break symmetries in the computation graph. (2) σ-Asymmetric networks use a new nonlinearity (FiGLU) that does not act elementwise, and hence does not induce symmetries such as permutations.These two approaches are inspired by previous work, which shows that both symmetries of computation graphs [39] and equivariances of nonlinearities [20] induce parameter symmetries in standard neural networks. We theoretically prove that both of our approaches remove parameter symmetries under certain conditions. Our Asymmetric networks are similar structurally to standard networks and can be trained with standard backpropagation and first-order optimization algorithms like Adam. Thus, they are a reasonable "counterfactual" system for studying neural networks that are similar to standard neural networks, but that do not have as many parameter symmetries.With our Asymmetric networks, we run a suite of experiments to study the effects of removing parameter symmetries on several base architectures, including MLPs, ResNets, and graph neural networks. We investigate linear mode connectivity, Bayesian deep learning, metanetworks, and monotonic linear interpolation. Through the lenses of linear mode connectivity and monotonic linear interpolation, we see that the loss landscapes of our Asymmetric networks are remarkably more well-behaved and closer to convex than the loss landscapes of standard neural networks. When using our Asymmetric networks as the base model in a Bayesian neural network, we find faster training and better performance than using standard neural networks that have many parameter symmetries. When using metanetworks to predict properties such as test accuracy of an input neural network, we see that all tested metanetworks more accurately predict the accuracy of Asymmetric networks than standard networks. Overall, our Asymmetric networks provide valuable insights for empirical study and hold promise for advancing our understanding of the impact of neural parameter symmetries. 
background and definitions: 
 Let Θ be the space of parameters of a fixed neural network architecture. For any choice of parameters θ ∈ Θ, we have a neural network function f θ : X → Y from an input space X to an output space Y.We call a function ϕ : Θ → Θ a parameter space symmetry if f θ (x) = f ϕ(θ) (x) for all inputs x and parameters θ ∈ Θ (i.e. if f θ and f ϕ(θ) are always the same function).For instance, consider a two-layer MLP with no biases, parameterized by matrices θ = (W 2 , W 1 ) with an elementwise nonlinearity σ. Then f θ (x) = W 2 σ(W 1 x). Let P be a permutation matrix, and let ϕ(θ) = (W 2 P ⊤ , P W 1 ). Then for any input x,EQUATIONso ϕ is a parameter space symmetry. A key step is the second equality, which holds because P σ(x) = σ(P x): any elementwise nonlinearity σ is permutation equivariant. Any other equivariance of σ also induces a parameter symmetry; for instance, if σ(x) = max(0, x) is the ReLU function, then ασ(x) = σ(αx) for any α > 0, so there is a positive-scaling-based parameter symmetry [49, 11, 20] . 
related work: 
 Characterizing parameter space symmetries. While many works spanning several decades have noted specific parameter space symmetries in neural networks [24, 61] , some works take a more systematic approach to deriving parameter space symmetries. Godfrey et al. [20] characterize all global linear symmetries induced by the nonlinearity for two-layer multi-layer perceptrons with pointwise nonlinearities. Zhao et al. [78] study several types of symmetries, and derive nonlinear, data-dependent parameter space symmetries. Lim et al. [39] show that graph automorphisms of the computation graph of a neural network induce permutation parameter symmetries, which captures hidden neuron permutations in MLPs and hidden channel permutations in CNNs.Constraints and post-processing to break parameter space symmetries. Several works develop methods for constraining or post-processing the weights of a single neural network to remove ambiguities from parameter space symmetries. This includes methods to remove scaling symmetries induced by normalization layers or positively-homogeneous nonlinearities such as ReLU [6, 55, 54, 36] , methods to remove permutation symmetries [55, 54, 71, 36] , and methods to remove sign symmetries induced by odd activation functions [71] .Unlike these previous works, we develop neural network architectures that have reduced parameter space symmetries. Our models are optimized using standard unconstrained gradient-descent based methods like Adam. Hence, our networks do not require any non-standard optimization algorithms such as manifold optimization or projected gradient descent [6, 55] , nor do they require post-trainingprocessing to remove symmetries or special care during analysis of parameters (such as geodesic interpolation in a Riemannian weight space [54] ). These methods based on constraining weights or post-processing have significantly different optimization and loss landscape properties (for instance, linear interpolation is not even well-defined on general nonlinear parameter manifolds), so they are less suitable than our Asymmetric networks for studying phenomena that may generalize to standard neural networks.Aligning multiple networks for relative invariance to parameter symmetries. One way to reduce the impact of parameter symmetries in certain settings, especially for model merging, is to align the parameters of one network to another. Several methods have been proposed for choosing permutations that align the parameters of two neural networks of the same architecture, via efficient heuristics or learned methods [4, 69, 63, 13, 1, 53, 47, 67] . Other approaches relax the exact permutation-parameter-symmetry constraint or do additional postprocessing to achieve effective merging of models in parameter space [59, 28, 29, 60, 56] . As our Asymmetric networks have removed parameter symmetries, they can often be successfully merged and linearly interpolated between without any alignment. 
asymmetric networks: 
 We develop two methods of parameterizing neural network architectures without parameter symmetries, both of which are justified by theoretical results. We can prove that our σ-Asym networks have permutation and scale symmetries removed, and that our W-Asym networks have permutation symmetries removed. Although we have not formally proven that W-Asym networks have scale symmetries removed, we believe that they do (intuitively, the fixed weights fix a scale).We first focus on the case of fully-connected MLPs with no biases, which take the formf θ (x) = W L σ(W L-1 • • • σ(W 1 x))for weights θ = (W L , . . . , W 1 ) and nonlinearity σ. Then in Section 4.3, we discuss how we use these approaches to remove parameter symmetries in other architectures (e.g. CNNs, GNNs) as well. : Depiction of our W-Asymmetric approach to removing parameter symmetries. Entries with a black outline are untrained. Note that the W-Asym linear map has 2 nonzeros per row, the W-Asym convolution with fixed entries has 8 fixed entries for its single output channel, and the W-Asym convolution with fixed filters has a single input filter fixed. We often use a constant number of fixed entries per row or output channel in our experiments.Our first approach to developing neural networks with greatly reduced parameter space symmetries relies on their computation graph. In particular, we can write a feedforward neural network architecture as a DAG G = (V, E) with neurons as nodes V and connections between them as edges E.For a choice of parameters θ ∈ R |E| , we get a function f θ from input neuron space to output neuron space [19, 49, 39] Lim et al. [39] showed that neural DAG automorphisms ϕ, which are graph automorphisms of the DAG G that preserve types of nodes and weight-sharing constraints, induce permutation parameter symmetries ϕ that leave the function unchanged: f θ = f ϕ(θ) . Thus, any feedforward neural network architecture that has no permutation parameter symmetries must necessarily have a computation graph with no nontrivial neural DAG automorphisms.To modify MLPs so they have no nontrivial neural DAG automorphisms, we mask edges in the computation graph, by setting certain edge weights to constant values that are not updated during training. For an MLP, we can do this by enforcing that every linear layer T : R d1 → R d2 takes the form of a matrix W ∈ R d2×d1 , where each row has a unique pattern of untrained weights. To achieve this, define a mask M ∈ {0, 1} d2×d1 such that W ij is a trainable parameter if and only if M ij = 1, and the rows of M are pairwise distinct binary vectors in {0, 1} d1 . We call any neural network with linear maps masked as such a W-Asymmetric neural network. In Appendix B.1, we show that masking these entries so that they are not trained is sufficient to remove all nontrivial neural DAG automorphisms.Theorem 1. If each mask matrix M has unique nonzero rows, then W-Asymmetric MLPs with fixed entries set to zero have no nontrivial neural DAG automorphisms.In practice, we generate a binary mask M by randomly selecting a subset of n fix fixed elements for each row. For the fixed entries, we sample them from a normal distribution N (0, κI) with standard deviation κ > 0 that is a hyperparameter that we tune. Our asymmetric linear layer can be written asEQUATIONwhere W ∈ R d2×d1 is a matrix of trainable parameters, and F ∈ R d2×d1 is a matrix of fixed elements, sampled from N (0, κI). The only trainable parameters are the unmasked entries of M ⊙W, of which there are d 2 • (d 1 -n fix ). We empirically find that having κ be significantly larger than the standard deviation of typical initializations for weight matrices (e.g. κ = 1 while the trained coefficients have standard deviation about 1/ √ 1000) is important for breaking parameter symmetries.Another approach for removing parameter symmetries is to change the nonlinearity. As studied by Godfrey et al. [20] , equivariances of the nonlinearity induce parameter symmetries in MLPs with elementwise nonlinearities. Recall that an elementwise nonlinearity acts by using the same function on each coordinate of the input; σ : R d → R d is elementwise if it takes the form σ(x) = (σ 1 (x 1 ), . . . , σ 1 (x d )) for some real function σ 1 : R → R. Any elementwise nonlinearity is permutation equivariant, and hence induces a permutation parameter symmetry.Thus, in contrast to most neural network architectures, for Asymmetric networks we must use a nonlinearity that does not act elementwise. Likewise, the nonlinearity cannot have any linear symmetry itself, since if σ • A = B • σ for A, B ∈ GL(d), then for a two-layer network:EQUATIONSo (W 2 , W 1 ) and (W 2 B -1 , AW (x) = W 2 σ(W 1 x). If σ has no linear equivariances, then f θ1 = f θ2 if and only if θ 1 = θ 2 .In other words, there are no nontrivial parameter space symmetries.Motivated by Proposition 1, we define a non-elementwise nonlinearity that does not have the equivariances of standard nonlinearities. Letting η be the sigmoid function η(x) = 1 1+e -x , we define our nonlinearity asEQUATIONfor a randomly sampled, untrained matrix F. Similarly to W-Asym nets, we sample F as an i.i.d Gaussian matrix with variance that we tune. This nonlinearity is similar to Swish / SiLU [57, 26] with an additional matrix F to mix feature dimensions (to break permutation equivariance), and it is also similar to a gated linear unit (GLU) with no trainable parameters [9] . Thus, we call our nonlinearity FiGLU: the Fixed Gated Linear Unit.In Appendix B.2.1, we prove that FiGLU does not have permutation equivariances or diagonal equivariances, which are the only equivariances for most elementwise nonlinearities [20] .Proposition 2. With probability 1 over the sampling of F, FiGLU has no permutation equivariances or diagonal equivariances.We call any network with our symmetry-breaking FiGLU nonlinearity a σ-Asymmetric Network.The graph-based approach (W-Asymmetric Networks) works naturally for neural network architectures with "channel" dimensions, such as convolutional neural networks (CNNs), graph neural networks (GNNs) [22] , Transformers [66] , and equivariant neural networks based on equivariant linear maps [16] . In these types of networks, permutations of entire channels induce permutation parameter symmetries [39] . For such networks, we thus mask entire connections between channels, e.g. entire filters in CNNs. For CNNs, we also experiment with randomly masking some number of entries in each filter (instead of masking entire filters), and find that this also works well in removing parameter symmetries. For neural networks with linear layers that include bias terms, we do not modify the biases in any way, as they do not introduce new computation graph automorphisms [39] .The nonlinearity-based approach (σ-Asymmetric Networks) can be straightforwardly applied to many general architectures as well. Though, the fixed matrix F may have to be changed to a structured linear map; for instance, in CNNs we take F to be a 1D convolution. Our two approaches remove parameter symmetries from standard neural networks, but still intuitively retain much of the structure of standard networks. One important property of widely-used neural network architectures is universal approximation -for any target function of a certain type, there exists a neural network of the given architecture that approximates the target to an arbitrary accuracy [8, 25, 42, 75] . In Appendix B.3, we show that W-Asymmetric MLPs retain this property: 4 ), where n is the hidden dimension, W-Asymmetric MLPs are Universal Approximators with probability 1 over the choice of hardwired entries.EQUATIONWe have not been able to prove a similar result for σ-Asymmetric networks. Classical universal approximation results for standard neural networks do not apply to σ-Asym nets, as they tend to assume elementwise nonlinearities. 
linear mode connectivity without permutation alignment: 
 Background. Under certain conditions, neural networks have been found to exhibit linear mode connectivity, which is when all networks on the line segment in parameter space between two well-performing trained networks are also well-performing. Starting with Frankle et al. [18] , who coined the term and provided the first in-depth analysis, many works have studied this phenomenon [44, 13, 76, 14] . When the two networks are randomly initialized and trained independently, linear mode connectivity generally does not hold [13, 1] . However, if one of the two networks is permuted with a parameter symmetry that does not change its function, but that aligns its parameters with the other network, then linear mode connectivity empirically and theoretically holds for many more model / task combinations [13, 1, 79, 14] . In fact, Entezari et al. [13] conjectures that if all permutation symmetries are accounted for, then linear mode connectivity generally holds. Since our Asymmetric networks remove parameter space symmetries, we may expect linear mode connectivity to hold, without any post-processing or alignment step.Hypothesis. Asymmetric networks are more linearly mode connected than standard networks, and do not require post-processing or alignment of pairs of networks before merging.Experimental Setup. We consider several networks and tasks: MLPs on MNIST, ResNets [23] on CIFAR-10, and Graph Neural Networks [73] on ogbn-arXiv [27] . For each architecture and task, we compute the midpoint test loss barrier:L( 1 2 θ 1 + 1 2 θ 2 ) -1 2 (L(θ 1 ) + L(θ 2 )). This measures how much worse the interpolated network with parameters 1 2 θ 1 + 1 2 θ 2 is than the original networks with parameters θ 1 and θ 2 . We measure this barrier for standard networks, pairs of networks aligned with Git-Rebasin [1] , and networks with our two approaches (σ-Asym and W-Asym) applied. To be clear, whenever we interpolate between the weights of two Asymmetric networks, they have the same exact fixed weights F (for both W-Asym and σ-Asym) and the same exact mask M (for W-Asym).Results. Figure 3 plots interpolation curves and Table 1 displays midpoint test loss barriers of various methods. Our σ-Asymmetric approach lowers the test loss barrier compared to standard networks, but falls short of the alignment approach of Git-Rebasin. On the other hand, our W-Asymmetric approach achieves strong (and sometimes perfect) interpolation, and interpolates better than standard networks aligned via Git-ReBasin. This may be caused by failure of the Git-ReBasin approaches to find the optimal permutations, importance of other parameter symmetries besides layer-wise permutations, or other properties of W-Asymmetric networks. ( 1 2 θ 1 + 1 2 θ 2 ) -1 2 (L(θ 1 ) + L(θ 2 )) . We use different methods of breaking symmetries in each column; from left to right: no symmetry breaking, Git-Rebasin [1] , our σ-Asym approach, and our W-Asym approach. We report mean and standard deviation of the barrier across at least 5 pairs of networks, and bold lowest barriers. Background. Bayesian deep learning is a promising approach to improve several deficits of mainstream deep learning methods, such as uncertainty quantification and integration of priors [30, 51] . However, parameter symmetries are problematic in Bayesian neural networks, as they are a major source of statistical nonidentifiability [30] . Parameter symmetries introduce modes in the posterior p(θ|D) that make the posterior harder to approximate [2, 34, 72] , sample from [50, 71] , and otherwise analyze [36] . For instance, one common technique for training Bayesian neural networks is variational inference via fitting a Gaussian distribution to the true posterior p(θ|D). This approach suffers because the Gaussian distribution has only one mode, whereas the true posterior has at least one mode for every parameter symmetry. As such, some approaches treat kernel matrices are random variables, which has less symmetries than treating features or weights as random variables, and allows better approximation by unimodal posteriors [74, 43] . Instead, we consider traditional, commonly-used Bayesian deep learning techniques applied on the features or weights of our Asymmetric networks. Hypothesis. Using Asymmetric networks as the base model improves Bayesian neural networks, as the posterior will have less modes.Experimental setup. We train Standard Bayesian and Asymmetric Bayesian Networks for image classification using variational inference. We use the method of [64] for variational inference, which fits a Gaussian approximate posterior with a diagonal plus low-rank covariance. We train 10 instances of each model and then report train loss, test loss, test accuracy, and Expected Calibration Error (ECE) [45] , which is a measure of calibration.Results. See training curves in Figure 4 , and quantiative results in Table 3 : Metanetwork performance for predicting the test accuracy of small ResNets and our W-Asym ResNets. Each row is a different metanetwork. Reported are R 2 and Kendall τ on the test set -higher is better.ResNet W-Asym ResNet R 2 τ R 2 τMLP .330 ± .04 .389 ± .03 .594 ± .12 .864 ± .01 DMC [12] .950 ± .01 .787 ± .02 .967 ± .01 .911 ± .01 DeepSets [77] .855 ± .01 .617 ± .03 .936 ± .00 .858 ± .00 StatNN [65] .976 ± .00 .866 ± .00 .978 ± .00 .935 ± .01Background. Metanetworks [39] -also referred to as deep weight-space networks [46, 58] , meta-models [35] , or neural functionals [81, 82, 83] -are neural networks that take as inputs the parameters of other neural networks. Recent work has found that making metanetworks invariant or equivariant to parameter-space symmetries of the input neural networks can substantially improve metanetwork performance [46, 81, 39, 32] .Hypothesis. Asymmetric networks are easier to train metanetworks on because they do not have to explicitly account for symmetries.Experimental setup. We experiment with metanetworks on the task of predicting the CIFAR-10 test accuracy of an input image classifier, which many metanetworks have been tested on [65, 12, 81, 39] . We use metanetworks based on simple MLPs, 1D-CNN metanetworks [12] , and metanetworks that are exactly invariant to permutation parameter symmetries: DeepSets [77] and StatNN [65] . We train two separate datasets of 10,000 image classifiers: one dataset of small ResNet models, and one dataset of W-Asymmetric ResNet models. More information on the data, metanetworks, and training details are in Appendix F.3.In Table 3 , we see that metanetworks are signficantly better at predicting the performance of our W-Asymmetric ResNets than standard ResNets. Interestingly, simple MLP metanetworks, which view the input parameters as a flattened vector, can predict the test accuracy of Asymmetric Networks quite well, but fail on standard networks. Also, the permutation equivariant metanetworks (DeepSets and StatNN) both improve on W-Asym ResNets compared to on ResNets, even though the permutation symmetries of standard ResNets do not affect these metanetworks; thus, it may be possible that other symmetries in standard ResNets (but not Asym-ResNets) harm metanetwork performance, or they may be other factors besides symmetries that improve metanetwork performance for Asym-ResNets. Background. One common method of studying the loss landscapes of neural networks is by studying the one-dimensional line segment of parameters attained by linear interpolation between parameters at initialization and parameters after training. Many works have observed monotonic linear interpolation (MLI), which is when the training loss monotonically decreases along this line segment [21, 17, 41, 68] . Loss landscapes of convex problems have this property as well, so presence of the monotonic linear interpolation property has been used as a rough measure of how well-behaved the loss landscape is. However, with many types of models, tasks, or hyperparameter settings, monotonic linear interpolation does not hold [41, 68] , or there is a large plateau where the loss barely changes for much of the line segment [17, 70] ; neither of these properties can happen for convex objectives trained to completion. To the best of our knowledge, there has been little work on the role of parameter symmetries -or lack thereof -in monotonic linear interpolation (besides one minor experiment in [41] Appendix C.9). Nonetheless, since removing parameter symmetries substantially improves linear interpolation between trained networks (Section 5.1), one may expect removing parameter symmetries to improve monotonic linear interpolation.Hypothesis. The training loss along the line segment between initialization and trained parameters is more monotonic and convex for Asymmetric networks.Experimental setup. For the learning task, we follow the setup used for creating the dataset of image classifiers in Section 5.3. In particular, we train 300 standard ResNets and 300 W-Asymmetric ResNets with varying hyperparameters sampled from the same distributions as used for the dataset of image classifiers (see Appendix Table 13 ). For each of these networks, we linearly interpolate between its initial parameters θ 0 and its final trained parameters θ T : (1 -α)θ 0 + αθ T for 25 uniformly spaced values 0 = α 1 < α 2 < . . . < α 25 = 1. To measure monotonicity, we record the maximum increase between adjacent networks ∆ = max(L(α i+1 ) -L(α i )), and the percentage of networks that have ∆ ≤ 0 i.e. the percentage of networks that satisfy monotonic linear interpolation. To measure convexity, we consider a local convexity measure (the proportion of α i where the centered difference second derivative approximation is nonnegative) and a global convexity measure (the proportion of α i such that L(α i ) lies below the line segment between the endpoints, i.e.L(α i ) ≤ (1 -α i )L(0) + α i L(1)).Results. Table 4 shows the measures of monotonicity and convexity for standard, σ-Asymmetric, and W-Asymmetric ResNets. Remarkably, every single one of the 300 W-Asymmetric ResNets satisfies monotonic linear interpolation and has a trajectory that lies underneath the line segment between the endpoints. Qualitatively, we can see in Figure 5 that W-Asymmetric ResNets do not have any clear loss barriers from initialization, nor any loss plateaus that indicate nonconvexity. In contrast, the majority of standard ResNets have non-monotonic trajectories, and the monotonic trajectories seem to be more nonconvex. σ-Asymmetric network trajectories are signficantly more convex and monotonic than standard network trajectories, but there are some non-monotonic or nonconvex trajectories still.In Appendix A, we note other interesting differences in optimization and loss landscape properties of Asymmetric and standard neural networks. These can be summarized as:1. Even though Asymmetric networks interpolate much better than standard networks, the parameters of trained Asymmetric networks are often basically the same distance away from each other in weight space as standard networks. 2. Asymmetric networks do not tend to overfit as much: the difference in train performance and test performance can be substantially lower than that of standard networks. 3. Asymmetric networks can take longer to train, especially when choosing hyperparameters that make them more dissimilar to standard networks. 
discussion: 
 While many properties of Asymmetric networks are in line with our hypotheses and intuition about the impact of removing parameter symmetries, there are many unexpected effects and unanswered questions that are promising to further investigate. For instance, we did not extensively explore Asymmetric networks in the context of model interpretability, generalization measures in weight spaces, or optimization improvements, all of which are known to be influenced to some extent by parameter symmetries. Further studying the properties in Section 5.5, the dependence of behavior on the choices of Asymmetry-inducing hyperparameters, and other design choices in making networks asymmetric could also bring more insights into parameter space symmetries.Also, it is interesting that our σ-Asymmetric networks do not appear to break parameter symmetries as well as our W-Asymmetric networks. We have run preliminary empirical tests on several variants of σ-Asym networks, such as: σ • F • σ as the nonlinearity, sparsifying F, adding instead of multiplying the gate, using cosine instead of sigmoid, squaring instead of using sigmoid, and putting a LayerNorm in the nonlinearity. However, none of these approaches worked well. We believe that such failures may be because σ-Asymmetry breaks symmetries at the activation / neuron level, whereas W-Asymmetry breaks symmetries in the larger space of weights (for more evidence, see Appendix E, where we show that fixing biases at neurons also fails to effectively remove symmetries). These curiosities provide interesting directions for future work.All in all, we believe that future theoretical and empirical study of Asymmetric networks could garner many insights into the role of parameter symmetries in deep learning. We see that more-asymmetric networks need more time to train.There are several other interesting differences in the optimization and loss landscape properties of Asymmetric and standard neural networks. For one, even though Asymmetric networks generally interpolate significantly better than standard networks, this cannot be seen by measuring distances in parameter space. For instance, in GNN experiments following the setup of Section 5.1, pairs of standard GNNs have a distance per parameter of .000174 on average, whereas W-Asymmetric GNNs have .000159, which is only slightly lower. However, the average test loss barrier is 1.448 for standard GNNs while it is only 0.069 for W-Asymmetric GNNs. Likewise, in our datasets of 10,000 standard and W-Asymmetric ResNets, the average distance per parameter between the weights of trained standard classifiers is .0034, which is actually lower than the distance per parameter of .0051 for W-Asymmetric ResNets (estimated on 20,000 pairs of networks). Thus, although we sometimes imagine well-interpolating pairs of networks to lie in the same local basin of parameter space, W-Asymmetric networks are actually rather far apart in parameter space, but nonetheless have linear segments of low loss between them.We also find that Asymmetric networks often do not overfit as much as standard networks. For instance, in the GNN setup of Section 5. Further, in Figure 6 , we see that training speed is slower for W-Asymmetric ResNets when we increase the amount of asymmetry (by increasing the number of fixed entries and the standard deviation of the fixed entries). While standard ResNets take on average 11 epochs to reach 70% training accuracy on CIFAR-10, W-Asymmetric ResNets with the most extreme hyperparameters take up to 72 epochs.Here, we prove that as long as each mask matrix M in our W-Asymmetric MLPs with fixed entries set to zero has unique nonzero rows, then our architecture has no nontrivial neural DAG automorphisms.In practice, we find that setting the standard deviation κ of the fixed entries F to be positive (and in fact orders of magnitude larger than the standard deviation that we typically initialize trainable weights with) is important to achieve properties such as linear mode connectivity that Asymmetric networks have but standard networks do not have. When κ = 0 (i.e. when fixed entries are set to zero), we can directly work in the framework of Lim et al. [39] that connects parameter symmetries to computation graph automorphisms. To work towards generalizing our result to κ > 0, we would have to modify the definitions and results of Lim et al. [39] ; for instance, we would need to add edges associated to untrainable parameters in the computation graph, and redefine the concept of neural DAG automorphisms. We leave such exploration to future work. Theorem 3. If each mask matrix M has unique nonzero rows, then W-Asymmetric MLPs with κ = 0 have no nontrivial neural DAG automorphisms.Proof. Consider an L-layer W-Asymmetric MLP with fixed entries set to zero. Denote its weights as W L , . . . , W 1 and the corresponding binary masks as M L , . . . , M 1 . The forward pass of such a network on an input x is thenEQUATIONfor some elementwise nonlinearity σ. The dimension ofW i is d i × d i-1 .In the framework of Lim et al. [39] , this is a feedforward neural network with a computation graph defined as follows.The node set isV 0 × V 1 × . . . × V L, where V i has d i nodes, and no nodes are shared between different V i . If a node v is in V i , then we say that layer(v) = i. V 0 contains the input nodes and V L contains the output nodes. The adjacency matrix can be written asEQUATION)Every block besides the ones containing masks is zero. There are L + 1 × L + 1 blocks, and the (i, j) block is of sized i × d j .Recall that a neural DAG automorphism is a relabelling of nodes τ : V → V such that τ is bijective, (i, j) ∈ E if and only if (τ (i), τ (j)) ∈ E, and every input node and output node is a fixed point of τ . Now, let τ : V → V be a neural DAG automorphism. Further, let P be the corresponding permutation matrix. We will show that τ is the identity, i.e. that P = I. By Lemma 1, we know that τ preserves layer number of nodes, meaning layer(τ (i)) = layer(i). Thus, P is a block diagonal permutation matrix:EQUATIONwhere P i is d i × d i . Morever, P 0 = I and P L = I because input nodes and output nodes are fixed points. Applying this to the adjacency matrix, we see thatτ (A) = P AP ⊤ =      0 P 1 M 1 P ⊤ 0 0. . .EQUATION)Since τ is a neural DAG automorphism, we have that τ (A) = A. Equating blocks, this means thatP 1 M 1 P ⊤ 0 = M 1 . As P 0 = I, we have P 1 M 1 = M 1 .But M 1 has unique rows, so P 1 = I as well.For the inductive step, assume P i = I for some i. Then P i+1 M i+1 P ⊤ i = P i+1 M i+1 = M i+1 , so since M i+1 has unique rows, we have that P i+1 = I. As this holds for any i by induction, this means that P = I, so τ is a trivial neural DAG automorphism and we are done.Lemma 1. Neural DAG automorphisms preserve layer number in W-Asymmetric MLPs that have masks with nonzero rows.Proof. Let τ be a neural DAG automorphism. This means that P AP ⊤ = A, where P is the permutation matrix associated to τ . Then, using P AP ⊤ = A for the first equality and the definition of P in the second, we have thatEQUATIONWe proceed by induction on layer number l. For any input node i we know that τ (i) = i, so of course layer(τ (i)) = layer(i).Now, suppose that layer(τ (i)) = layer(i) for any i in layer l ≥ 1. If node j is in layer l + 1, then there is some i in layer l such that (i, j) ∈ E because M l+1 has no nonzero rows. We have that A τ (i),τ (j) = A i,j , so τ (i) is connected to τ (j). As we know that τ (i) is in layer l, we have that τ (j) is in layer l + 1.Proposition 3. Let the parameter space Θ be all pairs of square invertible matrices θ = (W 2 , W 1 ) for W 2 , W 1 ∈ GL(d), and let f θ (x) = W 2 σ(W 1 x). If σ has no linear equivariances, thenf θ1 = f θ2 if and only if θ 1 = θ 2 .In other words, there are no nontrivial parameter space symmetries.Proof. If θ 1 = θ 2 , then clearly f θ1 = f θ2 . For the other direction, suppose f θ1 = f θ2 , and denote θ 1 = (W 2 , W 1 ) and θ 2 = ( W 2 , W 1 ). Then for any input z ∈ R n , we haveEQUATIONEQUATIONNow, choose an arbitrary x ∈ R n . We let z in the above equation ( 11) be W -1 1 x, so we haveEQUATIONThis holds for any x, soW -1 2 W 2 • σ = σ • W 1 W -1 1 , i.e.we have found a linear equivariance of σ. Since σ has no linear equivariances,EQUATIONmeaning that W 2 = W 2 and W 1 = W 1 , i.e. θ 1 = θ 2 , so we are done.Now, we study the properties of our FiGLU nonlinearity σ(x) = η(Fx) ⊙ x, where η is the sigmoid function η(x) = 1 1+e -x . For proving Proposition 2, we want to prove that with probability 1 over samples of F, σ has no permutation or diagonal equivariances.We say that σ has no permutation equivariances if whenever P 2 • σ = σ • P 1 for permutation matrices P 1 and P 2 , then P 1 = P 2 = I. Likewise, we say that σ has no diagonal equivariances if whenever B • σ = σ • A for invertible diagonal matrices A and B, then A = B = I.We will show that these two properties hold for any F that has no permutation symmetries and no zero entries. We say that F has no permutation symmetries if P 2 FP 1 = F for permutation matrices P 1 and P 2 implies that P 1 = P 2 = I. Note that if F has distinct entries, then it has no permutation symmetries. Thus, F satisfies both of these conditions with probability 1, since the set of matrices with nondistinct entries or with at least one zero entry are of Lebesgue measure zero, so they have zero probability under the Gaussian distribution. We now proceed to show that σ has no permutation or diagonal equivariances under these conditions on F. Proposition 4. If F is a square matrix with no permutation symmetries, then σ(x) = η(Fx) ⊙ x has no permutation equivariances. Proof. Suppose σ • P 1 = P 2 • σ for permutation matrices P 1 , P 2 . We will show that P 1 = P 2 = I. For any input x, we have η(FP 1 x) ⊙ P 1 x = P 2 η(Fx) ⊙ x (14)EQUATIONEQUATIONwhere we used permutation equivariance of η, which acts elementwise. Let x = e i , the standard basis vector that is 1 in the ith coordinate and 0 elsewhere. If i is not a fixed point of the permutation P ⊤ 2 P 1 , then let j be the index that it is mapped to. Then equation (16) gives thatEQUATIONEQUATIONIn the ith coordinate of this equality of vectors, we see that η(Fe i ) = 0, which is impossible, since η is the sigmoid function. Thus, i cannot be a fixed point of P ⊤ 2 P 1 , so P ⊤ 2 P 1 = I is the identity permutation. Now, let x be an arbitrary vector with no zero entries. Equation (16) gives thatEQUATION) Since x is nonzero, we can divide by x i in the ith coordinate of this vector equality for each i to get thatEQUATIONEQUATIONBecause this holds for all x with no zero entries (and in particular for a basis of the input space), we know thatEQUATIONas matrices. But since F has no permutation symmetries, we have that P 1 = P 2 = I, so we are done.Proposition 5. If F is a square matrix with no zero entries, then σ(x) = η(Fx) ⊙ x has no diagonal equivariances.Proof. Let A = Diag(α) and B = Diag(β) be invertible diagonal matrices, and suppose that σ • A = B • σ. We will show that A = B = I. For any input x, we haveEQUATIONLet x = ce i , where e i is the ith standard basis vector and c ̸ = 0 is any nonzero number. Then η(Fcα i e i ) ⊙ cα i e i = β ⊙ η(cFe i ) ⊙ ce i .(24) At the ith coordinate of this equality, we haveEQUATIONThus, the right hand side is constant in c. We must have that α i > 0, because if not, then increasing c would increase either the numerator or denominator and decrease the other, hence contradicting the equality (here we use that F has no zero entries, so cFe i is nonzero in every entry). Thus, letting c → ∞, we see that αi βi = 1, so α i = β i . Plugging this back into Equation ( 26), we haveEQUATIONwhere in the third line we used the fact that η is invertible. We have shown that α i = β i = 1 for each i, so A = B = I and we are done.We note that the proofs of these two results about FiGLU are reminiscent of some proof techniques from Godfrey et al. [20] , such as those used in their analysis of GELU nonlinearities.Here, we prove the universal approximation result for our W-Asymmetric MLPs.Theorem 4. Let η be any nonpolynomial elementwise nonlinearity with η(x) -η(-x) = x (e.g. ReLU, GELU, swish), let Ω ⊆ R D be a compact domain, and let f target : Ω → R be a continuous target function. Fix ε > 0 and δ > 0.There exists a width n ′ such that for all n > n ′ , with probability 1-δ, for a randomly sampled 4-layer W-Asymmetric MLP f with η nonlinearity, hidden dimensions 24n → n → 24n, and n fix ∈ o(n 1/4 ) hardwired entries per neuron, there will exist θ ∈ Θ such that the W-Asymmetric MLP f : R n → R approximates f target to ε:EQUATIONImportantly, we require that the input to f θ be padded with n -D zeroes, so [x; 0] ∈ R n .To approximate f target to ε > 0, we will leverage the universal approximation for standard MLPs with nonlinearity η to first obtain a standard 2-layer MLP that approximates f target to within ε, meaning f target (x) -f MLP (x) < ε for all x ∈ Ω. Then we will exactly represent f MLP using an Asymmetric Network f . This will be done by approximating each linear map W of f MLP by two layers of an Asymmetric network:W ′ 2 • η • W ′ 1 = W for Asymmetric linear maps W ′ 2 and W ′ 1 .For the sake of exposition, we will show how to do this first when both W ′ 2 and W ′ 1 have no Asymmetric mask (i.e. fitting a linear map W using a standard two-layer η-MLP), then when only W ′ 2 has an Asymmetric mask, and finally when both W ′ 2 and W ′ 1 have an Asymmetric mask.Let W ∈ R n×n be the target linear map, and let B ∈ R 2n×n and A ∈ R n×2n be parameters of a two-layer MLP, defined by f A,B (x) = Aη(Bx). We will choose A and B such that f A,B (x) = W x for all x ∈ R n .Denote the ith row of W by W i , so thatEQUATIONWe set A and B as follows, where I n is the n × n identity matrix:EQUATION)Then we can see that f A,B exactly computes the linear transformation W x.EQUATIONLet n fix > 0 and let each row of N ∈ {0, 1} n×6n have n fix entries equal to 0, selected at random. Let B ∈ R 6n×n and A ∈ R n×6n . Define A ′ to be an Asymmetric linear map:A ′ = A⊙N+(1-N)⊙P,where N is a randomly sampled binary mask, and P a randomly sampled Gaussian matrix. We consider a two-layer network with one Asymmetric and one standard linear map: f A,B (x) = A ′ η(Bx). We want f A,B (x) = W x for all x. For the remainder of this proof, we will assume that N never has three consecutive entries in a row set to zero; we will later show that this holds with high probability over the sampling of N.First, we define B in a similar way to the purely linear setting, but with additional copies of entries to allow for error correction of the random noisy entries fixed in A ′ .EQUATION)Recall that each row A ′ i of A ′ has n fix entries that are randomly hardwired to constants. Ideally, we would wantA ′ i to pick out η(W i • x) -η(-W i • x) = W i •x, but because of the hardwired constants, A ′ i might randomly add c * η(W j • x). However, since there are three copies of η(W j • x) in η(Bx), as long as not all three corresponding entries in A ′ i are fixed, one of the un-fixed copies can be changed such that the coefficients sum to 0. Since by our assumption N never has three consecutive entries all set to 0, the coefficients of A can be picked such that A ′ η(Bx) = W x. For example, a possible A ′ matrix would beA ′ =   1 0 0 -1 0 0 | 1.1 .37 -1.47 0 0 0 | . . . | 0 0 0 0 0 0 .89 -.89 0 0 0 0 | .37 .63 0 -1 0 0 | . . . | 0 0 0 0 0 0 0 0 0 0 0 0 | 0 0 0 0 0 0 | . . . | 1 0 0 -1 0 0  Thus we have shown that under the assumption that N never has three consecutive entries equal to 0, A can be picked such that A ′ η(Bx) = W x. We will now show that P(N never has three consecutive entries equal to 0) can be made arbitrarily small by increasing the width n while keeping n fix to be o(n 1/3 ).The probability there are three consecutive entries in a given row of N that are zero is O(n 3 fix n 2 ). By the union bound, the probability that any row has 3 consecutive hardwired entries is O(n 3 fix n ). For any n fix ∈ o(n 1/3), this tends towards 0. Thus, with probability ≥ 1 -O(n 3 fix n ), A can be picked such that A ′ η(Bx) = W x.Once again let n fix > 0, and let each row of M have n fix entries equal to 0, selected at random. Let B ∈ R 24n×n and A ∈ R n×24n . Further, define Asymmetric mapsEQUATION)where N, M are randomly sampled masks, and P, Q are normal matrices. Then we let f A,B (x) = A ′ η(B ′ (x)), and we once again desire choices of parameters A and B such that f A,B (x) = W x.Consider the randomly drawn mask M ∈ {0, 1} 24n×n , and denote the ith row by M i .EQUATIONwhere M i ∈ {0, 1} n . We partition M's rows into n blocks of 24 rows. β 1 = {M 0 . . . M 23 }, . . . β i = {M 24i . . . M 24i+23 }. Now, consider β 1 , the first 24 rows of M. Definition B.1. We say two rows M j , M k are intersecting if there is some column index α such that M j,α = M k,α = 0. That is, two rows of are intersecting if they share a 0 at the same index.Note that for any two given rows of M , the probability that they share a 0 in the same location is≤ n 2 fixn . We assume that β i contains no more than one pair of intersecting rows; later, we show this to hold with high probability. Then, every β i can be broken into two disjoint sets of 12 rows, β i,0 and β i,1 , such that neither set of 12 contains a single pair of intersecting rows. Intuitively, this means that each row in B corresponding to β i,0 will have unique fixed indices. Our goal will be for the rows in β i to mimic the row W i . We will show how to do this for each i. Fix an arbitrary index i ∈ {0, . . . , n -1}.Without loss of generality, assume β i,0 and β i,1 are continguous, so β i,0 = M 24i:24i+11 and β i,1 = M 24i+11:24i+23 . By our assumption, for j, k ∈ β i,0 (i.e. j, k ∈ {0, . . . , 11}), the mask rows M 24i+j and M 24i+k are never 0 in the same two column indices. Similarly, for j, k ∈ β i,1 (i.e. j, k ∈ {12, . . . , 23}), the mask rows M 24i+j and M 24i+k are never 0 in the same two column indices.Next, we define c i,j as the difference between B ′ and B in the (24i + j)th row:EQUATIONIn particular, we have thatEQUATION)Lemma 2. For any indices j ̸ = k such that j, k ∈ β i,0 or j, k ∈ β i,1 , we have thatEQUATIONProof. By the definition of c i,j , we know that c i,j is only nonzero at indices where M 24i+j is equal to zero. Since j, k are either both in β i,0 or β i,1 , we know that M 24i+k cannot also be zero at indices where M 24i+j is zero. Thus, M 24i+k is equal to 1 at every index where c i,j is nonzero, so c i,j ⊙ M 24i+k = c i,j as desired.Next, we construct B, by constructing this block of 24 rows. Let [c i,0 , c i,1 , c i,2 , c i,3 ] be continguous sums of length-3 segments of c i,: :EQUATIONEQUATIONEQUATIONWe assign the first 12 rows of B as follows.(0EQUATION)With our above construction, each block of the 24 rows inEQUATIONImportantly, each row here is n wide. Recall thatA ′ ∈ R n×24n . Denote the ith row of A ′ by A ′ i ∈ R 24n with A ′ i ∈ R 24n . If A ′ had 0 hardwired entries, then setting A i = 1 24i -1 24i+3 - (1 24i+6 -1 24i+9 ) would give A i η(B ′ • x) = W i • x, by the same argument as in Appendix B.3.2.Unfortunately this is not the case, so we have to use the construction in Appendix B.3.3. Recall, that A ′ has n fix fixed entries in each row. This means that N i has n fix entries equal to 0. Since every entry of B ′ x has three copies, as long as N i does not have three elements set to 0 in a row, A ′ i can be madeequivalent to A i = 1 24i -1 24i+3 -(1 24i+6 -1 24i+9). This is because, as in Appendix B.3.3, if at most 2 elements out of any 3 three copies are hardwired, then the third can be changed arbitrarily to offset the hardwiring.Further, just as in Appendix B.3.3, the probability that a given row of A ′ has three items hardwired in a row is O(n 3 fix n 2 ). Thus, by the union bound, the probability that some row of A ′ has three items hardwired in a row is O(n 3 fix n ). So, with large enough width n, A can be chosen such that A ′ η(B ′ x) = W x.Similarly, any linear map in R ñ×n for ñ < n can also be fit using this method.We have shown that a W-Asymmetric MLP with hidden dimension 24n can exactly fit an n × n linear map with high probability over the choice of Asymmetric masks M . It is known by [8] that for any continuous function f target : Ω ⊆ R D → R and any ε > 0, there exists a width k ′ such that 2-layer MLPs of width k ′ can approximate f to within ε.Let k be sufficiently big so that the probability that the masks do not satisfy the conditions of Appendix B.3.4 is less than δ. Such a k exists as long as n fix ∈ o(k1 4 ). Let m ≥ max(k, k ′ , D).Importantly, if a 2-layer MLP of width k ′ can approximate f target to within ε, a 2-layer MLP of width m with m ≥ k ′ can also approximate f to within ε. Let f MLP be a width m MLP that approximates f target to within ε.We now pad the input x to f target , with m -D zeros. This allows us to define a new function f 0 target : R m → R by f 0 target ([x; 0]) = f (x). Clearly f 0 target can also be approximated by a width m MLP.Let f 0 MLP denote the width m MLP that approximates f 0 to within ε. Now, f 0 MLP has dimensions m → m → 1, with corresponding linear maps W 1 ∈ R m×m , W 2 ∈ R 1×m . Each of these maps can be exactly fit using a 2-layer W-Asymmetric MLP, since their corresponding matrices have at least as many columns as rows. Concatenating these two exact fits yields an asymmetric MLP whose output exactly matches f 0 MLP and thus approximates f 0 to within ε. Thus, setting n ′ = m, there exists a width n ′ such that for all n > n ′ , with probability 1 -δ, for a randomly sampled 4-layer W-Asymmetric MLP f with η nonlinearity, hidden dimensions 24n → n → 24n, and n fix ∈ o(n 1/4 ) hardwired entries per neuron, there will exist θ ∈ Θ such that the W-Asymmetric MLP f : R n → R approximates f target to ε.As an aside, this procedure of mapping 2-layer standard MLPs to 4-layer W-Asymmetric MLPs implies that these 4-layer W-Asymmetric MLPs have at least as many symmetries as 2 layer standard MLPs. To fix this, we may want to consider a nonlinearity η such that η(x) -η(-x) ̸ = x.Although our W-Asymmetric and σ-Asymmetric networks are motivated by removing parameter space symmetries, their distinct empirical behavior may be caused by other factors besides just parameter space symmetries. For instance, the fixed entries F for the W-Asymmetric approach are taken to be much larger than the standard initialization of linear maps, which could cause several changes to optimization and loss landscapes besides just parameter symmetry breaking. Also, our theoretical results could be strengthened by future work in several ways. For instance, for the σ-Asymmetric approach, Proposition 1 only gives a guarantee of no parameter symmetries in the two-layer network case with square invertible weights. Future work could also give tighter analysis of the required width and depth for universal approximation using our W-Asymmetric architecture.This work does not focus on any particular application area. Instead, we study fundamental phenomena and theory of deep learning in general. Our work has potential to improve known deficits of neural networks: by making neural network loss landscapes more similar to convex landscapes, we can improve our understanding of them, and by improving Bayesian neural networks we advance one paradigm for bettering uncertainty quantification in neural networks. However, unlike standard neural networks, which have millions of papers studying them, we have only scratched the surface of Asymmetric networks. Important properties such as generalization, robustness to distribution shifts, and adversarial robustness have not been extensively studied for Asymmetric networks, and the interaction of parameter symmetries with these properties is not clear. Future research should further explore these important properties.Here, we present experiments with various ablations. These were in-part suggested by anonymous reviewers for the NeurIPS 2024 conference (thanks!).For the experiments in the main text, our W-Asym networks often had less learnable parameters than the standard networks that they were compared against (since we take them to have the same architecture, but the W-Asym approach fixes certain learnable parameters to constants). See Table 5 for the learnable parameter counts. In this section, we control for this, by decreasing the width of the standard networks so that they have the same number of parameters as the W-Asymmetric networks they are compared against. Results are shown in Tables 6 and 7 . We see that there is little to no change in metanetwork performance or Bayesian neural network performance for these smaller standard networks, so our results from the main text all still hold.Altintas et al. [3] showed that amount of learning rate warmup can affect the extent of linear mode connectivity between training runs. In Table 8 , we see that varying the number of warmup epochs between 1 and 20 does not change the qualitative results much on test loss barrier for our linear mode connectivity experiments.Intuitively, one way to break permutation symmetries of an MLP is to order the hidden neurons in some way. One possible way to do this is to fix the biases (so that they are untrained). As shown in Table 9 , a basic attempt at this fails, and has a much larger test barrier than our W-Asym and σ-Asym networks. For MNIST we use no data augmentation, and for CIFAR-10 we use random cropping and horizontal flipping. For the Git-ReBasin tests, we use the weight matching algorithm from [1] . For MLPs on MNIST, we used the Asymmetry hyperparameters in Table 10 . Table 11 gives the Asymmetric hyperparameters for ResNet20 on CIFAR-10, and Table 12 lists the same for ResNet20 with 8x larger width.For the GNN experiments, we use a GNN architecture similar to GIN [73] with mean aggregation.The base GNN has three message passing layers and a hidden dimension of 256, which gives 176,424 trainable parameters. The dataset is ogbn-arXiv [27] , which is a citation network of computer science arXiv papers with 169,343 nodes and 1,166,243 edges. The task is transductive node classification, where the label of each paper node is the primary subject area of the paper.As is common in transductive node classification on modestly sized graphs, we train each network with full-batch gradient on the whole graph. Thus, the randomness in training is purely from the initialization -there is no noise from minibatch selection in SGD. We use the Adam optimizer [31] with a peak learning rate of .001. The learning rate is linearly warmed up for 25 epochs to the peak, and then is held constant. Each network is trained for 500 epochs.For the Git-ReBasin alignment, we implement the activation matching approach. For the σ-Asymmetric GNN, we take σ to be FiGLU, in which we randomly initialize each fixed matrix F as a standard normal matrix with standard deviation .01/ √ d where d is the number of hidden channels; we found that having small standard deviation helped with training and interpolation. For the W-Asymmetric GNN, we fix 6 constants in each row of each linear map, and randomly initialize these constants from a normal distribution with standard deviation .5.For training Bayesian neural networks, we use the variational inference approach of Tomczak et al. [64] , which fits an approximate posterior that is Gaussian with a diagonal plus rank-4 covariance matrix structure. For the W-Asym ResNet tests, we train ResNet20s with the same Asymmetric hyperparameters as in Table 11 , though with κ = .5. For the CIFAR-100 experiments, we use aWe trained two datasets of image classifiers on CIFAR-10: one consisting of 10,000 small ResNet-like convolutional neural networks, and one consisting of 10,000 networks with a similar architecture, that use our graph-based approach to removing parameter symmetries. For fast training of many image classifiers, we use the FFCV package [37] . In particular, we use their CIFAR-10 sample script https://github.com/libffcv/ffcv/tree/main/examples/cifar, which includes data augmentation (random horizontal flips, random translations, and Cutout [10] ), label smoothing [62] , and a linear learning rate warmup and decay. In total, training all 20,000 classifiers takes just under 400 GPU hours (about 2 GPU-weeks) on NVIDIA RTX 2080 Ti GPUs.See Table 13 for the hyperparameters and ranges that we varied across the networks in our datasets. In each dataset, the trained networks all have the same architecture.Each ResNet has 78,042 trainable parameters, and each W-Asym ResNet has 60,634 trainable parameters. Both have the same architecture, except the W-Asym ResNet has certain filters that are fixed to constants to break the parameter symmetries. The ResNets each have 8 convolution layers, LayerNorm [5] , and a final fully-connected linear classification layer after average pooling across spatial dimensions. We trained several types of metanetworks for our experiments. All of these metanetworks are trained for 50 epochs using the AdamW optimizer [40] . For each metanetwork, on each dataset, we choose the learning rate in {10 -5 , 10 -4 , 5 • 10 -4 , 10 -3 , 5 • 10 -3 , 10 -2 } that gives the best validation R 2 performance on one training run. Then we run train each type of metanetwork 5 times on each dataset, and report the mean and standard deviation for each metric in Table 3 .For the monotonic linear interpolation experiments, we used the same setup as in the training of the datasets of CIFAR-10 image classifiers in Section 5.3. For each architecture, we sample 300 sets of hyperparameters from the distributions in Table 13 , and train one network for each set of these sampled hyperparameters. When evaluating training loss, we include the labeling smoothing term.For the σ-Asymmetric networks, we initialize the FiGLU F with a standard deviation of 1/ √ d, where d is the number of channels in the layer. Note that this is considerably larger than the standard deviation of .01/ √ d used in the GNN experiments of Section 5.1; we found this setting to train better (note that this initialization is in line with standard initializations of trainable parameters). Further, for the σ-Asymmetric networks, 24 out of the 300 networks diverged during training (giving NaNs), so we exclude them from the computation of statistics in Table 4 . From manual inspection, this divergence seems to happen when the learning rate is high (greater than .1). In contrast, none of the standard or W-Asymmetric networks diverged.The datasets we use are MNIST [38] , CIFAR-10 [33], CIFAR-100 [33] , and ogbn-arXiv [27] , which are all widely used in machine learning research. The first three appear to not have licenses and are open to use, while the last dataset is from the Open Graph Benchmark, which has an MIT License in the Github repository.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We generally do several runs of experiments, and report error bars that measure the variance in the results. We also conduct statistical significance tests in Table 4 . Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not train models or use data that have a high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We credit code and data used in Appendix F.5. For other types of models, we cite papers in which they were introduced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper studies (1) how to construct neural network architectures without "trivial" parameter space symmetries (e.g. permutation or scale invariance) and (2) the downstream, empirical affect of having fewer symmetries on applications ranging from posterior inference to Bayesian NNs to studying loss landscapes.   All reviewers were in favor of acceptance to some degree ( 2 x accept, 1 x weak accept, 1 x borderline accept), applauding the paper's clarity, importance of topic, connection to downstream applications.  The reviewers' most critical issues were about connections to / comparison with optimization-based approached and how the training conditions for the reduced-symmetry networks are substantially different than typical training (jeopardizing the generalizability of the results).  Yet both I and reviewers believe the authors have adequately addressed these concerns in their rebuttal, as the authors' general rebuttal has promised to add more discussion of constrained optimization to the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper studies how removing parameter symmetry of neural networks affects the loss landscape, Bayesian Neural Networks and meta-networks. The authors proposed two ways to remove the parameter symmetry: one is similar to pruning that making some parameters untrainable, the other is to adopt non-elementwise activations. Both ways can remove the permutation symmetry in parameter space. After that, the authors empirically demonstrate that removing permutation symmetry substantially 1) make LMC easier to satisfy 2) make BNN's training more efficient 3) improve the performance of meta-network (I am not quite familiar with meta-network or neural functionals.) 4) make training loss along the line segment between initialization and trained parameters more monotonic.
strength_weakness: strength_weakness: 1. A major issue about this paper is that some findings are not new. Especially, some studies have already demonstrated that asymmetric networks are more easily to satisfy LMC. [Cite 1] showed that pruning solutions are within the same basin and pruned networks can be viewed as a special case of W-asymmetric networks.

[Cite 1] Evci, Utku, Yani Ioannou, Cem Keskin, and Yann Dauphin. "Gradient flow in sparse neural networks and how lottery tickets win." In Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 6, pp. 6577-6586. 2022.

2. Another issue is that in experimental part, the four hypotheses seemingly do not relate to each other. There is not a unified conclusion drawn from the experiments.
3. The most important question about the permutation symmetry remains unresolved. As permutation symmetry holds in most neural network architectures, should we remove permutation symmetry or not? Is the success of deep learning related to the permutation symmetry? This paper seemingly cannot give an insightful answer.
3. In Sec. 5, despite each experiment are motivated from some simple intuition, there is no rigorous theoretical foundation for each hypothesis, which could potentially lower the value of this study.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes to study the effect of parameter symmetries on the neural networks' training and final properties by analyzing the behavior of networks without such symmetries (or with fewer of them). To do so, the authors develop two methods of parameterizing neural network architectures without parameter symmetries: W-asymmetric parametrization fixes the different subsets of weights in each row of weight matrices to be constant and untrainable, and a $\sigma$-asymmetric network uses a new FiGLU nonlinearity. The paper analyzes the asymmetric properties of both proposed methods theoretically and demonstrates empirically that asymmetry improves the behavior of neural networks in several setups. Specifically, asymmetric neural networks have better linear mode connectivity after training and more stable monotonic linear interpolation between initialization and trained model, are more effective for Bayesian deep learning, and are easier to model with metanetworks.
strength_weakness: strength_weakness: My main concerns are related to the limited discussion on the benefits of the proposed perspective in comparison to previous works, asymmetric properties of the proposed methods in practical setups, and poor results of $\sigma$-asymmetric networks:
1. As mentioned in the Related Work section, previous works examined the influence of parameter symmetries on neural network training by changing the optimization schemes instead of the network parametrization. Even though I find the idea of studying the behavior of asymmetric neural networks interesting, it is not clear to me how this perspective is beneficial compared to the constrained optimization one. Adding a more thorough discussion on that in the Related Work section would improve the paper. 
2. It is not clear from the paper if W-asymmetric and $\sigma$-asymmetric networks are fully asymmetric or just have fewer symmetries in practical setups. A more accurate discussion on which symmetries are removed by the proposed methods in the experimental setups should be added to the paper. For example, the paper does not cover the normalization layer symmetries, even though layer and batch normalization are used in the experiments. It seems W-asymmetric networks remove normalization symmetries, while $\sigma$-asymmetric ones do not. It may be one of the reasons why $\sigma$-asymmetric networks show weaker results in linear mode connectivity and linear interpolation sections. The ReLU scale symmetry is also not adequately discussed.
3. The analysis of the $\sigma$-asymmetric networks is limited. There is no proof or discussion of the universal approximation for this method. The paper does not explain poor results on linear mode connectivity and linear interpolation and does not include the results on Bayesian neural networks and metanetworks. A more detailed analysis of the $\sigma$-asymmetric networks, or at least a discussion of its suboptimal results, would benefit the paper.

Additionally, I have some minor concerns regarding the experiments:
1. In the Bayesian neural network experiment, the W-asymmetric networks have fewer parameters than standard ones, which may influence the optimal training hyperparameters. Hence, it may be the case that the difference in the results is due to, e.g., different effective learning rates and not the asymmetric network structure.
2. In the metanetwork experiments, standard and W-asymmetric networks have different numbers of parameters. Hence, it may be the case that the difference in the results is due to the different dimensionality of the input space and different optimal training hyperparameters for metanetworks and not the asymmetric network structure.
questions: questions: I would kindly ask the authors to address the concerns from the Weaknesses section and focus on the following questions:
1. Could you please elaborate on how the asymmetric network perspective differs from the constrained optimization one and in which cases it shows new insights, in your opinion? 
2. Could you please clarify whether the W-asymmetric and $\sigma$-asymmetric networks are fully asymmetric in the experiments or not? Which of the known symmetries (neuron permutations, ReLU scaling, pre-normalization parameters scaling, etc.) does each method remove in practice?
3. Could you please comment on the poor results of $\sigma$-asymmetric networks in linear mode connectivity and linear interpolation experiments? Is there any specific reason why the Bayesian neural networks and metanetworks experiments are not conducted for $\sigma$-asymmetric networks?
4. Could you please comment on the technical differences between standard and W-asymmetric networks (optimal hyperparameters and parameter count) in Bayesian neural networks and metanetworks experiments and whether they can affect the conclusions? 

Minor questions:
1. Do I understand correctly that the standard deviation $\kappa$ used in W-asymmetric and $\sigma$-asymmetric networks is a hyperparameter tuned separately from the trainable weights? (lines 134 and 163).
2. Do the non-trainable weights of all W-asymmetric networks have the same values in the metanetwork experiment? Does a metanetwork take only the trainable parameters of W-asymmetric networks as input?
limitations: limitations: The authors adequately discuss the limitations of the paper in the appendix.
paper_summary: paper_summary: The paper suggested asymmetric neural networks in terms of an asymmetric nonlinearity and weight. It demonstrated some tasks including linear mode connectivity without permutation alignment, Bayesian neural networks, training metanetworks, and monotonic linear interpolation to show the role of permutation symmetry inherited in the standard neural networks.
strength_weakness: strength_weakness: 1. The number of learnable parameters of the standard NN and the asymmetric NN is reported only for the metanetwork task. It should also be reported for the LMC, BMA, and MLI tasks.
1. Comprehensible reason is required for the proposition “the posterior will have less modes” in the hypothesis of the BMA task in order to understand the purpose of the BMA task. No symmetry does not imply less modes. Thus, the improved performance of BMA may be induced from another reason other than asymmetry.
questions: questions: What do you mean by “loss” in BNN. Is it an ELBO? Why don’t you report the negative log-likelihood (NLL)?
limitations: limitations: 1. Fixing some weights or adding anisotropic activation function implies constraining not only symmetry but also distorted parameter manifold (like equivariant NNs) that also leads to increasing correlation between solutions. The asymmetric NN is limited to directly correspond to the standard networks with a fixed permutation.
1. The BMA and MLI tasks are based on the strong assumption, where its evidence is absent, that no symmetry induces the reduced number of modes.
1. Although the paper partially gives insights on the loss surface, it does not suggest a new practical method utilizing the insights.
paper_summary: paper_summary: This paper develops two types of modifications to neural networks to remove permutation symmetries: fixing certain weights, or using a non-elementwise activation function. The resulting "asymmetric" neural networks are found to improve on certain metrics that are observed in networks with permutation symmetries, and thought to be caused by such symmetries: lack of linear mode connectivity between randomly initialized/trained networks, reduced Bayesian network performance, and reduced metanetwork performance. The asymmetric networks also are more likely to have monotonic linear interpolation between their initial and trained weights.
strength_weakness: strength_weakness: The main issue is that the paper relies heavily on experimental rather than theoretical results, but the strongest results are for a single method ($W$-asymmetry) with settings much more different than "standard" training than seems at first glance - specifically, lower learning rates, longer warmup, and larger fixed weights that all differ from standard training by order(s) of magnitude. These settings, plus a lack of available code, seriously impact the relevance/reproducibility of the methods/results.

There should be a comparison with standard models initialized to the same weights as the $W$-asymmetric networks. This would make certain metrics like $L^2$ weight distance comparable (currently the $W$-asymmetric networks are more likely to have extremely large magnitude weights which could increase the $L^2$ distance between independently initialized/trained networks). This would also eliminate some other variables such as reduced performance due to suboptimal weight initialization.

Low learning rates and long warmup are likely to improve similarity and reduce barriers, particularly between identically initialized neural networks. An analysis of how learning rate/warmup affects the experimental results (particularly barriers after training) is needed to determine if the observed results are mainly due to asymmetry.

Some additional details are needed to properly replicate the asymmetry methods, specifically in tables 5-7.
- how are $n_{\text{fix}}$ and $\kappa$ determined for each architecture?
- what do the names of the blocks refer to? There are not enough for a 1-to-1 mapping onto ResNet20 layers or permutations, so more explanation is needed.
questions: questions: Are the results in table 1 for networks with different random initializations, or the same identical initialization? If the former, are the masks and fixed weights different or the same between the two networks? I find it highly unlikely that networks with different fixed weights or different initializations would have low barrier after training without post-processing.

The fixed weights in $W$-asymmetry are distributed very differently to the non-fixed weights. If the fixed linear transform $F$ in $\sigma$-asymmetry is similarly shifted in distribution, would that make its results closer to $W$-asymmetry?

Why not fix biases to get asymmetry instead?
limitations: limitations: The reproducibility issues discussed previously, combined with the experimentally-focused framing, detract from an otherwise very strong and impactful paper.

The results for $\sigma$-asymmetry are considerably weaker than $W$-asymmetry (and rely on even smaller learning rates). Another work has found that networks mainly use linear transformations [1]. Since $\sigma$-asymmetry is effectively a non-elementwise sigmoid activation function, I wonder if the network's activations mainly stay in the approximately linear region of the sigmoid, allowing many permutations to be approximately equivalent. Some other comments or experiments on why $\sigma$-asymmetry is less effective in practice would be helpful, especially since theoretically speaking, both methods are clearly asymmetric.

[1] Mehmeti-Göpel & Disselhoff. Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think. ICML 2023. https://proceedings.mlr.press/v202/ali-mehmeti-gopel23a.html

Comments: title: Rebuttal
Reviewer's response
comment: I have reviewed the code and raised my score accordingly. All the best and good luck!
Hello, the LMC results use layer norm, as is done in e.g. Git Rebasin. We will also update you with batch norm results for comparison.
Hi, could the authors clarify whether the LMC results (e.g. table 1) use layer or batch normalization? I noticed the batch normalization is commented out in the code in favour of layer normalization.
Thank you for the comment. However, we would like to clarify very important differences between lottery tickets and our networks.

**Lottery tickets have a fixed initialization, W-Asymmetric Networks don't**. Once a pruning mask is found, lottery tickets **must maintain the same fixed initialization** when retrained from scratch. Thus, the only difference between different training runs of a lottery ticket come from things like SGD noise: the two lottery tickets share the same initialization. This is in contrast to our W-Asymmetric networks, which can be trained from any initialization: in our paper's experiments, we always randomly initialize W-Asymmetric networks' learned weights, and never share the initialization of learned weights.

**Thus, lottery tickets cannot be used for studying many of the phenomena we study with Asymmetric Networks**: metanetworks are often used on networks with different initialization, we cannot use lottery tickets to study standard linear mode connectivity between networks with different initialization, and we cannot use lottery tickets to study monotonic linear interpolation across different initializations. Lottery tickets thus cannot be used for the main purpose of our paper: to explore these diverse phenomena in deep learning from the perspective of parameter symmetries.

**Moreover, lottery tickets are generally found by training on one task and dataset.** In many regimes, lottery tickets can fail to transfer to other datasets. Morcos et al. 2019 show that lottery tickets can sometimes transfer between datasets for image classification, but there are regimes in which a lottery ticket found one dataset does not do well on other datasets; moreover, we expect this effect to be much worse when changing to different tasks. In contrast, our Asymmetric networks can just be simply initialized, without any special expensive procedure per dataset / task.

**Our $\sigma$-Asymmetric networks are not similar at all to lottery tickets.** Although the review says "some findings are not new", it does not mention our $\sigma$-Asymmetric networks at all, which are substantially different in structure to pruned networks. This is another valid way of removing some parameter symmetries, which the other reviewers (ceUS, tX9W)
 find is novel: "The methods appear both novel and logical"

Given these points, we ask the reviewer to reconsider their review. Let us know if you have any further questions or topics for discussion!
Thank you for your detailed response and I will maintain my current score. 

Also, I would like to clarify that

> The pruning methods for standard neural networks require specialized training algorithms that differ significantly from standard training (e.g. lottery tickets require repeated training and resetting, dynamic sparse training requires updating connectivity during training)

The process of finding the lottery tickets requires repeated training and resetting, however, once the lottery ticket (or subnetwork) is found, the training process of the subnetwork is indifferent from the training of its original network.
Hi, I've received the code and will update my score tomorrow. No issues so far.
Great! We are glad that we could reach consensus on this point.

We believe we have addressed all of the points brought up in your review. Do you have any other questions or concerns that we could address, which account for the relatively low score that you still give? If so, we can try to address those points in the remaining time of the discussion period.
I can finally understand more of the purpose of the BNN task. I think you should emphasize in the paper that W-Asym have the same architecture specification and have fewer parameters. Thanks for clarification. I will raise my score.
Hello, looking at the comments again, we realize why there may be some confusion. In our original rebuttal, the Bayesian experiments of the number of learnable parameters table has a typo, and the two columns are switched.
In particular, in every row, the W-asym experiment should have fewer parameters (because as mentioned above, it has the same width, depth, and modules as the standard network). Here's the updated table. Sorry for any confusion this may have caused.

|  Experiment / Architecture | Standard / sigma-Asym | W-Asym|
| --- | --- | --- |
| Sec 5.1 MLP |  935,434 | 834,570  |
| Sec 5.1 Resnet 1x  | 272,474 | 230,024 |
| Sec 5.1 Resnet 8x  | 17,289,866 | 16,273,946 |
| Sec 5.1 GNN | 176,424 | 171,576 |
| Sec 5.2 MLP-8 |3,324,466 | 3,242,146 |
| Sec 5.2 MLP-16 | 5,960,242 | 5,796,002 |
| Sec 5.2 Resnet-20 1x | 1,356,098 | 1,143,858 |
| Sec 5.2 Resnet-20 2x | 5,410,386 |  5,044,756 |
| Sec 5.2 Resnet-110 1x | 8,620,418 | 7,371,378 |
| Sec 5.2 Resnet-110 2x | 34,512,276 | 32,014,996 |
| Sec 5.2 Resnet-20 2x | 5,410,386 | 5,044,756 |
| Sec 5.4 MLI ResNet | 78,042 | 60,634 |
You said W-Asym has more learnable parameters in the BNN task (Sec 5.2) as in the table below (brought from your first rebuttal) but W-Asym should have less parameters to have the same architecture with the standard NN. I don't understand why your experiments already follow the exact same architectures. Furthermore, I think that the standard and sigma-Asym are not the same architecture even when their learnable parameters are the same because of the additional fixed matrix inside of the non-linearity.

| Experiment / Architecture | Standard / sigma-Asym | W-Asym     |
|---------------------------|-----------------------|------------|
| Sec 5.1 MLP               | 935434                | 834570     |
| Sec 5.1 Resnet 1x         | 272,474               | 230,024    |
| Sec 5.1 Resnet 8x         | 17,289,866            | 16,273,946 |
| Sec 5.1 GNN               | 176,424               | 171,576    |
| Sec 5.2 MLP-8             | 3,242,146             | 3,324,466  |
| Sec 5.2 MLP-16            | 5,796,002             | 5,960,242  |
| Sec 5.2 Resnet-20 1x      | 1,143,858             | 1,356,098  |
| Sec 5.2 Resnet-20 2x      | 5,044,756             | 5,410,386  |
| Sec 5.2 Resnet-110 1x     | 7,371,378             | 8,620,418  |
| Sec 5.2 Resnet-110 2x     | 32,014,996            | 34,512,276 |
| Sec 5.2 Resnet-20 2x      | 5,044,756             | 5,410,386  |
| Sec 5.4 MLI ResNet        | 78,042                | 60,634     |
Hello Reviewer tX9W, we were wondering whether you have received the code from the AC, given that you wanted to take a look at it (we sent it as soon as we submitted the rebuttal). Let us know if there are any questions!
Thanks for the clarification!

We would like to clarify that we already have empirical results in **both of the experimental regimes that you discuss here**.

1. In all of the experiments of our original submission, when comparing a standard and Asymmetric network, both networks have the same exact base architecture (width, depth, modules), besides the fact that the Asymmetric network has either some weights fixed (W-Asymmetric), or uses FiGLU nonlinearities (with an additional F matrix). We believe this is what you desire when you say "I would recommend using the exact same architecture for both the standard NN and the asymmetric NN, even if it causes fewer parameters." **All experiments in our original submission already followed this recommendation!**
2. In new experiments included in the rebuttal, as suggested by Reviewer ajZW, we matched the number of parameters of the standard and Asymmetric networks for Bayesian NNs and Metanetworks, by making the standard networks have less width.

Given that our empirical results are essentially the same in both regimes, we believe the empirical evidence is strong.

> "the number of modes can be uncountably many when ReLU nonlinearity (e.g., ResNet) is used due to its scaling symmetry"

In the case of continuous symmetries, we can use the dimension of the symmetry group (as a lie group) as an analogous measure of "number of modes". Then, since $\sigma$-Asymmetry removes scaling symmetries provably, and $\mathbf{W}$-Asymmetry removes any obvious scaling symmetries, we can again argue that there are less symmetries. We will be more careful with our wording around "less modes" in the revision.


> "I believe a comprehensive (theoretical) justification is necessary."

We agree with the reviewer that this would be nice. We only have a full understanding of the theory in the case considered in Proposition 3 (two-layer MLPs with square invertible weights), where we can say that, as long as only one neural network function is a minima of the loss, then there is exactly one mode of the loss landscape. Further theoretical results have been more difficult to derive, but we believe that future work could build on this.
Thank you very much for the detailed clarification. While I understood some of them, I still have concerns regarding the proposition.

As you and the cited papers mentioned, when we fix an architecture (both in width and depth) and then fix some weights, the total number of modes will obviously be reduced. However, in a fair comparison, where the standard NN and the asymmetric NN have the same number of learnable parameters, the architectures, and thus their parameter spaces, will be different. Since we have no idea about the total number of modes for each architecture, it cannot be definitively stated that the number of modes will be reduced. In fact, the modes could potentially increase due to the added nodes. Furthermore, **the number of modes can be uncountably many** when ReLU nonlinearity (e.g., ResNet) is used due to its scaling symmetry. For these reasons, I believe a comprehensive (theoretical) justification is necessary.

Although you claimed that the experiment does not rely on this assumption, I believe it does. If the number of modes is not reduced, the improved BNN performance in the asymmetric NN could simply be a result of the increased number of learnable parameters, as you mentioned in the rebuttal table. If you intend to demonstrate the hypothesis regarding the number of modes, I would recommend using the exact same architecture for both the standard NN and the asymmetric NN, even if it causes fewer parameters.
Thank you for the detailed response and additional ablations! I find most of my concerns adequately addressed. After reading other reviews and responses, my evaluation of the paper remains very positive. Considering the clarifications on the asymmetric properties of the proposed methods in practice and the low performance of $\sigma$-asymmetric networks, as well as the new ablations, I am raising my score to 7. 

Generally, I really enjoyed reading this paper =)
Thank you for the detailed response and additional ablations, as well as releasing the source code.

Apologies for the confusion regarding the learning rate, I did not notice you were using Adam. The ablations are very promising, and it is good to know the method does not depend on the warmup schedule. Although the performance between fixed and non-fixed W-asymmetric weights is not as different as with regular networks, it is still important to know that fixing the weights significantly reduces barriers, which proves the method's utility is not just in the differing weight initialization. I am willing to significantly raise my score after reviewing the source code.

Thank you for entertaining the many possible explanations for differences between $W$- and $\sigma$-asymmetry. I believe such negative results are important both for building asymmetric experiments, and for understanding how networks train in practice.