REPAIR: REnormalizing Permuted Activations for Interpolation Repair
Authors: Keller Jordan
Hanie Sedghi
Olga Saukh
Rahim Entezari
Behnam Neyshabur
Venue: ICLR 2023 poster (2023)
Keywords: Permutation
Invariance
Mode Connectivity
Barrier
Loss landscape
Deep Learning

Abstract: In this paper we empirically investigate the conjecture from Entezari et al. (2021) which states that if permutation invariance is taken into account, then there should be no loss barrier to the linear interpolation between SGD solutions. We conduct our investigation using standard computer vision architectures trained on CIFAR-10 and ImageNet.  First, we observe a general phenomenon in which interpolated deep networks suffer a collapse in the variance of their activations. We demonstrate that an appropriate rescaling of the pre-activations of the interpolated networks ameliorates this problem and significantly reduces the barrier. Second, by combining this with an algorithm for finding permutations based on maximizing correlations between the activations of matched neurons, we are able to reduce the interpolation barrier for a standard ResNet18 trained on CIFAR-10 to 1.5% absolute test error. We explore the interaction between our method and the choice of normalization layer, and demonstrate its robustness across a variety of architectures and training sets.

Published as a conference paper at ICLR 2023
REPAIR: RENORMALIZING PERMUTED ACTIVATIONS FOR INTERPOLATION REPAIR
Keller Jordan', Hanie Sedghi", Olga Saukh Rahim Entezari & Behnam Neyshabur Hive AI' Google Research² TU Graz/ / CSH Vienna
kell 1 ee@thehiiee.i.. {hsedghi, neyshabur /8google.com {or Colga.saukee saukh, rahim.entezari @@gaai 1 com
ABSTRACT
In this paper we look into the conjecture of Entezari et al. (2021) which states that if the permutation invariance of neural networks is taken into account, then there is likely no loss barrier to the linear interpolation between SGD solutions. First, we observe that neuron alignment methods alone are insufficient to establish low- barrier linear connectivity between SGD solutions due to a phenomenon we call variance collapse: interpolated deep networks suffer collapse in the variance of their activations, causing poor performance. Next, we propose REPAIR (REnor- malizing Permuted Activations for Interpolation Repair) which mitigates variance collapse by rescaling the preactivations of such interpolated networks. We explore the interaction between our method and the choice of normalization layer, network width, and depth, and demonstrate that using REPAIR on top of neuron alignment methods leads to 60%- 100% relative barrier reduction across a wide variety of architecture families and tasks. In particular, we report 74% barrier reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on CIFAR 10. Our code is available at hittt:://///. /tte:://ggtttaa http:///c...eeeeeeeeeraaooo
INTRODUCTION
Training neural network corresponds optimizing highly non-linear function by navigating complex loss landscape with numerous minima, symmetries and saddles (Zhang et al. [2017] Keskar
al., 2017 Draxler et al. 2018: Şimşek et al. 2021). Overparaneetrizzttion one of the reasons
behind the abundance of minima leading to different functions that behave similarly on the training
data (Neyshabur al. 2017: Nguyen et al., 20188: et et al.) 2018 Liu et al. (020). Another reason is the existence of permutation and scaling invariances which lead to functionally identical minima that differ in the weight space (Brea et al. 2019 Entezari et al., 202T). Due the relationship of the loss landscape with generalization and optimization, a large body of recent works (Li et al. 2017 Mei et al.] 2018: Geiger et al., 2019 Nguyen et al. 20188 Fort et al., 2019 Simşek et al. Juneja et al.) 2022) study the loss landscape of deep neural networks with the goal of navigating the optimizer to a region with desired properties, e.g., with respect to flatness around the SGD
2021
solution (Baldassi et al. 2020 Pittorino et al. 2020).
Early work conjectured the existence non-linear path of non-increasing loss between solutions found by SGD (Freeman and Bruna 2016, Draxler et al., 2018) and empirically showed how to find it (Garipov et al., 2018) Tatro et al. 2020 Pittorino et al. 2022). Recently, Entezari et al. (202T) conjectured the existence of such linear path between SGD solutions if the permutation invariance of neural networks' weight space is taken into account. That is, with high probability over SGD solutions, for each pair of trained networks A and B there exists a permutation of the hidden units in each layer of B such that the linear path between A and the permuted network B (B') is of non-increasing loss relative to the endpoints. This conjecture is important from both theoretical and empirical perspectives. Theoretically, it leads to a drastic simplification of the loss landscape, reducing the complexity obstacle for analyzing deep neural networks. Empirically, linear interpolation between neural network weights has become an important tool, having recently been used to set state of the art accuracy on ImageNet (Wortsman et al., 2022a). improve robustness of finetuned models (Wortsman et al.] 2022b I]harco et al. 2022), build effective weight-space model ensembles (Izmailov et al.) 2019 Frankle et al. 2020 Guo et al. (022), and constructively merge models trained separate data splits (Wang al. 2020 Ainsworth et al. (022). Therefore, any improvements toward reducing the obstacles to interpolation between trained models has the potential to lead to empirical progress in the above areas.

---

Published as a conference paper at ICLR 2023
Veriance collpso intepolated retworks
Nes00 / ImageNet
Berreer reduction due to REPAIR
Re 10-
Gefoo REAA
VOD CIAATOO
FEEFAR (Ou-2N)
deetro REPAI (NG661)
[-vr)) Gonsyy
Ader FLEFWRRT (NGS TI
Postlet theg
Pitton dERN (Rodthel)))
74% ros soon
Resher33 WNNIT-
Norra 1
Bickno RE RESUU
RarNecto SAAAk-
06
Loyar
Figure 1: REPAIR improves the performance of interpolated networks by mitigating variance collapse. In each experiment, we interpolate hetween the weights of two independently trained networks whose hidden units have heen aligned using the method described in Section 2.3 We then compare the interpolated network before and after applying our correction method REPAIR. Left: The variance of activations in interpolated networks progressively collapses. We report the average variance across each layer, normalized by that of the corresponding layer in the original endpoint networks. REPAIR is designed correct this phenomenon. Middle: REPAIR reduces the barrier linear interpolation between aligned ResNet50s independently trained on ImageNet by 74% (from 76% to 20%). Right: REPAIR reduces the interpolation barrier across many choices of architecture, training dataset, and normalization layer. For cach architecturc/dataset pair we vary the network width; larger markers indicate wider networks.
Prior and concurrent works on linear interpolation (Singh and Jaggi 2020 Entezari et al. 1021 Ainsworth et al.] 2022) have focused on improving the algorithms used to bring the hidden units of two networks into alignment, in order reduce the barrier to interpolation between them. (Singh and
Jaggi (2020) develop a strong optimal transport-based method which allows linear interpolation be- Iween pair of Re (He al., 2016) networks such that the minimum accuracy attained along the path is 77%. This constitutes a "barrier" of 16% relative to the original endpoint networks which achieve over 93% accuracy on the CIFAR-10 test set. EEntezari al. (2021) use an approach based on simulated annealing (Zhan et al. 2016) in order to find permutations such that wide multi-layer per- ceptrons (MLPss) (Rosenblatt 1958) trained on MNIST (LeCun 1998) can be linearly interpolated with a barrier of nearly zero. Ainsworth et al. (202)) make the first demonstration of zero- zero-barrier connectivity between wide ResNets trained on CIFAR-10 by replacing" the standard Batch Normal- ization (loffe and Szegedy, 2015) layers with Layer Normalization (Ba et al. 2016), and develop several novel alignment methods. Further discussion of related work can be found in Appendix [ In this paper are interested in understanding why alignment of the endpoint networks alone has so far been insufficient to reach now-barrier linear connectivity between standard deep networks.
Contributions In this work we focus on understanding the source of the poor performance of standard deep networks (ResNet18, VGG11) whose weights have been linearly interpolated from between pairs of networks with aligned neurons. Our contributions are as follows:
We find that such interpolated networks suffer from a phenomenon of variance collapse in which their hidden units have significantly smaller activation variance compared to the corresponding units of the original networks from which they were interpolated. We further identify and explain the reason behind this variance collapse. (Figure (left) and Scction
We propose REPAIR (REnormalizing Permuted Activations for Interpolation Repair), method that corrects variance collapse by rescaling hidden units in the interpolated network such that their statistics match those of the original networks. (Section
We demonstrate that applying REPAIR to such interpolated networks leads to significant barrier reductions across wide variety of architectures, datasets, normalization techniques, and network width/depth (Section 5 and Figure [[ (middle and right)).
2 PRELIMINARIES
In this section we give preliminary definitions and algorithms to be used throughout the paper.
2.1 LINEAR INTERPOLATION OF NEURAL NETWORKS
We consider the problem of interpolating between independently trained neural networks. That is, if we let f1, 8g be the weight vectors of two such networks, then we are interested in networks whose
See the code release, https://gtt httpps://gttthb..ooo/ hwwps//oo.ooooomoaaaaoa/o htteee/aaaattt///////////////rer
2

---

Published as a conference paper at ICLR 2023
weights are of the form 8a. (1 a)) 81 + 002 for 0 interpolated networks, and to 81. 2g as the endpoint networks.
< 1. We refer to such networks 8a as
The loss barrier (0, 02) between pair of networks is defined (Entezari et al. 2021) as the maxi- mum increase in loss along the linear path between 01 and 02, relative to the corresponding convex combination of the two endpoint losses. In notation:
[c(( a)0, + =02)] [ (1--)) + +
B(01,02) = sup
(I)
ac[o,1]
The barrier between pairs of unmodified, independent SGD solutions 0,, 02 is typically large. We next discuss permutation invariance in neural networks.
2.2 PERMUTATION INVARIANCE
For typical neural architectures, the neurons in each layer can be permuted without functionally changing the network; this is known the permutation invariance property. In a simple feedforward network, this amounts to the observation that we can replace the Lth weight matrix by PW, and the (L 1)th weight by W for a permutation matrix P without changing the function represented by the network. Therefore, even if two networks 01. Bg learn functionally identical sets of neurons at each layer, is possible for their neurons to be arbitrarily permuted or misaligned.
Entezari et al. 2021 conjecture that if permutation invariance is taken into account, then all SGD solutions for a sufficienlly wide network trained on the same task become linearly mode connected. For completeness we provide the formal statement of the conjecture below.
Conjecture 1. (Entezari et al. 2027) For a given neural architecture, let P be the of all valid permutations of hidden units, and P * x + R2 be the function that applies a given permutation weight vector and returns the permuted version. Then for sufficiently wide networks, there exists a high-probability set of SGD solutions S C IRt and function Q S + P such that for any 01, 02 E S, we have small interpolation barrier (PP ((0)))) (2,0)) 82)
2.3 NEURON ALIGNMENT ALGORITTMS
A number of works have proposed methods of finding such alignments between the hidden units of a pair of neural networks. Li et al. (2015) propose to maximize the sum of correlations between the
activations of paired neurons across batch of training data. That is, if we let (0) and x(() be
random variables corresponding to the activations of the i-th hidden units of the I-th layer (across
batch of training data), then we optimize the permutation P/ to minimize following cost:
Σ coorr(X (2) (LP.)))
This amounts to a lincar sum assignment problem corresponding to the matrix of correlations be- tween pairs of hidden units in the two networks which can be solved via the Hungarian algo- rithm (Kuhn) 1955). Recent works have proposed alternative approaches: He et al. (2018) compute a Hessian approximation to align functionally similar neurons, and Singh and Jaggi (2020) develop an optimal transport- baase method of soft-alignment. [Ainsworth et al. (2022) compare three methods, including one based Li et al. (2015) and two novel approaches.
Tatro et al. 2020) also perform alignment based on minimizing Equation 2 in order to reduce the barrier non-linear interpolation. They show using proximal alternating minimization scheme that alignments found using this method are nearly optimal for their purposes. In this paper, we continue use this alignment method which was originally introduced by Li et al. (015)).
We note that for networks with residual connections, care must be taken to restrict the set of permu- tations such that the function represented by the network does not change. particular, the same permutation of hidden units must be applied to all layers which feed into single residual stream.
VARIANCE COLLAPSE IN INTERPOLATED NETWORKS
We begin our study of interpolated networks by picking up where previous works left off:
The barrier between aligned networks decreases with width, including to nearly zero for very wide MLPs trained on MNIST. The barrier also increases sharply with depth, becom- ing large for ML.Ps or simple CNNs of more than a few layers (Entezari al. 2021
3

---

Published as a conference paper at ICLR 2023
MI P MNIST
VGDO1 / CIFAR-
RosNet50/ ImageNet
10
12
Valiine rodas
68
Drroct Bufor RETAIR
Derett into Bufax REFAIR
Direct imoopeiaiinn
Before
0.4
02:
0U
U2
DG
6
10
Layer Irdex
Layerinone
k lydea
Figure 2: Variance collapse phenomenon in averaged networks. find that hidden units of weight- space averaged neural networks suffer from variance collapse: as we progress through the network, variance of neuron activations reduces, with neurons in deeper layers becoming nearly constant while varying the input data. Before REPAIR refers networks which interpolated from endpoint networks whose hidden units have been aligned (Li et al.) 2015) (Singh and Jaggi 2020 Ainsworth al. 2022), before our correction method REPAIR applied. REPAIR is applied on top this baseline in order to restore the internal statistics of averaged networks back to the level of the endpoint networks.
Even strong optimal transport-based methods, which go far beyond alignment by allowing each neuron network A be matched to a weighted sum of neurons in network B, are insufficient to achieve low-barrier (below 5% test-error) connectivity between standard ResNets (Singh and Jaggi 2020).
We focus on understanding the source of this sharp increase in barrier between aligned networks as a function of depth observed in EEtezaai et al. (202T). which we hypothesize to be the same phenomenon that causes high barrier to interpolation for standard ResNets.
3.1 IDENTIFYING THE PROBLEM: VARIANCE COLLAPSE
What causes this rapid drop-off in the performance of interpolated networks which are deeper than a few layers? To answer this question, we investigate the internal behavior of such networks, focusing on the statistics of hidden units (Figure 2}. We find that for deep MLPs, interpolated from pairs of aligned endpoint networks which have high accuracy on the MÄIST test-set, hidden units undergo a variance collapse. That is, the variance of their activations progressively decays as we move deeper into the network, with activations in later layers becoming nearly constant. For each layer, we quantify this decay as follows. First, we measure the variance of each neuron across a batch of training data. Then we sum these variances across the neurons in the layer. Finally, if we let this sum be denoted Vo U1, D2 for the layer in the interpolated and two endpoint networks, respectively, then we return the ratio (v1-22)/2- We report this ratio for each layer, yielding the sequence of values in Figure 2 (left), where we see variance collapse occur in MLPs, VGG networks, and ResNets. For an example of the set of per-neuron variances in single layer of interpolated ResNet18, see Figure 3 (left).
Ve
We observe that variance decays nearly zero by the final layer of interpolated 35-layer MLP, in- dicating that the activations in these last layers have become nearly constant. This effect seems to be further exacerbated when directly interpolating between unaligned networks. We repeat this experi- ment for VGG (Simonyan and Zisserman 2014) and ResNet50 architectures, trained on CIFAR-10 10 and ImageNet respectively, and find that variance by the final layers decays by more than 10x (Fig- ure 2 (middle) and Figure 2 (right)). This is a problem: if these networks have nearly constant activations in their final layers, then they will no longer even be able to differentiate between inputs.
3.2
WHY DOES THIS PHENOMENON OCCUR?
We argue that this phenomenon can be understood through the following statistical calculation. Consider a hidden unit or channel in the first layer of the interpolated network. Such unit will be functionally equivalent to the linear interpolation between the respective units in the endpoint networks. That is, if we represent the unit's 'ss preactivation by X, in the interpolated network, and X2 in the two endpoint networks (as random variables over the input data distribution), then the equality X, = (1 a X, + aX2 holds. We will argue that the variance of X,, is typically reduced as compared to that of X, or X².
4

---

Published as a conference paper at ICLR 2023
If the two endpoint networks perfectly aligned and have learned the same features, then should have corr(X1, X2) ) = 1. But in practice, it is more typical for pairs of aligned units (whose alignment minimizes the cost function given by Equation to have correlation of
X2) so 0.4. When considering the midpoint interpolated network (a 0.5), the variance of Xa is given by
Var(Xn) Var (X1 X2
2
Var(Xi) + Var(X2) + 2Co
-slld(x:) + sld(X2) -1- (orr(r(X1, X2)) std(())std(Xp))
2
We typically have std(X,) x (td( so that this simplifies to Var(Xa) = (0.5 + 0.5 1orr(X1,, Var(X1). With our typical value of corr(X1, x 0.4 for pairs of neurons in aligned networks, this yields Var(X.) = 0.7 Var(X1): ). a 30% reduction compared to the endpoint networks. This analysis cannot be rigorously extended to deeper layers of the interpolated network, but intuitively we expect this decay to compound with depth. This intuition matches our experi- ments, where we see that variance collapse becomes worse as we progress through the layers of MLP, VGG, and ResNet50 networks (Figure
REPAIR
We now propose a method which mitigates variance collapse by rescaling the preactivations of hidden units in interpolated networks. We call this method REPAIR (REnormalizing Permuted Activations for Interpolation Repair).
Given an interpolated network 0a = a1 01 a 02 for some 0 < a < (between aligned endpoint networks 02, 02), we select the set of hidden units or channels whose statisties we aim to correct. For example, for VGG networks we correct the outputs (preactivations) of every con- volutional layer. For ResNets we correct both these convolutional outputs and the outputs of each residual block.
Our goal will be to compute affine (rescale-and-shift) coefficients for every selected channel, such that the statistics of all selected channels are corrected. Let us consider a particular channel, egg. the 45th convolutional channel of the 8th layer in an interpolated ResNet18. Similar to the analysis in the last section, let X1 and X2 be the values of the channel in the two endpoint networks, viewed as random variables over the input training data, and let Xa be the same channel in the interpolated network. Then after rescaling, we want the following two conditions to hold:
E[Xa] (1 a1) a:) E[X,] + a [[[2),, std(X.) = (1 a)) std(Xi) + a std(X2).
(4)
Whereas before any correction, typically have std(X..) < (ttsst((x),,sd((())) due vari- ance collapse. In the following, we present an algorithm which computes exact affine coefficients for each selected channel such that conditions (3), (4) become true. We also propose an approximate algorithm which avoids the use of forward passes in the interpolated network in Appendix C
This algorithm depends first upon the computation of the statistics I[ [[[x]], std(X1), std( for each selected channel in the endpoint networks. We present a PyTorch-based approach to per- form this computation in Appendix B.2 Given these values, the algorithm proceeds follows. For each module in the interpolated network whose outputs we have identified as targets for statistical correction. we apply a wrapper, which adds Batch Normalization layer after the wrapped module that is initially set to "train" mode. PyTorch pseudocode for this wrapper can also be found in Ap- pendix B.2 Each such added BatchNorm layer contains affine (i.e., per-channel rescale-and-shift parameters. For a given channel incoming to the BatchNorm layer, we set the respective affine weight to (1 (1-a) std(X,) + a sted(X2) and the bias to (1 - a [ + a IE [x]] where X₁ and X2 the respective channels in the endpoint networks in conditions
With the added BatchNorm layers in training mode, during execution they first renormalize their inputs to have zero mean and unit variance per channel, and then apply the given affine transforma- tion, such that our statistical conditions B) 4 are exactly induced with respect to each batch of input data. Next, we pass of training data through the network (~5,000 examples suffices) so
5

---

Published as a conference paper at ICLR 2023
Heskwett Stwiayer statotics
MI P/ MNSST
Reshvet CIFARR0
130
Exspponn metocrta 5,.
REW
80
Toxe ACDUACy
Endpe reporaaa B, & Brfort REPUR REFAIR
interpiation Bulase RENUR
THE
40
Alka NEAUI
20-
54
05
129
100
192
224
200
6.
Iraepclation
Figure 3: REPAIR restores the internal statistics of averaged neural networks. Left: We visualize the statistics of different channels in 9th layer of interpolated ResNet18 CIFAR- 10. The uncorrected network undergoes variance collapse, whereas REPAIR restores the internal statistics of the network be similar the endpoint networks. Middle: Permuted interpolation without statistical correction (hefore REPAIR) only performs well when limited to MI. of few layers (Entezari et al. (02T). REPAIR enables relatively high- performance weight-space averaging between much deeper aligned MLPs. Right: Using REPAIR, the barrier interpolation between aligned ResNet18s trained on CIFAR-10 10 is reduced from 29% 1.5%.
that the running mean and variance parameters of these BatchNorm layers will be accurately esti- mated. During this pass, any BatchNorm layers which already existed in the original network are kept frozen. Finally, we set the added BatchNorm layers to evaluation mode, so that they behave as affine layers which do not recompute statistics. At this point, the resulting network is functionally equivalent to one in which the weights of our selected set of channels have been rescaled and biases shifted. If we wish to generate new parameter vector 0' which is compatible with the original network architecture (i.e., lacks these added BatchNorm layers), then we can perform BatchNorm layer fusion (Markus 2018) in which appropriate rescaling and bias-shifting values are computed from each added NotchNorm layer, and then applied to the preceding convolutional filters.
The networks resulting from this process have their internal statistics corrected, so that all selected channels satisfy conditions (3). (4) In Figure and Figure (left) we observe that REPAIR resolves variance collapse, bringing the statistics of the interpolated units back up the level of the endpoint networks. In Figure 3 (left) we apply REPAIR to MIPs and in Figure (right) we apply REPAIR to networks where the weights are linearly interpolated between a pair of ResNet18s whose hidden units have been brought into alignment. Before the correction, networks near the midpoint have reduced accuracy of 66.0% on the CIFAR-I0 test set, while the endpoints accuracy is 95.5%. After correction, all checkpoints along the linear path have significantly boosted accuracy, with the midpoint performing at 94. In comparison, [Singh and Jaggi (2020) report linear midpoint accuracy of 77.0% using strong optimal-transport based alignment methods.
Throughout the rest of the paper, we refer to the above method as REPAIR. We note that the utility of this method is orthogonal to any improvements in the algorithm used to align the hidden units of the endpoint networks. We explore this in Appendix Figure [5] where we report barrier curves for REPAIR applied to interpolations between unaligned networks as well. In the remainder of the paper, we demonstrate the effectiveness of REPAIR across a wide range of scenarios.
5 EXPERIMENTAL RESULTS
In this section we report the results of the following experiments.
We investigate the effectiveness of REPAIR for ResNets of varying width, trained on CIFAR-10 using three different normalization layers: standard NatchNorm, normalization- free using FixUp (Zhang et al. 2019), and LayerNorm. (Section 5.1
We apply REPAIR to the challenging scenario of interpolating between aligned ResNets trained on ImageNet. (Section 5.2)
Finally, we perform an improved replication using REPAIR of an experiment from Ainsworth et al. (2022), in which a pair of models trained on disjoint subsets of data is constructively merged. (Section 5.3)
We further explore the effect of width and depth across variety of datasets. (Figure 5
Extra figures replicating these experiments in terms of alternate metrics can be found in Appendix D

---

Published as a conference paper at ICLR 2023
RasNe20 CIFAR-13
Layerficm Restvats CIFAR-10
ReeNas0O / CIFAR-16
100
Belore Ho-HA (Bute N
40
After REPAIR (Bal:))om))
Alter REMPAR BaacHHorm
Rodirn SERMR 1.sperkem;
20-
Bolon RE NA Norrs)
After RERUR (LayerNom)
Aller REEARR Euppahoent
Defore REPAI Norm) After REPAR (archMemm
Balore RE (Flllp)))
Aller HELPAA slgl
Barlon RENUR (Laye Norm))
After REPUR (Laysshoo)
50
2+
24
10xx
30%
mulliperr
Iropptation
Figure 4: Effect of normalization layer. Left: Loss barriers with and without REPAIR for ResNet20s traindd with BatchNorm, LayerNorm, and normalization free via FixUp, varying the width multiplier from 16. Middle: LayerNorm networks are unique in reaching relatively low barrier hefore REPAIR. Right: Performance curves of networks interpolated between aligned ResNer20s. Without REPAIR, the midpoint BatchNorm-hee basnd network achieves 34.7% accuracy, compared to 88.4% for LayerNorm. After REPAIR, hoth variants attain 92.0% accuracy.
5.1 NORMALIZATION LAYER AND NETWORK WIDTII
We begin by considering ResNet20s trained on CIFAR-10 using three different types of normaliza- tion layer: BatchNorm (loffe and Szegedy 2013), LayerNorm (Ba et al. (016)). and normalization- free via Fixup (Zhang et al. (019). We combine this ablation with study on the effect of width. For each class of networks, we vary the width multiplier from 1 1x, where the final block has 64 channels, up to 16x 1024 channels. In Figure 4 we report barriers to interpolation, both with and without REPAIR. We find that REPAIR is effective for all three cases, shrinking the loss barrier to below 0.05 for BatchNorm and LayerNorm- networks when width is scaled to 16x.
The LayerNorm-basee ResNets are a special case that deserves comment. We first note that Layer- Norm is nonstandard in ResNets, and has worse performance when compared to BatchNorm. With our training configuration, LayerNorm-based ResNer20s achieve 92.4% CIFAR-10 10 test set accuracy, 93.6% for the BatchNorm baseline. But these networks also have a unique property: when inter- polating between aligned LayerNorm- -oreerNNet-Reeeettt.. the midpoints already perform relatively well before REPAIR is applied, see Figure 4 (middle and right). The midpoint between our pair of 8x-width MayerNorm-ResNetee
attains 90.3% test-set accuracy, which is boosted to 93.2% by an applica- tion of REPAIR. In contrast, for standard BatchNorm ResNets of the same width, the interpolated network obtains only 32.3% accuracy, which is boosted to 94.4% by REPAIR.
We hypothesize that LayerNorm partially mitigates variance collapse because it applies test-time renormalization after each layer, which may prevent per-layer reductions in variance from com- pounding between layers. In contrast, BatchNorm is functionally just an affine layer during test- time, so that variance reductions do compound.
We claim that this observation regarding LayerNorm-based ResNets explains the contradiction be- tween the results of Singh and Jaggil (2020) and Ainsworth et al. (2022). In the former, the authors develop a strong aptimal-transpon based method of aligning networks, which contains the approach of et al. (2015) as special case. They show that even this method is insufficient to attain low- barrier connectivity between standard ResNets before fine-tuning; their best result is a barrier of 16% error between aligned ResNet18 18s trained on CIFAR-10. In contrast, Ainsworth al. (2022) report that a variety of alignment methods, including the method of Li et al. (2015) which they call "activation matching", suffice establish nearly zero-barrier connectivity between wide ResNets trained on CIFAR-I0. How can both results be true? We claim that this contradiction is resolved by the observation that Ainswwrth et al. al. (022) replace standard BatchNorm layers with LayerNorm in their ResNets, as is evident in the code release" In our experiments, the use of LayerNorm in ResNets uniquely allows for low-barrier linear connectivity without the requirement of a statistical correction such as REPAIR, at the cost of reduced performance on most datasets.
5.2 IMAGENET
Next, we explore the impact of REPAIR on the barrier to interpolation between standard ResNet models trained from scratch on ImageNet (Deng al. 2009). We test ResNet18, ResNet50, and

---

Published as a conference paper at ICLR 2023
10-layr VEP
ReSNIGOO
VC3 / GIFAR11
WNIST
Faseion SIUST
60-
StiiN
Akcoria baaier
CIFAR-10
Bulan REFAIR (bun))
Delore REFARR
KNNIST
After REFA than)
Fashhonk98ST
Bittore REFAR
SU--N
Arher REPP
CIFAR:10
Liefore KELMR
0
012
1024
2015
12x
VOG
VGG13
VOG16
VSG19
Depth
Figure 5: Network width and depth. Left: We investigate the effect of REPAIR on the barrier for 10-layer MLPs, trained MNIST, FashionMNIST (Xiao et al. (017). SVHN (Netzer et al. 2 and CIFAR- 10. In each case, the baseline (interpolation between aligned networks) is shown with dotted curve, and the solid curve refers to cases where REPAIR is applied on top the haseline. We vary the width from 32 to 2048 hidden units per layer. Middle: We conduct the same experiment using ResNet20s trained on the same four datasets. We vary the width multiplier from to 16, producing models whose final block ranges from 64 to 1024 channels. Right: We investigate the effect of REPAIR the barrier for VGG networks trained CIFAR-10. We vary the network depth from 11 to 19 layers, and observe that without REPAIR the barrier increases with depth, whereas with REPAIR it is close to constant in depth.
ResNel50 ImagaNet
ResNets/ / ImageNet
80
80
Aocrrsy (bop"" -)
60-
Rocuracy (os-- Bop-1)
60-
Direct interpolation
40-
Before REFAIR
RasNot18
KEPAIR (onty conv layers)
ResNet50
After REPAI
20
ResNitt0 acttleneck
Before RE PAIR
0.6
0.2
0.4
0.6
Inile collioni
Figure 6: REPAIR significantly reduces the barrier between ResNet50s trained on ImageNet. Left: Without REPAIR. interpolations between two aligned, independently trained ResNet50s attain less than 1 1% test accuracy ImageNet. After applying REPAIR to convolutional layer outputs, the midpoint boosted to 53.2%. Using full REPAIR, which is also applied to the outputs of residual blocks, this further boosted to 56.5%. Right: Larger and wider ResNet architectures have smaller barrier. Dashed lines indicate the baseline of interpolation between aligned networks (Singh and Jaggi 2020) Ainsworth ct al. 2022). and solid lines refer to REPAIR on top of baselinc.
double-width variant of ResNet50 in Figure 6 (right). Without REPAIR, the interpolated midpoints between each aligned pair of networks perform at below 1% (top-1) 1) accuracy on the ImageNet validation set. After REPAIR, the midpoint ResNet18 improves to 41.1%, ResNet50 to 56.5%, and double-width ResNe150 to 64.2%.
We find in Figure 6 (left) that it is important to apply REPAIR not just to all convolutional layer outputs, but also to the outputs of every residual block. For the case of ResNet50, REPAIRing these these extra channels boosted the performance of the midpoint from 53.2% to 56.5%, which is a 14% reduction in the barrier (from 23.4% to 20.1%). We note that for architectures which contain BatchNorm after convolutional layers, applying REPAIR only convolutional layer outputs is equivalent to resetting the BatchNorm statistics. Performing such a reset on averaged networks from a single training trajectory goes back to Izmailov et al.) (2018). but as far as we are aware, has not been applied to interpolations between independently trained networks until now. We provide a comparison between the ImageNet results of our work and those of Ainsworth et al. (2022) in Appendix Figure 9
In general, the barriers for these architectures on ImageNet are still relatively high. The standard ResNet50 architecture has a final-block bottleneck width of 1024, and we measure the barrier after REPAIR to be 20.1 % in terms of test error. The double-width ResNet50 variant has a final-block bottleneck width of 2048, reducing the barrier to 12.9%. In comparison, the widest ResNet20 we studied on CIFAR-10 10 had final-layer width of 1024, and a barrier of nearly zero Therefore, it may be the case that for more difficult datasets, larger widths are required in order to reach low barriers.
8

---

Published as a conference paper at ICLR 2023
5.3 SPLIT DATA TRAINING
In this section we study setting where the two end- point networks are trained on disjoint splits of the training dataset. We aim to improve the correspond- ing experiment of Ainsworth et 2 by using REPAIR applied to standard BatchNorm networks.
CIFAR- 100. Sp data training
1.4
12:
Interpolating with REPAIR
We first split the CIFAR- 100 training set, consisting of 50,000 images distributed across 100 classes, into two disjoint sets of 25,000 images. The first split contains random 80% of the images in the first 50 classes, and 20% of the second 50 classes; the sec- ond split has the proportions reversed. We then train one network on each split. The result is that the first network is more accurate on the first 50 classes, and the other more accurate on the second. Both net- works underperform either their ensemble or a net- work trained on the full training set.
Text
Ensembliigg
Corr bined data training
1.0
08-
0.2
0.4
0.0
0.8
Interpolstions
Figure 7: Split data training. When two net- works are trained disjoint, biased subsets of CIFAR-100, their REPAIRed interpolations out- perform either endpoint with respect to the com- bined test set.
We next align the hidden units of these two networks, and generate a series of weight-space inter- polations, applying REPAIR to each. We find that many of these interpolated networks significantly outperform either of the two endpoints in terms loss on the full CIFAR- test set (Figure (7). In this sense, the two networks can be said to have been constructively merged. The best interpolated network of the corresponding experiment in Ainsworth et al. (2022) was reported to obtain a loss of 1.73 with mixing coefficient a 0.3. Using REPAIR our best interpolated network achieves loss of 1.30, also with a = 0.3. We attribute this improvement partially to REPAIR, and partially to the increased performance of standard ResNets compared to the SoyyeNoommbased variants used in Ainsworth et al. (2022). In Appendix D we compare these results against a strong baseline.
6 DISCUSSION AND FUTURE WORK
In this paper proposed REPAIR, method of mitigating the variance collapse phenomenon that we identified in interpolated networks. We demonstrated that REPAIR improves the performance of interpolated networks across wide variety of architectures and datasets. For example, we used REPAIR to reduce the barrier to permuted interpolation for standard ResNet18s trained on CIFAR- 10 from 16% (Singh and Jaggi 2020) to 1.5%, and to improve the performance of interpolated ResNet50s from below 1 5% to 56.5% on ImageNet. REPAIR is effective for networks trained with many choices of normalization, including BatchNorm, LayerNorm, and normalization-free.
To explain these results, we provided an analysis of the variance collapse phenomenon and how REPAIR mitigates it. We also demonstrated that LayerNorm-basse ResNets are unique in attaining a relatively low barrier to aligned interpolation before the use of REPAIR, resolving the contradiction between the results of Singh and Jaggi (222)) and Ainsworth et (2022).
In so far we establish low-barrier permuted interpolation for further scenarios, these results pro- vide support for the conjecture of Entezari et al. (202T). On the other hand, to practically do so we needed to rescale the preactivations of the interpolated network, moving us out of the realm of strictly permuted linear interpolation. Our correction method REPAIR was developed as a gen- eralization of the method of resetting BatchNorm statisties of averaged networks (Izmailov et al.) 2018). Such a correction appears to be necessary in order to establish low-barrier permuted linear connectivity, at practical widths and without the use of test-time normalization layers.
On the applications side, we demonstrated that REPAIRed interpolations between two networks which were trained disjoint, biased dataset splits are able to outperform either network from which they were interpolated. We hope that further such applied results will be possible, including improvements to weight-space ensembling, checkpoint averaging, and robust finetuning.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Thanks to Luke Johnston for his insightful comments on the manuscript. This collaboration was facilitated by ML Collective,
REFERENCES
Agueh, M. and Carlier, G. (2011). Barycenters in the wasserstein space. SIAM Journal Mathe- matical Analysis, 43(2):904-924,
Ainsworth, S. K., Hayase, J., and Srinivasa, S. (2022). Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.0444 04836.
Ashmore, S. and Gashler, M. (2015). A method for finding similarity between multi-layer percep- trons by forward bipartite alignment. In 2015 International Joint Conference on Neural Nenworks (IJCNN) pages 1-7. IEEE.
Ba. J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint wrXiv:1607006450
Baldassi, C., Pittorino, F., and Zecchina, R. (2020). Shaping the learning landscape in neural net- works around wide flat minima. Proceedings of the National Academy of Sciences, 117(1):161- 170.
Brea, J., Simsek, B., Illing, B., and Gerstner, W. (2019). Weight-space symmetry in deep networks gives rise permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint
Deng. J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. (2009). Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. IEEF
Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. (2018). Essentially no barriers in neural network energy landscape. In International conference machine learning, pages 1339-1318 PMLR.
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. (2021). The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint 1rXiv:2110.062900
Fort, S., Hu, H., and Lakshminarayanan, B. (2019). Deep ensembles; A loss landscape perspective. arXiv preprint arXiv: 1
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. (2020). Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pages 3259-3269. PMLR.
Freeman, C. and and Bruna, J. (2016). Topology and geometry of half-rectified network optimization. arXiv preprini arXiv: 1X1.:111
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. (2018). Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31.
Geiger, M., Spigler, S., d' Ascool, S., Sagun, L.,, Baity-Jesi, M., Biroli, G., and Wyart, M. (2019). Jamming transition as a paradigm to understand the loss landscape of deep neural networks. Phys- ical Review E, 10:11115..
Guo, H., Jin, J., and Liu, B. (2022). Stochastic weight averaging revisited.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
He, X., Zhou, Z., and Thicle, L. (2018). Multi-task zipping via layer-wis wise neuron sharing. arXiv preprint (ttt:088000999
10

---

Published as a conference paper at ICLR 2023
Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Korbllith, S., Farhadi, A., and Schmidt, L. (2022). Patching open ppeabbocab models by interpolating weights. arXiv preprint arXiv:2208.05592.
loffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448- 456. PMLR.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. (2018). Averaging weights leads to wider optima and better generalization. arXiv preprint
Izmailov, P.. Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. (2019). Averaging weights leads to wider optima and better generalization.
Juneja, J., Bansal, R., Cho, K., Sedoc, J., and Saphra, N. (2022). Linear connectivity reveals gener- alization strategies. arXiv preprint vrXiv:2205.12411
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima.
Kuhn, H. W. (1955). The hungarian method for the assignment problem. Naval Research Logistics (NRL), 52.
LeCun, Y. (1998). The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/.
Li, D., Ding, T., and Sun, R. (2018). Over-parameterizer deep neural networks have no strict local minima for any continuous activations. arXiv preprint arXiv:
Li, H., Xu, Z. Taylor, G., Studer, C., and Goldstein, T. (2017). Visualizing the loss landscape of neural nets. arXiv preprint /rXiv:17/2.099))
Li, Y., Yosinski, J.. Clune, J., Lipson, H., and Hopcroft, J. (2015). Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.07543.
Liu, C., Zhu, L.,, and Belkin, M. (2020). Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. arXiv preprint arXhv:2003.00307.
Markuš, N. (2018). Fusing batch normalization and convolution in runtime. https:// Accessed: 2022-09-28.
henadma rkus http://////uuongaaaaaaaaaaooooom
Mei, S., Montanari, A., and Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115633).E766-E7777.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning.
Neyshabur, B.. Bhojanapalli, S.. McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 3949-5955
Nguyen, Q. Mukkamala, M. C., and Hein, M. (2018). On the loss landscape of class of deep neural networks with no bad local valleys. arXiv preprint arXiv:1809.10744
Pittorino, F,, Ferraro, A., Perugini, G., Feinauer, C., Baldassi, C., and Zecchina, R. (2022). Deep networks toroids: Removing symmetries reveals the structure of flat regions in the landscape geometry. arXiv preprint arXiv:2202.03038,
Pittorino, F., Lucibello, C., Feinauer, C., Malatesta, E. M., Perugini, G., Baldassi, C., Negri, M., Demyanenko, E.,, and Zecchina, R. (2020). Entropic gradient descent algorithms and wide flat minima. arXiv preprint Stti:200060089977
Rosenblatt, E. (1958). The perceptron: a probabilistic model for information storage and organiza- tion in the brain. Psychological review, 65(6):386.

---

Published as a conference paper at ICLR 2023
Simonyan. K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X.::140911556
Şimgek, B., Ged, Jacot, A., Spadaro, F., Hongler, C., Gerstner, W., and Brea, J. (2021). Geometry of the loss landscape in overparameterizee neural networks: Symmetries and invariances. arXiv preprint 1rXiv:2105.12222
Singh, S. P. and Jaggi, M. (2020). Model fusion via optimal transport. Advances in Neural Informa- fion Processing Systems, 33:22045-2200
Tatro, N. J., Chen, P.-Y., Das, P., Melnyk, L, Sattigeri, P., and Lai, R. (2020). Optimizing mode connectivity via neuron alignment. arXiv preprint wxXv:2000.02439.
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. (2020). Federated learning with matched averaging. arXiv preprint arXiv:2002.064400.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lppe R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., al. (2022a). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 33965-23998.. PMLR.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Korbliihh, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., al. (2022b). Robust fine-tuning of zero-shot models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Putlern Recognition, pages 7959-7971.
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mmin novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.0774
Zhan, S.-h., Lin, J., Zhang, Z.-j.. and Zhong, Y.-w. (2016). List-based simulated annealing algorithm for traveling salesman problem. Intell. Neuroscience, 2016:8.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In Proceedings of the International Conference on Learning Representations.
Zhang, H., Dauphin, Y. N., and Ma. T. (2019). Fixup initialization: Residual learning without normalization. arXiv preprint (rxiv:9900 1001119192922 09321..
12

---

Published as a conference paper at ICLR 2023
APPENDIX
A FURTHER DISCUSSION OF RELATED WORK
Entezari et al. (222T) try to find low-barrier permutation between two SGD solutions by searching in the set of all permutations using simulated annealing. Neuron alignment methods use efficient heuristics to find a matching that maximizes defined similarity measure given two neural networks (Li et al. 2015: Ashmore and Gashler 2015 Singh and Jaggil 2020; Tatro et al. 2020; Pittorino ct al., 2022). This similarity measure is often based on correlation between weights or activations of the neurons in the same layer. Prior work was unable to achieve low-bariier for many standard architectures and/or challenging tasks (e.g., ImageNet classification).
He al. (2018) propose neuron alignment method in the context of multi-task model compres- sion. The algorithm leverages layer-wise Hessian approximation to match neurons by computing a similarity measure based their functional difference. (Singh and Jaggi (2020) propose model fusion algorithm which leverages optimal transport to perform neuron alignment in the Wasserstein space (Agueh and Carlier (00I), achieving barrier of 16% for ResNet18 8 trained on CIFAR-10; furthermore they show that by finetuning such interpolated networks, the original performance is recovered.
Tatro et al. (020)) point out that special care must be taken for networks that contain batch normal- ization layers when it comes to neuron alignment. They match neurons by optimizing a curve in the weight space between two models by aligning correlations of post-activations. The running statistics are normalized by training the model for one epoch, while freezing learnable parameters of the model. However, the crucial role of statistics reset in the context of linear interpolation of the models is not investigated. recent work by Pittorino et al. (2022) uses LayerNorm as part of symmetry removal, so that the network behavior remains unchanged.
B IMPLEMENTATION DETAILS
B.1 TRAINING HYPERPARAMETERS
Table
summarizes the hyperparameters we used to train the neural networks which appear in this work.
We train all networks using SGD with momentum 0.9. The weight decay and learning rates differ for each task, and are specified below.
For our MLP trainings, we keep the hyperparameters below constant across varying widths, depths. and datasets.
When training on MNIST and SVHN, remove cutout and horizontal flip from the list of data augmenations used by our ResNet20 training. Otherwise, we keep the below ResNet20 hyperparameters constant across different choices of width, dataset, and normalization layer.
Table 1: Training hyperparameters
Hyper-paue parameters
MLP
VGG
ResNet20
ResNet560/mageNett
Batch Size
2000
500
500
512
Epochs Learning Rate
100
100
200
300
Linear 0.2
Cosine 0.08
Cosine 0.4
Linear 0.5
Weight decay
0.0
0.0005
0.0001
0.0001
Data augmentation
Translate
Flip/Trann Flip/Translate
Flip/ hhtp/TranslateCCutout
Flip/RRC
B.2
PSEUDOCODE FOR PYTORCH MODULES
1. The first step of REPAIR is to measure the statistics of identified channels in the endpoint networks. There are many ways to do this, but we found that the following was efficient and had low code-complexity in Pytorch environment. For each module in the interpolated
13

---

Published as a conference paper at ICLR 2023
network whose outputs wish to REPAIR, wrap the corresponding modules in the endpoint networks with the following:
class TrackLayer (nn. Module):
def _s_niit__eeef, layer):
()
self.layer layer
self.bn hdm..ccchooo (len (layer. (layer..
self.bm.train
self.layez.evald def return (self.bn.running_ running self. ron.inniiggg_ar sqrt ()
(sel))
def (sel,, Soffiinpuuee inputs):
outputs self. (inputs)
Apply BatchNorm so that the running mean/variance are updated; discard the output. self. on (outputs)
return outputs
It then suffices pass small set of training data (~5,,000 examples) through the tracked endpoint networks. At this point, the statistics of each tracked module's output can be retrieved from the wrapping TrackLayer.
2. Next, we wrap each module in the interpolated network that we wish to REPAIR with the following:
class ResetLayer (nn (odulee
def _inii________ layer): super () ()__init_____ inii_
self.layer layer
self.bn B.n.Bacchhoor22d1een (layer. weight))
def st goal_mean, goal_std):
goal_mean
self. a.f.bn.wer
wwwpw.twww...................... data goal_std
def forward (self, x): return self.bn (self. layer(x))
The next step to iterate over each triple of corresponding (Tracklaaerr, TrackLayer, Re- setLayer modules coming from the two endpoint networks and the interpolated network. For each triple, we set the statistics of the ResetLayer to be the interpolation of the statistics from the TrackLayers. Working and minimal code to accomplish this be found in our code release.
C CLOSED-FORM APPROXIMATE VARIANT OF REPAIR
We present an alternative correction algorithm which generates channel-wist affine coefficients using a closed-form approximation, eliminating the need for forward passes in the interpolated network. This alternative is effective for MLP networks, but we found that it did not work well for deep convo-
MLP MNIST
80
Toot accuracy
Endpoint
Bafaro REPAIR
lutional architectures. In Figure g we compare the affine coefficients produced by this variant with those of the primary method in Section finding substantial correlation.
Attr REPAIR iloryard pass)
Añoo REPAIR (clceed-forn)
The variant proceeds as follows. Consider hidden unit the first layer of the interpolated network. As in Section 4 let X, represent the unit in the inter- polated network, and X, the same unit in the two endpoint networks, respectively. Condition 4
2
10
15
20
30
35
40
45
50
Figure Closed-form variant of REPAIR. flosed-form, approximate variant of REPAIR per- forms similarly to the full algorithu for the pur- pose of correcting deep MLPs which were trained MNIST.
14

---

Published as a conference paper at ICLR 2023
will already be satisfied for this unit by virtue of the equation Xa = (1 a) X1 + a X2 (this holds only because the unit is in the first layer). Given the values (,), Var Var(X2), and 1,, which can be computed only using forward-passes in
the endpoint networks, is possible compute the variance of X, exactly according the formula
Var(X.) = (1 - Var a² Var(2)) 2ax(1 - (o),
(5)
Therefore, to satisfy condition 4 for this unit, the rescaling coefficient / must be
(1 (1-o0 a) - sti( + a std((2)
8 = VLLaa)Var(X;) a Var(( + 2a(1 Co
which is simply the desired standard deviation divided by the standard deviation of Xa. For each unit in the first layer, this factor is exactly correct in order to obtain the desired statistics. For deeper layers, this factor
approximation which empirically test.
In Figure (right), we apply this rescaling to every hidden unit of MLPs of depth between 2 and 50 hidden layers, which are linearly interpolated (a 0.5) between aligned endpoints networks trained on MNIST. We find that this rescaling significantly improves the performance of such interpolated networks. In particular, we obtain interpolated checkpoints of up to 27 layers that achieve over 90% accuracy, whereas without a correction, we hit this limit after only 6 layers. For this case, the approximate rescaling performs similarly the full exact method presented in Section
D ADDITIONAL PLOTS
LayerNorm ResNets
ResNetbU ImageNet
REFAIR
After RE PAIR
Gii Re Basin
Loss Lossbama bamar
Itop
interpolation
Before REPAIR
After REPAIR
1x
2x
Ax
Ex
16x
32x
6.
Width multiplier
Inlerpotetion
Figure 9: Comparison to Ainsworth al. (002)). Left: We compare barrier values of LayerNorm bayeeroomm-baedd ResNets, for our baseline and REPAIR, the barriers reported Ainsworth al. (2022). We find that for networks of width up to 8x, our baseline barrier values are comparable, indicating that our baseline alignment method (Li et al. 2015) performs similarly to those explored in Ainsworth et al. (002).. With REPAIR, our
barrier values are lower. For the case of 3-w-whtth networks, our barrier value is higher than that reported in results ImageNet using REPAIR with those Ainsworthh al. (2022). We note that their ImageNet experiments, [Ainwwrthh
Ainsworth al. (022). Right: We compare
(2022) reset the BatchNorm statistics their interpolated ResNet50s based on our suggestion to do so. The extra barrier reduction our
results can therefore be attributed to the fact that REPAIR also statistically corrects the outputs of cach residual block.
15

---

Published as a conference paper at ICLR 2023
Figure 10: Aligned convolutional filters for the first layer in ResNet50. We train two ResNet5Os Im- ageNet independently, calling these models and B. Left: The first-layer convolutional filters of model A. Middle: The filters of model B, having been permuted order to maximize the total activation-wis correla- tion to the corresponding filters of model A. Some paired filters appear nearly identical, while for others there was no close match. Right: The filters of model B in their original positions.
ResNet20/ CIFAR-10
ResNet20 / CIFAR-10
Before REPMIR (BaichNarm)
2.
After REPAIR (BatehWom))
Accurany barte"
80
After REPAIR (LayerNor)
(LayerNom)
Before REPAIR (BatshNorm)
Before REPAIR (FixUo)
Ahhr REPAIR (BatchNorm)
-0-
Aftor RERAIR ("ocUp)
Text
1.0
Befare REPAIR (LayerNorm)
Noet REPAIR (LayeNNrm))
0.5
2x
16x
0,
Witth Witth multipterr
Interolaatinn
Figure 11: Effect of normalization layer. Left: We replicate Figure 4 (left) with barriers measured in terms of test accuracy. Right: We replicate Figure with barriers measured in terms of test loss.
(iecllse500 Insepatie
(eeelabe nocaaber
Reehhel88 (2E A
intopor Eweore RM RMR
Eied eneepcadion
Bittre li muut
Als REFARR (olly can bopss)
Debuttered
Alter RDPNIR
REFAI
2.2
2.4
02
C.4
0.5
0.5
6.
0.2
0.4
0.9
0.2
55
Ietppcstoon
penpolaton
1130
Figure 12: Barrier curves in terms of loss. We report the performance of interpolated ResNets in terms of test loss on ImageNet and CIFAR-10 10. The corresponding ligures terms of accuracy are Figure and Figure (middle).
ResNe120
VGS CIFAR
MNIST
26
2.-
Tartvor-M4S
-+- CITA3-11
Defore REFAIR (ran)
1:81
Belorw REEARR
Ale RE 1E%.R trant
Bufine REFAI Qua(
100
Alee 1514 (koi)
85-
60
250
612
Vilone
V9619
15647h
With mutiier
Degit sinior
Figure 13: Network width and depth. We replicate Figure with barriers measured in terms of test loss instead of test accuracy.
16

---

Published as a conference paper at ICLR 2023
CIFAR 100, Spiit data training
CIFAR-100, Split data training
Interpoiating
Ensembiling
Texi aes may
Flas-comected 6
Tool loss
-2-
Combined data fraining
Interpollting wilh REPAI
Ensemaling
Biss-coractedd 8,
66
Combined data training
0.2
0.4
0.6
QU
8, 0.2
0.4
0.6
0.8
6,
Interaclation
Intorpolation
Figure 14: CIEAR-100 split data experiment against a baseline. Two networks are independently trained on disjoint, biased subsets of the CIFAR-100 training set. With respect to performance on the full CIFAR-100 test set, report the performance of ensembling/mixing the logit outputs of the two models (orange), and the performance of interpolations in weight-space between two aligned models (blue), with REPAIR applied to every interpolated network. baselines, we report the performance of a single model which trained on the full CIFAR- 100 training set (red), and the performance of the two endpoint networks, after distribution- shift correction has been applied (green). We find that there exist REPAIRed interpolated checkpoints which outperform either endpoint network in terms of both test loss and accuracy, showing that constructive merging of models possible. The distribution-shift correction (green line) is follows. Consider an endpoint network A; was trained with training set was biased 4:1 towards the first classes, so that the set, network A over pver-predicts these first 50 and under- preeirtteeictss the second 50 classes, leading to increased loss. Our baseline is then scale down the first 50 logits of network A such that its predictions on the test set become balanced over the 100 classes. We find that this haseline, applied separately to either endpoint network (thus two green lines), outperforms the best gain that be obtained from constructive merging of the two models via REPAIRed interpolation.
Drect marpoletaon
Allgned Interpalation
Direct marp. with R.:AU
Alignos intorpp with REPAIR
CIFAR-10
VGGLL CIFAR-
ResNet 50 ImageNet
100
160
1.00
Thtt ACCUPNC
Thrst ACCUEACY
60
80
60
20-
20
23
0.4
Inttar polal.un
Interpalation
interpecations
Figure 15: REPAIR without alignment. the preceding experiments, have always first aligned the
neurons of two endpoint networks before interpolating. this figure also report accuracy curves interpolations between the original unaligned endpoint networks, both with and without REPAIR. We note that in the case of ResNet18. we apply REPAIR to convolutional outputs, which is exactly equivalent to resetting BatchNorm statistics. For VGG11, the architecture does not contain normalization layers, and REPAIR applied to convolutional outputs. For ResNet50, we apply REPAIR both convolutional outputs and residual block outputs, as described in Section 5.2 It appears that REPAIR effective towards the endpoints, and neuron-alignment becomes essential near the midpoint. Both methods are necessary in order to have a high- performing midpoint.
17


Reviews:
[{"review_id": "f_X0OhkwCQ", "reviewer": "Reviewer_SpdK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I thank the authors for their responses. I am especially happy about the added section 3.2. that tries to provide some theoretical reasons for the observed phenomenon of variance collapse. I am raising my score to 6.\n\nThe authors in this paper observe that interpolating between neural network weights decreases the variance of activations. By counteracting this phenomenon they show that much of the performance drop in interpolated models can be recovered. They show this across multiple architectures, normalisations and datasets.", "summary_of_the_review": "The paper is clear, simple and proposes an effective modification to interpolation variance collapse. There is some concurrent work, to which it would be great to provide a more detailed comparison. Moreover, it would be great if the paper would go beyond noticing that some statistics changed, modifying this and obtaining better performance. Ideally, after reading this paper, we would have a slight advance (even preliminary, anecdotal) about why variance collapses in interpolated models, how the modification changes the interpolation path and why this works.", "strength_and_weaknesses": "Strengths: The paper is very clear and well written. It is easy to follow and presents a concise observation leading to a simple modification to recover the performance in interpolated models. The experiments are straightforward and clear. They also do a good job in relating to existing research on this subject.\n\nWeaknesses: It would be great if some more theoretical analysis were provided in an attempt to understand why variances collapse, how their modifications change the standard interpolation. While it is great to make a simple observation, propose a fix and demonstrate that it increases performance; one is left wondering, after reading the paper, what exactly we have learned from this scientific study.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aRmEnktTcw", "reviewer": "Reviewer_zoWW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the linear mode connectivity problem empirically and uncovers a new phenomenon called variance collapse (where activation variances of interpolated networks between two parent networks reduces significantly). Test accuracy barrier reduction is shown successfully for the interpolated networks (for CIFAR-10, barrier goes down to about 1.5% gap in test accuracy between parent models and the interpolated network).", "summary_of_the_review": "The paper can be significantly stronger if it actually shows improved results on downstream applications and manages to reduce the ImageNet barrier further with more advanced BN tuning methods.\n\n==== UPDATE AFTER REBUTTAL ====\n\nI have read the author response and other reviews. I am raising my score to 6 since the authors have reasonably addressed my main concerns. ", "strength_and_weaknesses": "The paper has following strengths:\n\n1. It is interesting to see the variance collapse phenomenon on the interpolated networks.\n\n2. Barrier reduction for CIFAR-10 down to 1.5% is significantly better than existing techniques.\n\n3. The paper is well-written and nicely explained. Related work is cited and discussed very well.\n\n\nThe paper has following weaknesses:\n\n1. The correction method presented by the authors is trivial. It is well-known that tuning batchnorm can significantly improve accuracy (sometimes even in completely unsupervised, domain adaptation settings, e.g., see: TENT: https://openreview.net/forum?id=uXl3bZLkr3c). Hence, it is no surprise that fixing batch norm statistics would help in this case. While the proposed method achieves great results on CIFAR-10, we can see that on ImageNet (despite the proposed method being the current best), the gap in interpolation accuracy is very significant (making this method not too useful on large-scale datasets). I was wondering if instead of just initializing affine parameters and re-computing the running mean/variance statistics, the authors were to finetune (or learn) the BN stats similar to TENT above, would we see significant reduction in the ImageNet barrier (compared to their current results)? Having better numbers for large scale datasets like ImageNet would indeed make this work useful.\n\n2. In the final sentence, the authors state “a better understanding of linear interpolation would result in improvements in applications like weight-space ensembling, checkpoint averaging, robust finetuning, etc.”. I think the current paper lacks the use of proposed ideas (e.g., fixing variance collapse, etc.) within a real application framework. It would make the paper significantly stronger if the authors could pick a couple of the above applications and show that fixing this subtle issue significantly improves the current solutions in those domains.\n\n3. There are many figures in the paper that do not contribute much to other results. Example, training loss barrier, test loss barrier, etc. (Fig. 6ab, Fig. 7a), do not provide any new intuitive information over the test accuracy barrier results. I suggest to remove the redundant information and include more results on downstream applications and other experiments suggested above (e.g., TENT on linear interpolations, etc.).\n\n4. Finally, I am a bit unclear on what happens when the model already has BNs (e.g., ResNets). Do you add a new BN layer in your correction method (even if a BN layer is already present)? Page 6, “We present the following practical…” paragraph seems to suggest that you add a new BN layer. If you do add an extra BN layer: When you do interpolation, wouldn’t you already set the affine parameters (of existing BN) to the respective interpolated means and interpolated variances from parent networks (similar to all other weights in the network)? In that case, I am not sure how the new BN layer adds anything new (other than recomputing the running mean and running variance statistics with training data, which is similar to other methods like TENT that also tune the BN layer (but they also do some more learning on the BN layers)). Also, when you do the permute invariance methods, do you consider the BN layers (for somehow permuting the affine parameters, running mean, running variances)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JU-08Vot7z", "reviewer": "Reviewer_LkzC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes the striking empirical phenomenon that, accounting for permutations, two independently trained networks of the same architecture will have no energy barrier on the linear path between them. Furthermore, the authors demonstrate an issue with naive interpolation for deeper networks where the activations experience a variance collapse on this linear path, creating an 'artificial' barrier on this connected path. To resolve this, the authors propose a simple renormalization procedure that re-establishes mode connectivity (albeit no longer linear). ", "summary_of_the_review": "The empirical validation of the LMC phenomenon is important, and this work does a good job of diagnosing the issues with LMC when applied in practice. The renormalization method appears novel and significantly improves the loss barrier along the bath between the two models.", "strength_and_weaknesses": "Strengths:\n- Renormalization without permutation significantly lowering the barrier is an interesting finding complementary to the concurrent Ainsworth et al. 2022 result.\n- Renormalization technique is an improvement over Ainsworth et al. 2022, which requires the replacement of BatchNorm layers with LayerNorm layers to alleviate the barrier in Residual networks.\n\nWeaknesses:\n- The pattern of increasing test error corresponding to a larger barrier is an interesting finding. But there doesn't seem to be an intuition on why this is true. Without an explanation, it's unclear why this pattern is viewed as expected or \"simple\"\n\nFrom the paper:\n\"the high barrier of ImageNet models stems in part simply from the relatively higher test error of said models\"\n\nComment:\nCould the authors elaborate on the connections between robustness and linear mode connectivity? Is there an expectation that linear mode connectivity will indicate robustness?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster