Optimizing Loss Landscape Connectivity via Neuron Alignment
Authors: N. Joseph Tatro
Pin-Yu Chen
Payel Das
Igor Melnyk
Prasanna Sattigeri
Rongjie Lai
Venue:  (2020)
Keywords: deep learning
optimization
non-convex optimization

Abstract: The loss landscapes of deep neural networks are poorly understood due to their high nonconvexity. Empirically, the local optima of these loss functions can be connected by a simple curve in model space, along which the loss remains fairly constant. Yet, current path finding algorithms do not consider the influence of symmetry in the loss surface caused by weight permutations of the networks corresponding to the minima. We propose a framework to investigate the effect of symmetry on the landscape connectivity by directly optimizing the weight permutations of the networks being connected. Through utilizing an existing neuron alignment technique, we derive an initialization for the weight permutations. Empirically, this initialization is critical for efficiently learning a simple, planar, low-loss curve between networks that successfully generalizes. Additionally, we introduce a proximal alternating minimization scheme to address if an optimal permutation can be learned, with some provable convergence guarantees. We find that the learned parameterized curve is still a low-loss curve after permuting the weights of the endpoint models, for a subset of permutations. We also show that there is small but steady performance gain in performance of the ensembles constructed from the learned curve, when considering weight space symmetry.

Under review as a conference paper at ICLR 2020
OPTIMIZING Loss LANDSCAPE CONNECTIVITY VIA NEURON ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
The loss landscapes deep neural networks are poorly understood due to their high nonconvexity. Empirically, the local minima of these loss functions can be connected by learned curve in model space, along which the loss remains nearly constant. Yet, current path finding algorithms do not consider the influence of symmetry in the loss surface caused by weight permutations of the networks cor- responding the minima. We propose framework to investigate the effect of symmetry on landscape connectivity by directly optimizing the weight permuta- tions of the networks being connected. To learn locally optimal permutation, we introduce a proximal alternating minimization scheme and prove some con- vergence guarantees. Using an neuron alignment technique, we derive an inex- pensive heuristic for approximating locally optimal weight permutation. Em- pirically, optimizing the permutation critical for efficiently learning simple, planar, low-loss curve between networks that successfully generalizes. As an ap- plication, we also show that there is performance gain, particularly for models with less parameters, in the accuracy of ensembles constructed from the learned curve with neuron alignment permutation.
INTRODUCTION
Loss surfaces of neural networks have been of recent interest in the deep learning community both from a numerical (Dauphin al. 2014); Sagun et al. (0144) and theoretical (Choromanska et al. (2044; Safran & Shamir (2015) perspective. Their optimization yields interesting examples of a high-dimensional non-convex problem, where counter-intuitively gradient descent methods suc- cessfully converge to non-spurious minima. Practically, recent advancements in several applications have used insights on loss surfaces to justify their approaches. For instance, Moosavi-Dezionn et al. (2019) investigates regularizing the curvature of the loss surface to increase the adversarial robust- ness of trained models.
One interesting question about these non-convex loss surfaces is to what extent trained models, which correspond to local minima, are connected. Here, connection denotes the existence of a path between the models, parameterized by their weights, along which loss nearly constant. There has been conjecture that such models are connected asymptotically, with respect to the width of hidden layers. Recently, Freeman & Bruna (2016) proved this for rectified networks with one hidden layer.
When considering the connection between two neural networks, it is important for us to consider what properties of the neural networks are intrinsic. There is permutation ambiguity in the indexing of units in given hidden layer of a neural network, and as result, this ambiguity extends to the network weights themselves. Thus, there are numerous equivalent points in model space that correspond to a given neural network. This creates weight symmetry in the loss landscape. It is possible that the minimal loss paths between a network and all networks equivalent to a second network could be quite different. If we do not consider the best path among this set, we could fail to see to what extent models are intrinsically connected. Therefore, in this work we propose to develop technique for more robust model interpolation / optimal connection finding by investigating the effect of weight symmetry in the loss landscape. The analyses and results will give us insight into the geometry of level sets of the loss surfaces of deep networks that often hard to study theoretically.

---

Under review as a conference paper at ICLR 2020
Related Work Freeman & Bruna (016)) is one of the first studies rigorously prove that one hid- den layer rectified networks are asymptotically connected and established relevant bounds. Several recent numerical works have shown that parameterized curves along which loss is nearly constant can be successfully learned. Concurrently, Garipov et al.] (2018) proposed learning Bezier curves and polygonal chains and and[Draxles et al. (2018) proposed learning curve using nudged elastic band energy between two models. Gotmare et al. (2018) showed that these algorithms work even for mod- els trained using different hyperparameters, excluding network architecture. Recently, Kuditipudi et al. (2019) analyzed the connectivity between c-dropout stable networks. This body of work can be seen as the extension of the linear averaging of models studied in (Goodfellow et al. 2014). In fact, concurrent with our work, Singh & Jaggi (2019) applied neuron alignment to model averaging.
The symmetry groups in neural network weight space have long been formally studied (Chen al., 1993). While permutation ambiguity in the weights has been acknowledged, ostensibly ambiguity due scaling in the weights has received more attention in research. Numerous regularization approaches based on weight scaling such as in (Cho & Lee 2017) have been proposed to improve the performance of learned models. Recently, (Brea et al. (2019) studied the existence of permutation plateaus in which the neurons in the layer of network can all be permuted at the same cost.
second line of work studies network similarity. Kornblith al. (2019) gives a comprehensive review on the topic while introducing centered kernel alignment (CKA) for comparing the behavior of different neural networks. CKA an improvement over the canonical correlation analysis (CCA) technique introduced in (Raghu et al. 2017) and explored further in MMrcoos et al. 201 8). Another contribution in this direction is the neuron alignment algorithm from (Li et al. 2016)) which showed empirically that two networks of the same architecture learn subset of similar features.
Contributions We summarize the main contributions of this work as follows:
1. We formalize this problem, and apply proximal alternating minimization (PAM) scheme to split the problem into iteratively optimizing the permutation of the second model weights and optimizing the curve parameters. We prove convergence of this scheme to a local critical point for feed-forward neural networks which piece-wiss analytic functions and continuously diffcrentiable.
2. Motivated by the neuron alignment technique of (Li et al., 2016) and our PAM framework, we introduce heuristic for approximating the optimal weight permutation in order to learn aligned curves connecting networks up to a symmetry in their weights.
3. We perform experiments on datasets and 4 architectures affirming that more optimal curves can be learned faster with neuron alignment. We observe that this aligned permutation is close to a locally optimal permutation that PAM converges to under the same initialization.
4. For simple networks, we observe notable improvement in ensemble accuracy when constructing them by sampling the aligned as opposed to the unaligned curve or a set of independent models.
For the structure of this paper, we first review pertinent background on curve finding and neuron alignment Section [] Then, we introduce our proposed optimization models and algorithms for curve finding up to weight permutation in Section [3] In Section E we discuss our experiments detail. Section 5
we explore the effect of alignment on the performance of model ensembles constructed through sampling along the curve.
BACKGROUND ON CONNECTIVITY AND ALIGNMENT
In this section we review the existing approaches for loss optima connectivity and neuron alignment.
Loss Optima Connectivity To learn the minimal loss path connecting two N-dimensional neural networks, 0 and 02, we utilize the curve finding approach introduced in (Garipov et al.) 2018).
Here search for the path, r [0, 1] NR,, that connects the two models while minimizing the average of the loss function, L, along the path. This problem is formalized in equation] [
r arg min SEeoo.1 ((r(()))r (Dlldt
subject to r(0) = 0,, r() 02.
(1)
Steto.n) [[)(((]]]
For tractability, r" can be approximated by a parameterized curve Tæ, where ó denotes the curve parameters. For instance, described in Section 4 this paper will be using the quadratic Bezier
2

---

Under review as a conference paper at ICLR 2020
curve. Computationally, arclength parameterization, that is [t)ll) 1 for all t, is assumed to make optimization more computationally feasible. Note that if the endpoint networks are global minima and flat loss path does exist, then the optimal objective of equation is unchanged.
An equivalent view under the arclength parameterization is that we are minimizing E (C(r, (t)) where U is the uniform distribution the unit interval. This view is taken in Algorithm 2 in Appendix C For clarity, we emphasize that Te denotes the curve on the loss surface between two networks while (t))) denotes a point on that curve which a neural network.
Neuron Alignment We give an overview of the neuron alignment framework in (Li et al. 2016).
Given input d drawn from the input data distribution D, let X (d) R represent the activation values of channel i in layer of network 01. where & is the number of units in the channel. As an example, a channel could correspond to one unit in a hidden state or one filter output by a convolutional layer, where i would be the number of pixels in the filter respectively.
Given networks, 0 and 02, we define the channel wiie mean and standard deviation for 81 in equa- tion 2 We also define the cross-correlation matrix, c(1.2), denoting the cross-correlation between each channel in 02 and 0g in layer L.
k 14, = Een k
k v Buw Σ h a=1
Lia (d))
(x() (d)- -4.))
0
Eaw Ex=
[ (d) M1,
(2) (X(j.a()) (d) - mi,
(1122) [,,.]
(2)
(2) 0,1 1.5
To align the activations in layer between networks 0, and 02, the neuron alignment algorithm maximizes the sum of cross-correlation between aligned activations. Equivalently, this finds the permutation, P. that maximizes the trace of P c(1.) which is an instance of the linear assignment problem. We formalize this optimization model in equation 3 below, where K, represents the index set of activations layer I. We recommend Burkard & Cela (1999) reference for those unfa- miliar with the assignment problem.
trace CT
max P
1, 1,1-1, PE Kil
subject to The alignment technique is visualized in Figure [a This displays the cross-correlation matrix for the TinyTen network and CIFAR 100 dataset that we discuss later in Section 4 It is clear that the values along the diagonal are much stronger after alignment. Figure ThjureeThdisppp displlas the mean correlaononn at each layer between corresponding neurons. This figure also shows the standard deviation of this signal over a set of 3 network pairs. With this correlation signature being consistent over different pairs and being increased highly with alignment, we can feel confident that some subset of highly correlated features are being matched.
3 OPTIMA CONNECTIVITY CONSIDERING WEIGHT SYMMETRY
We clarify the idea of weight symmetry a neural network. 02 neural network on the loss surface parameterized by its weights. A permutation Pt is in K,)- the set of permutations Ki, the index set of channels in layer I. For simplicity suppose we have an L layer feedwffowwrrd network with activation function a, weights (Wi)/= and input X Then the weight permutation ambiguity becomes clear when we introduce the following set of permutations to the feedforward equation:
:= Y oo Pt. Wr..
oo P.W Xo (4)
Then we can define the network weight permutation P as the block diagonal matrix, blockdiag(P,, P Pr.-i). Additionally, PO denotes the network parameterized by the weights W,, P.W.PT.... W W.P.). Note that we omit permutations P. and Pz. as the input and out- put channels of neural networks have fixed ordering, they correspond to the identity I. Without much difficulty this framework generalizes for more complicated architectures. We discuss this for residual networks in Appendix
3

---

Under review as a conference paper at ICLR 2020
Tiny'an Behro Abgmmgnt
RuSNACS2
VOGLO
Before Alignment
After Alignment
Sefiic
Neetwork Activation
Netwrokk Activottons
Clps- Coss-Coreurr on
Crpss Loos-Comeede en 8 % Pa
MAp Wiy
10-
10
15
10
Network 2 Activations
Networkc Activations
(a) Cross-correlation between neurons
(b) Mean cross-correlation at each layer
Figure 1: la Cross-correlation between the activations in the first layer of a TinyTen model for CIFAR100. The plot on the left uses the original indices of the second network, while the plot on the right uses the reindexing of the second model consistent with alignment the first. Note the diagonal of the second matrix is much more positive than the first matrix, which implies meaning- ful correspondence between aligned units. (ib) The mean cross-correlation between corresponding units for each layer before and after alignment. The standard deviation of this correlation signature over a set of different network pairs is displayed. This shows that the quality of the correspondence between the average pair of units at each layer can be strongly improved through alignment.
3.1 CURVE FINDING UP TO SYMMETRY
From equation it becomes clear that the networks 02 and P0 share the same structure and intermediate outputs up to indexing. Taking weight symmetry into account, we can find the optimal curve connecting two networks up to symmetry with the model in equation
4
$" P
arg min 6.P
(5)
subject to
(0) = ro P02. P blockdiag P1, P2...., PL-1)
P. 1l for E {1, L 1}
3.1.1 PROXIMAL ALTERNATING MINIMIZATION AS A FRAMEWORK
We first introduce framework to solve the generalized problem in equation 5 Theoretically. this problem is fairly complicated and hard to analyze. Numerically, approaching the problem directly need to store gradients of and P
with first order methods could be computationally intensive simultaneously. The problem can be more easily addressed using the method of proximal alternating minimization (PAM) (Attouch et al., 2010). The PAM scheme involves iteratively solving the two subproblems in equation 6 Here we let (ΦØ, P) denote the objective function in equation 5 We only consider parameterized forms of that satisfy the endpoint constraints for all ø and P. For generality, we let R denote a regularization term Ø.
pk+1 arg minp
Q( P) + 2vp
1 PP- 111
such that
P. E HT |a for 1 EE {1,2, 1-
P= blockdiag( (1, P2, Pt.-))
g++ arg min,
@ pk+1) )+ + R(6) + 2.1l 11
Computing the unaligned curve is equivalent to solving the PAM scheme with a very large value of vp. In fact, able to prove local convergence results for certain class of networks.
Theorem 3.1 (Convergence). Let (a++1 pk+1] } be the sequence produced by equation 6 Assume feed-forward neural network with activation function for I E (0, 1). Assume that C, ræz and are all piece-wise analytic functions in C and locally Lipschitz differ- entiable in & and P. Additionally, assume R is piece-wise analytic in the primal variables and bounded below. Lastly, assume that the input data is bounded and the norm of the network weights are constrained to be bounded above. Then the following statements hold:
that rl(l) corresponds
1. Q(0++1, pk+1) R(o++1) + 200 1111ot11
p
pk S Q(0 ph
20P
R( 0
2. f0t pb) converges critical point of Q(1, P) + R(
4

---

Under review as a conference paper at ICLR 2020
Proof
See Appendix D
Remark Theorem 3.1 does not extend to neural networks with ReLU activation functions. In Appendix D we address technique utilizing this theorem for learning curve connecting rectified networks while still generating sequence of iterates with monotonic decreasing objective value.
3.1.2 NEURON ALIGNMENT AS AN INITIALIZATION
In spite of convergence guarantees. PAM still requires good initialization the loss landscape is nonconvex. This is critical for avoiding convergence to non-giobal optima. Conceptually, neuron alignment introduced in (Li al. 2016 able to match subsets of similar feature representations. Thus, we believe that the permutation on the network weights induced by neuron alignment could be meaningful enough to provide good initialization of P.
In practice, we solve the linear sum assignment problem formulated in equation 3 using the Hun- garian algorithm. See Kuhn (9555) for further reading on the Hungarian algorithm. Algorithm summarizes the process for efficiently computing permutation of the network weights from neu- ron alignment. For an L layer network with a maximum layer width of M, we compute P using a subset of the training data. Then the cost of computing the cross-correlation matrices for all layers is dominated by the forward propogation through the network accumulate the activations. The running time needed to compute all needed linear assignments is O(LM*). This is the order of the running time associated with one iteration of forward propagation. Then neuron alignment is relatively cheap as the time complexity of computing curves using neuron alignment is on the same order as traditional curve finding. We refer to these different curves as aligned and unaligned.
Data: Trained Neural Networks 0 and 02. Subset of Training Data Xo
Result: Aligned Neural Networks 02 and P0₂ Initialize P02 = [W7. W ... W as [W?, W W for each layer l (1, 2, ..., 1} do
for each network j in (1,2} do
compute activations, x a w
for each element in the batch, vectorize X if applicable:
compute, z(²), the Z-score normalization of the activations
end
compute the cross-correlation matrix, C² = z (2)
compute P: by solving the assignment problem associated with c(1.2) using the Hungarian algorithm;
update W? P.W., W W2I1!
end
Algorithm 1: Computing Permutation via Neuron Alignment
4 EXPERIMENTS
Datasets In experiments, trained neural networks to classify images from CIFARI0 and CIFAR 100 (Krizhevsky et al. 2009), as well as STL.10 (Coates et al. (001).. The loss function is the cross entropy loss the softmax of the logits output by the networks. 20% of the images in the training set are used for computing alignments between pairs of models. We augment the data using color normalization, random horizontal flips, random rotation, and random cropping to prevent the models from overfitting on the training set.
Architectures Four different model architectures are used. Table summarizes relevant properties of these architectures. The first architectures considered were the TinySix and TinyTen models. TinyTen, introduced in (Kornblthh al., 2019), is a narrow 10 layer convolutional neural network that uses batch-normalizatioo ReLU activations, and global average pooling. TinySix is equivalent
5

---

Under review as a conference paper at ICLR 2020
to TinyTen with layers 2, 4. 5, and 7 removed. These useful models for concept testing and allow us to gain insight to networks that are underparameterized. We also include ResNet32 (He) et al. 2016) in experiments to understand the effeet of skip connections curve finding with alignment. VGG16-BN is the third architecture that we considered in our experiments (Simonyan & Zisserman 2014). VGG16 has significantly more parameters compared to other models. We chose this set of architectures for its varying properties and because of their prevalence in numerical experiments in related literature.
All models used as curve endpoints are trained using stochastic gradient descent. We set a learning rate of 1E-1 that decays by a factor of 0.5 every 20 epochs. Weight decay of 5E-4 was used for regularization. Each model was trained for 250 epochs, and all models were seen to converge.
Table 1: Properties of models used in this study
Number Parameters
Accuracy CIFARI00
Model
Depth
CIFARIO
STLI0
TinySix
34,596
77.0
40.1
65.1 = 0.1
TinyTen
86,778
10
88.7
58.1
73.8 0.3
ResNet32
466,906
32
92.9
6..1
76.5 0.3
VGG16-BN
15,253,578
16
93.1 0.2
0.3
72.5 1.5
Quadratic Bezier curves All curves parameterized quadratic Bezier curves. Bezier curves are popular in computer graphics as they can be defined by their control points. In the current study, refer endpoint models 01 and 0₂ well as the control point, O.. Then is defined in equation]] 7
(t) (1 (1-
+
Then 0g is the learnable parameter in d. Of course could consider more complicated curve parameterizations. In practice, we find a simple curve to be enough for our experiments, and consider the learning of planar curve along which loss is nearly constant to be significant in itself.
We explicitly define this curve for use in the PAM algorithm in equation g Here the curve has been
reparameterized that the control point is function of the permutation P. Ó, captures the devia- tion the control point from the linear midpoint between 0 and P02. For PAM, O, is the learnable curve parameter Ø. zero initialized so that the initial curve linear interpolation between models as traditional curve finding. This coupling of the control point with the permutation was seen to be critical for the success of PAM.
0
P02
r(t: Ge P) = =1 (1-)) 0) + 2(1 - t)t
+ 0 + PP2
(8)
2
4.1 TRAINING CURVES
For each architecture, we train 12, and different models using different random initializations for CIFARIO, CIFAR100, and STL10 respectively. Thus we have or 3 independent model pairs
for a dataset. We learn four classes of curves:
Unaligned: Solution algorithm given networks 0 and 0₂
PAM Unaligned: Solution to equation given networks 02 and 02 with p(0)
PAM Aligned: Solution to equation given networks 0, and 02 with p(0) PAi
Aligned: Solution algorithm given networks 0, and PA
where PAi denotes the permutation learned by neuron alignment (algorithm
We learn PAM curves for all architectures except VGG16, as its size made this computationally prohibitive. We train two sets of each curve class. One set involves the curves learned when the
random seed for curve finding is fixed for all model pairs. The other set consists the curves learned when the random seed is different for each model pair. We find that the learned curves for

---

Under review as a conference paper at ICLR 2020
different seeds are similar up reindexing the endpoints. For Figures of curves so that interesting geometric features on the loss surface are not averaged out. For tables and other figures,
2 and
the first set
the second set of curves they more general.
The average accuracy along the curve with standard deviation is reported for each com- bination of dataset, network architecture, and curve class. This shows that aligned curves not only outperform the unaligned curves which do not consider the permutation ambiguity, they perform as well as the PAM curves which learn locally optimal permutation Note that aligned accuracies are typically as high as the trained model accuracies in Table 1
Table 2:
Model
Curve Class
CIFARIO
CIFARI00
STL10
Unaligned
73.8 0.4
38.8
6.31
TinySix
PAM Unaligned
75.8
0.3
6.3
PAM Aligned
76.3 0.3
65.4 4.5
Aligned
76.3 + 0.3
4.5
65.4+00.2
Unaligned
87.2
0.2
73.7
TinyTTn
PAM Unaligned
87.9
5.11
73.9 + 0.3
PAM Aligned
88.6 0.1
58.
74.0
Aligned
88.6 - 0.1
5.2
74.1
Unaligned
92.4 0.2
0.2
0.2
ResNet32
PAM Unaligned
92.5 0.1
6.18
0.3
PAM Aligned
92.5 0.1
0.2
76.9 0.2
0.2
Aligned
92.9 + 0.2
6..7
76.7 + 0.2
Unaligned
93.0 0.1
0.1
74.5 + 1.0
VGG16
Aligned
93.3 0.1
0.1
74.7 + 0.8
4.1.1 NEURON ALIGNMENT
First, we investigate the effects using neuron alignment as a heuristic for curve finding up to symmetry. That is, between networks 0 and PacO2. The unaligned and aligned curves were both trained for 200 epochs using stochastic gradient descent with an annealing learning rate. The training of these curves share the same hyperparameters as the training of the individual models.
determining some weight permutation PAI and then finding the curve
TryTen
TryTen
22.5
3.6
1.0
62
G6
VG016
2.5
Unaigeed
(a)
(b)
Figure 2: Test loss (left) and accuracy (right) of the learned quadratic Bezier curve between model endpoints trained on CIFARI00. Results are compared for aligned (blue) and unaligned (green) curves. This shows that aligned curves have better generalization performance and do not suffer from large drops in accuracy typical for unaligned curves.
The test accuracy can be seen for each dataset and curve class in Table Clearly, the aligned curves outperform the unaligned curve. In many cases, the average accuracy along the aligned curves in
'Strictly speaking, the algorithm converges to local optima in the convex relaxation of the domain. The learned permutation the projection of this optima the feasible

---

Under review as a conference paper at ICLR 2020
TinyTen
ResNet32
VSG16
70
30
80
1
30
20-
30
30
10
20
30
6. 8,
Figure 3: Test accuracy CIFARIOO across the plane containing 0,, 02. and Pat22, where Pa: determined using neuron alignment. This plane contains the two different intializations used in our curve finding experiments. The default initialization, 02 01, and the aligned initialization, Pat2- 0I. This shows that the aligned initialization notably better.
TinyTen Aligned Endpolnts
ResNet32 Aligned Endpoints
Aligned Endipoints
70
50
60
40
-3
30
20-
10-
20
5-
10
10
Model
TinyTen
ResNer32
VGG16
Unaligned Endpoints
Unaligned Endpaints
Unaligned Endpoints
10
50
60
40
40
40
20
30
10
10
Model
20
Figure 4: Test accuracy on CIFAR100 across the plane containing the bezier curve, (t)
comparable to the trained models used endpoints. Table contains the minimum accuracy along the curve, indicating that while aligned curves do not suffer from the same generalization gap that unaligned curves are prone to. Finally, Table 5 the training loss for each case at convergence. Overall, it is clear that the strongest gains from using alignment are in the case of underparameter ized networks, As seen in Table 2 the largest increase in performance is for TinySix on CIFARIO while the smallest gain made for STL10 on VGG16. This is inline with observations by (Freeman & Bruna 2016).
The test loss and accuracy along the learned curves for CIFAR100 are shown in Figure The corresponding Fourier transform of the loss along the curve for assessing curve smoothness dis- played in Figure [T] We observe that, as expected, the accuracy each point along the aligned curve exceeds that of the unaligned curve, while the loss along the curve is also smoother with neu- ron alignment. We are comparing loss and accuracy at the curve parameter, prominent presence of the accuracy barrier along the unaligned curve around at 0.8 for all models. This accuracy barrier corresponds to a clear loss barrier for Tiny- 10 and ResNet32. contrast, for VGG16 there is valley in the loss function this point the unaligned curve with worse gener- alization performance, Overall, we find that loss along the aligned curves varies more smoothly as seen in Figure TT and this leads to better generalization of the interpolated models.
Noteworthy is the
8

---

Under review as a conference paper at ICLR 2020
Figure displays the planes which contains the initializations for curve finding. It is clear that the aligned initialization has better objective value. This can also be seen for the other datasets in Figure The planes containing the learned curves displayed in Figures and [3] These are the planes containing 0,. P02, and c.. although the control point is out of bounds of the figure. The axis is determined by Gram-Schmidt orthonormalization. The loss displayed the planes containing the
7
linear initializations and the Bezier curves can be seen in Figures and 12
TiryTen
Resitet32
VGGG1
TinyTTe
Reshiet32
VGG16
200
Algnad
Aigaed
Algred
60.5 vo.
300
14-
17.
Accurscy
Accurscy
Loz
135
19
38+
545
34-
tpoch
spoch
spoch
Epach
Eppac
tpoch
(a) CIFAR100 training loss
(b) CIFAR100 training accuracy
Figure 5: The training loss and training accuracy for learning the quadratic Bezier curve between model endpoints. These are compared for aligned and unaligned curves. This shows that the training of aligned curves converges to lower loss value in less epochs than the training of unaligned curves.
Practically, the neuron aligmment heurisite for determining the permutation P may be enough and avoids performing more complicated optimization. We see this by noting the relative flatness of the accuracy along the aligned curves in Figure 26 Additionally, Figure 5 indicates much faster convergence when learning & using neuron alignment, which is quite impressive. For example, the aligned curve takes 100 epochs less to achieve the training accuracy that the unaligned curve converges to, when TinyTen used on CIFARI00 Even for VGG16, the aligned curve reaches the milestone 40 epochs earlier. Figures g displaay these training curves for the additional datasets.
While these observations are promising, we intend to provide insight into why neuron alignment works, whih is addressed in Appendix F where we investigate how the alignment is preserved along the learned curve. We find that the midpoints of the unaligned curves are highly aligned to each endpoint even though the endpoints themselves are weakly aligned best. Then, we find that curve finding essentially trying smoothly interpolate similar feature representations. Sensibly, neuron alignment of the endpoints makes this task easier.
4.1.2 PROXIMAL ALTERNATING MINIMIZATION
Proximal alternating minimization provides comprehensive formulation for learning the weight permutation P directly, coupled with some convergence guarantees. We find that curves learned using PAM perform better than the unaligned curves as seen in Table 2 As was the case for the aligned curves, this performance gain is more notable underparameterized models. Notably, the aligned curves perform comparably to PAM aligned. This indicates that PAi is already close to the locally optimal permutation when PAI is chosen as the initialization for PAM. Additionally, the performance gain of PAM Aligned over PAM Unaligned shows that this permutation is not easy to learn when ((0) is not necessarily close. Then training aligned curves is inexpensive way to approximate the solution to rigorous optimization method with good initialization. We stress that this observation shows that the gain from neuron alignment is not trivial.
We now address some technical points in the training of the PAM curves. To learn each eurve, we perform iterations of PAM. The permutation subproblem entails 20 epochs of projected stochastic gradient descent to the set doubly stochastic matrices. This done as the set of doubly stochas- tic matrices is the convex relaxation of the set of permutations. This projection through 20 iterations of alternating projection of the updated permutation to the set of nonnegative matrices and the set of matrices with row and column sum of After the 20 epochs of projected
accomplished
gradient descent, each layer permutation is projected to the set of permutations, I Kil- The curve
9

---

Under review as a conference paper at ICLR 2020
parameter subproblem, which optimizes 0. from equation 8 entails 40 epochs of stochastic gradient descent. The same hyperraaaameerss are used as in training the endpoint models. The learning rates are annealed with each iteration of PAM. This training can be seen for CIFAR 100 in Figure 10
The outlier case in Table 2 is for ResNet32 where Aligned notably outperforms PAM Aligned. This result is due to the integral nature of the problem. Essentially, during PAM Aligned, the scheme learns a more optimal doubly stochastic matrix that actually projects to a worse permutation at the end of the permutation subproblem. This is a known possible negative, when solving a problem in the convex hull and then projecting back to the feasible set.
APPLICATION: CONSTRUCTING ENSEMBLES ALONG ALIGNED CURVES
We investigate if the diversity of models along the suffers due alignment through the lens of constructing network ensembles. We consider the simple ensemble that performs classification by averaging the probability distributions output by the individual networks. In our experiment, we look at four cases. The ensemble formed by considering the curve endpoints, the ensembles formed by the curve endpoints and the midpoint, and an ensemble of the curve endpoints and a third inde- pendent model. We consider ensembling only on the CIFARIO0 dataset with results for the aligned and unaligned case being summarized in Table 3 We see that at best, ensembles constructed by sampling along the unaligned curve perform as well as the independent ensemble. When construct- ing ensembles by sampling along the aligned curve, those outperform the independent ensemble for the TinySix, TinyTen, Ten, and Resnet32 case and show comparable performance for VGG16. The enhanced performance is most obvious on TinyTen. This result makes sense since ensembling has been reported to lead to better performance increase for simpler networks (Ju et al., 2018). This result is encouraging as ensembles of simple models are common in practice. We also note that the performance gain from the aligned ensemble is comparable in magnitude to the gains of Fast Geometric Ensembling in (Garipov et al. 2018) over Snapshot Ensembling (Huang et al. 2017).
Table Accuracy of model ensembles constructed from the curve connecting trained models on the CIFAR 100 dataset. Standard deviation is reported as well. All ensembles function by averaging the predictions of their component models. The curve ensembles are constructed by sampling the curve at the values of t listed. The independent ensemble consists of three independently trained models.
Curve Ensembles (%)
Independent Ensembles
curve parameter
{0.0, 1.0}
{0.0, 0.5, 1.0)
Unaligned
Aligned
(%)
TinySix
41.80 + 0.15
42.66 + 0.48
43.82 + 0.78
42.05 + 0.26
TinyTen
61.23 0.36
61.39 + 0.25
62.40 + 0.30
61.76 + 0.01
ResNet32
71.35 + 0.30
72.13
+ 0.12
72.33
0.12
72.03 + 0.08
VGGT6
74.00 + 0.17
74.69
+ 0.22
74.91 + 0.10
74.88 + 0.06
6 DISCUSSION AND FUTURE WORK
We generalize the curve finding problem by removing the weight symmetry ambiguity associated with the endpoint models. The optimal permutation of these weights can be approximated using neuron alignment. We find empirically that this approximation performs comparably to a proximal alternating scheme with the same initialization which learns a locally optimal permutation. Addi- tionally, we prove that this PAM scheme has some convergence guarantees. Empirically, we show that neuron alignment can be used to successfully and efficiently learn optimal connections between neural nets. Addressing the ambiguity of weight symmetry is critical for learning planar curves on the loss surface along which accuracy is mostly constant. Our results hold true over a range of datasets and network architectures. With neuron alignment, these curves can be trained in less epochs and to higher accuracy. We also see a modest to notable increase in the performance of ensembling.
10

---

Under review as a conference paper at ICLR 2020
Future work will include gaining deeper theoretical understanding of how neuron alignment affects curve finding dynamics. We plan to further explore how similar feature representations between models smoothly interpolated by these learned curves, and how this relates network training.
REFERENCES
Hédy Attouch, Jérôme Bolte, Patrick Redont. and Antoine Soubeyran. Proximal alternating mini- mization and projection methods for nonconvex problems: An approach based on the kurdyka- lojasiewicz inequality. Mathematics of Operations Research, 35(2):438 457, 2010.
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss land- scape. arXiv preprint (XXi::907.029)) 2019.
Rainer E Burkard and Eranda Cela. Linear assignment problems and extensions. In Handbook of combinatorial optimization, pp. 75-149. Springer, 1999.
An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network error surfaces. Neural computation, 5(6):910-927, 1993.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, pp. 5225-5235, 2017.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The Joss surfaces of multilayer networks, 2014.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223, 2011.
Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization, 2014.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, pp. 1308- 1317, 2018
Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprini arXiv: 1.01440, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in Neural Information Processing Systems, pp. 8789-8798, 2018.
Ian J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems, 2014.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Using mode con- nectivity for loss landscape analysis. arXiv preprint arXiv:1806.066. 06977, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. arXiv preprint arXiv:1704.00/00 2017.
Cheng Ju, Aurélien Bibaut, and Mark van der Laan. The relative performance ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45(15):2800-281 2018.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, 2019.
11

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, and Rong Ge. Explaining landscape connectivity of low-cost solutions for multilayer nets, 2019.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hoperoft. Convergent learning: Do different neural networks learn the same representations? In Iclr, 2016.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9078-9086, 2019.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances Neural Information Processing Systems, pp- 5727-5736, 2018.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076-6085, 2017.
Itay Safran and Ohad Shamir. On the quality of the initial basin in overspecified neural networks, 2015.
Levent Sagun, V. Ugur Guney, Gerard Ben Arous, and Yann LeCun. Explorations on high dimen- sional landscapes, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport, 2019.
Lou van den Dries and Patrick Speissegger. O-minimal preparation theorems. Model theory and applications, 11:87-116, 2002.
Yangyang Xu, Ioannis Akrotirianakis, and Amit Chakraborty. Proximal gradient method for huber- ized support vector machine. Pattern Analysis and Applications, 1944),999-005, 2016.
12

---

Under review as a conference paper at ICLR 2020
Supplementary Material for Optimizing Loss Landscape Connectivity via Neuron Alignment
A
ADDITIONAL FIGURES
A.I PLANES CONTAINING LINEAR INITIALIZATIONS
TinyTen
Renflet22
8y 9
62-0)
8) 6.
(a) CIFAR100
ThyTen
Reclet32
VDG16
04
6,
82-0h
(b) CIFARIO
Restlet32
(c) STL10
Figure Test loss on plane containing 01, 02. and Pa
220
7u0:
25
9
CIFAR10
TinyTon
Recher33
V6616
-0
6y 8:
9--0)
(b) STL.10
Figure 7: Test accuracy plane containing 0,, 02, and Par
13

---

Under review as a conference paper at ICLR 2020
A.2 TRAINING
Tiny Ten
1es
VGG116 Unaigaa
TinyTen
Readlet 12
VGS11
Liaagaed
D11-
Arcusayy
Accuraty
ACCurSC
012
439
Epoch
Epoch
Epoch
Epach
Epoch
Epoch
(a) Training loss
(b) Training accuracy
Figure 8: CIFAR10
TinyTen
ResNet32
VGSL6 Unailper
TinyTen
RedNet32
VGS16
009
grad
Actuney
Accuracy
Accunacy
Epoch
Epoch
Spach
Epocc
(a) Training loss
(b) Training accuracy
Figure 9: STL10
ISM Unalggeee 1982 JOOE tnaligseed
3.5
Tex:: IAN Aligned
L00
130
250
Figure 10: Log loss over run of the proximal alternating minimization scheme TinyTen for CI- FAR100. The scheme consists 20 epochs of projected SGD to solve the permutation subproblem, followed by 40 epochs of SGD to solve the curve parameter subproblem. Vertical lines denote the change different subproblem iterations. This shows that neuron alignment provides a much better intialization for PAM, and this permutation initialization is close being locally optimal.
14

---

Under review as a conference paper at ICLR 2020
TinyTen Ten
ResNet32
VGG16
0.020
Aigned Unaligned
Mignesi
Alignedi Unaiigned
Unaligned
0.025
0.025
0.015
0.020
0.015
0.010
0.015
0.010
0.005
0.005
0.000
0.000
Wavenumiberr
Figure 11: Fourier transform of CIFARI00 loss curve. Notice that the absolute value of the trans- form lower for the aligned case at higher modes/wavenumberss spectral terms, this shows that the average aligned curve is less oscillatory than the unaligned curve. This is a rigorous way to measure the smoothness of curve.
15

---

Under review as a conference paper at ICLR 2020
A.3 PLANES CONTAINING LEARNED BEZIER CURVES
Resdlt322 Algned Endoolnns
VI216
Aligtee Enspalinss
Alggned Endpoints
Pocel
8
0:-01
8
TeyTon
Unaligned Endpoints
Unaligned Enopoints
Lnaigred Erdccints
56
15
16
53
20
Hooil
(a) CIFARIO0 100
Alined Endboiins
Alined Endpoints
Aligheed Endpoints
15
ThnyTon
ResHet32
VSG16
25
0
CIFARIO
TinyTen
VGG10
Enopoints
26
6,-6)
TinyTen
Rcolet32 Iinelgeed Ineepinns
VOG16
Unaligned Fratoninis
Unollgged Fradpiints
-9
8:
(c) STL10
Figure 12: Test loss plane containing learned curve, n (t).
16

---

Under review as a conference paper at ICLR 2020
Tayyle Algreed Enopoints
Reshar32 Alighed Endgrannts
VIS138
Alipned Endpoints
6:-B: -t
0g -
VOG16 Unaligned Erdgpeinss
Unaligned Endoo itss
Unaligned Enopeiins
Pecs
Pece
(a) CIFARIO
Aligned Enspolrts
Aligned Endpoinns
Aligned Endoolints
TinyTen
Restle132
VGG16
Unaiigned
20
(b) STLIO
Figure 13: Test accuracy on plane containing learned curve, (t).
B ADDITIONAL RESULTS
Table 4: The minimum accuracy along the curve with standard deviation is reported for each com- bination of dataset, network architecture, and curve class.
Model
Endpoints
CIFARIO
CIFARI00
STLI0
Unaligned
7..6
2.3
0.6
TinySix
PAM Unaligned
0.5
0.2
6..4
PAM Aligned
75.7
0.3
6.1
Aligned
7.5
3.2
6..8
Unaligned
8.2
1.1
0.6
TinyTen
PAM Unaligned
87.2
5.1
0.5
PAM Aligned
88.0
0.3
73.5
Aligned
88.6 0.1
5.4
73.4
Unaligned
91.2
6..7
0.3
ResNet32
PAM Unaligned
91.7
6..1
7.40
PAM Aligned
91.8
0.6
76.1
Aligned
92.6 + 0.2
6..6
76.0 + 0.2
Unaligned
92.6
6.4
71.3 1.8
VGG16
Aligned
93.0 0.1
0.1
71.3 + 1.8
17

---

Under review as a conference paper at ICLR 2020
Table 5: The training loss with standard deviation is reported for each combination of dataset, net- work architecture, and curve class.
Model
Endpoints
CIFARIO
CIFARI00
STL10
Unaligned
0.896 + 0.008
2.636 +.0008
0.931 + 0.014
TinySix
PAM Unaligned
0.826 = 0.006
2.501
0.914
PAM Aligned
0.813 = 0.008
2.489
0.018
0.863 0.005
Aligned
0.817 + 0.006
2.490 0.018
0.864 = 0.004
Unaligned
0.460 + 0.004
1.839 0.010
0.479 = 0.018
Tiny
PAM Unaligned
0.419 = 0.005
1.733 0.013
0.469 0.000
PAM Aligned
0.387 =
1.662
0.007
0.393 = 0.007
Aligned
0.392 + 0.003
1.693 + 0.008
0.389 0.007
Unaligned
0.212 = 0.005
1124
0.292 0.007
ResNet32
PAM Unaligned
0.191 + 0.002
1.093 + 0.011
0.283 0.001
PAM Aligned
0.192 + 0.002
1.086 0.011
0.283 0.001
Aligned
0.180 = 0.002
1.011 0.002
0.256 + 0.002
Unaligned
0.212 = 0.002
0
0.372 0.029
VGG16
Aligned
0.198 = 0.001
0.676 - 0.005
0.366 + 0.015
C ALGORITHMS
This section contains algorithms described in Section 2
Data: Two trained models, 0 and 0₂
Result: A parameterized curve, ra. connecting 0 and 02 along which loss is flat Initialize (4) as t(0 01):
while not converged do for batch in dataser do
sample point in in [0, 1|:
compute loss (((r=(to))=
optimization step on network (to) to update Φ
end end
Algorithm Curve Finding (Garipov al. 2018
In this algorithm, the optimization step can correspond to variety of techniques. In this paper, use traditional stochastic gradient descent to update the curve parameters Φb. Notice that stochasticity introduced by the sampling of as well as the training data.
For the purpose of computing validation loss and test loss for ræ. important care must be given for networks that contain batch normalization layers. This is because batch normalization aggregates running statistics of the network output that are used when evaluating the model. Though, r (ta)) gives the weights for the model point fg. the running statisties need to be aggregated for each normalization layer. In practice, this can be done by training the model for one epoch, while freezing all learnable parameters of the model. Since batch statistics would need be computed for each point sampled along the curve, it happens that computing the validation or test loss of the curve r more expensive than an epoch of training.
D PROOFS
For the following proofs, we first establish and more rigorously define some terminology. We first discuss important abuse of notation. For clarity the parameterized curve connecting networks under some permutation P that has been written as rt(t) will now sometimes be referred to as r(: o, P).
18

---

Under review as a conference paper at ICLR 2020
Feed-forward neural networks In this section, will be analyzing feed-forward neural net- works. We let X E Rm be the imput to the neural network, d samples of dimension mg- Then let W; Xm denote the network weights mapping from layer - 1 to layer l. Addition- ally, a denotes the pointwise activation function. Then we can express the output of a feed-forward neural network, Y, as:
Y = WLO WL-10 WL-2...0 W
(9)
To include biases, (b: / we simply convert to homogeneous coordinates,
X, = X
W
bj
Y 1
W;
=
Y
(10)
In all proofs, these terms are interchangeable.
Huberized ReLU The commonly used ReLU function is defined as (t)) == max(0, t). However, this function is not in C' and hence not locally Lipschitz differentiable. This makes conducting analysis with this function difficult. Thus, we will approach studying it through the lens of the huberized ReLU function, defined as:
for S
(a(() =
1 2 28
for 0 S t S 5
(11)
for 8 S t
is clear that ds is a C² approximation of such that || Tblloo = 9. Using huberized forms of
loss functions for analysis is a fairly common technique such as in (Xu et al. 2016) which studies huberized support vector machines.
Kurdyka-Loojssewiec property The function J is said to have the Kurdyka-Lojasiewicc (KL.) property if there exist (0, +oo), a neighhorhood U of x and continuous concave function فi: : [0, v) + 14 such that:
(0) =
+ is Ct on (0,
vs ('s))
(eEUn(f()< < f(z) + v/. the Lurdyka-Lojasiewies inequality holds
Φ' ('(((x) - (())) (f))) 2
(12)
Here 0 denotes the subdifferential of f. Informally, a function that satisfies this inequality is one whose range be reparameterized such that kink occurs at its minimum. More intuitively, if th has the form, s1-0 and f is diffcrentiable on (0, v),, then the inequality reduces to
1
(I - OT- 1f(²)² < [[v((z)]
(13)
A subset of _ is semialgebraic if it can be written as a finite union of Semialgebraic function sets the form
{a Ex (z(z) 0, (x) < 0, --
where and qa are real polynomial functions. function R - R U {+0o} is said to be semialgebraic if its graph is semialgebraic subset of 1=+11
Subanalytic function Globally subanalytic sets are sets the form ((x,t) t) E [-1,1]" x
f(z) = t} where f [| - R is analytic function that can be extended analytically neighborhood the interval [-1, 1|". A function subanalytic if its graph is a globally subanalytic set.
19

---

Under review as a conference paper at ICLR 2020
D.1 PROOF OF THEOREM 3.1
To prove this, we need that our problem meets the conditions required for local convergence of proximal alternating minimization (PAM) described in (Attouch et al. 2010). This requires the following:
1. Each term in the objective function containing only one primal variable bounded below and lower semicontinuous.
2. Each term in the objective function which contains both variables is C' and locally Lipschitz differentiable.
3. The objective function satisfies the Kurdyka-Loojsiewiicz (KL) property.
First we reformulate the problem so that it becomes unconstrained. Let x denote the indicator function, where:
0, 2 +00,
for t E C otherwise
(c())
(14)
This problem contains two hard constraints. First, each permutation matrix, Pt. must clearly be restricted to the set of permutation matrices of size |i], [[ Additionally, it is assumed that the norm of the weights are bounded above. Without loss of generality, let Kw denote upper bound valid for the weights. We denote the set of weights that satisfy the norm constraint as {A: ||All2 I K. Then 5 with added regularization is equivalent to:
L-1
0² P"
= min 6.P
Q P)
Σ X
(P)) + Σ
X( (15)
1=1
E=
We now address each requirement for local convergence.
1. By the theorem, R is assumed to be bounded below and lower semicontinuous. It is easy to see from equation 14 that the sum of indicator functions are bounded below and lower semicontinuous.
2. Now we consider the form of the function, Q(o, P). It has been defined as
C(r(t;o, o, P))dt
We know that rt; P) corresponds feed-forward neural network. Then can be expressed as:
(We(tt: o, P)a o W, P)...r () dt
(16)
with weight matrices W. and activation function a.. It becomes clear that for Q(6, P) to be
in C¹ and locally Lipschitz differentiable, the same must be true for L, a, and [W The first two are true as they are assumptions of the theorem. Since, T C and locally Lipschitz differentiable in the primal variables, then this is also true for all W.. Thus, Qό) P) is in C' and locally Lipschitz differentiable.
3. To satisfy the KL property, the objective function must be tame function Attouch al. 2010). Rigorously, this means that the graph of the function belongs to an o-minimal structure, concept from algebraic geometry. We refer curious readers to (van den Dries & Speissegger 2002) for further reference.
First, we note that Q(ò, P) is piece-wise analytic. This because Q composition of piece-wise analytic functions, L. a, and rg- Additionally, because the input data is bounded and the norm of the weight matrices are bounded, follows that the domain of Q is bounded. Since, Q is piece-wise analytic function with bounded domain, it follows that Q is a subanalytic function. The boundedness of the domain is an important detail here. This is because analytic functions are not necessarilly subanalytic unless their domain bounded; a popular example of such a function is the exponential function.
20

---

Under review as a conference paper at ICLR 2020
The regularization function, R, is assumed to be piece- piece-wise analytic function. It follows from the previous reasoning that R has bounded domain. Thus, R is a subanalytic function.
We now consider the constraints associated with this problem, which have been re- expressed as indicator functions in the objective. The set of permutation matrices, [ is finite and thus it is clearly a semi-algebraic set. Notice that the set weight matrices satisfying the norm bound is equivalent to {A ||Ali Kw < O). The function that defines this set is a polynomial, so is a semi-algebraic set. Indicator functions on semi-algebraic sets are semi-algebraic functions. Thus, the indicator functions in the objective are semi-algebraic.
The graphs of semi-algebraic functions and subanalytic functions both belong to the logarithnic-expooeniia structure, o-minimal structure. A basic algebraic property of o-minimal structures is that the graphs of addition and multiplication are also elements of the structure (van den Dries & Speissegger 2002). Since our objective function is linear combination of allebrrgc functions and subanalytic functions, it follows that the graph of our objective function is an element of the logaritmmic-eppoentiaa structure. Therefore, our objective function is a tame function and it satisfies the KL property.
D.2 CONSIDERING RECTIFIED NETWORKS
Theorem 3.1 does not extend to the class of rectified networks. However, we are still interested in contructing sequence of iterates fst Pt} such that the objective value, E, [C(r(t; d², Pb))], monotonic decreasing. The following theorem will introduce a technique for constructing such a sequence.
Lemma D.1 (C restricted possible network outputs is Lipschitz continuous). For a wooddffowwrdd neural network, assume that L is continuous and that the neural network input, Xo. is bounded. Additionally, assume that the spectral norm all weights, (W:) is bounded above by Kw, and the activation functionn, a, is continous with ||a||| < 1. Let Sy denote the set of Y where
W WL-10 WL-2...0 W1
(17)
such that [W Kw Vi (1,2,.... L}
Then C restricted to the set Sy is Lipschitz continuous with some Lipschitz constant K.
Proof. Since Xo is bounded, it follows that there exists some constant Kx x such that ||oo|| S Kx- Since, the spectral norm of W, is bounded above by Kw, it is easy to see that [[W, Ill S Kw Kx- Now since the pointwise activation function is non-expansive map, it immediately follows that ||a o Xoll < Kw Kx. Following this process inductively, we see that the network output, Y, is bounded and that:
|| S K44 Kx
(18)
Since Y is arbitrary, it follows that this is bound for Sy. Then RT xd radius K Kx. This ball is compact and L is continuous, so follows that restricted to this ball is Lipschitz continuous. Thus, there exists some Lipschitz constant K. Clearly, Sy is contained in this ball. Therefore, L Lipschitz continuous on the set of all possible network outputs with Lipschitz constant K.
restrict L to the ball in
Let 81 and fg be feed-forward neural networks with ReLU activation function. Assume that L and rg are piece-wise analytic functions in C¹ and locally Lipschitz differentiable. Assume that R is piece-wise analytic, lower semi-continuous, and bounded below. Assume that the maximum network width at any layer is M units. Additionally, assume that the network weights have spectral norm bounded above by Kw, and that this a hard constraint when training the networks. Finally, any point rs must be equivalent to affine combination of neural networks (Bezier curves, polygonal chains, etc.) satisfying the previously stated spectral norm bound.
Create the parameterized curve ra(t; o, P) by substituting the huberized ReLU function, ds, into all ReLU functions in r(t: o, P). We refer to the objective values associated with these curves as Qa(ó. P) and Q(ò, P) respectively.
21

---

Under review as a conference paper at ICLR 2020
Theorem D.2 (Monotonic Decreasing Sequence for Rectified Networks). For a feed-forward net- work, assume the above assumptions have been met. Additionally, assume that Xu is bounded, so that L restricted to the set of possible network outpus is Lipschitz continuous with Lipschitz constant Kr, by Lemma D.I Now generate the sequence f Pkj by solving equation ofor To ro(t; Φ. P). On this sequence impose the additional stopping criteria that
1
1
p
p²1 KLVM M
8
Σ
L-1
KW
V4 0.
(19)
200
2vp
Then, the sequence of curves r(t; $ ph) connecting rectified networks has monotonic decreasing objective value.
Proof. First we consider the approximation error from replacing with az. It is straightforward to see that
6
s 2
(20)
Then it follows that for any input æ.
o Wow- - W1 S VM 8
2
Since the spectral norm of W; are bounded above by Kw, then we see that
|22 o Wix W208 Wialle s Kw M 8
2
Now notice that
||o W20 Wis j - ag W20 Wi S ||a o W20 Wi a W205 o Wiæl|
(21)
+ ||or W205 Wiæ aa W205 o Wiæl|-
Since the ReLU function a non-expansive map, it must be that the first term bounded above by the previous error, / The second term corresponds once again to the error associated with the huberized form of the ReLU function, VM Thus the total error can be bounded by (Kw + 1)
Following this inductively, it can be seen that the this error grows geometrically with the number of layers. Additionally, the loss function is Lipschitz continuous when restricted to the set of possible network outputs. So we find the following bounds:
||Y - Yoll S VM
Σ
Ki
2
[|((Y) -(-))) s KLV M
M
K
(22)
Since any point the curve is affine combination of networks with the Kn bound the spectral norm of their weights, it immediately follows this spectral norm bound also holds for the weights for any point on the curve. Then [[Q(,, P) (o( P)]| also bounded above by the bound in equation 22
Then let { _Pk be the sequence generated by solving equation 6 using the curve rg. o a piece- wise analytic function in C' and is locally Lipschitz differentiable. Additionally, the spectral norm constraint on the weights is semi-algebraic and bounded below, so Theorem 3.1 can be applied.
then follows that
((*tt)pt*+)) pkt1) + R(0+t1) +
110ht1 - 11112 + 2up
p - p 1
(23)
2v
< <0(0", ph1 + R(ON) + KL KLVM
M
Ki,,
Vk 20
2
22

---

Under review as a conference paper at ICLR 2020
Thus, r(t; Ø², P() ) is sequence of curves, connecting rectified networks, with monotonic decreas- ing objective value as long as
1 2vp
110
+1
||
P*
KLVM 2
KLV
Kw
4k 2 0
2
Since the above equation is stopping criterion introduced in the theorem statement, it follows that we have constructed a sequence of curves, connecting rectified networks, with monotonic decreasing objective value.
E RESIDUAL NETWORK ALIGNMENT
Algorithm applies to networks with a typical feed-forward structure. In this section, we discuss how we compute alignments for the ResNet32 architecture as it is more complicated. It is important to align networks such that the network structure is preserved and network activations are not altered. In the context of residual networks, special consideration must be given to skip connections.
Consider the formulation of a basic skip connection,
X = a (W XB) + X
(24)
In this equation, we can see that X and X share the same indexing of their units. This becomes clear when you consider permuting the hidden units in Xx-1 without permuting the hidden units of X there is essentially the use of an identity mapping from X to X
impossible to do so without breaking the structure of the equation above, where
We consider a traditional residual network that decomposed into residual blocks. In each block the even layers have skip connections while the odd layers do not. So, compute the alignment as usual for odd layers. For all even layers within a given residual block, we determine a shared alignment. We do this by solving the assignment problem for the average of the cross-correlation matrix over the even layers in that residual block.
F ALIGNMENT ALONG CURVES
Clearly, alignment is useful method for learning better flat loss curves between models. An inter- esting question is how curve finding itself relates to alignment, Until now, we have only considered the alignment between the endpoint models, r0) and r(1). Now, consider how points along the curve, rtt), align to the endpoints. To study this numerically, we will use the curve midpoint r (0.5). that this is the point the quadratic Bezier curve that is roughly linearly
From Figure 4 connected to both endpoints.
F.1 CORRELATION SIGNATURE
First, we consider how the correlation signature changes along the curve. Figure 14 displays the correlation signature between the curve midpoint and each endpoint in blue. To gain a better un- derstanding of this signature, we require some context. Thus, the correlation signature between the linear midpoint and each endpoint displayed in green. This allows us to understand how the correlation signature changes over the curve finding optimization. Additionally, we display the correlation signature between the curve midpoint and each endpoint, where the midpoint has been aligned to the given endpoint, in yellow. This essentially gives us context on how highly the mid- point is aligned to each endpoint. This is because the yellow curve acts as an upper bound for the blue curve.
There are several observations to be made about Figure [4] The correlation signature between the endpoint and the curve midpoint is fairly high. For unaligned endpoints, the correlation is only slightly lower than the signature computed when the curve midpoint aligned to the endpoint. In the case where the endpoints aligned, the signatures are curve finding algorithm is finding the quadratic curve along which similar feature representations being interpolated.
coincide. This suggests that the
23

---

Under review as a conference paper at ICLR 2020
Msan Cross-Corelation of Conespanninng Activations Between anaig Endpoinns and Curve Midpoint
Endlocnn TinTTen
Endosint Reshut32
VOG16
Biclarc algrerent
0.8
Bocfare mentt
After alignment
Afhee de dligmment
Linca Wilpont
Linoar dilpent
64-
0.4
0.4
0.-
Algerrint
0.2
0.-
3.0
7.5 Lager
10.0
12.5
15.0
Layer
Endocint
Endpoint
Endpalitt
TinyTen
Reshet32
VGG16
0.5
Bofore Algronott
Afher Aigiment
Beflri Alyy more
frhee aigmment
Midpoent
Cros-Comnatto
6.-
6.-
02
Micpo
125
Nean Cross Correlation of Correspording Activations Uetween Aligned Encpoints and Curve Midpoint
GIFARUUU
TinyTen
ResNet32
WGG16
0.9
Detfore Algrmant After A aigmeett
0.8
0.0
inwar nivipiine
6.7
0.7
0.7
0.0
0.6
Barore Aligemeet Algemmnt
0.4
Beffree Aligerrent
0.5
After Aligement
0.4
Nolport
Hicipo
10
2.2
3.0
7.5
10.
12.5
15.0
Layer
Leyer
Undpoint
Endpoint Reshett2
Endpoint VGG16
TinyTen
0.0
0.9
Delore Al gomment
0.R.
0.0
Afler Algreniae, Lincar Vödpoin
Coav-rretee on
0.7
0.7
0.6
0.-
0.
Angeeeent
ilggnreet
03-
0.4
10.0
Figure 14: The mean cross-correlation between units in the curve midpoint model and each endpoint model. For context, the mean cross cross-correlation between the linear midpoint and each endpoint is displayed. Additionally, the mean cross-correlatio between the curve midpoint and each endpoint after being aligned to the respective endpoint is displayed.
Concerning the linear midpoint, the correlation at the linear midpoint decays to 0 when endpoints are unaligned as the network goes deeper. When endpoints are aligned, the correlation signature at the linear midpoint is similar to the correlation signature at the curve midpoint. Since these linear connections between the endpoints the initializations for the curve finding algorithm, this gives some intuition on how alignment works to give better initialization.
24


Reviews:
[{"review_id": "SylrU09ptS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the connection between symmetries in the neural network architectures and the loss landscape of the corresponding neural networks. In the previous works, there was shown that the two local minima of a neural network can be connected by a curve with the low validation/train loss along the curve. Despite the loss on the curve being close to the loss at the end points, there are segments of the curve on which the loss in higher than loss at the local minima. To overcome this problem, the authors proposed two-step procedure: 1. Align weights between two neural networks 2. Launch the path finding algorithm between aligned weights. In other words, the authors proposed to connect not original local minima but local minima that parametrize the same functions as the original ones, but have a different order of neurons. The authors also proposed PAM algorithm where they iteratively apply path finding algorithm and weights alignment algorithm.\n\nNovelty and significance. The idea to combine the symmetrization of NN architectures with path finding algorithms is new to the best of my knowledge. Experimentally, the authors showed that ensembling the midpoint and endpoints of the curve found via path finding  algorithm coupled with the neural alignment algorithm delivers a better result than simple averaging of three independent networks. This is a new and significant result, since before the ensembling of points along the curve had the same quality as the ensemble of three independent networks or marginally better.  \nThe weak side of the paper is the PAM algorithm that occupies a significant part of the paper and does not deliver a significantly better result than the simple application of the neural alignment procedure before launching the path finding algorithm.\n\nClarity. Overall, the related work section contains all relevant references to the previous works to the best of my knowledge. The paper is well written, excluding the section Neuron Alignment that lacks notation description.  \n\nThe paper contains several typos and inaccuracies:\n1. “However, We find its empirical performance is similar to standard curve finding, and we advocate the latter for practical use due to computational efficiency.” The word “We” should start with the lowercase letter.\n2. In the sentence “The first question to address is how to effectively deal with the constraints in equation 6” the index i should be replaced with the index l.\n3. Notation Π|Kl| introduced after equation 5.\n4. In the section describing neuron alignment  algorithm Lie et al. [1] used a different notation. So I would recommend to further extend this section and add all necessary notations. Also, I would recommend to add a direct link to the paper where the problem is described in matrix form.\n5. In the Algorithm 1 “Initialize P θ2 := [Wˆ 2 1 ,Wˆ 2 2 , ...,Wˆ 2 L ] as [W2 1 ,W2 2 , ...,W2 L ] for k ∈ {1, 2, ..., L − 1};”  k is not used anywhere in notation.\n6. In Figure 3, “model 2 aligned ” sing is out of the plot box for ResNet-32 and VGG-16 architectures.\n7. The appendix contains the sketch of the proof that is quite difficult to follow. I would recommend giving all necessary definitions as it is done in the [2] and extend the proof. \n\nOverall, it is quite an interesting paper but it contains some drawbacks.\n[1] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hopcroft.  Convergent learning: Do different neural networks learn the same representations?  In ICLR, 2016\n\n[2]  H́edy Attouch, J́erˆome Bolte, Patrick Redont, and Antoine Soubeyran.  Proximal alternating minimization and projection methods for nonconvex problems:  An approach based on the kurdyka-łojasiewicz inequality.Mathematics of Operations Research, 35(2):438–457, 2010\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeZKdchKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Given two parameters theta_1 and theta_2 of the same architecture, the authors propose to learn a minimal loss curve between theta_1 and P theta_2, where P is a permutation matrix yielding another equivalent parameterization of the same network. The authors show that either by initializing P with neuron alignment or by optimizing P jointly with the curve, one can find a parameter-space with better accuracy and loss than by just learning a curve between theta_1 and theta_2. The authors also show that initializing P is sufficient to obtain these gains, avoiding the complexity of also optimizing for P. Furthermore, they show that ensembles across models from these curves have a very mild gain in accuracy to those of non-aligned curves.\n\nThe main qualm I have about this paper is about the significance of the contributions and the motivation.\nAt the core, the authors propose to find a curve between theta_1 and P theta_2 where P comes from aligning theta_1 and theta_2 as in (Li et al. 2016.). This is lacking almost any motivation, or discussion on what problem they are trying to solve. Are they trying to find ensembles with lower error? If that is the case, well the results are evidence of a negative result in this respect, which is okay but given how ad-hoc and poorly motivated the method is to that objective it's not much of a contribution. Are they trying to better understand theoretically the loss landscape of neural networks? I don't think there's any theoretical gain in that regard from this paper either. They show that the curves between aligned networks are better, but they don't show how this relates to anything else in the published literature or open questions in the theoretical deep learning field.\n\nRegarding contribution 2, PAM is in the end shown to not converge to better curves than simply initializing with alignment. Also, doesn't the convergence result to a critical point also apply for standard descent methods? The convergence theorem doesn't seem to be much of a contribution in my opinion, either to the optimization or the deep learning community.\n\nRegarding contribution 3, I agree that better curves can be learned faster, but why is this a meaningful contribution? What problem that the deep learning or the theoretical machine learning cares about does this help solve?\n\nRegarding contribution 4, as the authors themselves admit, the improvement is very dim in comparison to non-aligned curves, and comparisons to other ensemble methods are not present.\n\nI want to acknowledge that while the motivation and contributions are in my opinion dim, I find the experimental methodology of this paper very solid. All the claims are to my extent very well validated with experiments, and the experiments are in general well designed to validate those claims. My problem is sadly not with the validity of the claims but with their significance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxTPBxM9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines neuron alignment with mode connectivity. Specifically, it applies paths neuron alignment to the calculation of mode-connecting and empirical results show that alignment helps in finding better curves.\nCombining neuron alignment with mode connectivity seems to be a good idea, but the message the authors want to convey is somewhat vague. Some key details are not presented clearly, and some contents seem to be irrelevant. The following are some detailed comments and questions:\n1. One main contribution of this paper is the observation that the observation that alignment helps in finding better curves. An observation is excellent if it brings significant performance improvements in practice, or if it brings deep insights in the understanding of the field. However, for the former, the improvement in the performance is not that much; for the latter, there is hardly any insight conveyed by this paper. Therefore, this observation itself is not strong enough.\n2. One contribution of this paper is applying proximal alternating minimization (PAM) when optimizing the parameters and proving its convergence. Nonetheless, PAM is only used in one model (TinyTen) and does not bring any improvement in the performance. It seems that there is no point in applying PAM and the contents related to PAM are all somewhat irrelevant.\n3. Usually sufficient details help in good understandings, but in this paper, some key details are unfortunately missing. For example, in Algorithm 2, details on the optimization step is not clear: what is the optimization method the authors use other than PAM? Also, no comments are addressed on Figure 7 to Figure 10, either in the main body or in the appendix. I would like to see more explanations details. If the source code is provided, it will be better.\nIn sum, the idea seems to be interesting, but the overall quality of the paper is still yet to be improved, and some key details need to be addressed more clearly before it can be accepted as a qualified submission.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the loss landscape of neural networks by taking into consideration the symmetries arising from the parametrisation. Specifically, given two models $\theta_1$, $\theta_2$, it attempts to connect $\theta_1$ with the equivalence of class of $\theta_2$ generated by weight permutations. 
Reviewers found several strengths in this work, from its intuitive and simple idea to the quality of the experimental setup. However, they also found important shortcomings in the current manuscript, chief among them the lack of significance of the results. As a result, this paper unfortunately cannot be accepted in its current form. The chairs encourage the authors to revise their work by taking the reviewer feedback into consideration. 
Decision: Reject