The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

In this paper , we conjecture that if the permutation invariance of neural networks is taken into account , SGD solutions will likely have no barrier in the linear interpolation between them . Although it is a bold conjecture , we show how extensive empirical attempts fall short of refuting it . We further provide a preliminary theoretical result to support our conjecture . Our conjecture has implications for lottery ticket hypothesis , distributed training and ensemble methods . The source code is available at https : //github.com/AnonymousMLSubmission/PermutationInvariance . 1 INTRODUCTION . Understanding the loss landscape of deep neural networks has been the subject of many studies due to its close connections to optimization and generalization ( Li et al. , 2017 ; Mei et al. , 2018 ; Geiger et al. , 2019 ; Nguyen et al. , 2018 ; Fort et al. , 2019 ; Baldassi et al. , 2020 ) . Empirical observations suggest that loss landscape of deep networks has many minima ( Keskar et al. , 2017 ; Draxler et al. , 2018 ; Zhang et al. , 2017 ) . One reason behind the abundance of minima is over-parametrization . Over-parametrized networks have enough capacity to present different functions that behave similarly on the training data but vastly different on other inputs ( Neyshabur et al. , 2017 ; Nguyen et al. , 2018 ; Li et al. , 2018 ; Liu et al. , 2020 ) . Another contributing factor is the existence of scale and permutation invariances which allows the same function to be represented with many different parameter values of the same network and imposes a counter-intuitive geometry on the loss landscape ( Neyshabur et al. , 2015 ; Brea et al. , 2019a ) . Previous work study the relationship between different minima found by SGD and establish that they are connected by a path of non-increasing loss ; however , they are not connected by a linear path ( Freeman & Bruna , 2016 ; Draxler et al. , 2018 ; Garipov et al. , 2018 ) . This phenomenon is often referred to as mode connectivity ( Garipov et al. , 2018 ) and the loss increase on the path between two solutions is often referred to as ( energy ) barrier ( Draxler et al. , 2018 ) . Understanding linear mode connectivity ( LMC ) is highly motivated by several direct conceptual and practical implications from pruning and sparse training to distributed optimization and ensemble methods . The relationship between LMC and pruning was established by Frankle et al . ( 2020 ) where they showed the correspondence between LMC and the well-known lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) . In short , LTH conjectures that neural networks contain sparse subnetworks that can be trained in isolation , from initialization , or early in training to achieve comparable test accuracy . Frankle et al . ( 2020 ) showed that solutions that are linearly connected with no barrier have the same lottery ticket . They further discuss how linear-connectivity is associated with stability of SGD . This view suggests that SGD solutions that are linearly connected with no barrier can be thought of as being in the same basin of the loss landscape and once SGD converges to a basin , it shows a stable behavior inside the basin1 . Because of the direct correspondence between LMC and LTH , any understanding of LMC , has implications for LTH , stability of SGD and pruning techniques . Linear mode connectivity has also direct implications for ensemble methods and distributed training . Ensemble methods highly depend on an understanding of the loss landscape and being able to sample from solutions . Better understanding of mode connectivity has been shown to be essential in devising better ensemble methods ( Garipov et al. , 2018 ) . Linear mode connectivity between solutions or checkpoints also allows for weight averaging techniques for distributed optimization to be used as effectively in deep learning as convex optimization ( Scaman et al. , 2019 ) . 1This notion of basin is also consistent with the definition proposed by Neyshabur et al . ( 2020 ) . In this paper , we conjecture that by taking permutation invariance into account , the loss landscape can be simplified significantly resulting in linear mode connectivity between SGD solutions . We investigate this conjecture both theoretically and empirically through extensive experiments . We show how our attempts fall short of refuting this hypothesis and end up as supporting evidence for it ( see Figure 1 ) . We believe our conjecture sheds light into the structure of loss landscape and could lead to practical implications for the aforementioned areas . Contributions . This paper makes the following contributions : • We study linear mode connectivity ( LMC ) between solutions trained from different initializations and investigate how it is affected by choices such as width , depth and task difficulty for fully connected and convolutional networks ( Section 2 ) . • We introduce our main conjecture in Section 3 : If invariances are taken into account , there will likely be no barrier on the linear interpolation of SGD solutions ( see the left panel of Figure 1 ) . • By investigating the conjecture theoretically , we prove that it holds for a wide enough fully-connected network with one hidden layer at random initialization ( Section 3 ) . • In Section 4 , we provide strong empirical evidence in support of our conjecture . To overcome the computational challenge of directly evaluating the hypothesis empirically , which requires searching in the space of all possible permutations , we propose an alternative approach . We consider a set of solutions corresponding to random permutations of a single fixed SGD solution ( our model ) and show several empirical evidences suggesting our model is a good approximation for all SGD solutions ( real world ) with different random seeds ( see the middle and right panel of Figure 1 ) . Further related work . Permutation symmetry of neurons in every layer results in multiple equivalent minima connected via saddle points . Few studies investigate the role of these symmetries in the context of connectivity of different basins . Given a network with L layers of minimal widths r∗1 , ... , r ∗ L−1 that reaches zero-loss minima at r1 ! , ... , rL−1 ! isolated points ( permutations of one another ) , Şimşek et al . ( 2021 ) showed that adding one extra neuron to each layer is sufficient to connect all these previously discrete minima into a single manifold . Fukumizu & Amari ( 2000 ) prove that a point corresponding to the global minimum of a smaller model can be a local minimum or a saddle point of the larger model . Brea et al . ( 2019b ) find smooth paths between equivalent global minima that lead through a permutation point , i.e. , where the input and output weight vectors of two neurons in the same hidden layer interchange . They describe a method to permute all neuron indices in the same layer at the same cost . Tatro et al . ( 2020 ) showed that aligning the neurons in two different neural networks makes it easier to find second order curves between them in the loss landscape where barriers are absent . 2 LOSS BARRIERS . In this section , we first give a formal definition for linear mode connectivity and study how it is affected by different factors such as network width , depth , and task difficulty for a variety of architectures . 2.1 DEFINITIONS . Let fθ ( · ) be a function presented by a neural network with parameter vector θ that includes all parameters and L ( θ ) be the any given loss ( e.g. , train or test error ) of fθ ( · ) . Let Eα ( θ1 , θ2 ) = L ( αθ1 + ( 1 − α ) θ2 ) , for α ∈ [ 0 , 1 ] be the loss of the network created by linearly interpolating between parameters of two networks fθ1 ( · ) and fθ2 ( · ) . The loss barrier B ( θ1 , θ2 ) along the linear path between θ1 and θ2 is defined as the highest difference between the loss occurred when linearly connecting two points θ1 , θ2 and linear interpolation of the loss values at each of them : B ( θ1 , θ2 ) = sup α [ L ( αθ1 + ( 1− α ) θ2 ) ] − [ αL ( θ1 ) + ( 1− α ) L ( θ2 ) ] . ( 1 ) The above definition differs from what was proposed by Frankle et al . ( 2020 ) in that they used 0.5L ( θ1 ) + 0.5L ( θ2 ) instead of αL ( θ1 ) + ( 1 − α ) L ( θ2 ) in our definition . These definitions are the same if L ( θ1 ) = L ( θ2 ) . But if L ( θ1 ) , L ( θ2 ) are different , we find our definition to be more appropriate because it assigns no barrier value to a loss that is changing linearly between θ1 and θ2 . We say that two networks θ1 and θ2 are linear mode connected if the barrier between them along a linear path is ≈ 0 ( Frankle et al. , 2020 ) . It has been observed in the literature that any two minimizers of a deep network can be connected via a non-linear low-loss path ( Garipov et al. , 2018 ; Draxler et al. , 2018 ; Fort & Jastrzebski , 2019 ) . This work examines linear mode connectivity ( LMC ) between minima . Next , we empirically investigate the effect of task difficulty and choices such as architecture family , width and depth on LMC of SGD solutions . 2.2 EMPIRICAL INVESTIGATION : BARRIERS . In this section , we look into barriers between different SGD solutions on all combinations of four architecture families ( MLP ( Rosenblatt , 1961 ) , Shallow CNN ( Neyshabur , 2020 ) , ResNet ( He et al. , 2015 ) and VGG ( Simonyan & Zisserman , 2015 ) ) and four datasets ( MNIST ( LeCun & Cortes , 2010 ) , SVHN ( Netzer et al. , 2011 ) , CIFAR-10 ( Krizhevsky et al. , 2009 ) and CIFAR-100 ( Krizhevsky et al. , 2009 ) ) . The main motivation to use Shallow CNN is to move from fully connected layers ( MLP ) to convolutions . The main difference between Shallow CNN and VGG16 is depth and the main difference between ResNet18 and VGG16 is existence of residual connections . We empirically investigate how different factors such as architecture family , width , depth and task difficulty impact the barrier size2 . We refer to training loss barrier as barrier . For loss barriers on a test set see E.4 . For train and test errors see A.2 . Width : We evaluate the impact of width on the barrier size in Figure 2 . We note that for large values of width the barrier becomes small . This effect starts at lower width for simpler datasets such as MNIST and SVHN compared to CIFAR datasets . A closer look reveals that the barrier increases with width up to a point and beyond that increasing width leads to lower barrier size . This effect is reminiscent of the double descent phenomena ( Belkin et al. , 2019 ; Nakkiran et al. , 2019 ) . Checking the test error ( Figure 8 ) indicates that in our experiments the barrier peak happens at the same size that needed to fit the training data . This phenomena is observed for both fully-connected and convolutional architectures . MLP architectures hit their peak at a lower width compared to CNNs 2In all plots the barrier is evaluated using training loss across 5 different random pairs ( 10 random SGD solutions ) . For easier comparison between all figures , we report the train accuracy barrier in all barrier plots . In our experiments , we observed that evaluating the barrier at α = 1 2 is a reasonable surrogate for taking the supremum over α ( the difference is less than 10−4 ) . Therefore , to save computation , we report the barrier value at α = 1 2 . and a decreasing trend starts earlier . For ResNets the barrier size is saturated at a high value and does not change . The barrier value for VGG architecture on different datasets is also saturated at a high value and does not change by increasing the width . Such similar behavior observed for both ResNets and VGG architectures is due to the effect of depth as discussed in the next paragraph . Depth : We vary network depth in Figure 3 to evaluate its impact on the barrier between optimal solutions obtained from different initializations . For MLPs , we fix the layer width at 210 while adding identical layers as shown along the x-axis . We observe a fast and significant barrier increase as more layers are added . For VGG architecture family we observe significant barriers . This might be due to the effect of convolution or depth . In order to shed light on this observation , we use Shallow CNN ( Neyshabur , 2020 ) with only two convolutional layers . As can be seen in Figure 3 when Shallow CNN has two layers the barrier size is low , while keeping the layer width fixed at 210 and adding more layers increases the barrier size . For residual networks we also consider three ResNet architectures with 18 , 34 and 50 layers and observe the same barrier sizes as VGG for all these depth values . The main overall observation from depth experiments is that for both fully-connected and convolutional architectures , increasing depth increases the barrier size significantly so the effect of depth is not similar to width . This can also be attributed to the observations that deeper networks usually have a less smooth landscape ( Li et al. , 2017 ) . Task difficulty and architecture choice : In Figure 4 we look into the impact of the task difficulty provided by the dataset choice ( MNIST , SVHN , CIFAR-10 , CIFAR-100 , and ImageNet ( Deng et al. , 2009 ) ) and the architecture type ( one-layer MLP with 210 neurons , Shallow CNN with two convolutional layer and width of 210 , VGG-16 with batch-normalization , ResNet18 and ResNet50 ) . Each row in Figure 4a and Figure 4b shows the effect of task difficulty , e.g. , fixing the task to SVHN and moving from MLP to Shallow CNN gives lower test error hence lower barrier size . Each column also represents the effect of architecture on a specific dataset , e.g. , fixing the architecture to Shallow CNN and moving from CIFAR10 to CIFAR100 presents an increase in test error , hence increase in the barrier size . Although deep architectures like VGG16 and ResNet18 present low test error , the discussed effect of depth saturates their barrier at a high level . Figure 4c aggregates the correlation between test error and size of the barrier . For MLP and Shallow CNN we observe a high positive correlation between test error and barrier size across different datasets . Deeper networks ( VGGs , ResNets ) form a cluster in the top-left , with low test error and high barrier size .

Summary: This paper studies the optimization landscape of neural networks. In particular, this paper touches on the question of whether two local minima of the optimization landscape are connected by a “line” of linearly-interpolated neural networks. To probe this question, this paper evaluates the loss gap (so called “barrier”) between two local minima and their linear interpolation. The finding is that this barrier is non-zero. Next, the authors conjecture that if one takes into account permutation invariance of a local minimum, the barrier should be reduced to zero. To support this conjecture, the authors considered a simulated annealing algorithm to search for permutations of the weight matrices, to show that the barrier reduces after applying the simulated annealing algorithm