When Are Solutions Connected in Deep Networks?

√ N neurons , and ( ii ) if subsets of features at each layer are linearly separable , then almost no overparameterization is needed to show the connectivity . Our experiments confirm that the proposed condition ensures the connectivity of solutions found by stochastic gradient descent , even in settings where the previous requirements do not hold . 1 Introduction . The aim of this work is to provide further theoretical insights into the mode connectivity phenomenon which has recently attracted some considerable attention from the research community ( i.e . training a neural network via stochastic gradient descent ( SGD ) from different random initializations often leads to solutions that could be connected by a path of low loss ) [ 2 , 9 , 11 , 12 , 15 ] . Existing theoretical insights from optimization landscape analysis suggest that , for over-parameterized networks , the loss function has connected sublevel sets , and hence the solutions found by SGD are also connected [ 32 , 37 , 43 ] . However , in order for such a result to hold , the network needs to have one wide layer with at least N + 1 neurons ( N being the number of training samples ) [ 33 ] . This bound is also known to be tight for 2-layer networks . Here , we would like to understand the above phenomenon for deep and mildly over-parameterized networks ( i.e . we aim to prove weaker guarantees under milder conditions ) . The idea is that , even when the sublevel sets are disconnected , SGD may only concentrate in a ( large ) region with connected minima , see e.g . Figure 1 . This leads to the question that whether one can characterize the properties of the solutions found by SGD which capture the mode connectivity phenomenon , for mildly over-parameterized networks . Along this line , a recent work [ 24 ] proposed the so-called “ dropout stability ” condition , that is , removing half the neurons from every hidden layer leads to almost no change in the value of the loss . As shown in the prior 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . works ( also in our experiments in Section 6 ) , this condition holds for very wide networks or when dropout is used as a regularizer during training , but it does not hold under other practical settings . The aim of this work is to provide a new characterization of SGD solutions which capture their connectivity property , and to improve the previous assumptions ( width Ω ( N ) , and dropout stability ) for networks of practical sizes , learned without the support of dropout training . As a remark , although our work is motivated by the original phenomenon concerning mainly the solutions found by SGD , all the theoretical results are still valid for other pairs of solutions in parameter space . Main contributions . Our main Theorem 4.1 provides an upper bound of the loss along a path connecting two given solutions in parameter space . This bound is characterized by the loss of L optimized and sparse subnetworks , L being the network depth . These L subnetworks connect subsets of at least half the neurons at each layer to the output ( see Figure 2 ) , and their weights can be optimized independently of the weights of the original network , which leads to a significant reduction of the loss . This is the key difference to the previously proposed dropout stability property [ 24 ] . Next , we specialize our theorem to different settings to demonstrate the improvement over prior works : • Corollary 4.2 shows that our condition is provably milder than dropout stability [ 24 ] . In fact , dropout stability implies that the loss does not change much when we remove at least half of the neurons at each layer and rescale the remaining ones . Here , we allow to optimize again all the weights of the subnetworks connecting the subsets of features to the output . • As the weights of the subnetworks can be optimized , we establish a formal connection between mode connectivity and memorization capacity of neural nets . In particular , Corollary 4.3 proves that , if the last two hidden layers have Ω ( √ N ) neurons , then under some additional mild conditions on the features of intermediate layers , any two given solutions can be connected by a path of low loss . This improves the width condition of prior works [ 32 , 33 , 43 ] from Ω ( N ) to Ω ( √ N ) at the cost of ensuring a weaker guarantee ( i.e . the connectivity of sublevel sets proved in prior works implies the connectivity of any two pairs of solutions , but in this paper we only show the connectivity for certain pairs ) . • Corollary 4.4 proves that , for a classification task , if subsets of features at each layer are linearly separable , then no over-parameterization is required to prove the connectivity . The message of this paper is that , in order to ensure the connectivity , a smooth trade-off between feature quality and over-parameterization suffices . The two extremes of this trade-off are captured by Corollary 4.3 and 4.4 mentioned above . More generally , at each layer l , we pick a subset Il containing at least half the neurons . If the features indexed by Il are good enough , in the sense that there exists a small subnetwork ξl achieving small loss when trained on these features as inputs , then all layers k > l in the original network do not need to be larger than twice the widths of the subnet ξl . Hence , the better the features at lower layers , the less over-parameterized the higher layers need to be . Finally , our numerical results show that , if the learning task is sufficiently simple ( e.g . MNIST ) and the network is sufficiently over-parameterized , then dropout stability holds and the SGD solutions can be connected as in [ 24 ] . However , when the learning task is more difficult ( e.g . CIFAR-10 ) , or for networks of moderate sizes , dropout stability does not hold and the path of [ 24 ] exhibits high loss . In this challenging setting , we are still able to find low-loss paths connecting the SGD solutions . Proof techniques . We remark that , while previous work [ 32 , 33 , 43 ] focused on the connectivity of all the sublevel sets ( i.e . a property of the global landscape ) , here we are only interested in the connectivity of a particular subset of solutions , especially those that are discovered by SGD . This leads to several differences in the proofs . First , previous analysis relies on the fact that the first hidden layer has at least N + 1 neurons [ 33 ] , and thus the training inputs can essentially be mapped to a feature space where they become linearly independent . This property allows the network to realize arbitrary outputs at each layer , which leads to the connectivity of sublevel sets . This property is no longer true for the networks considered here , where the layer widths are allowed to be sublinear in N . Second , the analysis of sublevel sets often do not take into account the bias of SGD , whereas here we focus more on understanding the properties of these solutions which lead to their connectivity . More closely related is the work by [ 24 ] , with which we provide a detailed discussion in Section 4 . 2 Further Related Work . Theoretical research on the loss surface of neural networks can be divided into two main categories : one that studies the global optimality of local minima , see e.g . [ 7 , 13 , 14 , 17 , 18 , 19 , 21 , 22 , 25 , 27 , 28 , 29 , 30 , 34 , 38 , 42 , 45 , 46 , 47 , 48 , 50 , 51 ] , and the other that studies landscape connectivity . This work falls into the latter category . Specifically , the works of [ 12 , 37 ] suggest that the loss function of a two-layer network becomes more and more connected as the number of neurons increases . In [ 43 ] and [ 32 ] , it is proved that networks with one sufficiently wide layer have no spurious valleys and connected sublevel sets , respectively . All these works require networks with at least one layer of order N neurons , whereas Corollary 4.3 requires widths of order √ N neurons . In [ 5 ] , the authors study the connectivity of equivalent minima ( i.e . representing the same network function ) which arise by permuting neurons from the same layer . A similar property is investigated in [ 20 ] for all local minima lying in a cell or activation region ( i.e . those sharing the same activation patterns across all the hidden neurons and training samples ) . In contrast , here we consider a more general setting : Theorem 4.1 provides upper bounds on the loss when connecting two arbitrary points in parameter space . In particular , these points need not be local minima , nor share the same activation patterns , or even represent the same function . This allows us to study the mode connectivity phenomenon of SGD solutions [ 24 ] , which are typically not captured by the previous assumptions . Other works discuss lower bounds on over-parameterization to ensure the existence of descent paths in shallow nets [ 39 ] , the satisfiability of the dropout stability condition in the mean field regime [ 40 ] . Memorization capacity is an old and well-studied topic [ 4 , 8 ] . Networks with Ω ( N ) parameters are known to memorize N arbitrary samples , and this type of results has been studied for a variety of settings , see [ 3 , 6 , 16 , 31 , 44 , 49 ] . Nevertheless , how memorization capacity is related to the mode connectivity phenomenon remains unclear , especially in the absence of a layer of width Ω ( N ) . Our work draws a formal connection between the two , which opens up new directions for future works . 3 Preliminaries . Given two nonnegative integers i , j , let [ i , j ] = { i , . . . , j } and [ i ] = { 1 , 2 , . . . , i } . Given a set I , we denote by |I| its cardinality . We consider an L-layer neural network with weight matrices Wl ∈ Rnl−1×nl for l ∈ [ L ] , and bias vectors bl ∈ Rnl for l ∈ [ L−1 ] . Here , n0 is the input dimension , nL the output dimension , and n1 , . . . , nL−1 the widths of all the hidden layers . Let θ be the set of all the network parameters . Given an input x ∈ Rn0 , the feature vector fl ∈ Rnl at layer l is defined as fl ( θ , x ) = x , l = 0 , σ ( WTl fl−1 ( θ , x ) + bl ) , l ∈ [ L− 1 ] , WTL fL−1 ( θ , x ) , l = L , ( 1 ) where σ : R→ R is some activation function . Let Ω be the parameter space of the network . For a data distribution ( x , y ) ∼ D , the population loss Φ : Ω→ R is defined as Φ ( θ ) = ED [ Ψ ( fL ( θ , x ) , y ) ] , ( 2 ) where Ψ ( ŷ , y ) is assumed to be convex with respect to the first argument . Typical examples of Ψ which satisfy this assumption include the standard square loss and cross-entropy loss . By letting D be the uniform distribution over a fixed set of N samples , the loss Φ reduces to the empirical loss . In the following , we frequently encounter situations where we are given a layer l ∈ [ 0 , L− 1 ] , and we want to study a subnetwork from layer l to the output layer L. The widths and the inputs to this subnet are defined as follows . Let Ip ⊆ [ np ] be a subset of neurons at layer p , for p ∈ [ l , L− 1 ] . By convention , let I0 = [ n0 ] . Let fp , Ip ∈ R|Ip| be obtained from fp by taking the neurons indexed by Ip . The layer widths of this subnet ( associated to l ) are given by m ( l ) l = |Il| , m ( l ) p = np − |Ip| for p ∈ [ l + 1 , L− 1 ] , and m ( l ) L = nL . The associated weight matrices and bias vectors are { U ( l ) p } Lp=l+1 and { v ( l ) p } L−1p=l+1 , where U ( l ) p ∈ Rm ( l ) p−1×m ( l ) p and v ( l ) p ∈ Rm ( l ) p . Let ξl = ( ( U ( l ) p ) Lp=l+1 , ( v ( l ) p ) L−1 p=l+1 ) , see Figure 2 for an illustration . Given an input z ∈ Rm ( l ) l to this subnetwork , the corresponding output , denoted by hL ( ξl , z ) ∈ RnL , is recursively defined for p ∈ [ l , L ] as hp ( ξl , z ) = z , p = l , σ ( ( U ( l ) p ) Thp−1 ( ξl , z ) + v ( l ) p ) , p ∈ [ l + 1 , L− 1 ] , ( U ( l ) L ) ThL−1 ( ξl , z ) , p = L. ( 3 )

Summary: The authors propose a theoretical explanation to the mode connectivity phenomena observed in neural networks. Specifically, it is shown that any two points in parameter space may be connected by a piece-wise linear path of low loss.  Technically this is done by assuming that there exists a tradeoff between feature quality and over parameterization at every layer. The authors note that this tradeoff assumption is milder than the dropout stability assumption made in previous work. Given this assumption, the authors improve upon the conditions for the existence of level set connectivity by requiring that the last two hidden layers only contain O(\sqrt{N}) neurons, instead of O(N).