Mode Connectivity and Sparse Neural Networks

1 INTRODUCTION . The lottery ticket hypothesis ( Frankle & Carbin , 2019 ) conjectures that neural networks contain sparse subnetworks that are capable of training in isolation from initialization to full accuracy . The sole empirical evidence in support of the lottery ticket hypothesis is a series of experiments using a procedure called iterative magnitude pruning ( IMP ) . IMP returns a subnetwork of the original , randomly initialized network by training the network to completion , pruning the lowest-magnitude weights ( Han et al. , 2015 ) , and resetting each remaining weight to its original initialization . On small networks for MNIST and CIFAR-10 , IMP subnetworks can match the accuracy of the full network ( we refer to such subnetworks as matching subnetworks ) at sparsity levels far beyond those at which randomly reinitialized or randomly pruned subnetworks can do the same . The lottery ticket hypothesis offers a new perspective on the role of overparameterization and raises the tantalizing prospect that there may exist much smaller neural networks that are capable of replacing the larger models we typically train today . Unfortunately , in more challenging settings , there is no empirical evidence that the lottery ticket hypothesis holds . IMP subnetworks of VGG and Resnet-style networks on CIFAR-10 and ImageNet perform no better than other kinds of sparse networks ( Liu et al. , 2019 ; Gale et al. , 2019 ) . In this paper , we describe a new framework called instability analysis , which measures whether the outcome of optimizing a network is robust to SGD noise ( in which case we call it stable ) . Instability analysis offers a range of new insights into the behavior of unpruned networks . For example , the outcome of optimization becomes stable to SGD noise early in training ( 3 % for Resnet-20 on CIFAR-10 and 20 % on Resnet-50 for ImageNet ) . Moreover , it distinguishes known cases where IMP succeeds and fails to find a matching subnetwork ; namely , IMP subnetworks are only matching when they are stable . It also allows us to identify new scenarios where sparse , matching subnetworks emerge early in training in more challenging settings , including Resnet-50 and Inception-v3 on ImageNet . In doing so , our results demonstrate that instability analysis is a valuable scientific tool for investigating the behavior of neural networks . W0 Wk W 1T W 2 TInstability Figure 1 : A diagram of instability analysis ( text left ) . Instability analysis . Instability analysis is a technique to determine whether the outcome of optimization is robust to SGD noise . Figure 1 visualizes this process . We train two copies of the same network from initialization ( W0 ) on different data orders ( which models different samples of SGD noise ) . We then linearly interpolate ( dashed line ) between the trained weights ( W 1T and W 2 T ) and examine the error along this path ( blue curve ) . The instability of the network to SGD noise is the maximum increase in train or test error along this path ( red line ) . We say a network is stable if error does not increase along the path , i.e. , instability is 0 . To examine instability at a later iteration k , we first train the network to iteration k ( Wk ) and make two copies afterwards . Instability is a property of a network with respect to an optimization procedure ; we focus on the standard procedure prescribed for the networks we examine . Instability analysis assesses a linear form of mode connectivity , a phenomenon where the minima found by two networks are connected by a path of constant error . Draxler et al . ( 2018 ) and Garipov et al . ( 2018 ) show that the modes of standard vision networks trained from different initializations are connected by piece-wise linear paths of constant error or loss . Based on this work , we expect that our networks will be connected by such paths . However , the modes found by Draxler et al . and Garipov et al . are not connected by linear paths . The only extant example of linear mode connectivity is by Nagarajan & Kolter ( 2019 ) , who train MLPs from the same initialization on disjoint subsets of MNIST and find that the resulting networks are connected by linear paths of constant test error ; we explore linear mode connectivity from points throughout training , we do so at a larger scale , and we focus on different samples of SGD noise rather than disjoint samples of data . Results . We begin by examining the instability of unpruned networks for MNIST , CIFAR-10 , and ImageNet . All but the smallest MNIST network we study are unstable at initialization . However , by a point early in training ( 3 % for Resnet-20 on CIFAR-10 and 20 % for Resnet-50 on ImageNet ) , all networks become stable . In other words , from this point forward , the outcome of optimization is determined modulo linear interpolation . In fact , the entire trajectory of a stable network is so determined : when we train two copies of the network on different data orders , the states of the networks at each epoch are connected by linear paths over which test error does not increase . In the lottery ticket context , we find that extremely sparse IMP subnetworks are matching only when they are stable , providing the first basis for understanding the mixed results in the literature . In doing so , we make a new connection between lottery ticket behavior and the optimization dynamics of neural networks . Inspired by our full network results , we modify IMP to rewind subnetwork weights to their values at iteration k rather than resetting them to initialization . For values of k that are early in training ( in fact , earlier than the full networks ) , IMP subnetworks become stable in all cases we consider . Correspondingly , they also become matching . At these sparsity levels , randomly reinitialized and randomly pruned networks are neither stable nor matching . This connection between stability and accuracy suggests that linear mode connectivity is fundamental to sparse neural networks found by IMP and , thereby , to our current knowledge of the lottery ticket hypothesis . Contributions . We make the following contributions : • We introduce instability analysis , which measures the maximum increase in error along the linear path between minima found by training the same network on different data orders . • On a range of image classification benchmarks including Resnet-50 on ImageNet , we observe that networks become stable to SGD noise early in training . • We show that stable networks are stable throughout the training process . • We use instability analysis to distinguish successes and failures of IMP ( the core algorithm behind the lottery ticket hypothesis ) as identified in previous work . Namely , extremely sparse IMP subnetworks are only matching when they are stable . • We augment IMP with rewinding to study subnetworks initialized after iteration 0 . We show that IMP subnetworks become stable and matching early in training if not at initialization . • Using rewinding , we show how to find sparse , matching subnetworks in much larger settings than in previous work by setting the weights of IMP subnetworks to their values from early in training . 2 PRELIMINARIES AND METHODOLOGY . Instability analysis via linear mode connectivity . Instability analysis evaluates whether the minima found when training two copies of a neural network on different samples of SGD noise ( i.e. , the random data order used during SGD ) are linearly connected by a path over which error does not increase . The neural network in question could be randomly initialized ( W0 in Figure 1 ) or the result of k training iterations ( Wk ) . To perform instability analysis , we make two copies of the network and train them to completion with different random data orders ( W 1T and W 2 T ) . We then linearly interpolate between the trained weights ( dashed line ) and compute the train or test error at each point ( blue curve ) to determine whether it increased ( minima are not linearly connected ) or did not increase ( minima are linearly connected ) . Formally , we capture training with SGD ( or a variant ) by a function As→t : RD ×U → RD , which maps weights Ws at iteration s and SGD randomness u ∼ U to updated weights Wt at iteration t by training for t− s steps ( for s , t ∈ { 1 , .. , T } and s < t ) . Algorithm 1 describes our procedure : Algorithm 1 Stability analysis from iteration k. 1 : Create a neural network with randomly initialized weights W0 ∈ Rd . 2 : Train W0 to Wk under SGD noise u ∼ U . That is , Wk ← A0→k ( W0 , u ) . 3 : Train Wk to W 1T under SGD noise u1 ∼ U . That is , W 1T ← Ak→T ( Wk , u1 ) . 4 : Train Wk to W 2T under SGD noise u2 ∼ U . That is , W 2T ← Ak→T ( Wk , u2 ) . 5 : Evaluate E ( αW 1T + ( 1− α ) W 2T ) for α ∈ [ 0 , 1 ] . We describe the result of linear interpolation ( step 5 ) with a quantity that we term instability . Let E ( W ) denote the train or test error of a network parameterized by W . Let Ē = mean ( E ( W 1T ) , E ( W 2T ) ) be the average test error of W 1T and W 2T . Let Emax = supα∈ [ 0,1 ] E ( αW 1T + ( 1−α ) W 2T ) be the highest test error when linearly interpolating betweenW 1T andW 2T . The instability is Emax − Ē ( red line in Figure 1 ) . When instability ≈ 0 , the minima are linearly connected and the network is stable . In practice , we average the instability from three initializations and three data orders per initialization ( nine combinations in total ) . We use 30 evenly-spaced values of α ∈ [ 0 , 1 ] . Networks and datasets . We study image classification networks on MNIST , CIFAR-10 , and ImageNet as specified in Table 1 . All hyperparameters listed are the standard values for these networks from reference implementations or prior work as cited in Table 1 . The warmup and low variants of Resnet-201 and VGG-16 are adapted from hyperparameters chosen by Frankle & Carbin ( 2019 ) . 1Frankle & Carbin ( 2019 ) mistakenly refer to Resnet-20 as “ Resnet-18 , ” which is a separate network . 3 NEURAL NETWORK INSTABILITY TO SGD NOISE . In this section , we study whether the outcome of optimization becomes robust to SGD noise after a certain amount of training . Concretely , we perform instability analysis ( Algorithm 1 ) on the standard networks in Table 1 from many points during training to understand when , if ever , networks become stable to SGD noise . We find that , although only Lenet is stable at initialization , every network becomes stable early in training . Neural networks are unstable at initialization . We begin by studying the instability of neural networks at initialization . We do so by training two copies of the same , randomly-initialized network under different samples of SGD noise ( that is , Algorithm 1 with k = 0 ) . Figure 2 shows the train error ( purple ) and test error ( red ) when linearly interpolating between the minima found by these copies . With the exception of Lenet on MNIST , none of the networks we study are stable at initialization . In fact , both training and test error rise to the point of random guessing when linearly interpolating between the minima found under different data orders . Lenet ’ s error does rise slightly , but the increase is a small fraction of a percentage point . We conclude that , in general , larger-scale image classification networks are not stable at initialization . Stability improves early in training . Although nearly all networks are unstable at initialization , they will inevitably become stable at some point . In the limit , they will be stable by the end of training , and it seems reasonable to expect that the final few steps of SGD are too insignificant to cause the network to enter linearly unconnected minima . In this experiment , we ask how early neural networks become stable . In other words , after what point in training is the outcome of optimization determined modulo linear interpolation regardless of the sample of SGD noise ? To explore this behavior , we train a single copy of the network for k iterations or epochs before making two copies that we train to completion on different data orders ( Algorithm 1 with k ≥ 0 ) . Figure 3 plots the instability of the networks for various values of k. We measure instability as the maximum error during interpolation ( the peaks in Figure 2 ) minus the mean of the errors of the two networks ( the endpoints in Figure 2 ) . In all cases , instability decreases as k increases , culminating in networks that are stable ( i.e. , instability≈ 0 ) . The iteration at which stability emerges is surprisingly early . For example , it occurs from approximately iteration 2000 for Resnet-20 and VGG-16 ; in other words , after 3 % of training , SGD noise can not affect the final minimum modulo linear interpolation . Stability occurs later for Resnet-50 : about epoch 18 ( 20 % into training ) . Instability is essentially identical when measured in terms of train or test error ( although train instability is slightly higher than test instability for Resnet-50 ) , indicating that the minimum becomes determined on both the train and test surfaces around the same time . Going forward , we present all results with respect to test error for simplicity . Stable networks are linearly connected throughout training . Stable networks arrive at minima that are linearly connected , but do the trajectories they follow throughout training also have this property ? In other words , when training two copies of the same network with different noise , is there a linear path over which test error does not increase connecting the states of the networks at each iteration ? To study this behavior , we linearly interpolate between the networks at every epoch of training and compute the test error instability . That is , we compute instability throughout training . Figure 4 plots instability throughout training for Resnet-20 and VGG-16 from different starting iterations k. For k = 0 ( blue line ) , instability increases rapidly . In fact , it follows the same pattern as test error : as the test error of each network decreases , the maximum possible instability increases ( since instability never exceeds random guessing ) . With larger values of k , instability increases more slowly throughout training . When k is sufficiently large that the networks are stable at the end of training , they are stable at every epoch of training ( k = 2000 , pink line ) . In other words , after iteration 2000 , the networks follow identical optimization trajectories modulo linear interpolation . Discussion . Our observations implicitly divide training into two phases : an initial , unstable phase in which the final “ linearly connected ” minimum is undetermined on account of SGD noise and a subsequent , stable phase in which the final linearly connected minimum becomes determined . From this perspective , our observations contribute to a growing body of literature suggesting that training experiences a noisy initial phase and a less stochastic second phase . For example , the eigenspectrum of the Hessian settles into a bulk of small eigenvalues and a few large outlier eigenvalues after some amount of training ( Gur-Ari et al. , 2018 ) , and networks trained with large batch sizes and high learning rates benefit from learning rate warmup during the first part of training ( Goyal et al. , 2017 ) . One possible way to exploit our observations could be to explore changing aspects of the optimization process ( e.g. , learning rate schedule or optimizer ) similar to Goyal et al . ( 2017 ) once the network enters the stable phase in order to improve the performance of training ; instability analysis makes it possible to evaluate the consequences of doing so . As a scientific tool , we also believe instability analysis provides a framework for studying topics related to the scale and distribution of SGD noise , e.g. , the relationship between batch size , learning rate , and generalization ( Keskar et al. , 2017 ; Smith & Le , 2018 ; Smith et al. , 2018 ) and the efficacy of alternative learning rate schedules ( Smith , 2017 ; Smith & Topin , 2018 ; Li & Arora , 2019 ) .

Summary: This paper works on empirically demonstrating the connection between model connectivity and the lottery ticket hypothesis, which are individually explored in the literature. Here the model connectivity refers to the fact that SGD produces different solutions (from the randomness, such as data ordering) that are connected through model parameter transition paths of approximately equal loss/accuracy. The lottery ticket hypothesis tells that there exist sparse subnetworks of the corresponding full dense network which can attain as strong loss / accuracy as the full dense network.