Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory

1 Introduction . Deep learning techniques utilize the architecture of large model size and over-parameterization to achieve state-of-the-art performance in a wide range of applications [ 17 , 18 ] . The performance improvements can be owed to the availability of large-scale neural networks and massive computational resources to train such complex models . However , large model size and extreme over-parameterization typically lead to high training cost , slow inference speed , and large memory consumption . This hinders the applicability of deep learning models in resource-intensive scenarios with the requirement of low latency and energy consumption , such as Internet of Things [ 2 ] and mobile computing [ 51 , 55 ] . We have witnessed a large number of research advances in neural network pruning for reducing the time and space requirements of deep neural networks both at training and test time , with minimal loss in accuracy [ 31 , 20 , 34 ] . These techniques identify non-essential weights , filters , and other structures ∗Equal contributors †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . from neural networks that do not contribute significantly to the model performance , and remove them for compressing original dense neural networks [ 49 , 68 , 52 , 57 , 3 , 30 , 35 , 64 , 39 , 27 , 29 , 4 , 40 , 61 , 33 , 56 , 16 , 21 ] . The goal of neural network pruning is to reduce computational cost and memory consumption without a significant accuracy drop , compared with the original dense networks . However , the majority of neural network pruning methods require to train a dense neural network to generate a pruned sparse network , as it is difficult to train a pruned network from scratch [ 20 , 62 , 34 , 23 , 41 , 67 , 49 , 68 , 52 , 57 , 3 , 30 , 74 , 24 , 25 , 63 , 35 , 64 , 39 , 27 , 29 , 4 , 40 ] , although there are few pruning before training approaches [ 32 , 61 , 33 , 56 , 16 , 21 ] . Thus , these methods still suffer from the non-trivial training cost on over-parameterized networks . Most importantly , traditional network pruning techniques are based on ad-hoc heuristic pruning strategies [ 20 , 74 , 63 , 29 ] , leaving some critical theoretical questions unanswered on how much we can prune a neural network given a specified tolerance of accuracy drop , and how to achieve it with a practical and efficient procedure . Therefore , it is difficult to theoretically guarantee that the pruned networks achieve test accuracy comparable to the original dense ones . In addition , a recent literature has demonstrated that traditional neural network pruning often fall-short compared to the Lottery Ticket Hypothesis techniques [ 33 ] . A recent report has empirically identified and validated a surprising finding of the Lottery Ticket Hypothesis ( LTH ) : a randomly-initialized dense neural network contains an extremely sparse subnetwork ( i.e. , a winning lottery ticket ) such that , when trained from scratch with weights being reset to its initialization , can achieve similar performance to the original dense network within similar training iterations [ 13 , 14 ] . Compared with the original dense networks , the winning tickets are able to save a lot of memory for retraining [ 13 , 73 , 14 , 15 , 70 , 42 , 53 , 11 , 8 ] . The existence of winning tickets indicates that deep learning models can be trained in low-resource environments . While the LTH techniques exhibit promising practical implications for scaling over large models and datasets , they fail to provide an efficient algorithm to search for the winning ticket subnetworks for deep learning models in practice , due to the lack of theoretical evidence and analysis . They often utilize an Iterative Magnitude Pruning ( IMP ) strategy to gradually find winning tickets , which needs multiple rounds of expensive training , pruning , and resetting over large neural networks . Although existing LTH techniques raise very intriguing observations , most of them provide only empirical evidence to verify the LTH [ 71 , 12 , 1 , 47 , 69 , 54 , 5 , 53 , 26 , 8 , 7 , 11 ] . In this case , it is difficult to identify the optimal pruning strategy without performance drop , i.e. , how and where a neural network should be pruned . Therefore , two fundamental unanswered questions are raised : ( 1 ) How to verify the existence of winning lottery tickets , i.e. , sparse trainable subnetworks inside a dense neural network , with theoretical evidence and analysis ? ( 2 ) How to provide efficient algorithms to discover winning lottery tickets , without multiple rounds of expensive training and pruning in the IMP process ? To our best knowledge , this work is the first to theoretically verify the LTH and the existence of winning lottery tickets by leveraging dynamical systems theory and inertial manifold theory . We reformulate gradient descent optimization of a neural network as a gradient dynamical system regarding W ( i.e. , parameters of the neural network ) and analyze its dynamics with dynamical systems theory . Since inertial manifoldsMW contain all local minimum points W ∗ of loss function L regarding W ( i.e. , ∇L ( W ∗ ) = 0 ) , we reduce the original high-dimensional gradient system ϕ onto inertial manifolds to obtain a low-dimensional system regarding W+ , i.e. , parameters of the winning ticket subnetwork . We demonstrate that there must exist suchMW for neural network pruning , when the gradient dynamical system meets a certain condition about its spectrum and Lipschitz constant , for cutting W into W+ and W− ( i.e. , redundant parameters of the original neural network ) . In comparison with existing neural network pruning and LTH techniques , our dynamical systems and inertial manifold-based method exhibits two unique advantages : ( 1 ) All local minimum points W ∗ lie inMW and thus W+ is theoretically lossless pruning of W . ( 2 ) The computation ofMW and W+ is a one-time operation , such that our method is able to avoid the cost of multiple rounds of expensive training and pruning on large networks in the LTH methods . Due to large size of W in real scenarios , an approximate method is designed to estimate the Lipschitz constant for maintaining the efficiency . Extensive evaluation on real datasets demonstrates the superior performance of our proposed IMC model against several state-of-the-art neural network pruning and LTH methods . More experiments , implementation details , and hyperparameter setting are presented in Appendices A.3-A.5 . 2 Background . 2.1 Lottery Ticket Hypothesis . The Lottery Ticket Hypothesis ( LTH ) states that a randomly-initialized , dense neural network contains a subnetwork that is initialized such that , when trained in isolation , it can match the test accuracy of the original network after training for at most the same number of iterations [ 13 , 14 ] . Given a dense neural network f ( x ; W ) with randomly-initialized weight parameters W = W0 ∈ Rd , when optimizing with stochastic gradient descent ( SGD ) on a training set , f reaches minimum loss L at iteration I with test accuracy a . A sparse subnetwork of f is denoted as f ( x ; M W ) using a binary pruning maskM ∈ { 0 , 1 } d , ‖M‖0 d on its parameters such that its initialization isM Wi , where denotes the Hadamard product and Wi are the generated parameters of the original model f ( x ; W ) at iteration i . When optimizing with SGD on the same training set , f ( x ; M W ) , where W is initialized with Wi , can be trained from M W to reach minimum loss L at commensurate iteration Ī with commensurate accuracy ā ≥ a to the original model f ( x ; W ) . The LTH techniques often utilize an Iterative Magnitude Pruning ( IMP ) strategy to find winning tickets by gradually training the network and pruning its smallest-magnitude weights [ 13–15 ] . It operates in T rounds of expensive training , pruning , and resetting over large neural networks for guaranteeing the acceptable accuracy drop . Each of multiple rounds in the IMP consists of four steps : ( 1 ) Randomly initialize a dense neural network f ( x ; W ) with weight parameters W = W0 ; ( 2 ) Train W of f for I iterations , generating updated parameters WI ; ( 3 ) Prune p % of the parameters with the smallest magnitude in WI , producing a mask M ; and ( 4 ) Reset the remaining weight parameters back to their initial values in W0 , creating the winning ticket f ( x ; M W0 ) . 2.2 Dynamical Systems Theory . In mathematics , an ordinary differential equation ( ODE ) is a differential equation containing one or more functions of one independent variable and the derivatives of those functions [ 75 ] . A system of first-order ODE is defined as follows . x′ ( t ) = f ( x ( t ) ) , x ∈ X ( 1 ) where X is a Banach space and f : X → X is a function of x ( t ) . The unknown function x ( t ) at time t appears on both sides of the differential equation . Definition 1 . [ Globally Lipschitz ] f ( x ) is globally Lipschitz in X if there exists a constant K ≥ 0 for all x , y ∈ X , it holds ‖f ( x ) − f ( y ) ‖X ≤ K‖x− y‖X , where K is the Lipschitz constant of f . Definition 2 . [ Global Solution of ODE ] A solution x ( t ) : R→ X of the ODE in Eq . ( 1 ) is said to be global if x ( t ) exists for any t ∈ R. Lemma 1 . If f ( x ) is globally Lipschitz in X , then the ODE in Eq . ( 1 ) has a unique global solution in X for any initial condition x ( 0 ) = x0 [ 9 ] . Definition 3 . [ Flow of ODE ] ϕ : R × X → X , defined by ϕ ( t , x0 ) = x ( t ) is the flow ( i.e. , solution map ) of the ODE in Eq . ( 1 ) if ϕ ( t , x0 ) is the unique solution of the ODE with initial condition ϕ ( 0 , x0 ) = x ( 0 ) = x0 . For any given initial value x0 , the sets { ϕ ( t , x0 ) : t ∈ R } , { ϕ ( t , x0 ) : t ∈ R+ } , { ϕ ( t , x0 ) : t ∈ R− } are called the complete orbit , positive orbit , and negative orbit starting from x0 , respectively . In this work , we only consider the case of positive time and use the positive orbit starting from x . The orbit { ϕ ( t , x0 ) : t ∈ { R+ ∪ 0 } } of the flow is also called the trajectory of the point x under the flow . Definition 4 . [ Continuous Dynamical System ] If a flow ϕ ( t , x0 ) satisfies the following properties : • ϕ ( s+ t , x0 ) = ϕ ( s , x0 ) ϕ ( t , x0 ) for any s , t ∈ R ; • ϕ ( t , x0 ) is continuous in x0 for any t ∈ R ; • ϕ ( t , x0 ) is continuous in t for any x0 ∈ X , then the flow ϕ ( t , x0 ) together with the space X is a continuous dynamical system . If f ( x ) is globally Lipschitz , then the flow of the ODE is a continuous dynamical system . Dynamical system is a mathematical concept that is used to describe the time dependence of a point ’ s position in its ambient space . Dynamical System Theory is a mathematical tool to capture long-time behavior of solutions . Definition 5 . [ Equilibrium of Dynamical System ] A point x∗ is called an equilibrium of a dynamical system if ϕ ( t , x∗ ) = x∗ for all t ∈ R , i.e. , f ( x∗ ) = 0 . An equilibrium is a solution that does not change with time . This means if the systems starts at an equilibrium , the state will remain at the equilibrium forever . Definition 6 . [ Invariant Set/Manifold ] A set or manifold A is invariant under a dynamical system ϕ if ϕ ( t , A ) ⊂ A for any t ≥ 0 . That is , if a is an element of A then so is ϕ ( t , a ) for all t ≥ 0 . A manifold is a topological space that locally resembles Euclidean space near each point . Examples of invariant manifolds include stable manifold , unstable manifold , and inertial manifold . Definition 7 . [ Attractor of Dynamical System ] A set A ⊂ Rn is an attractor that attracts a set B ⊂ Rn under a dynamical system ϕ if A is an invariant set under ϕ and lim t→∞ dist ( ϕ ( t , B ) , A ) = 0 ( 2 ) where B denotes the basin of attraction for A. dist represents the Hausdorff semi-distance defined by dist ( B , A ) = sup x∈B inf y∈A ‖x− y‖X ( 3 ) An attractor is a set of states ( i.e. , points in the phase space ) , invariant under the dynamics , towards which neighboring states in a given basin of attraction asymptotically approach in the course of dynamic evolution . An attractor can be a point , a finite set of points , a curve , a manifold , or even a complicated set with a fractal structure . lim t→∞ dist ( ϕ ( t , B ) , A ) = 0 implies that flows starting from any point b ∈ B enter A in the limit t→∞ . Definition 8 . [ Global Attractor of Dynamical System ] A nonempty compact set A ⊂ X is a global attractor under a dynamical system ϕ if A is an invariant set under ϕ and A attracts all bounded sets B ⊂ X . A global attractor represents the maximal compact invariant set which attracts the orbits of all bounded sets at a uniform rate . Definition 9 . [ Stable/Unstable Manifold of Dynamical System ] Given an equilibrium x∗ of a dynamical system , the stable manifoldMs ( x∗ ) of x∗ is a set of points in phase space if Ms ( x∗ ) = { x ∈ X : ϕ ( t , x ) → x∗ as t→ +∞ } ( 4 ) and the unstable manifoldMu ( x∗ ) of x∗ is a set of points in phase space if Mu ( x∗ ) = { x ∈ X : ϕ ( t , x ) → x∗ as t→ −∞ } ( 5 )

Summary: Summary:   The paper is dedicated to validating the lottery tickets hypothesis and delivering an effective one-time pruning. The authors investigate dynamical systems theory and inertial manifold theory to offer a theoretical justification. Specifically, they reduce the high-dimensional system of dense networks onto inertial manifolds to obtain a low-dimensional system regarding pruned subnetworks. Experiments on CIFAR and ImageNet with ResNet-18 and ResNet-32 validate the effectiveness of the proposal.