Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory

1 Introduction . Deep learning techniques utilize the architecture of large model size and over-parameterization to achieve state-of-the-art performance in a wide range of applications [ 17 , 18 ] . The performance improvements can be owed to the availability of large-scale neural networks and massive computational resources to train such complex models . However , large model size and extreme over-parameterization typically lead to high training cost , slow inference speed , and large memory consumption . This hinders the applicability of deep learning models in resource-intensive scenarios with the requirement of low latency and energy consumption , such as Internet of Things [ 2 ] and mobile computing [ 51 , 55 ] . We have witnessed a large number of research advances in neural network pruning for reducing the time and space requirements of deep neural networks both at training and test time , with minimal loss in accuracy [ 31 , 20 , 34 ] . These techniques identify non-essential weights , filters , and other structures ∗Equal contributors †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . from neural networks that do not contribute significantly to the model performance , and remove them for compressing original dense neural networks [ 49 , 68 , 52 , 57 , 3 , 30 , 35 , 64 , 39 , 27 , 29 , 4 , 40 , 61 , 33 , 56 , 16 , 21 ] . The goal of neural network pruning is to reduce computational cost and memory consumption without a significant accuracy drop , compared with the original dense networks . However , the majority of neural network pruning methods require to train a dense neural network to generate a pruned sparse network , as it is difficult to train a pruned network from scratch [ 20 , 62 , 34 , 23 , 41 , 67 , 49 , 68 , 52 , 57 , 3 , 30 , 74 , 24 , 25 , 63 , 35 , 64 , 39 , 27 , 29 , 4 , 40 ] , although there are few pruning before training approaches [ 32 , 61 , 33 , 56 , 16 , 21 ] . Thus , these methods still suffer from the non-trivial training cost on over-parameterized networks . Most importantly , traditional network pruning techniques are based on ad-hoc heuristic pruning strategies [ 20 , 74 , 63 , 29 ] , leaving some critical theoretical questions unanswered on how much we can prune a neural network given a specified tolerance of accuracy drop , and how to achieve it with a practical and efficient procedure . Therefore , it is difficult to theoretically guarantee that the pruned networks achieve test accuracy comparable to the original dense ones . In addition , a recent literature has demonstrated that traditional neural network pruning often fall-short compared to the Lottery Ticket Hypothesis techniques [ 33 ] . A recent report has empirically identified and validated a surprising finding of the Lottery Ticket Hypothesis ( LTH ) : a randomly-initialized dense neural network contains an extremely sparse subnetwork ( i.e. , a winning lottery ticket ) such that , when trained from scratch with weights being reset to its initialization , can achieve similar performance to the original dense network within similar training iterations [ 13 , 14 ] . Compared with the original dense networks , the winning tickets are able to save a lot of memory for retraining [ 13 , 73 , 14 , 15 , 70 , 42 , 53 , 11 , 8 ] . The existence of winning tickets indicates that deep learning models can be trained in low-resource environments . While the LTH techniques exhibit promising practical implications for scaling over large models and datasets , they fail to provide an efficient algorithm to search for the winning ticket subnetworks for deep learning models in practice , due to the lack of theoretical evidence and analysis . They often utilize an Iterative Magnitude Pruning ( IMP ) strategy to gradually find winning tickets , which needs multiple rounds of expensive training , pruning , and resetting over large neural networks . Although existing LTH techniques raise very intriguing observations , most of them provide only empirical evidence to verify the LTH [ 71 , 12 , 1 , 47 , 69 , 54 , 5 , 53 , 26 , 8 , 7 , 11 ] . In this case , it is difficult to identify the optimal pruning strategy without performance drop , i.e. , how and where a neural network should be pruned . Therefore , two fundamental unanswered questions are raised : ( 1 ) How to verify the existence of winning lottery tickets , i.e. , sparse trainable subnetworks inside a dense neural network , with theoretical evidence and analysis ? ( 2 ) How to provide efficient algorithms to discover winning lottery tickets , without multiple rounds of expensive training and pruning in the IMP process ? To our best knowledge , this work is the first to theoretically verify the LTH and the existence of winning lottery tickets by leveraging dynamical systems theory and inertial manifold theory . We reformulate gradient descent optimization of a neural network as a gradient dynamical system regarding W ( i.e. , parameters of the neural network ) and analyze its dynamics with dynamical systems theory . Since inertial manifoldsMW contain all local minimum points W ∗ of loss function L regarding W ( i.e. , ∇L ( W ∗ ) = 0 ) , we reduce the original high-dimensional gradient system ϕ onto inertial manifolds to obtain a low-dimensional system regarding W+ , i.e. , parameters of the winning ticket subnetwork . We demonstrate that there must exist suchMW for neural network pruning , when the gradient dynamical system meets a certain condition about its spectrum and Lipschitz constant , for cutting W into W+ and W− ( i.e. , redundant parameters of the original neural network ) . In comparison with existing neural network pruning and LTH techniques , our dynamical systems and inertial manifold-based method exhibits two unique advantages : ( 1 ) All local minimum points W ∗ lie inMW and thus W+ is theoretically lossless pruning of W . ( 2 ) The computation ofMW and W+ is a one-time operation , such that our method is able to avoid the cost of multiple rounds of expensive training and pruning on large networks in the LTH methods . Due to large size of W in real scenarios , an approximate method is designed to estimate the Lipschitz constant for maintaining the efficiency . Extensive evaluation on real datasets demonstrates the superior performance of our proposed IMC model against several state-of-the-art neural network pruning and LTH methods . More experiments , implementation details , and hyperparameter setting are presented in Appendices A.3-A.5 . 2 Background . 2.1 Lottery Ticket Hypothesis . The Lottery Ticket Hypothesis ( LTH ) states that a randomly-initialized , dense neural network contains a subnetwork that is initialized such that , when trained in isolation , it can match the test accuracy of the original network after training for at most the same number of iterations [ 13 , 14 ] . Given a dense neural network f ( x ; W ) with randomly-initialized weight parameters W = W0 ∈ Rd , when optimizing with stochastic gradient descent ( SGD ) on a training set , f reaches minimum loss L at iteration I with test accuracy a . A sparse subnetwork of f is denoted as f ( x ; M W ) using a binary pruning maskM ∈ { 0 , 1 } d , ‖M‖0 d on its parameters such that its initialization isM Wi , where denotes the Hadamard product and Wi are the generated parameters of the original model f ( x ; W ) at iteration i . When optimizing with SGD on the same training set , f ( x ; M W ) , where W is initialized with Wi , can be trained from M W to reach minimum loss L at commensurate iteration Ī with commensurate accuracy ā ≥ a to the original model f ( x ; W ) . The LTH techniques often utilize an Iterative Magnitude Pruning ( IMP ) strategy to find winning tickets by gradually training the network and pruning its smallest-magnitude weights [ 13–15 ] . It operates in T rounds of expensive training , pruning , and resetting over large neural networks for guaranteeing the acceptable accuracy drop . Each of multiple rounds in the IMP consists of four steps : ( 1 ) Randomly initialize a dense neural network f ( x ; W ) with weight parameters W = W0 ; ( 2 ) Train W of f for I iterations , generating updated parameters WI ; ( 3 ) Prune p % of the parameters with the smallest magnitude in WI , producing a mask M ; and ( 4 ) Reset the remaining weight parameters back to their initial values in W0 , creating the winning ticket f ( x ; M W0 ) . 2.2 Dynamical Systems Theory . In mathematics , an ordinary differential equation ( ODE ) is a differential equation containing one or more functions of one independent variable and the derivatives of those functions [ 75 ] . A system of first-order ODE is defined as follows . x′ ( t ) = f ( x ( t ) ) , x ∈ X ( 1 ) where X is a Banach space and f : X → X is a function of x ( t ) . The unknown function x ( t ) at time t appears on both sides of the differential equation . Definition 1 . [ Globally Lipschitz ] f ( x ) is globally Lipschitz in X if there exists a constant K ≥ 0 for all x , y ∈ X , it holds ‖f ( x ) − f ( y ) ‖X ≤ K‖x− y‖X , where K is the Lipschitz constant of f . Definition 2 . [ Global Solution of ODE ] A solution x ( t ) : R→ X of the ODE in Eq . ( 1 ) is said to be global if x ( t ) exists for any t ∈ R. Lemma 1 . If f ( x ) is globally Lipschitz in X , then the ODE in Eq . ( 1 ) has a unique global solution in X for any initial condition x ( 0 ) = x0 [ 9 ] . Definition 3 . [ Flow of ODE ] ϕ : R × X → X , defined by ϕ ( t , x0 ) = x ( t ) is the flow ( i.e. , solution map ) of the ODE in Eq . ( 1 ) if ϕ ( t , x0 ) is the unique solution of the ODE with initial condition ϕ ( 0 , x0 ) = x ( 0 ) = x0 . For any given initial value x0 , the sets { ϕ ( t , x0 ) : t ∈ R } , { ϕ ( t , x0 ) : t ∈ R+ } , { ϕ ( t , x0 ) : t ∈ R− } are called the complete orbit , positive orbit , and negative orbit starting from x0 , respectively . In this work , we only consider the case of positive time and use the positive orbit starting from x . The orbit { ϕ ( t , x0 ) : t ∈ { R+ ∪ 0 } } of the flow is also called the trajectory of the point x under the flow . Definition 4 . [ Continuous Dynamical System ] If a flow ϕ ( t , x0 ) satisfies the following properties : • ϕ ( s+ t , x0 ) = ϕ ( s , x0 ) ϕ ( t , x0 ) for any s , t ∈ R ; • ϕ ( t , x0 ) is continuous in x0 for any t ∈ R ; • ϕ ( t , x0 ) is continuous in t for any x0 ∈ X , then the flow ϕ ( t , x0 ) together with the space X is a continuous dynamical system . If f ( x ) is globally Lipschitz , then the flow of the ODE is a continuous dynamical system . Dynamical system is a mathematical concept that is used to describe the time dependence of a point ’ s position in its ambient space . Dynamical System Theory is a mathematical tool to capture long-time behavior of solutions . Definition 5 . [ Equilibrium of Dynamical System ] A point x∗ is called an equilibrium of a dynamical system if ϕ ( t , x∗ ) = x∗ for all t ∈ R , i.e. , f ( x∗ ) = 0 . An equilibrium is a solution that does not change with time . This means if the systems starts at an equilibrium , the state will remain at the equilibrium forever . Definition 6 . [ Invariant Set/Manifold ] A set or manifold A is invariant under a dynamical system ϕ if ϕ ( t , A ) ⊂ A for any t ≥ 0 . That is , if a is an element of A then so is ϕ ( t , a ) for all t ≥ 0 . A manifold is a topological space that locally resembles Euclidean space near each point . Examples of invariant manifolds include stable manifold , unstable manifold , and inertial manifold . Definition 7 . [ Attractor of Dynamical System ] A set A ⊂ Rn is an attractor that attracts a set B ⊂ Rn under a dynamical system ϕ if A is an invariant set under ϕ and lim t→∞ dist ( ϕ ( t , B ) , A ) = 0 ( 2 ) where B denotes the basin of attraction for A. dist represents the Hausdorff semi-distance defined by dist ( B , A ) = sup x∈B inf y∈A ‖x− y‖X ( 3 ) An attractor is a set of states ( i.e. , points in the phase space ) , invariant under the dynamics , towards which neighboring states in a given basin of attraction asymptotically approach in the course of dynamic evolution . An attractor can be a point , a finite set of points , a curve , a manifold , or even a complicated set with a fractal structure . lim t→∞ dist ( ϕ ( t , B ) , A ) = 0 implies that flows starting from any point b ∈ B enter A in the limit t→∞ . Definition 8 . [ Global Attractor of Dynamical System ] A nonempty compact set A ⊂ X is a global attractor under a dynamical system ϕ if A is an invariant set under ϕ and A attracts all bounded sets B ⊂ X . A global attractor represents the maximal compact invariant set which attracts the orbits of all bounded sets at a uniform rate . Definition 9 . [ Stable/Unstable Manifold of Dynamical System ] Given an equilibrium x∗ of a dynamical system , the stable manifoldMs ( x∗ ) of x∗ is a set of points in phase space if Ms ( x∗ ) = { x ∈ X : ϕ ( t , x ) → x∗ as t→ +∞ } ( 4 ) and the unstable manifoldMu ( x∗ ) of x∗ is a set of points in phase space if Mu ( x∗ ) = { x ∈ X : ϕ ( t , x ) → x∗ as t→ −∞ } ( 5 )

Summary: This work aims to theoretically verify the lottery ticket hypothesis and the existence of winning lottery tickets. Dynamical systems and inertial manifold theory are used.  Lots of mathematical proof and definitions are included in the papers, and experiments are conducted to test the performance of the proposed method.