Plant 'n' Seek: Can You Find the Winning Ticket?

1 INTRODUCTION . Deep learning has achieved breakthroughs in multiple challenging areas pertaining to machine learning , in particular in areas for which we lack competitive hand-crafted algorithms . The benefits of overparameterization for training with SGD ( Belkin et al. , 2019 ) seem to call for ever wider and deeper neural network ( NN ) architectures , which are computationally demanding to learn and deploy . Training smaller , adequately regularized NNs from scratch could be a remedy but it commonly seems to fail due to inadequate parameter initialization , as Frankle & Carbin ( 2019 ) noted in their seminal paper . As proof of concept that this problem is solvable , they proposed the lottery ticket ( LT ) hypothesis , which states that a small , well trainable subnetwork can be identified by pruning a large , randomly initialized NN , opening the field to discover such subnetworks or ’ winning tickets ’ . Based on the findings of Zhou et al . ( 2019 ) , Ramanujan et al . ( 2020b ) went even further and conjectured the existence of strong lottery tickets , i.e. , subnetworks of randomly initialized NNs that do not require any further training . This strong LT hypothesis holds the promise that training NNs could potentially be replaced by efficient NN pruning , which simultaneously performs structure learning by identifying a task specific sparse neural network architecture . The existence of strong LTs has also been proven formally for networks without ( Malach et al. , 2020 ; Pensia et al. , 2020 ; Orseau et al. , 2020 ) and with potentially nonzero biases ( Fischer & Burkholz , 2021 ) . While these types of proofs show existence in realistic settings , the sparsity of the constructed tickets is likely not optimal , as they represent a target parameter by multiple neurons of degree 1 . The construction and proof of the generalized strong LT hypothesis raises two question – is the suboptimal sparsity merely an artifact of existence proofs or a general limitation of the pruning approach ? And , if very sparse tickets exist , are current algorithms able to find them or are further improvements needed to achieve effective network compression ? These questions can not be answered by comparing LT pruning algorithms solely on standard benchmark datasets ( Frankle et al. , 2021 ) , but demand the comparison with known ground truth LTs . The lack thereof was raised as an issue by Frankle et al . ( 2021 ) . To fill this gap and generate baselines with known ground truth , we here propose an algorithm to plant and hide arbitrary winning tickets in randomly initialized NNs and construct sparse tickets that reflect common challenges in machine learning . We use this experimental set-up to compare state-of-the-art pruning algorithms designed to search for lottery tickets . Our results indicate that state-of-the-art methods achieve only sub-optimal sparsity levels . This suggests that previous challenges to identify highly sparse winning tickets as subnetworks of randomly initialized dense networks ( Frankle et al. , 2020 ; Ramanujan et al. , 2020b ) can be explained by algorithmic limitations rather than fundamental problems with LT existence . In our experiments , the qualitative trends how methods compare to each other are consistent with previous results on image classification tasks ( Tanaka et al. , 2020 ; Frankle et al. , 2021 ) indicating that our experimental set-up exposes pruning algorithms to realistic challenges . In addition , we identify an opportunity to improve state-of-the-art pruning algorithms in order to find strong LTs of better sparsity . Our proposed planting framework will enable the evaluation of future progress in this direction . Contributions 1 ) We prove the existence of strong lottery tickets with sparse representations . 2 ) Inspired by the proof , we derive a framework that allows us to plant and hide strong tickets in neural networks and thus create benchmark data with known ground truth . 3 ) We construct sparse representations of four types of tickets that reflect typical machine learning problems . 4 ) We systematically evaluate state-of-the-art pruning methods that aim to discover tickets on these three problems against the ground truth tickets and highlight key challenges . 1.1 RELATED WORK . LT pruning approaches for neural networks can be broadly categorized into three groups , pruning before , during , or after training . While methods that sparsify the network during ( LeCun et al. , 1990a ; Mozer & Smolensky , 1989 ; Han et al. , 2015 ; Frankle & Carbin , 2019 ; Srinivas & Babu , 2016 ; Lee et al. , 2020 ) , or after training ( Savarese et al. , 2020b ; LeCun et al. , 1990b ; Hassibi & Stork , 1992 ; Dong et al. , 2017 ; Li et al. , 2017 ; Molchanov et al. , 2017 ) help in reducing computational resources required for inference , they are , however , less helpful in reducing resources at training time but can make a difference if they prune early aggressively ( You et al. , 2020 ) . They are most useful for structure learning at lower sparsity levels ( Su et al. , 2020 ; Lee et al. , 2020 ) . The LT hypothesis ( Frankle & Carbin , 2019 ) has also promoted the development of neural network pruning algorithms that prune before training ( Wang et al. , 2020 ; Lee et al. , 2019 ; Verdenius et al. , 2020 ; Tanaka et al. , 2020 ; Ramanujan et al. , 2020b ) . Usually , these methods try to find LTs in a ‘ weak ’ ( but powerful ) sense , that is to identify a sparse neural network architecture that is well trainable starting from its initial parameters . These methods score edges in terms of network flow , which can be quantified by gradients at different stages of pruning , or based on edge weights , and prune all edges with the lowest scores until the desired sparsity is achieved . Frankle et al . ( 2021 ) . Strong LTs are sparse sub-networks that perform well with the initial parameters , hence do not need to be trained any further ( Zhou et al. , 2019 ; Ramanujan et al. , 2020b ) . Their existence has been proven by providing lower bounds on the width of the large , randomly initialized neural network that contains them ( Malach et al. , 2020 ; Pensia et al. , 2020 ; Orseau et al. , 2020 ; Fischer & Burkholz , 2021 ) . In addition , it was shown that multiple candidate tickets exist that are also robust to parameter quantization ( Diffenderfer & Kailkhura , 2021 ) . Beyond LT pruning , many more methods have been developed to reduce computational resources and perform structure learning , including dynamic sparse training ( Evci et al. , 2020 ; Liu et al. , 2021b ) , adaptations ( Frankle et al. , 2020 ; Renda et al. , 2020 ; Liu et al. , 2021a ; Liu et al . ) of Iterative Magnitude Pruning ( IMP ) Han et al . ( 2015 ) ; Frankle & Carbin ( 2019 ) and sparse regularization techniques ( Weigend et al. , 1991 ; Savarese et al. , 2020a ) . As these approaches do not identify LTs as subnetworks of randomly initialized NNs , they do not rely on the existence of planted tickets and are therefore beyond the scope of our experimental analysis . However , the ground truth tickets which we derived for planting could still provide an interesting baseline to explore whether sparse training of deep NNs can identify extremely sparse , hand designed NN architectures . Dense NNs are known to find NN representations that are less sparse than hand crafted architectures ( Denker et al. , 1987 ) , yet , the the explicit objective of sparse training is to address this issue . We provide the tools to evaluate progress in this direction by planting known ticket architectures . While the ultimate goal of deep learning is to solve problems with otherwise unknown solutions like image classification ( Frankle et al. , 2021 ) or protein structure prediction ( Tunyasuvunakool et al. , 2021 ) , the design of NN ar- chitectures for human solvable problems has already in the past provided important insights into NN properties , including universal approximation ( Scarselli & Tsoi , 1998 ; Yarotsky , 2018 ) or the importance of algorithmic alignment ( Xu et al. , 2020 ) . NNs that compute polynomials ( Scarselli & Tsoi , 1998 ; Yarotsky , 2018 ) , xor gates ( Rumelhart et al. , 1986 ) , discrete fast fourier transformation ( Velik , 2008 ) , symmetry groups ( Sejnowski et al. , 1986 ) , general piecewise linear functions ( Arora et al. , 2018 ) , or argmax ( Xu et al. , 2020 ) could also present interesting candidates for planting in future investigations . 1.2 NOTATION AND TERMINOLOGY . Let f ( x ) denote a bounded function , without loss of generality f : [ −1 , 1 ] n0 → [ −1 , 1 ] nL , that is parameterized as a deep neural network with architecture n̄ = [ n0 , n1 , ... , nL ] , i.e. , depth L and widths nl for layers l = 0 , ... , L with ReLU activation function ϕ ( x ) : = max ( x , 0 ) . It maps an input vector x ( 0 ) to neurons x ( l ) i as x ( l ) = ϕ ( W ( l ) x ( l−1 ) + b ( l ) ) , where W ( l ) ∈ Rnl−1×nl is the weight matrix , and b ( l ) ∈ Rnl is the bias vector of Layer l. We will establish approximation results with respect to the supremum norm ∥g∥∞ : = supx∈ [ −1,1 ] n0 ∥g∥2 defined for any function g on the domain [ −1 , 1 ] n0 . Assume furthermore that a LT fϵ can be obtained by pruning a large mother network f , which we indicate by writing fϵ ⊂ f0 . If fϵ achieves a similar performance after training the non-zero parameters as training all parameters of f0 , we call fϵ a weak LT . If fϵ does not require any further training , we call it a strong LT . Note that every strong LT is automatically also a weak LT and any algorithm that prunes the initial network f0 identifies a LT that could , in principle , be a strong LT . The sparsity level ρ of fϵ is then defined as the fraction of non-zero weights that remain after pruning , i.e. , ρ = ( ∑ l ∥∥∥W ( l ) ϵ ∥∥∥ 0 ) / ( ∑ l ∥∥∥W ( l ) 0 ∥∥∥ 0 ) , where ∥·∥0 denotes the l0-norm , which counts the number of non-zero elements in a vector or matrix . Another important quantity that influences the existence probability of lottery tickets is the in-degree of a node i in layer l of the target f , which we define as the number of non-zero connections of a neuron to the previous layer plus 1 if the bias is non-zero , i.e. , k ( l ) i : = ∥∥∥W ( l ) i , : ∥∥∥ 0 + ∥∥∥b ( l ) i ∥∥∥ 0 , where W ( l ) i , : is the i-th row of W ( l ) . The maximum degree of all neurons in layer l is denoted as kl , max .

Summary: The authors identify that contemporary methods for finding very sparse subnetworks in deep neural networks (DNNs), including both methods for finding either "weak tickets" (after training) or "strong tickets" (before training), do not find very sparse (and good) solutions. They question if this is a fundamental limitation (i.e. these very sparse solutions don't exist), or if this is a limitation of current methods in finding such solutions. To answer this question, the authors propose to either "plant" known good very sparse subnetworks in DNNs, or to try and find solutions for toy problems for which the authors could identify good hand-made very sparse subnetworks - including a classification problem (Circle), function regression (ReLU), and manifold-learning problem (Helix) . The experimental results analyze the performance of strong and weak sparse subnetwork search methods, including GraSP, SNIP, SYNFLOW, alongside magnitude and random pruning in finding these planted tickets. None of the strong ticket methods are able to find the "planted" tickets for any sparsity level, while the weak methods are able to find tickets, but not at extreme sparsity. Further analysis reveals layer collapse in particular to be a problem. The authors conclude that rather than a fundamental limitation, current methods (in particular for strong tickets) are limited in finding very sparse tickets even if they are known to exist at that sparsity level.