Dual Lottery Ticket Hypothesis

1 INTRODUCTION . While over-parameterized networks perform promisingly on challenging machine learning tasks Zagoruyko & Komodakis ( 2016 ) ; Arora et al . ( 2019 ) ; Zhang et al . ( 2019 ) , they require a high cost of computational and storage resources Wang et al . ( 2020a ) ; Cheng et al . ( 2017 ) ; Deng et al . ( 2020 ) . Recent pruning techniques aim to diminish the model size by discarding irrelevant weights of well-trained models based on different criteria Gale et al . ( 2019 ) ; He et al . ( 2017 ) ; Han et al . ( 2015a ; b ) . Decisive weights are preserved and finetuned for obtaining final compressed model with acceptable performance loss . Following this line , several series of research works have been done to explore effective pruning criterion for better pruning performances . For example , regularization based pruning approaches Liu et al . ( 2017 ) ; Han et al . ( 2015a ; b ) leverage a penalty term during training for network pruning . Also , many researches take advantages of Hessian information to build more proper criteria for pruning LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Wang et al . ( 2019a ) ; Singh & Alistarh ( 2020 ) . However , regular pruning methods still requires full pretraining with high computational and storage costs . Pruning at initialization ( PI ) attempts determining the sparse network before training and maintains the final performances . For instance , Single-shot Network Pruning ( SNIP ) Lee et al . ( 2018 ) uses a novel criterion called connection sensitivity to measure the weights importance and decide which weights should be removed . Gradient Signal Preservation ( GraSP ) Wang et al . ( 2020a ) regards the gradient flow as an importance factor and correspondingly use it to design PI criterion . Regular pruning and PI techniques both achieve promising results on sparse network training and model compression . They only focus on designating criteria to find a specific sparse network but ignore exploring the internal relationships between the dense network and its eligible subnetwork candidates , which impedes a full understanding of the sparse network training . Lottery Ticket Hypothesis ( LTH ) Frankle & Carbin ( 2018 ) hits this problem with the first shot . It claims a trainable subnetwork exists in the randomly initialized dense network and can be found by pruning a pretrained network . In other words , as long as the dense network is initialized , the good subnetwork has been implicitly decided but needs to be revealed by the pruning process . The LTH has been validated by training the subnetwork from scratch with mask obtained from corresponding iterative magnitude pruning Han et al . ( 2015b ; a ) . The subnetwork is regarded as the winning ticket among the lottery pool given by dense networks . This valuable discovery naturally describes the relationship between a random initialized dense network and the trainable subnetwork hidden in it . However , LTH only focuses on finding one sparse structure at the expense of full pretraining , which is not universal to both practical usage and investigating the relationship between dense and its subnetworks for sparse network training . In this work , we go from a complementary direction of LTH and propose a new hypothesis called Dual Lottery Ticket Hypothesis ( DLTH ) . It studies a more challenging and general case and achieves promising performances ( see Fig . 1 ) for sparse network training . DLTH is described as any random selected subnetwork of a randomly initialized dense network can be transformed into a trainable condition . In other word , any ticket in a given lottery pool can be turned into a winning ticket . In this way , we construct a closed-loop research perspective from the dual side of LTH as shown in Fig . 2 . Compared with LTH , our proposed DLTH considers a general case - studying any subnetwork instead of a specific one – with a simple training strategy proposed to validate it . The comprehensive relationship between dense and sparse networks is inquired for sparse network training . Our contributions can be summarized as follows : • We investigate a novel view of studying sparse network training . Specifically , we present a Dual Lottery Ticket Hypothesis ( DLTH ) , a dual problem of Lottery Ticket Hypothesis ( LTH ) , articulated as : Any randomly selected subnetwork of a randomly initialized dense network can be transformed into an appropriate condition with admirable trainability . • We propose a simple sparse network training strategy , Random Sparse Network Transformation ( RST ) , to validate our proposed DLTH and achieve promising performance for sparse network optimization . Concretely , we achieve the information extrusion from the weights which will be removed to enhance sparse training by introducing a regularization term . Any subnetwork will be transformed into a trainable condition in this way . • Extensive experiments are conducted based on benchmark datasets and comparisons with competitive approaches . Our method obtains promising and consistent performance which solidly validates our DLTH and demonstrates the model effectiveness . • Our DLTH considers a more general and challenging case for sparse network training compared with LTH . In this way , we expect our work inspires a novel perspective to study sparse network training in a more flexible and adjustable way . 2 RELATED WORK . 2.1 SPARSE NETWORK TRAINING . Sparse Evolutionary Training ( SET ) Mocanu et al . ( 2018 ) proposes an evolutionary strategy to randomly add weights on sparse network and achieve better training . Dynamic Sparse Reparameterization ( DSR ) Mostafa & Wang ( 2019 ) presents a novel direction to dynamically modify the parameter budget between different layers . In this way , the sparse structure can be adaptively adjusted for more effective and efficient usage . Sparse Networks from Scratch ( SNFS ) Dettmers & Zettlemoyer ( 2019 ) designates a momentum based approach as criterion to grow weights and empirically proves it benefits the practical learning performance . Deep Rewiring ( DeepR ) Bellec et al . ( 2017 ) introduces the sparsity during the training process and augments the regular SGD optimization by involving a random walk in the parameter space . It can achieve effectively training on very sparse network relying on theoretical foundations . Rigging the Lottery ( RigL ) Evci et al . ( 2020 ) enhances the sparse network training by editing the network connectivity along with the optimizing the parameter by taking advantages of both weight magnitude and gradient information . 2.2 NETWORK PRUNING . Network pruning is a relevant research topic to our work . Related approaches aim to remove unnecessary weights based on different well-designed criteria and obtain sparse network for model compression . Diversified pruning methods can be categorized from different angles . To better present the background of our work , we customize the existing pruning methods into Training-Relevant Pruning and Training-Free Pruning groups as follow . Training-Relevant Pruning . Most pruning methods require a pre-trained network Louizos et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Ye et al . ( 2018 ) ; Han et al . ( 2015b ; a ) . Algorithms utilize different ranking strategies to pick redundant weights with low importance scores , and remove them to achieve pruning along with acceptable performance drop . Pioneer magnitude-based method Han et al . ( 2015b ; a ) regards weights with low values as unnecessary ones , which is straightforward but may remove important low-value weights . Hessian-based approaches measure the weight importance by computing the their removal effects on the final loss LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) . Recently published technique also achieves Hessian-free pruning Wang et al . ( 2020b ) by adding regularization which is more computational friendly . On the other hand , pruning process can be also conducted during the network training instead of relying on completed pre-trained network . To name a few , a special dropout strategy is utilized in Srinivas & Babu ( 2016 ) to adjust the dropout during training and obtain a pruned network after training . A L0 norm regularization based method is proposed for network pruning Louizos et al . ( 2017 ) . Above algorithms requiring network training are seen as Training-Relevant Pruning . Training-Free Pruning . This group of research investigates how to prune randomly initialized network without any training . Pruning at initialization is one typical direction deriving sparse network by remove part of randomly initialized weights Lee et al . ( 2018 ; 2019 ) ; Wang et al . ( 2020a ) . For instance , Single-Shot Neural Network Pruning ( SNIP ) algorithm Lee et al . ( 2018 ) first proposes a learnable sparse network strategy according to the computed connection sensitivity . An orthogonal initialization method is proposed to investigate pruning problem in signal propagation view Lee et al . ( 2019 ) . Gradient Signal Preservation ( GraSP ) considers to preserve the gradient flow as an efficient criterion to achieve pruning at initialization . On the other hand , Lottery Ticket Hypothesis ( LTH ) claims that the winning ticket subnetworks exist in randomly initialized network and can be found by deploying conventional pruning algorithm . Then the selected sparse network can be efficiently trained from scratch and achieve promising performance . Based on LTH , Early-Bird ( EB ) ticket You et al . ( 2020 ) proposes an efficient way to find winning ticket . Above algorithms obtaining sparse network from random network are seen as Training-Free Pruning . 3 LOTTERY TICKET PERSPECTIVE OF SPARSE NETWORK . Lottery Ticket Hypothesis ( LTH ) Frankle & Carbin ( 2018 ) articulates the hypothesis : dense randomly initialized networks contain subnetworks , which can be trained in isolation and deliver performances comparable to the original network . These subnetworks are treated as winning tickets in the given lottery pool with good trainability . Subnetworks are discovered by iterative magnitude pruning algorithms , which requires training a full network and removes redundant weights by specific criterion . The mask of the pruned network illustrates the sparse structure of winning tickets . Problem Formulation . We start from introducing general network pruning problem . The neural network training process can be seen as a sequence of parameter updating status using stochastic gradient descent ( SGD ) Wang et al . ( 2021 ) ; Bottou ( 2010 ) : { w ( 0 ) , w ( 1 ) , w ( 2 ) , · · · , w ( k ) , · · · , w ( K ) } , ( 1 ) where w is the model parameter with superscript k as training iteration number . For a general case , the sparse network structure can be mathematically described by a binary mask m which has the same tensor shape as w. The process of obtaining m can be formulated as a function m = Fm ( w ; D ) , where D is the training data . Further , the weights of the sparse structure are modified based on different pruning strategies Hassibi & Stork ( 1993 ) ; Wang et al . ( 2020b ) , which is given by w∗ = Fw ( w ; D ) . The final pruned network can be integrated as w̃ = Fm ( w ( km ) ; D ) · Fw ( w ( kw ) ; D ) , ( 2 ) where wkm and wkw represent different parameter conditions for Fm and Fw . The conventional pruning requires that km = kw = K. The LTH needs km = K , kw = 0 , and Fw = I , where I is the identical mapping representing model directly inherits the randomly initialized weights . Against with traditional view that directly training sparse network can not fully exploit network capacity Wang et al . ( 2020b ; a ) , LTH validates there exists sparse network with better trainability ( winning ticket ) than other subnetworks ( other tickets ) . However , to uncover this property still needs first using pruning techniques for obtaining mask and the selected sparse structure must match with corresponding randomly initialized dense network - winning ticket matches with the given lottery pool . LTH provides a novel angle to understand and reveal the connections between a randomly initialized dense network and its subnetworks with admirable tranability . However , the proposed hypothesis is valid with restrict constraint and finding subnetworks still requires pruning on pre-trained model .

Summary: This work proposes the so called Dual Lottery Ticket Hypothesis which claims that every ticket in the ticket pool, i.e., every random sparse subnetwork in a randomly initialized dense neural network, can be transformed into a winning ticket with admirable trainability. This "transformation" proposed in the notation of Random Sparse Network Transformation (RST), in practice, comes in with the form of a squared L2 norm regularization on the masked weights, making them still involved during training while extruding the information contained in them and transferring into the unmasked ones. The empirical experiments show good empirical performance of RST compared to the vanilla LTH, pruning-at-initialization methods and LTH variants such as EB-LTH.