Dual Lottery Ticket Hypothesis

1 INTRODUCTION . While over-parameterized networks perform promisingly on challenging machine learning tasks Zagoruyko & Komodakis ( 2016 ) ; Arora et al . ( 2019 ) ; Zhang et al . ( 2019 ) , they require a high cost of computational and storage resources Wang et al . ( 2020a ) ; Cheng et al . ( 2017 ) ; Deng et al . ( 2020 ) . Recent pruning techniques aim to diminish the model size by discarding irrelevant weights of well-trained models based on different criteria Gale et al . ( 2019 ) ; He et al . ( 2017 ) ; Han et al . ( 2015a ; b ) . Decisive weights are preserved and finetuned for obtaining final compressed model with acceptable performance loss . Following this line , several series of research works have been done to explore effective pruning criterion for better pruning performances . For example , regularization based pruning approaches Liu et al . ( 2017 ) ; Han et al . ( 2015a ; b ) leverage a penalty term during training for network pruning . Also , many researches take advantages of Hessian information to build more proper criteria for pruning LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Wang et al . ( 2019a ) ; Singh & Alistarh ( 2020 ) . However , regular pruning methods still requires full pretraining with high computational and storage costs . Pruning at initialization ( PI ) attempts determining the sparse network before training and maintains the final performances . For instance , Single-shot Network Pruning ( SNIP ) Lee et al . ( 2018 ) uses a novel criterion called connection sensitivity to measure the weights importance and decide which weights should be removed . Gradient Signal Preservation ( GraSP ) Wang et al . ( 2020a ) regards the gradient flow as an importance factor and correspondingly use it to design PI criterion . Regular pruning and PI techniques both achieve promising results on sparse network training and model compression . They only focus on designating criteria to find a specific sparse network but ignore exploring the internal relationships between the dense network and its eligible subnetwork candidates , which impedes a full understanding of the sparse network training . Lottery Ticket Hypothesis ( LTH ) Frankle & Carbin ( 2018 ) hits this problem with the first shot . It claims a trainable subnetwork exists in the randomly initialized dense network and can be found by pruning a pretrained network . In other words , as long as the dense network is initialized , the good subnetwork has been implicitly decided but needs to be revealed by the pruning process . The LTH has been validated by training the subnetwork from scratch with mask obtained from corresponding iterative magnitude pruning Han et al . ( 2015b ; a ) . The subnetwork is regarded as the winning ticket among the lottery pool given by dense networks . This valuable discovery naturally describes the relationship between a random initialized dense network and the trainable subnetwork hidden in it . However , LTH only focuses on finding one sparse structure at the expense of full pretraining , which is not universal to both practical usage and investigating the relationship between dense and its subnetworks for sparse network training . In this work , we go from a complementary direction of LTH and propose a new hypothesis called Dual Lottery Ticket Hypothesis ( DLTH ) . It studies a more challenging and general case and achieves promising performances ( see Fig . 1 ) for sparse network training . DLTH is described as any random selected subnetwork of a randomly initialized dense network can be transformed into a trainable condition . In other word , any ticket in a given lottery pool can be turned into a winning ticket . In this way , we construct a closed-loop research perspective from the dual side of LTH as shown in Fig . 2 . Compared with LTH , our proposed DLTH considers a general case - studying any subnetwork instead of a specific one – with a simple training strategy proposed to validate it . The comprehensive relationship between dense and sparse networks is inquired for sparse network training . Our contributions can be summarized as follows : • We investigate a novel view of studying sparse network training . Specifically , we present a Dual Lottery Ticket Hypothesis ( DLTH ) , a dual problem of Lottery Ticket Hypothesis ( LTH ) , articulated as : Any randomly selected subnetwork of a randomly initialized dense network can be transformed into an appropriate condition with admirable trainability . • We propose a simple sparse network training strategy , Random Sparse Network Transformation ( RST ) , to validate our proposed DLTH and achieve promising performance for sparse network optimization . Concretely , we achieve the information extrusion from the weights which will be removed to enhance sparse training by introducing a regularization term . Any subnetwork will be transformed into a trainable condition in this way . • Extensive experiments are conducted based on benchmark datasets and comparisons with competitive approaches . Our method obtains promising and consistent performance which solidly validates our DLTH and demonstrates the model effectiveness . • Our DLTH considers a more general and challenging case for sparse network training compared with LTH . In this way , we expect our work inspires a novel perspective to study sparse network training in a more flexible and adjustable way . 2 RELATED WORK . 2.1 SPARSE NETWORK TRAINING . Sparse Evolutionary Training ( SET ) Mocanu et al . ( 2018 ) proposes an evolutionary strategy to randomly add weights on sparse network and achieve better training . Dynamic Sparse Reparameterization ( DSR ) Mostafa & Wang ( 2019 ) presents a novel direction to dynamically modify the parameter budget between different layers . In this way , the sparse structure can be adaptively adjusted for more effective and efficient usage . Sparse Networks from Scratch ( SNFS ) Dettmers & Zettlemoyer ( 2019 ) designates a momentum based approach as criterion to grow weights and empirically proves it benefits the practical learning performance . Deep Rewiring ( DeepR ) Bellec et al . ( 2017 ) introduces the sparsity during the training process and augments the regular SGD optimization by involving a random walk in the parameter space . It can achieve effectively training on very sparse network relying on theoretical foundations . Rigging the Lottery ( RigL ) Evci et al . ( 2020 ) enhances the sparse network training by editing the network connectivity along with the optimizing the parameter by taking advantages of both weight magnitude and gradient information . 2.2 NETWORK PRUNING . Network pruning is a relevant research topic to our work . Related approaches aim to remove unnecessary weights based on different well-designed criteria and obtain sparse network for model compression . Diversified pruning methods can be categorized from different angles . To better present the background of our work , we customize the existing pruning methods into Training-Relevant Pruning and Training-Free Pruning groups as follow . Training-Relevant Pruning . Most pruning methods require a pre-trained network Louizos et al . ( 2017 ) ; Liu et al . ( 2017 ) ; Ye et al . ( 2018 ) ; Han et al . ( 2015b ; a ) . Algorithms utilize different ranking strategies to pick redundant weights with low importance scores , and remove them to achieve pruning along with acceptable performance drop . Pioneer magnitude-based method Han et al . ( 2015b ; a ) regards weights with low values as unnecessary ones , which is straightforward but may remove important low-value weights . Hessian-based approaches measure the weight importance by computing the their removal effects on the final loss LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) . Recently published technique also achieves Hessian-free pruning Wang et al . ( 2020b ) by adding regularization which is more computational friendly . On the other hand , pruning process can be also conducted during the network training instead of relying on completed pre-trained network . To name a few , a special dropout strategy is utilized in Srinivas & Babu ( 2016 ) to adjust the dropout during training and obtain a pruned network after training . A L0 norm regularization based method is proposed for network pruning Louizos et al . ( 2017 ) . Above algorithms requiring network training are seen as Training-Relevant Pruning . Training-Free Pruning . This group of research investigates how to prune randomly initialized network without any training . Pruning at initialization is one typical direction deriving sparse network by remove part of randomly initialized weights Lee et al . ( 2018 ; 2019 ) ; Wang et al . ( 2020a ) . For instance , Single-Shot Neural Network Pruning ( SNIP ) algorithm Lee et al . ( 2018 ) first proposes a learnable sparse network strategy according to the computed connection sensitivity . An orthogonal initialization method is proposed to investigate pruning problem in signal propagation view Lee et al . ( 2019 ) . Gradient Signal Preservation ( GraSP ) considers to preserve the gradient flow as an efficient criterion to achieve pruning at initialization . On the other hand , Lottery Ticket Hypothesis ( LTH ) claims that the winning ticket subnetworks exist in randomly initialized network and can be found by deploying conventional pruning algorithm . Then the selected sparse network can be efficiently trained from scratch and achieve promising performance . Based on LTH , Early-Bird ( EB ) ticket You et al . ( 2020 ) proposes an efficient way to find winning ticket . Above algorithms obtaining sparse network from random network are seen as Training-Free Pruning . 3 LOTTERY TICKET PERSPECTIVE OF SPARSE NETWORK . Lottery Ticket Hypothesis ( LTH ) Frankle & Carbin ( 2018 ) articulates the hypothesis : dense randomly initialized networks contain subnetworks , which can be trained in isolation and deliver performances comparable to the original network . These subnetworks are treated as winning tickets in the given lottery pool with good trainability . Subnetworks are discovered by iterative magnitude pruning algorithms , which requires training a full network and removes redundant weights by specific criterion . The mask of the pruned network illustrates the sparse structure of winning tickets . Problem Formulation . We start from introducing general network pruning problem . The neural network training process can be seen as a sequence of parameter updating status using stochastic gradient descent ( SGD ) Wang et al . ( 2021 ) ; Bottou ( 2010 ) : { w ( 0 ) , w ( 1 ) , w ( 2 ) , · · · , w ( k ) , · · · , w ( K ) } , ( 1 ) where w is the model parameter with superscript k as training iteration number . For a general case , the sparse network structure can be mathematically described by a binary mask m which has the same tensor shape as w. The process of obtaining m can be formulated as a function m = Fm ( w ; D ) , where D is the training data . Further , the weights of the sparse structure are modified based on different pruning strategies Hassibi & Stork ( 1993 ) ; Wang et al . ( 2020b ) , which is given by w∗ = Fw ( w ; D ) . The final pruned network can be integrated as w̃ = Fm ( w ( km ) ; D ) · Fw ( w ( kw ) ; D ) , ( 2 ) where wkm and wkw represent different parameter conditions for Fm and Fw . The conventional pruning requires that km = kw = K. The LTH needs km = K , kw = 0 , and Fw = I , where I is the identical mapping representing model directly inherits the randomly initialized weights . Against with traditional view that directly training sparse network can not fully exploit network capacity Wang et al . ( 2020b ; a ) , LTH validates there exists sparse network with better trainability ( winning ticket ) than other subnetworks ( other tickets ) . However , to uncover this property still needs first using pruning techniques for obtaining mask and the selected sparse structure must match with corresponding randomly initialized dense network - winning ticket matches with the given lottery pool . LTH provides a novel angle to understand and reveal the connections between a randomly initialized dense network and its subnetworks with admirable tranability . However , the proposed hypothesis is valid with restrict constraint and finding subnetworks still requires pruning on pre-trained model .

Summary: This paper studies the Lotter Ticket Hypothesis (LTH) and proposes a Dual Lottery Ticket Hypothesis (DLTH). DLTH describes that any ticket in a given lottery pool can be transformed into a winning ticket. The paper uses a regularization-based method for this transformation. The experiments have been conducted on CIFAR10/100 with ResNets, which indicates a consistent empirical result with DLTH.