The Elastic Lottery Ticket Hypothesis

1 Introduction . Lottery Ticket Hypothesis ( LTH ) [ 13 ] suggests the existence of sparse subnetworks in overparameterized neural networks at their random initialization , early training stage , or pre-trained initialization [ 35 , 44 , 5 , 3 , 2 ] . Such subnetworks , usually called winning tickets , contain much fewer non-zero parameters compared with the original dense networks , but can achieve similar or even better performance when trained in isolation . The discovery undermines the necessity of over-parameterized initialization for successful training and good generalization of neural networks [ 46 , 31 ] . That implies the new possibility to train a highly compact subnetwork instead of a prohibitively large one without compromising performance , potentially drastically reducing the training cost . However , the current success of LTH essentially depends on Iterative Magnitude-based Pruning ( IMP ) , which requires repeated cycles of training networks from scratch , pruning and resetting the remaining parameters . IMP makes it extremely expensive and sometimes unstable to find winning ∗ Work was done when the author interned at Microsoft . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tickets at scale , with large models and large datasets [ 15 ] . To alleviate this drawback , many efforts have been devoted to finding more efficient alternatives to IMP that can identify sparse trainable subnetworks at random initialization , with little-to-no training [ 27 , 40 , 38 , 16 ] . Unfortunately , they all see some performance gap when compared to the winning tickets found by IMP , with often different structural patterns [ 16 ] . Hence , IMP remains to be the de facto scheme for lottery ticket finding . Prior work [ 33 ] found that a winning ticket of one dense network can generalize across datasets and optimizers , beyond the original training setting where it was identified . Their work provided a new perspective of reducing IMP cost – to only find one generic , dataset-independent winning ticket for each backbone model , then transferring and re-training it on various datasets and downstream tasks . Compared to this relevant prior work which studies the transferablity of a winning ticket in the same network architecture , in this paper , we ask an even bolder question : Can we transfer the winning ticket found for one network to other different network architectures ? This question not only has strong practical relevance but also arises theoretical curiosity . On the practicality side , if its answer is affirmative , then we will perform only expensive IMP for one network and then automatically derive winning tickets for others . It would point to a tantalizing possibility of once-for-all lottery ticket finding , and the extraordinary cost of IMP on a “ source architecture ” is amortized by transferring to a range of “ target architectures ” . A promising application is to first find winning tickets by IMP on a small source architecture and then transfer it to a much bigger target architecture , leading to drastic savings compared to directly performing IMP on the latter . Another use case is to compress a larger winning ticket directly to smaller ones , in order to fit in the resource budgets on different platforms . On the methodology side , this new form of transferability would undoubtedly shed new lights on the possible mechanisms underlying the LTH phenomena by providing another perspective for understanding lottery tickets through their transferablity , and identify shared and transferable patterns that make sparse networks trainable [ 17 , 38 , 14 ] . Moreover , many deep networks have regular building blocks and repetitive structures , lending to various modelbased interpretations such as dynamical systems or unrolled estimation [ 19 , 42 , 20 , 1 ] . Our explored empirical methods seem to remind of those explanations too . 1.1 Our Contributions . We take the first step to explore how to transfer winning tickets across different architectures . The goal itself would be too daunting if no constraint is imposed on the architecture differences . Just like general transfer learning , it is natural to hypothesize that two architectures must share some similarity so that their winning tickets may transfer : we stick to this assumption in this preliminary study . We focus our initial agenda on network architectures from the same design family ( e.g. , a series of models via repeating or expanding certain building blocks ) but of different depths . For a winning ticket found on one network , we propose various strategies to “ stretch ” it into a winner ticket for a deeper network of the same family , or “ squeeze ” it for a shallower network . We then compare their performance with tickets directly found on those deeper or shallower networks . We conduct extensive experiments on CIFAR-10 with models from the ResNet and VGG families , and further extend to ImageNet . Our results seem to suggest an affirmative answer to our question in this specific setting . We formalize our observations by articulating the Elastic Lottery Ticket Hypothesis ( E-LTH ) : by mindfully replicating ( or dropping ) and re-ordering layers for one network , its corresponding winning ticket could be stretched ( or squeezed ) into a subnetwork for another deeper ( or shallower ) network from the same family , whose performance is nearly the same as the latter ’ s winning ticket directly found by IMP . Those stretched or squeezed winning tickets largely outperform the sparse subnetworks found by pruning-at-initialization approaches [ 27 ] , and show competitive efficiency to state-of-the-art dynamic sparse training [ 11 ] . We also provide intuitive explanations for the preliminary success . Lastly , we stress that our method has a pilot-study nature , and is not assumption-free . The assumption that different architectures come from one design “ family ” might look restrictive . However , we point out many state-of-the-art deep models are designed in “ families ” , such as ResNets [ 22 ] , MobileNets [ 23 ] , EfficientNets [ 37 ] , and Transformers [ 39 ] . Hence , we see practicality in the current results , and we are also ready to extend them to more general notions - which we discuss in Section 5 . 2 Related Work . 2.1 Lottery Ticket Hypothesis Basics . The pioneering work [ 13 ] pointed out the existence of winning tickets at random initialization , and showed that these winning tickets can be found by IMP . Denote f ( x ; θ ) as a deep network parameterized by θ and x as its input . A sub-network of f can be characterized by a binary mask m , which has exactly the same dimension as θ . When applying the mask m to the network , we obtain the sub-network f ( x ; θ m ) , where is the Hadamard product operator . For a network initialized with θ0 , the IMP algorithm with rewinding [ 15 , 14 ] works as follows : ( 1 ) initialize m as an all-one mask ; ( 2 ) train f ( x ; θ0 m ) for r steps to get θr ; ( 3 ) continue to fully train f ( x ; θr m ) to obtain a well-trained θ ; ( 3 ) remove a small portion p ∈ ( 0 , 1 ) of the remaining weights with the smallest magnitudes from θ m and update m ; ( 5 ) repeat ( 3 ) - ( 4 ) until a certain sparsity ratio is achieved . Note that when r = 0 , the above algorithm reduces to IMP without rewinding [ 13 ] . Rewinding is found to be essential for successfully and stably identifying winning tickets in large networks [ 15 , 14 ] . [ 44 ] identified Early-Bird Tickets that contain structured sparsity , which emerge at the early stage of the training process . [ 33 , 3 ] studied the transferability of winning tickets between datasets ; the former focuses on showing one winning ticket to generalize across datasets and optimizers ; and the latter investigates LTH in large pre-trained NLP models , and demonstrates the winning ticket transferability across downstream tasks . 2.2 Pruning in the Early Training , and Dynamic Sparse Training . A parallel line of works study the pruning of networks at either the initialization or the early stage of training [ 6 ] , so that the resulting subnetworks can achieve close performance to the dense model when fully trained . SNIP [ 27 ] proposed to prune weights that are the least salient for the loss in the one-shot manner . GraSP [ 40 ] further exploited the second-order information of the loss at initialization and prune the weights that affect gradient flows least . [ 38 ] unified these methods under a newly proposed invariant measure called synaptic saliency , and showed that pruning iteratively instead of in one-shot is essential for avoiding layer collapse . The authors then propose an iterative pruning method called SynFlow based on synaptic saliency that requires no access to data . However , it is observed in [ 16 ] that vanilla magnitude-based pruning [ 21 ] is as competitive as the above carefully designed pruning methods in most cases , yet all are inferior to IMP by clear margins . The pruning-at-initialization methods such as SNIP and GraSP were suggested to identify no more than a layer-wise pruning ratio configuration . That was because the SNIP/GraSP masks were found to be insensitive to mask shuffling within each layer , while LTH masks were impacted a lot by the same . Those suggest that existing early-pruning methods are not yet ready to replace IMP . Instead of searching for static sparse subnetworks , another line of influential works , called dynamic sparse training ( DST ) , focuses on training sparse subnetworks from scratch while dynamically changing the connectivity patterns . DST was first proposed in [ 32 ] . Following works improve DST by parameter redistribution [ 34 , 29 ] and gradient-based methods [ 8 , 11 ] . A recent work [ 30 ] suggested that successful DST needed to explore the training of possible connections sufficiently . 2.3 Network Growing . Broadly related to this paper also includes the research on network growing . Net2Net [ 4 ] provided growing operations that preserve the functional equivalence for widening and deepening the network . Network Morphism [ 41 ] further extended Net2Net to more flexible operations that change the network architecture but maintains its functional representation . Network Morphism is used in [ 10 ] to generate several neighbouring networks to select the best one based on some training and evaluation . However , this requires comparing multiple candidate networks simultaneously . A more recent work , FireFly [ 43 ] , proposed a joint optimization framework that grows the network during training . Despite our similar goal of transferring across architectures , the above methods do not immediately fit into our case of winning tickets . We face the extra key hurdle that we need to transfer ( stretch or squeeze ) not only general weights , but also the sparse mask structure while preserving approximately the same sparsity . For example , Net2Net and FireFly add ( near ) identity layers as the initialization for the extra layers , which are already highly sparse and incompatible with IMP without modification . Network Morphism decomposes one layer into two by alternating optimization , which will break the sparse structure of the source winning tickets . 3 Elastic Lottery Ticket Hypothesis . The overall framework of E-LTH is illustrated in Figure 2 . At the core of E-LTH are a number of Elastic Ticket Transformations ( ETTs ) proposed by us to convert the sparse architecture of a winning ticket to its deeper or shallower variants . We present rules of thumb for better performance , as well as discussions and intuitive explanation for why they do/do not work . For the simplicity of illustration , we will use the ResNet family as examples in this section . Terminology : Suppose we have identified a winning ticket fs ( θsr , ms ) on a source network using IMP , characterized by the rewinding weights θsr and the binary mask ms , where r is the rewinding steps and s stands for “ source ” . Our goal is to transform the ticket fs into a target network f t ( g ( θsr , m s ) , h ( θsr , m s ) ) with a different depth directly , avoiding running the expensive IMP on the target network again . Here , the superscript t stands for “ target ” . g ( · , · ) and h ( · , · ) are the transformation mapping for rewinding weights and the mask , respectively , taking the source weights and source mask as inputs .

Summary: This paper focused on exploring the transferability of a subnetwork obtained from the given dense network. Following the Lottery Ticket Hypothesis (LTH), a new hypothesis called Elastic Lottery Ticket Hypothesis (E-LTH) was proposed along with a corresponding validation method––Elastic Ticket Transformations (ETT). Extensive experiments on different benchmarks are provided to prove the effectiveness of ETT.