The Elastic Lottery Ticket Hypothesis

1 Introduction . Lottery Ticket Hypothesis ( LTH ) [ 13 ] suggests the existence of sparse subnetworks in overparameterized neural networks at their random initialization , early training stage , or pre-trained initialization [ 35 , 44 , 5 , 3 , 2 ] . Such subnetworks , usually called winning tickets , contain much fewer non-zero parameters compared with the original dense networks , but can achieve similar or even better performance when trained in isolation . The discovery undermines the necessity of over-parameterized initialization for successful training and good generalization of neural networks [ 46 , 31 ] . That implies the new possibility to train a highly compact subnetwork instead of a prohibitively large one without compromising performance , potentially drastically reducing the training cost . However , the current success of LTH essentially depends on Iterative Magnitude-based Pruning ( IMP ) , which requires repeated cycles of training networks from scratch , pruning and resetting the remaining parameters . IMP makes it extremely expensive and sometimes unstable to find winning ∗ Work was done when the author interned at Microsoft . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tickets at scale , with large models and large datasets [ 15 ] . To alleviate this drawback , many efforts have been devoted to finding more efficient alternatives to IMP that can identify sparse trainable subnetworks at random initialization , with little-to-no training [ 27 , 40 , 38 , 16 ] . Unfortunately , they all see some performance gap when compared to the winning tickets found by IMP , with often different structural patterns [ 16 ] . Hence , IMP remains to be the de facto scheme for lottery ticket finding . Prior work [ 33 ] found that a winning ticket of one dense network can generalize across datasets and optimizers , beyond the original training setting where it was identified . Their work provided a new perspective of reducing IMP cost – to only find one generic , dataset-independent winning ticket for each backbone model , then transferring and re-training it on various datasets and downstream tasks . Compared to this relevant prior work which studies the transferablity of a winning ticket in the same network architecture , in this paper , we ask an even bolder question : Can we transfer the winning ticket found for one network to other different network architectures ? This question not only has strong practical relevance but also arises theoretical curiosity . On the practicality side , if its answer is affirmative , then we will perform only expensive IMP for one network and then automatically derive winning tickets for others . It would point to a tantalizing possibility of once-for-all lottery ticket finding , and the extraordinary cost of IMP on a “ source architecture ” is amortized by transferring to a range of “ target architectures ” . A promising application is to first find winning tickets by IMP on a small source architecture and then transfer it to a much bigger target architecture , leading to drastic savings compared to directly performing IMP on the latter . Another use case is to compress a larger winning ticket directly to smaller ones , in order to fit in the resource budgets on different platforms . On the methodology side , this new form of transferability would undoubtedly shed new lights on the possible mechanisms underlying the LTH phenomena by providing another perspective for understanding lottery tickets through their transferablity , and identify shared and transferable patterns that make sparse networks trainable [ 17 , 38 , 14 ] . Moreover , many deep networks have regular building blocks and repetitive structures , lending to various modelbased interpretations such as dynamical systems or unrolled estimation [ 19 , 42 , 20 , 1 ] . Our explored empirical methods seem to remind of those explanations too . 1.1 Our Contributions . We take the first step to explore how to transfer winning tickets across different architectures . The goal itself would be too daunting if no constraint is imposed on the architecture differences . Just like general transfer learning , it is natural to hypothesize that two architectures must share some similarity so that their winning tickets may transfer : we stick to this assumption in this preliminary study . We focus our initial agenda on network architectures from the same design family ( e.g. , a series of models via repeating or expanding certain building blocks ) but of different depths . For a winning ticket found on one network , we propose various strategies to “ stretch ” it into a winner ticket for a deeper network of the same family , or “ squeeze ” it for a shallower network . We then compare their performance with tickets directly found on those deeper or shallower networks . We conduct extensive experiments on CIFAR-10 with models from the ResNet and VGG families , and further extend to ImageNet . Our results seem to suggest an affirmative answer to our question in this specific setting . We formalize our observations by articulating the Elastic Lottery Ticket Hypothesis ( E-LTH ) : by mindfully replicating ( or dropping ) and re-ordering layers for one network , its corresponding winning ticket could be stretched ( or squeezed ) into a subnetwork for another deeper ( or shallower ) network from the same family , whose performance is nearly the same as the latter ’ s winning ticket directly found by IMP . Those stretched or squeezed winning tickets largely outperform the sparse subnetworks found by pruning-at-initialization approaches [ 27 ] , and show competitive efficiency to state-of-the-art dynamic sparse training [ 11 ] . We also provide intuitive explanations for the preliminary success . Lastly , we stress that our method has a pilot-study nature , and is not assumption-free . The assumption that different architectures come from one design “ family ” might look restrictive . However , we point out many state-of-the-art deep models are designed in “ families ” , such as ResNets [ 22 ] , MobileNets [ 23 ] , EfficientNets [ 37 ] , and Transformers [ 39 ] . Hence , we see practicality in the current results , and we are also ready to extend them to more general notions - which we discuss in Section 5 . 2 Related Work . 2.1 Lottery Ticket Hypothesis Basics . The pioneering work [ 13 ] pointed out the existence of winning tickets at random initialization , and showed that these winning tickets can be found by IMP . Denote f ( x ; θ ) as a deep network parameterized by θ and x as its input . A sub-network of f can be characterized by a binary mask m , which has exactly the same dimension as θ . When applying the mask m to the network , we obtain the sub-network f ( x ; θ m ) , where is the Hadamard product operator . For a network initialized with θ0 , the IMP algorithm with rewinding [ 15 , 14 ] works as follows : ( 1 ) initialize m as an all-one mask ; ( 2 ) train f ( x ; θ0 m ) for r steps to get θr ; ( 3 ) continue to fully train f ( x ; θr m ) to obtain a well-trained θ ; ( 3 ) remove a small portion p ∈ ( 0 , 1 ) of the remaining weights with the smallest magnitudes from θ m and update m ; ( 5 ) repeat ( 3 ) - ( 4 ) until a certain sparsity ratio is achieved . Note that when r = 0 , the above algorithm reduces to IMP without rewinding [ 13 ] . Rewinding is found to be essential for successfully and stably identifying winning tickets in large networks [ 15 , 14 ] . [ 44 ] identified Early-Bird Tickets that contain structured sparsity , which emerge at the early stage of the training process . [ 33 , 3 ] studied the transferability of winning tickets between datasets ; the former focuses on showing one winning ticket to generalize across datasets and optimizers ; and the latter investigates LTH in large pre-trained NLP models , and demonstrates the winning ticket transferability across downstream tasks . 2.2 Pruning in the Early Training , and Dynamic Sparse Training . A parallel line of works study the pruning of networks at either the initialization or the early stage of training [ 6 ] , so that the resulting subnetworks can achieve close performance to the dense model when fully trained . SNIP [ 27 ] proposed to prune weights that are the least salient for the loss in the one-shot manner . GraSP [ 40 ] further exploited the second-order information of the loss at initialization and prune the weights that affect gradient flows least . [ 38 ] unified these methods under a newly proposed invariant measure called synaptic saliency , and showed that pruning iteratively instead of in one-shot is essential for avoiding layer collapse . The authors then propose an iterative pruning method called SynFlow based on synaptic saliency that requires no access to data . However , it is observed in [ 16 ] that vanilla magnitude-based pruning [ 21 ] is as competitive as the above carefully designed pruning methods in most cases , yet all are inferior to IMP by clear margins . The pruning-at-initialization methods such as SNIP and GraSP were suggested to identify no more than a layer-wise pruning ratio configuration . That was because the SNIP/GraSP masks were found to be insensitive to mask shuffling within each layer , while LTH masks were impacted a lot by the same . Those suggest that existing early-pruning methods are not yet ready to replace IMP . Instead of searching for static sparse subnetworks , another line of influential works , called dynamic sparse training ( DST ) , focuses on training sparse subnetworks from scratch while dynamically changing the connectivity patterns . DST was first proposed in [ 32 ] . Following works improve DST by parameter redistribution [ 34 , 29 ] and gradient-based methods [ 8 , 11 ] . A recent work [ 30 ] suggested that successful DST needed to explore the training of possible connections sufficiently . 2.3 Network Growing . Broadly related to this paper also includes the research on network growing . Net2Net [ 4 ] provided growing operations that preserve the functional equivalence for widening and deepening the network . Network Morphism [ 41 ] further extended Net2Net to more flexible operations that change the network architecture but maintains its functional representation . Network Morphism is used in [ 10 ] to generate several neighbouring networks to select the best one based on some training and evaluation . However , this requires comparing multiple candidate networks simultaneously . A more recent work , FireFly [ 43 ] , proposed a joint optimization framework that grows the network during training . Despite our similar goal of transferring across architectures , the above methods do not immediately fit into our case of winning tickets . We face the extra key hurdle that we need to transfer ( stretch or squeeze ) not only general weights , but also the sparse mask structure while preserving approximately the same sparsity . For example , Net2Net and FireFly add ( near ) identity layers as the initialization for the extra layers , which are already highly sparse and incompatible with IMP without modification . Network Morphism decomposes one layer into two by alternating optimization , which will break the sparse structure of the source winning tickets . 3 Elastic Lottery Ticket Hypothesis . The overall framework of E-LTH is illustrated in Figure 2 . At the core of E-LTH are a number of Elastic Ticket Transformations ( ETTs ) proposed by us to convert the sparse architecture of a winning ticket to its deeper or shallower variants . We present rules of thumb for better performance , as well as discussions and intuitive explanation for why they do/do not work . For the simplicity of illustration , we will use the ResNet family as examples in this section . Terminology : Suppose we have identified a winning ticket fs ( θsr , ms ) on a source network using IMP , characterized by the rewinding weights θsr and the binary mask ms , where r is the rewinding steps and s stands for “ source ” . Our goal is to transform the ticket fs into a target network f t ( g ( θsr , m s ) , h ( θsr , m s ) ) with a different depth directly , avoiding running the expensive IMP on the target network again . Here , the superscript t stands for “ target ” . g ( · , · ) and h ( · , · ) are the transformation mapping for rewinding weights and the mask , respectively , taking the source weights and source mask as inputs .

Summary: This paper describes a method for adapting lottery tickets (LT) found in one network architecture to a related architecture of different size. The motivation for this work is to study the connection between LTs of different architectures, and to reduce the cost of performing iterative magnitude pruning (IMP). If LTs can transfer between architectures, then it may be possible to find lottery ticket only once and adapting the size of the model as needed, without having to perform IMP for each model architecture. They restrict to adaptation to models of the same family (only differing in depth), and show that the adapted lottery tickets outperform pruning-at-initialization and RigL.  Networks are viewed at the level of "units", which are layers (+normalization), or residual blocks for ResNets. The adaptation method is to replicate nearby units of a smaller source model for each additional layer, or drop nearby units of a larger source model for each extra layer. They perform ablation studies on various choices for the replication/dropping of units.