Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective

1 INTRODUCTION . The lottery ticket hypothesis ( LTH ) for deep neural networks ( DNNs ) proposes that DNNs contain sparse subnetworks that can be trained in isolation and can reach performance that is equal to , or better than , that of the full DNN in the same number of training iterations ( Frankle & Carbin , 2019 ; Frankle et al. , 2020a ) . These subnetworks are called winning lottery tickets . The LTH has provided paradigm shifting insight into the success of dense DNNs , suggesting a key role for the emergence of winning tickets with increasing size , reminiscent of the maxim “ more is different ” ( Anderson , 1972 ) . In recent years , researchers have found an intriguing corollary : winning tickets found in the context of one task can be transferred to related tasks ( Desai et al. , 2019 ; Mehta , 2019 ; Morcos et al. , 2019 ; Soelen & Sheppard , 2019 ; Chen et al. , 2020a ; Gohil et al. , 2020 ; Chen et al. , 2021a ; Sabatelli et al. , 2021 ) , possibly even across different architectures ( Chen et al. , 2021c ) . In addition to having applications of practical interest , these results imply that winning tickets can be used to study how tasks and architectures are “ similar ” . However , to date , there exists no principled understanding of why winning tickets can be transferred between tasks , nor does there exist any way to know , without directly performing transfer experiments , which previously studied tasks a given winning ticket can be transferred to . Furthermore , there is a general lack of theoretical work on iterative magnitude pruning ( IMP ) ( Elesedy et al. , 2021 ) , the most commonly used method to find winning tickets . This is in striking analogy to the state of statistical physics in the early-to-mid–20th century . Empirical evidence suggested that disparate systems , governed by seemingly different underlying physics , exhibited the same , universal properties near their phase transitions . While heuristic methods provided insight ( Kadanoff , 1966 ) , a full theory from first principle on this universality was not realized until the development of the renormalization group ( RG ) ( Wilson , 1971a ; b ; 1975 ) . RG theory has not only provided a framework for explaining universal behavior near phase transitions , but also a scheme for grouping systems by that behavior . This classification , introduced via the notion of universality classes , has allowed for a detailed understanding of materials ( Tougaard , 1997 ; Winter & Mours , 1997 ; Durin & Zapperi , 2000 ; Bonilla et al. , 2010 ) , and provides immediate knowledge as to what other classified systems a given material behaves like . The analogy between universality in RG and LTH theories becomes strengthened under the following consideration . Recently , Rosenfeld et al . ( 2020 ) found that when the density ( i.e . the percentage of parameters remaining ) of a DNN being pruned via IMP is in a certain range , dL < d < dC , the network ’ s error scales according to the following power-law , e ∼ ( dC − d ) −γ = ∆d−γ . ( 1 ) Power-law scaling is well known to emerge in critical phenomena . For instance , when the temperature of a classical spin system ( e.g . two-dimensional Ising model ) is near and below the critical temperature , t < tC , many observables ( e.g . magnetization ) exhibit power-law scaling , m ∼ ( tC − t ) −β = ∆t−β . ( 2 ) These similarities hint at a more fundamental connection between the universality present in winning tickets and in physical systems near phase transitions ( see Fig . 1 and Table 1 ) . Indeed , we show that IMP is an RG scheme ( i.e . IMP satisfies the properties required to be considered an RG operator ) . By analyzing winning ticket transfer experiments ( Chen et al. , 2021a ) , we find that tasks that allow for successful transfer of winning tickets have similar behavior in their IMP flow ( i.e . the way in which the parameters of the DNN change after each iteration of IMP ) , supporting a key prediction of the theory . We additionally leverage the RG framework to interpret recent results on transferring winning tickets across differing architectures ( Chen et al. , 2021c ) . Our contributions are the following : • A theoretical basis for understanding IMP and the universality of winning tickets ; • Experimental support of the theory in large scale lottery ticket experiments , providing insight into previously known heuristic based results ; • A new set of tools for studying the LTH and associated phenomena . 2 THE RENORMALIZATION GROUP AND ITERATIVE MAGNITUDE PRUNING . The RG operator , R , can be viewed as a method for “ coarse-graining ” . That is , each application of R replaces local degrees of freedom with a composite of their values1 . An example of this for the two-dimensional Ising model is shown in the left-hand column Fig . 1 , where neighborhoods of four spins are replaced by their mode . The formal way to study R is to consider its action on a given Hamiltonian ( i.e . energy function ) . For classical spin systems ( e.g . two-dimensional Ising model ) , H has the general form H ( s , k ) = − ∑ i k1si − ∑ 〈i , j〉 k2sisj − ... , ( 3 ) where the si are the spins of the system ( e.g . si ∈ { −1 , +1 } ) , 〈· , ·〉 represents sites on the lattice that are nearest neighbors , and the ki are the strengths of the different coupling constants ( e.g . k2 is the strength of the nearest neighbor coupling ) . Due to the fact that coarse-graining amalgamates spins , the spin system resulting from an application ofR can be viewed as equivalent to the original , but with a new set of coupling constants . Therefore , the coarse-grained system has a different Hamiltonian , which is given by RH ( s , k ) = H ( s′ , T k ) = H ( s′ , k′ ) , ( 4 ) where s′ is the new set of spins and k′ is the new set of couplings determined by the operator T : RK → RK . Here K is the maximum number of couplings considered , usually introduced to keep the considered operators finite . R can be applied iteratively , defining a flow through the function space of Hamiltonians via Eq . 4 , with an associated flow in the space of coupling constants . The latter is commonly referred to as the RG flow . WhileR is often a complicated , non-linear function , T can be linearized near fixed points ( Goldenfeld , 2005 ) . The flow will grow in the direction of the eigenvectors vr with eigenvalues λr > 1 and will shrink along the direction of the eigenvectors vi with λi < 1 . These eigenvectors , which are linear combinations of coupling constants are called relevant and irrelevant , respectively . A schematic of these , and the general RG flow , is presented in Fig . 1 . By examining the effect of applying R to a given system , and its flow in the space of coupling constants , it is possible to find which components of the system are necessary for certain macroscopic behavior ( i.e those that are relevant ) , and which are not . Systems belonging to a particular universality class have the same relevant directions . Interestingly , many of the properties discussed above can be analogously found for IMP . To see this , consider a DNN with loss function L ( a , θ ) , where a is the unit activations and θ is the DNN parameters . Let I represent a single application of the IMP process . The DNN pruned via an application of I is given by the relation IL ( a , θ ) = L ( a′ , T θ ) = L ( a′ , θ′ ) , ( 5 ) where the new set of parameters , θ′ , are given by an operator T . This new set of parameters leads to a new , coarse-grained set of activations , a ’ . The similarities between Eqs . 4 and 5 are striking . We note here that , in the case of IMP , T is the composition of a masking operator , M , and a refining operator F . That is , T = F ◦ M : RN → RN , where N is the number of parameters of the 1Note that here we consider the block spin , or Kadanoff , RG . The momentum space , or Wilsonian , RG has a different , but related , interpretation . DNN . M is defined via the pruning procedure implemented ( e.g . magnitude pruning ) . While we are considering IMP here , because of its connection to the Lottery Ticket Hypothesis ( Frankle & Carbin , 2019 ; Frankle et al. , 2020a ) , there are many other pruning procedures [ e.g . Hessian based pruning ( LeCun et al. , 1989 ) ] , all of which have their own M. Similarly , F is defined via the refinement procedure used , making it dependent on the choice of optimizer and whether or not the DNN parameters are left alone or “ rewound ” to their value at a previous point during training ( Frankle et al. , 2020a ; Renda et al. , 2020 ) . For n iterations of I , the resulting network is given by InL ( a ( 0 ) , θ ( 0 ) ) = L ( a ( n−1 ) , T nθ ( 0 ) ) = L ( a ( n−1 ) , θ ( n−1 ) ) . ( 6 ) This defines a trajectory in parameter space , θ ( 0 ) → θ ( 1 ) → ... → θ ( n−1 ) , which we will refer to as the IMP flow . A schematic illustration of what the IMP flow might look like is given in Fig . 1 ( the axes of which will be explained in Sec . 3.1 ) . Just as in the case of RG theory , the directions of the IMP flow are determined by the eigenvectors of T , which grow or shrink exponentially by the magnitude of their associated eigenvalues . 2.1 CONNECTION BETWEEN THE RG AND STANDARD LTH FRAMEWORKS . Before we formally show that IMP is an RG scheme , we discuss how the standard theory that has emerged from studying the LTH is related to RG theory . In the familiar picture , the success of winning tickets is attributed to the surviving sparse DNN being able to find a “ good ” local minimum in the loss landscape ( Frankle & Carbin , 2019 ; Frankle et al. , 2020a ) . This may be possible because the training of DNNs via stochastic gradient descent ( SGD ) appears to be rapidly confined to a low-dimensional subspace ( Gur-Ari et al. , 2018 ) , implying that DNNs only “ feels ” changes to a small number of parameters during much of training . Parameters that are not in this low-dimensional subspace can , therefore , be removed with minimal impact . If a sparse DNN is initialized in this subspace ( as late rewinding aims to do ) , then it may be possible for training to find the same , or related , local minima as the full DNN ( Evci et al. , 2020 ; Maene et al. , 2021 ) . In this way , the parameters that do not lie in the low-dimensional subspace are irrelevant , and repeated applications of IMP are expected to remove them . On the other hand , the relevant parameters are those that span this subspace , and their removal changes the local minimum that the sparse DNN converges to . Therefore , certain observable functions of the DNN , such as error , are expected to only be sensitive to the relevant , but not irrelevant , parameters . If two models share the same low-dimensional subspace , then they should be able to transfer winning tickets . Note that this transferability is highly non-trivial . Indeed , it was only with the development of the LTH , and subsequent experimental work , that this was considered to be a possibility ( Frankle & Carbin , 2019 ; Mehta , 2019 ; Morcos et al. , 2019 ) . It is , in general , difficult to find these subspaces and accurately compare them across experiments . RG theory provides us with tools for finding relevant and irrelevant directions , which do not rely on explicitly constructing the underlying subspace . In particular , RG theory says that if two models have the same eigenvectors of T with eigenvalues > 1 , then they have the same relevant parameters . This means that , once the eigenvalues are computed , we will know whether winning tickets can be transferred between two models , without having to run any additional experiments . Note that the minimal sparsity a winning ticket can have , and still successfully be transferred , is not identified .

Summary: This paper seeks to explain empirical observations in the literature on the Lottery Ticket Hypothesis (LTH) based on ideas from renormalization group (RG) theory in physics. The authors focus on the particular case of iterative magnitude pruning (IMP) as the pruning scheme and view the flow on the space of model parameters during IMP as analogous to RG flow in the space of couplings of a Hamiltonian. The primary experimental evidence for the connection comes from two considerations. (1) Computing the percentage of all non-zero parameters that are in a particular residual block (of a ResNet model) at a particular time. For ResNet-50 models considered, this leads to four scalar, dynamical quantities. These four quantities are found to grow or shrink exponentially with time, suggesting they are eigenfunctions of the IMP map. For three different types of learning & dataset (ImageNet / simCLR / MoCo) and one architecture (ResNet-50), the authors find the same eigenvalue across the three settings, per block. From this, the authors conclude that this universality (shared exponent) across different settings is evidence for the analogy to RG. (2) Transferability of lottery tickets. The authors hypothesize that the relevance / irrelevance / marginality  of a particular block ought to be matched by the source and target architecture when transferring tickets. More specifically, there are three conclusions from the literature (#1-3 in Sec 3.2) that are explained (for example, why the smallest model has the weakest transferability).