Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective
Authors: William T Redman
Tianlong Chen
Akshunna S. Dogra
Zhangyang Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: lottery ticket hypothesis
winning tickets
renormalization group

Abstract: Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. While this has become of broad practical and theoretical interest, to date, there exists no detailed understanding of why winning ticket universality exists, or any way of knowing a priori whether a given ticket can be transferred to a given task. To address these outstanding open questions, we make use of renormalization group theory, one of the most successful tools in theoretical physics. We find that iterative magnitude pruning, the method used for discovering winning tickets, is a renormalization group scheme. This opens the door to a wealth of existing numerical and theoretical tools, some of which we leverage here to examine winning ticket universality in large scale lottery ticket experiments, as well as sheds new light on the success iterative magnitude pruning has found in the field of sparse machine learning.

Under review as a conference paper at ICLR 2022
UNIVERSALITY OF DEEP NEURAL NETWORK LOTTERY TICKETS: A RENORMALIZATION GROUP PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. While this has be- come of broad practical and theoretical interest, to date, there exists no detailed understanding of why winning ticket universality exists, or any way of knowing whether tickets can be transferred between two given tasks, without first perform- ing transfer experiments. To address these outstanding open questions, we make use of renormalization group theory, of the most successful tools in theoretical physics. We find that iterative magnitude pruning, the method used for discover- ing winning tickets, is a renormalization group scheme. This opens the door a wealth of existing numerical and theoretical tools, some of which we leverage here to examine winning ticket universality in large scale lottery ticket experi- ments, as well as sheds new light on the success iterative magnitude pruning has found generally in machine learning.
INTRODUCTION
The lottery ticket hypothesis (LTH) for deep neural networks (DNNs) proposes that DNNs contain sparse subnetworks that can be trained in isolation and can reach performance that is equal to, or better than, that of the full DNN in the same number of training iterations (Frankle & Carbin 2019 Frankle et al. 2020a). These subnetworks are called winning lottery tickets. The L.H has provided paradigm shifting insight into the success of dense DNNs, suggesting a key role for the emergence of winning tickets with increasing size, reminiscent of the maxim "more is different" (Anderson) 1972). In recent years, researchers have found intriguing corollary: winning tickets found in the
context of one task can be transferred to related tasks (Desai et al. 2019 Mehta 2019 Morcos et al.,
2019 Soelen & Sheppard 2019 Chen et al.) [2020a] Gohil al. 2020 Chen et al. 2021a (abatelli
et al. 2021), possibly even across different architectures (Chen et al. 2021c). In addition to having applications of practical interest, these results imply that winning tickets can be used to study how tasks and architectures are "similar". However, to date, there exists no principled understanding of why winning tickets can be transferred between tasks, nor does there exist any way to know, without directly performing transfer experiments, which previously studied tasks a given winning ticket can be transferred to. Furthermore, there is general lack of theoretical work on iterative magnitude pruning (IMP) (Elesedy et al. 2021), the most commonly used method to find winning tickets.
This is in striking analogy to the state of statistical physics in the early-to-me mid-mdddd""d century. Empir- ical evidence suggested that disparate systems, governed by seemingly different underlying physics, exhibited the same, universal properties near their phase transitions. While heuristic methods pro- vided insight (Kadanoff 1966). full theory from first principle this universality was not realized until the development of the renormalization group (RG) (Wilson b 1975).
RG theory has not only provided a framework for explaining universal behavior near phase transi- tions, but also a scheme for grouping systems by that behavior. This classification, introduced via the notion of universality classes, has allowed for detailed understanding of materials (Tougaard. 1997 Winter & Mours 1997] Durin & Zapperi| 20000 Bonilla et al.) (2010), and provides immediate knowledge to what other classified systems given material behaves like.

---

Under review as a conference paper at ICLR 2022
RG
IMP
0 6+1
Sparsified deep neural network
Coarse grained spin system
0
Pawar-law scaling
Power-lawe scaling
Maqnestaroo
Temperaturre
Dersiyy (16
RG flow
IMP flow
Relevant dimenaain
Relevant monsioon
Inelevant clmees
M.
Figure Summary of the similarities between the RG and IMP. Both the RG and IMP are applied iteratively to coarse-grain systems, revealing "relevant" features. Certain observables are known to have regimes where they follow power-law scaling. In the case of the RG, this scaling and its universality associated with properties of the flow the RG induces in the space of coupling constants. The nature of the flow IMP induces has not been previously studied, but none-the-less exists. See Table for the analogous quantities in each theory.
The analogy between universality in RG and LTH theories becomes strengthened under the follow- ing consideration. Recently, Rosenfeld et al. (220)) found that when the density (i.e. the percentage of parameters remaining) of DNN being pruned via IMP is in certain range, dp < de, the network's 's error scales according to the following power-law,
e ~ (dc - d Ad
Power-law scaling is well known to emerge in critical phenomena. For instance, when the temper- ature of classical spin system (e.g. two-dimensional Ising model) is near and below the critical temperature, t < tc, many observables (e.g. magnetization) exhibit power-law scaling,
m ~ (tc - 1) A
These similarities hint at more fundamental connection between the universality present in win- ning tickets and in physical systems near phase transitions (see Fig. [ and Table 1 Indeed, show that IMP is an RG scheme (i.e. IMP satisfies the properties required to be considered an RG operator). By analyzing winning ticket transfer experiments (Chen al., 2021a). we find that tasks that allow for successful transfer of winning tickets have similar behavior in their IMP flow (i.c. the way in which the parameters of the DNN change after each iteration of IMP), supporting a key prediction of the theory. We additionally leverage the RG framework to interpret recent results on transferring winning tickets across differing architectures (Chen al. 2021c). Our contributions are the following:
A theoretical basis for understanding IMP and the universality of winning tickets;
Experimental support of the theory in large scale lottery ticket experiments, providing in- sight into previously known heuristic based results;
A new set of tools for studying the L.TH and associated phenomena.
2

---

Under review as a conference paper at ICLR 2022
2 THE RENORMALIZATION GROUP AND ITERATIVE MAGNITUDE PRUNING
The RG operator, R, can be viewed as a method for "coarse- grrssiing"nnn"" That is, each application of R. replaces local degrees of freedom with composite of their values An example of this for the two-dimensional Ising model is shown in the left-hand column Fig. where neighborhoods of four spins are replaced by their mode.
The formal way to study R. is to consider its action on a given Hamiltonian (i.e. energy function). For classical spin systems (e.g. two-dimensional Ising model), H. has the general form
Hiss, k) - -'is: k1s;
k2518
(1.5)
where the s; are the spins of the system (e.g. s E [-1.+11), (.-)) represents sites on the lattice that are nearest neighbors, and the k; are the strengths of the different coupling constants (e.g. K2 is the strength of the nearest neighbor coupling).
Due to the fact that coarse- graining amalgamate spins, the spin system resulting from an application of R can be viewed as equivalent the original, but with new of coupling constants. Therefore, the coarse-grained system has different Hamiltonian, which is given by
RH(s, k) His', Tk) H(s' k'),
where s' is the new set of spins and k' is the new set of couplings determined by the operator T R - R Here K is the maximum number of couplings considered, usually introduced to keep the considered operators finite.
R. can be applied iteratively, defining a flow through the function space of Hamiltonians via Eq. with an associated flow in the space of coupling constants. The latter commonly referred to the RG flow. While R. is often a complicated, non-linear function, T can be linearized near fixed points (Goldenfeld] 2005). The flow will grow in the direction of the eigenvectors V, with eigenvalues Xy > 1 and will shrink along the direction of the eigenvectors v; with X; < 1. These eigenvectors, which are linear combinations of coupling constants are called relevant and irrelevant, respectively. A schematic of these, and the general RG flow, is presented in Fig. [ By examining the effect of applying R given system, and its flow in the space of coupling constants, it possible to find which components of the system are necessary for certain macroscopic behavior (i.e those that are relevant), and which are not. Systems belonging to a particular universality class have the same relevant directions.
Interestingly, many of the properties discussed above can be analogously found for IMP. To see this, consider DNN with loss function L(a. 0), where is the unit activations and 0 is the DNN parameters. Let I represent a single application of the IMP process. The DNN pruned via an application of I is given by the relation
(5) where the new set of parameters, 0' are given by an operator T. This new set of parameters leads to new, coarse-grained set of activations, a'. The similarities between Eqs. [add 5 are striking.
IL(a, 0) C(a', T0) C(a', 0'),
We note here that, the case of IMP, T is the composition of masking operator, M, and a refining operator F. That is, T F M RN + R where N is the number of parameters of the
Note that here we consider the block spin, or Kadanoff, RG. The momentum space, or Wilsonian, RG has different, but related, interpretation.
Table 1: Analogous quantities in RG and IMP theory.
RG
IMP
Spins (s;)
Unit activations (a:)
Coupling constants (k:))
Parameters (0,)
Hamiltonian (H(ss, k)
Loss function (C(a, 0))
3

---

DNN. M is defined via the pruning procedure implemented (e.g. magnitude pruning). While are considering IMP here, because of its connection to the Lottery Ticket Hypothesis (Frankle & Carbin 2019 Frankle et al., 2020a), there are many other pruning procedures [e.g. Hessian based pruning LeCCnn et al.) 1989), J. all of which have their own M. Similarly, F is defined via the the choice of optimizer and whether not Under review as a conference paper at ICLR 2022
refinement procedure used, making it dependent
the DNN parameters are left alone or "rewound" to their value at previous point during training
(Frankle al. 2020a Renda et al. 2020).
For n. iterations of I, the resulting network is given by
T L" (0)) = C(a n (T))())= = (n)) g(n-1)).
This defines trajectory in parameter space, ( - O(1) - O(n which we will refer to
as the IMP flow. A schematic illustration of what the IMP flow might look like is given in Fig. [ (the axes of which will be explained in Sec. 3.1). Just as in the case of RG theory, the directions of
the IMP flow are determined by the eigenvectors of 'T, which grow or shrink exponentially by the magnitude of their associated eigenvalues.
2.1 CONNECTION BETWERN THE RG AND STANDARD LTH FRAMEWORKS
Before we formally show that IMP is RG scheme, we discuss how the standard theory that has emerged from studying the LTH is related to RG theory.
In the familiar picture, the success of winning tickets attributed the surviving sparse DNN being able to find a "good" local minimum in the loss landscape (Frankle & Carbin 2019 Frankle et al. 2020a). This may be possible because the training of DNNs via stochastic gradient descent (SGD) appears to be rapidly confined to a fow-dimensional subspace (Gur-Ari et al. 20188. implying that DNNs only "feels" changes to a small number of parameters during much of training. Parameters that are not in this low-dimensional subspace can, therefore, be removed with minimal impact. If a sparse DNN is initialized in this subspace (as late rewinding aims to do), then may be possible for training to find the same, or related, local minima as the full DNN (Evci et al., 2020 Maene et al. 202T].
In this way. the parameters that do not lie in the low-dimensional subspace are irrelevant, and re- peated applications of IMP are expected to remove them. On the other hand, the relevant parameters those that span this subspace, and their removal changes the local minimum that the sparse DNN converges to. Therefore, certain observable functions of the DNN, such as error, are expected to only be sensitive to the relevant, but not irrelevant, parameters.
two models share the same oow-dimensional subspace, then they should be able to transfer winning tickets. Note that this transferability is highly non-trivial. Indeed, was only with the development of the I.TH, and subsequent experimental work, that this was considered to be a possibility (Frankle) & Carbin 2019 Mehtaa 2019. Morcos al. (219). and accurately compare them across experiments. RG theory provides us with tools for finding rele- vant and irrelevant directions, which do not rely on explicitly constructing the underlying subspace. In particular, RG theory says that if two models have the same eigenvectors of T with eigenval- ues > 1, then they have the same relevant parameters. This means that, once the eigenvalues are computed, we will know whether winning tickets can be transferred between two models, without having to run any additional experiments. Note that the minimal sparsity a winning ticket can have, and still successfully be transferred, is not identified.
in general, difficult find these subspaces
2.2 IMP AS AN RG SCHEME
To make the connection between IMP and RG more precise, we show that IMP fits the definition of an RG scheme. To do this, we consider the projection operator, P. that is associated with the RG operator. In the case of classical spin systems, the projection operator maps the spins, s,, to coarse-gras -aarse-griinddddee spin system, s', such that it satisfies
Tr(,,)((ss: P(ss, 81) exp [H(s,, ki] = exp [H(s)),
4

---

Under review as a conference paper at ICLR 2022
where (r.).) is the trace operator over the values that the 8; take (e.g. +1) (Goldenfeld 2005). For the two-dimensional Ising model, is standard to take
P(s;, st) = II 8
8 sign
Σ JET
8
(8)
where 8 is the Kronecker delta function and sign(.) :) is + if the argument is positive and - if negative (Goldenfeld] 2005). Eq. [8 formally defines the mapping from s; + 5 as
8/ sign
8j jEL
The projection operator is not unique, but must satisfy the following three properties:
1. P(s 2
2. P(s 8t) respects the symmetry of the system;
3. Σ P(s 84) = 1.
{ef}
To find projection operator associated with I, start by finding mapping between the activations of all the units, a, before and after an application of IMP. This is because a, and not 0, is the analogous quantity to s (Table and Fig. Dp. Without loss of generality, can consider the activation of unit j in layer as being defined by
af h
Σ 9* (a.
(10)
where ht is the activation function (e.g. ReLu, sigmoid) and the Ok are functions that determine how the different parameters and activations of other units affect aj
(8) For instance, in feedforward
DNN, the impact of the bias of unit j in layer i is given by go = =()), and the weighted input from
the previous layer is given by 01 EA=1 v(i-1 g(i) az (i-1) Here (((-1) is the number of units in layer
i
discussed above, IMP changes the parameters by the operator T, which is defined by a com- position of M and F. Therefore, the activation of unit j in layer i, after applying I, is given by
df ( h Σ ge F MO)
(11)
and thus, the projection operator associated with I is
N d Σ Fa MD] Moo
((a o
II 2=1
(12)
We can easily verify that the projection operator defined by Eq. 12 satisfies all three properties needed for an RG projection operator. First, property #1 is satistied. as Eq. [2] is a product of Kronecker delta functions. Property #2 satisfied, as IMP only removes parameters, so the form of (6) given in Eq. 11 and, thus, the loss function, will remain intact until layer collapse (i.e. when all the weights from one layer to another are removed). Finally, for property #3 to be satisfied, we can fix the test and training sample ordering for each epoch (as is done when the random seed is fixed).
aj
In such case, both the mask and refining operators in Eq. 12 are deterministic and will be unique.
of
Having shown that these three properties are satisfied by the projection operator associated with IMP, have found that I meets the criteria for being RG operator, making it an RG scheme.

---

Under review as a conference paper at ICLR 2022
IMP flow projected onto blocks and
IMP flow projected onto blocks and
Proportion of total remaining
Proportion of total remaining perameters block
0.65
CIFAR-12 CIFAR-100 Image/Mot
parameters in block
0.150
0.45
80%
d= 1.44%
0.30
0.050
0.02
0.08
0.02
0.06
Proportion of total remaining parameters block
Proportion of total remsining parameters in block
Figure 2: IMP flow. Two-dimensional projections of the IMP flow for ResNet-50, pre-trained ImageNet via self-supervised simCLR training (Chen et al.) 2020b), applied to three different tasks: ImageNet (triangles), CIFAR-10 (circles), and CIFAR-I00 (stars). Blue corresponds single application of IMP (density equal to 80%) and red corresponds to 77 applications of IMP (density 1.44%).
To the best of our knowledge, this has not been previously identified and provides new insight into why IMP has found success discovering winning tickets (Frankle 2020a), as well as has found general success in the study of DNNs (Frankle et al. 2020b).
& Carbin 2019 Frankle al.,
3 RESULTS
3.1
UNIVERSALITY IN THE IMP FLOW
We start by examining the IMP flow of computer vision models. The goal of this analysis to examine DNNs that known to allow for the successful transfer of winning tickets, and determine whether they have the same relevant and irrelevant parameters. If they do, this would confirm major prediction of the theory developed in Sec. 2.
For spin systems in statistical physics, the RG flow is studied in the space of coupling constants (Fig. []. Each coupling constant usually assumed be the same for all spins, greatly reducing the dimensionality of the underlying space. While most DNNs do not set all parameters of given type the same value, we can none-the-less estimate the "influence" the parameters of a given layer or residual block have on the full DNN, by considering the functions
N(
N
M,(n) Σ
my (n).
(13)
1=1
where (n) describes the percentage of all non-zero parameters that are layer or residual block i after IMP iteration n. The numerator sums over all (() the elements of the pruning mask restricted to layer or residual block i, (*) Each my
{0, The denominator sums the entire pruning mask, making it equivalent to the number of non-pruned parameters the DNN. Given the analogous role the M; play to the coupling constants of spin systems, we expect that they may be eigenfunctions of I, and, therefore, may encode the relevant and irrelevant components of a given DNN.
Because ResNet-50 has four residual blocks, IMP induces a four-dimensional flow the space of M,. To visualize this trajectory, examined awo-dimensional slices. Examples of these are plotted in Fig. for ResNet-50, pre-trained on ImageNet via simCLR self-supervised training (Chen et al. 2020b The flow for three different tasks are shown: ImageNet, CIFAR-10, and CIFAR-100. This data comes from experiments performed by Chen et al. (2021a).
If the M; are eigenfunctions of the IMP operator, then they will scale exponentially with respect to the number iterations I has been applied. That is, M,(n + 1) = IM,(n) = (nM = X M (0), where A; is the eigenvalue governing its exponential growth/decay. We can find X; by the simple inversion, d; Me(n (n) As long M,(n) * 0. this is well defined

---

Under review as a conference paper at ICLR 2022
Table 2: Computed eigenvalues of the eigenfunctions M, for ResNet-50, pre-trained on ImageNet using SimCLR (Chen et al. 2020b). Red (green) corresponds to those eigenvalues that are relevant (irrelevant). Any eigenvalues within standard deviation of À 1 are considered marginal and colored cyan.
Task
A1
X2
A
A
ImageNet
1.15 = 0.01
1.09 + 0.02
1.02 + 0.01
0.95
CIFAR-10
1.11 0.01
1.06 + 0.01
1.01 + 0.01
0.98 0.002
CIFAR-100
1.08 = 0.03
1.04 - 0.02
1.00 0.01
0.99
and we can compute A by doing this for each pair of data points. Performing this on our dataset yields estimates of eigenvalues with standard deviations < 5% (Table 2), suggesting that it is indeed reasonable to consider the M, as being eigenfunctions of I. Additionally, we find that ImageNet, CIFAR-10, and CIFAR- 100 have similar distributions eigenvalues (Table 2). In particular, all three identify the first two residual blocks being relevant and identify the last block as being marginal or irrelevant. From the theory developed in Sec. 2, these results imply that all three models are in similar universality classes and that winning tickets can be transferred between them. Chen et al. (2021a) indeed confirmed this, finding that winning tickets found on ImageNet could be successfully transferred both CIFAR-10 10 and CIFAR-100
This universality in IMP eigenvalues may be due to the fact that the first few residual blocks learn low level statistics of the data (Neyshabur et al. (020). which scale in specifie, non-trivial ways for natural images (Ruderman & Bialek] 1994] Saremi & Sejnowski| 2013). The sensitivity to the first two residual blocks, which are the furthest from the output layer, is in contrast to standard spin sys- tems. In the case of the usual two- wwo-dimensional Ising model, the RG removes long- range interactions (i.e. couplings between next-nearest neighbors, next next-nearest neighbors, etc.), which are typi- cally assumed, for physical reasons, to be weak. Systems with relevant long-range interactions have been found to have unique properties, such as the existence of a phase transition in one-dimension (Dyson 1969), which is known not to happen for the standard Ising model. The importance of these long-range interactions may therefore be another interpretation for the benefit of pre-training and normalized initialization (e.g. He initialization) for convolutional DNNs, as they are presumably part of what sets the ResNet-50 to be in this "long-range interaction" regime.
However, emphasize that the universal IMP cigenvalues (Table 2) are not merely trivial re- flection of the pre-training and normalized initialization. In particular, the specific task matters, each model starts from slightly different point in the four-dimensional space of M, [Ima- geNet: (0.0079, 0.0557, 0.3055 0.6309); CIFAR-10: (0.0075, 0.0556, 0.3176, 0.6193); CIFAR- 100: (0.0070, 0.0519, 0.3004, 0.6407)], implying that each task arrives at distinct distributions of param- eters, despite starting from the same initial set of parameters. While pre-training and normalized initialization likely constrain the degree of these differences, it conceivable that there are tasks, which would sufficiently re-organize the parameter distributions, such that earlier blocks could be- come irrelevant and/or later blocks could become relevant. In such cases, RG theory predicts that these tasks would not admit the transfer of winning tickets. Future work should attempt to identify such tasks.
Similar results as those reported in Table 2 were found for ResNet-50, pre-trained on ImageNet us- ing supervised learning (Huh et al. 2016) (Apppendix A), which was also previously shown to have successful winning ticket transfer between tasks (Chen et al. 202 la). While the theory developed in Sec. 2 made no assumption on architecture, we also computed IMP eigenvalues on large seale lottery ticket experiments using VGG-16 16 and BERT models, finding again that the M, could reason- ably be considered eigenfunctions of I (Appendix B). This further supports our claim that the RG framework is general and can be broadly used to study winning ticket transferability.
7

---

Under review as a conference paper at ICLR 2022
Table 3: Computed eigenvalues of the eigenfunctions M, for the first stage of various ResNet archi- tectures. Same coloring as in Table 2.
ResNet architecture
Stage 1
ResNet-11
1.07
ResNet-220
1.07, 1.07
ResNet-32
0,99, 1.06, 1.06, 1.08
ResNet-44
1.00, 0.98, 1.06, 1.06, 1.05, 1.07
ResNet-56
1.02, 0.98, 1.07, 1.06, 1.05, 1.06, 1.04, 1.08
3.2 ELASTIC LOTTERY TICKET HYPOTIESSI
Recent work on the "Elastic Lottery Ticket Hypothesis" (E-LTH) has extended the notion of tran- ferability by finding it possible to transform winning tickets found one DNN, to another with different architecture (Chen et al.] (021c). In particular, it was found that, for architectures in the same family (e.g. ResNets) trained on the same task (e.g. CIFAR-10), it was possible to either squeeze (by removing residual blocks) or stretch (by replicating residual blocks) winning tickets so that they could be transferred.
Three results from this study of the E-LTH are especially noteworthy:
1. The smallest ResNet considered (ResNet-14) had the weakest transferability properties;
2. The more unique residual blocks replicated, the better the performance;
3. Removing the later residual blocks, in the case of shrinking, or replicating the earlier resid- ual blocks, in the case of stretching, led to the best results.
While preliminary hypotheses on the origin of these results were developed by Chen al. (2021c). involving dynamical systems and unrolled estimation, detailed theoretical understanding was left to future work. Given that the RG framework developed above led to insight into the nature of the transfer of winning tickets in the case of different tasks, we wondered whether it could similarly help to explain the nature of transfer in the case of different architectures.
As discussed in Sec. 2, two tasks that have the same IMP eigenvalues can transfer winning tickets. Therefore, we expect that transforming a given winning ticket from one architecture (source) to another (target) will be successful if its IMP flow matches that of the target. In particular, the success (or lack-thereof) should rely on whether the number of relevant, irrelevant, and marginal residual blocks in the target architecture is matched by the transformed source ticket.
To test this, we computed the eigenvalues associated with the eigenfunctions Mj (Eq. 13 for each "normal" residual block of ResNet- 14, ResNet-20, ResNet-32, ResNet-44, and ResNet-56, trained on CIFAR-10 [data from Chen et al. (2021c)). As discussed in the original paper, the down- sampling residual block in each stage not transformed between winning tickets, they were not considered. The cigenvalues for the residual blocks in the first stage are presented in Table 3.
We find that, while the smaller ResNets (ResNet-14 and ResNet-20) have only relevant residual blocks, the larger ResNets have at least one non-relevant block. This offers a straightforward ex- planation as why ResNet-14 had particularly poor transferability: using it as the source ticket necessarily leads to the transformed ticket having only relevant residual blocks, which does not match the target tickets' structure. Having only relevant residual blocks may make the first stage too sensitive as the depth of the ResNet is increased, leading poorer performance. This is likely related to the idea that smaller ResNet models are not sufficiently expressive, but has the benefit of being theoretically backed and quantitative. Relatedly, ual blocks (i.c. replicating only block or blocks and 2) can lead to an over representation of either relevant non-relevant blocks. This mismatch is predicted by the theory to lead to worse transferability.
find that replicating few unique resid-
Finally, we find that most of the non-relevant residual blocks are in the early part of the first stage. Therefore, removing the first several blocks when shrinking a winning ticket (e.g. dropping the first four blocks to go from ResNet-56 to ResNet-32), replicating the last several blocks when
8

---

Under review as a conference paper at ICLR 2022
stretching a winning ticket (e.g. adding the last two blocks to go from ResNet-32 to ResNet-44), will lead to too many relevant residual blocks. Again, this will make the transformed source ticket have a different structure than the target ticket and thus, will lead to worse performance.
Note that for the later two stages, the nature of the distribution of eigenvalues is different (Tables and 8 of Appendix C). In particular, stage 2 has only relevant and marginal blocks, and stage has only marginal and non-relevant blocks. However, there is consistent evidence for the fact that the number of each type of block is related to the three results from the original E-LTH paper (Chen et al. 2021c) we highlighted.
4 DISCUSSION
Inspired by similarities between the current state of sparse machine learning and the state of statisti- cal physics in the early-to-mid-20m century, we proved that iterative magnitude pruning (IMP), the principle method used to discover winning tickets, is a renormalization group (RG) scheme. Given that the development of the RG led a first principled understanding of universal behavior near phase transitions, as well as way in which to characterize materials by such behavior, we reasoned that viewing IMP from an RG perspective may lead to new insight on the universality of winning tickets (Desai et al. 20199; Mehtal 2019 Morcos et al. [019 Soelen & Sheppard, (0119) [Chnn et al., 2020a Gohil et al., 2020 Chen et [0.||20222 Sabatelli et al. 2021) and the general success IMP has found in the study of DNNs (Frankle et al. 2020b).
By viewing IMP as inducing flow in DNN parameter space (Fig- 2 we showed that ResNet- 50, pre-trained ImageNet, has trajectories with similar properties for different tasks (Table 2). As winning tickets are known to be transferable between these tasks Chen et al. (2021a). these results support key prediction of the theory developed in Sec. 2. Applying the RG Tramework to recent work which extended the notion of winning ticket universality by transforming tickets between different architectures ("Elastic Lottery Ticket Hypothesis"), we again found that the IMP eigenvalues allowed for a new, quantitative perspective on experimental results (Table 3). While RG theory may be unfamiliar, and simpler explanations the phenomena may conceivably exist, to our knowledge, no current theory has provided as general and successful framework for understanding winning ticket universality, what we have present here.
A wealth of theory has been developed around the RG, including a number of numerical and the- oretical tools that go beyond what we have used, Given the connection between IMP and the RG made explicit in this work, we believe that there is considerable potential for collaboration between the two fields. Possible directions of future study include:
Bringing the RG framework to other sparsificiation methods, which may potentially simi- larly be viewed as RG schemes;
Bringing the RG framework to study winning tickets beyond computer vision models, such as in natural language processing (Chen et al., 2020a; Prasanna et al. 2020 Yu et al. 2020). reinforcement learning YYu et al. 2020), and lifelong learning (Chen et al. 202
Computing and classifying systems by their critical exponents (such as 7 in Eq. [p. We attempted to do this, but we were limited by having only a few independent seeds and scaling function with multiple free parameters (Rosenfeld et al.) (2020). This led to large differences between computed critical exponents of systems that displayed qualitatively similar scaling behavior (see Appendix D). Using more advanced methods, such as finite scaling (Goldenfeld. 2005), and increasing the number of independent seeds may alleviate this problem;
Exploring the connection between the RG framework developed here and work done to build an effective theory of DNN behavior (Roberts et al. 2021).
Finally, while RG theory provides a way to know whether two given tasks, or architectures, can have winning tickets transferred between them, knowing what the minimal density transferable winning ticket can have, remains an open question. In order to address this, finite size effects (Goldenfeld) 2005), differences in symmetries, and non-linear corrections to the IMP flow (Raju et al. 2019) may need to be taken into account. This interesting, and important, question will be the focus of future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
P. W. Anderson. More is different. Science, 177(4047):393-336, 1972.
Claudia Marcela Bonilla, Julia Herrero-Albillls,, Fernando Bartolomé, Luis Miguel García, María Parra-Borderian and Victorino Franco. Universal behavior for magnetic entropy change in mag- netocaloric materials: An analysis on the nature of phase transitions. Phys. Rev. B, 81:224424, Jun 2010.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 15834-15846. Curran Associates, Inc., 2020a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16306-16316, June 2021a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets lifelong learning. In International Conference on Learning Representations, 2021b.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé IIII and Aarti Singh (eds.), Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020b.
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, and Zhangyang Wang. The elastic lottery ticket hypothesis. arXiv preprint, 1cXiv22103.16547 2021c.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv,, arXiv: 1910. 12708, 2019.
Gianfranco Durin and Stefano Zapperi. Scaling exponents for barkhausen avalanches in polycrys- talline and amorphous ferromagnets. Phys. Rev. Lett., 84:4705 4708,, May 2000.
Freeman J. Dyson. Existence of phase-transition in one-dimensional ising ferromagnet. Com- munications Mathematical Physics, 12(2):91-107, 1969.
Bryn Elesedy, Varun Kanade, and Yee Whye Tch. Lottery tickets in linear models: An analysis of iterative magnitude pruning. arXiv preprint, arXiv:2007.08243, 2021.
Utku Evei, Yani A. Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. arXiv preprint, arXiv:2010.03533, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In Hal Daumé III and Aarti Singh (cds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3259-3269. PMLR, 13-18 Jul 2020a.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In International Conference on Learning Representations, 2020b.
Varun Gohil, S. Deepak Narayanan, and Atishay Jain. [Re] One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. ReScience 6(2)#44, May 2020.
Nigel Goldenfeld. Lectures on Phase Transitions and the Renormalization Group. Levant Books, 2005.
10

---

Under review as a conference paper at ICLR 2022
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arXiv preprint, arXiv:1812.04754, 2018.
2018.
Minyoung Huh, Pulkit Agrawal, and Alexei A. Efros. What makes imagenet good for transfer learning? arXiv preprint, (XXi::660.00611111 2016.
Eric Jones, Travis Oliphant, Pearu Peterson, et al. SciPy: Open source scientific tools for Python, 2001-. URL http://wwwy scipy. org/
Leo Kadanoff. Scaling laws for ising models near Tr. Physics Physique Fizika, 2:263-272, Jun 1966.
Yann LeCun, S. Denker, and Sara A. Solla. Optimal brain damage. In David S. Touretzky (ed.), Advances Neural Information Processing Systems 2, (NIPS Conference, Denver, Colorado, USA, November 27-30, 1989]. pp. 598-605, Morgan Kaufmann, 1989.
Jaron Maene, Mingxiao Li, and Marie-Francine Moens. Towards understanding iterative magnitude pruning: Why lottery tickets win. arXiv preprint, atXiv:2106.06955, 2021.
Rahul Mehta. Sparse transfer learning via winning lottery tickets. arXiv preprint, 190.:1995..0778.. 2019.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: gener- alizing lottery ticket initializations across datasets and optimizers. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Curran Associates, Inc., 2020.
Neural Information Processing Systems, volume 33, pp. 512-523.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are win- ning, 2020.
Archishman Raju, Colin B. Clement, Lorien X. Hayden, Jaron P. Kent-Dobias, Danilo B. Liarte, D. Zeb Rocklin, and James P. Sethna. Normal form for renormalization groups Phys. Rev. X. 9: 021014, Apr 2019.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruming. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. arXiv preprint, arXiv:2106.10165 2021.
Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. arXiv preprint, arXiv:2006. (tt.:2006..0611,, 2020.
Daniel L. Ruderman and William Bialek. Statistics of natural images: Scaling in the woods. Phys. Rev. Lett., 73:814-817, Aug 1994.
Matthia Sabatelli, Mike Kestemont, and Pierre Geurts, On the transferability of winning tickets in non-natural image datasets. In Proceedings of the 16th International Joint Conference Com- puter Vision, Imaging and Computer Graphics Theory and Applications Volume VISAPP, pp- 59-69. INSTICC, SeiTePress, 2021.
Saeed Saremi and Terrence J. Sejnowski. Hierarchical model of natural images and the origin of scale invariance. Proceedings of the National Academy of Sciences, 1100)):071-3376, 2013.
Ryan Van Soelen and John W. Sheppard. Using winning lottery tickets in transfer learning for con- volutional neural networks. In 20/9 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2019.
Sven Tougaard. Universality classes of inelastic electron scattering cross-sections. Surface and Interface Analysis, 25(3):137-154, 1997.
11

---

Under review as a conference paper at ICLR 2022
Kenneth G. Wilson. Renormalization group and critical phenomena. i. renormalization group and the kadanoff scaling picture. Phys. Rev. B, 4:3174-3183, Nov 1971a.
Kenneth Wilson. Renormalization group and critical phenomena. ii. chase-space cell analysis of critical behavior. Phys. Rev. B, 4:31 4:3184-3205, Nov 1971b.
Kenneth G. Wilson. The renormalization group: Critical phenomena and the kondo problem. Rev. Mod. Phys., 47:773-840, Oct 1975.
Horst Henning Winter and Marian Mours. Rheology of Polymers Near Liquid-Solic Transitions, pp- 165-234. Springer Berlin Heidelberg, Berlin, Heidelberg, 1997.
Haonan Yu. Sergey Edunov, Yuandong Tian, and Ari S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. In International Conference on Learning Representations, 2020.
Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data is more. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12380-12390. PMLR, 18-24 Jul 2021.
A ADDITIONAL RESNET-50 TRANSFER RESULTS
In the main text, computed IMP eigenvalues for ResNet-50, pre-trained ImageNet using simCLR self-supervised training (Chen et al., 2020b). These models were shown to have the best transferabilityy of winning tickets across tasks in Chen al. (2021a). However, other, pre-trained models were also found to have good transferability. Here, we present the same analysis for ResNet- 50, pre-trained on ImageNet using supervised training (Huh et al., 2016). again finding that the winning tickets for each task have similar distributions of eigenvalues (Table 4). The data used for this was also taken from Chen et (2021a).
Table 4: Computed eigenvalues of the eigenfunctions, M, for ResNet-50, pre-trained on ImageNet using supervised learning (Huh et al. 2016). Red (green) corresponds to those eigenvalues that are relevant (irrelevant). Any eigenvalues within standard deviation of X 1 are considered marginal and colored cyan.
Task
A
A
A4
ImageNet
1.13 0.01
1.08 0.01
1.02 0.01
CIFAR-10
1.04 = 0.04
1.06 + 0.03
1.03 + 0.03
0.96
CIFAR-100
1.04 = 0.05
1.06 0.02
1.03 + 0.004
0.97 0.01
B ADDITIONAL EVIDENCE OF IMP EIGENFUNCTIONS IN VGG-16 AND BERT
In the main text, we presented results for ResNets. However, there is no explicit assumptions of architecture present the theory developed in Sec. 2. Therefore, we explored whether the functions M, (Eq. 13 were similarly eigenfunctions for VGG-16, trained on CIFAR-10, and BERT, trained on the self-super mask language modeling (MIM) task. Data is from experiments performed by Zhang et al. (2021) and Chen et al. (2020a), respectively.
As can be seen Tables 5 and 6, for both pairs of architectures and tasks, we find that the com- puted eigenvalues have small (usually < 5%) standard deviations, suggesting that it is reasonable to consider them as eigenfunctions of Z. Note that we only considered the first 17 applications of IMP (matching that of the ResNet experiments presented in Table 2), in the case of VGG-16. Considering additional applications led to higher standard deviations, which is expected as the size of the system decreases and finite size effects start to become negligggig.
12

---

Under review as a conference paper at ICLR 2022
Interestingly, while VGG has layers that are relevant, marginal, and irrelevant, like ResNet-50, BERT has only marginal layers. This suggests that BERT may have been pushed into a fixed point of the IMP flow. Whether, and how, this is related to the challenges associated with going deeper with Transformers, will be studied of future work.
Table Computed eigenvalues of the eigenfunetions, M, for VGG-16 trained on CIFAR-10. Note that only the prunable layers had their IMP eigenvalues computed. Coloring is same as Table 4.
1.02
A
1.16
A
1.15
A11
1.14
A15
1.12
0.00
A21
1.007
A25
1.007
A28
0.95
A31
0.91 +0003
A35
0.89
Ags
0.88 0.04
A41
0.87 0.05
Table 6: Computed eigenvalues of the eigenfunctions, M, for BERT trained on the MLM task. Coloring same Table
A1
0.99
A2
1.00
A
1.00
A4
1.00
A5
1.01
A
1.01
AT
1.01
A
1.00
Ag
1.00 0.002
A10
+
X11
1.00
A12
1.00
13

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL E-LTH RESULTS
In the main text on the, the experimental results surrounding the "Elastic Lottery Ticket Hypothesis" focused only the cigenvalues corresponding to the cigenfunctions of residual blocks in the first stage. The models considered in [Cenn et al. (021cc) had two other stages, whose eigenvalues we report here in Tables[Jand]]
While the distribution eigenvalues is different than it in stage (i.e. considerably fewer relevant residual blocks), they again support the idea that the distributions related to the results from Chen et al. 2021c) that we focused on the main text.
Table 7: Computed eigenvalues of the eigenfunctions M, for the second stage of various ResNet architectures. Coloring is same as Table 4.
ResNet architecture
Stage 2
ResNet- 14
1.03
ResNet-20
1.02, 1.02
ResNet-32
1,02, 1.03, 1.01, 1.00
ResNet-44
1.04, 1.03, 1.03, 1.03, 1,00, 1.01
ResNet-56
1.03, 1.02, 1.02, 1.02, 1.01, 1.00, 0.99, 1.00
Table 8: Computed eigenvalues of the eigenfunctions M2 for the third stage of various ResNet architectures. Coloring is same as Table 4.
ResNet architecture
Stage 3
R4
0.95
ResNet-20
0.97, 0.95 0,99, 0.99, 0.96, 0.95
ResNet-32
ResNet-44
0.99, 0.98, 0.97, 0.95, 0.96, 0.96
ResNet-56
0.99, 0.98, 0.98, 0.97,0.96, 0.97, 0.98, 0.96
D CRITICAL EXPONENTS
Following the recent work on scaling laws for IMP developed by Rooeenelld et al. (2020), we fit the error, e, defined as 100% minus the top-1 accuracy, as a function density, d, defined as percent of weights remaining, via the functional form
d ip
ö(d, cnp; 7.P) enp
Eup
d ip
(14)
(d2 + p² (
7/2
Cmp
Cap
(d2 + +
where i i= V-T. Here Emp is interpreted as the error associated with not pruning, et as the asymp- totic error upon maximal pruning, 7 as the sensitivity of the combination of network architecture, task, activation function, and optimizer to pruning, and controlling how the transition from no change in error to power-law scaling takes place. Importantly, can be viewed as critical expo- nent under the RG framework. Because systems in the same universality class have the same critical exponents, we wondered if we could find evidence universality in the computed value of ạ for various different computer vision models.
We found the fits to be sensitive to Exp and, in some cases, there was not a clear single value for Cap- Therefore, we included 6mp free parameter (which Rosenfeld al. (2020) did not do),
14

---

Under review as a conference paper at ICLR 2022
Table 9: Critical exponent, 7, for ResNet-50 with either pre-training and 5% rewinding. or pre- training using ImageNet, simCLR, or MoCo. Bottom half gives ? computed on the pre-trained task after pre-training the parameters. Data from Chen al. (202la).
Data-set
No pre-train
ImageNet
simCLR
MoCo
CIFAR-I0
-0.14
? = -0.10
7= -0.16
-0.21
CIFAR-100 100
? = -0.11
?= -0.15
= -0.06
7= -0.24
SVHN
y -0.07
T= -0.06
? -0.03
Y= -0.29
ImageNet simCLR
7= -0.42
7 =1.00
Error as function of density scaling
0.000
Log, scaled error
CIFAR-00
SVHN
-0.008
-0.014
Logz density
Figure 3: Error function of density scaling of CIFAR-10/100 and SVHN. Overlaying the error as a function density curves for ResNet-50, trained from random initialization. on CIFAR- 10/100 SVHN, with 5% rewind. The overlaid curves shows very similar, but noisy, behavior.
but we tightly bounded the value so as to minimize instability in adding another free parameter. To numerically compute the fits, we used scipy's 's curve fitting function: scipy.optimize.cuvee.ek (Jones)
ct al. 2001-).
The computed values of for ResNet-50 evaluated on CIFAR- 10, CIFAR-100. and SVHN pre- sented in Table 9 The effect pre-training using various ofte-of-theeart methods is also given. The values vary, sometimes considerably, across tasks and method of pre-training (c.g. CIFAR-10 and SVHN pre-trained via simCLR). However, we note that the data came from only two or three seeds, making the fits susceptible noise and making the quantification of error in the computed difficult. In addition, the fitting function of Eq. [4 has multiple free parameters (including an additional one added), again making the fit susceptible noise.
Taking these possible complications into account, we overlaid the different error as a function of density scaling curves (an example of which given in Fig B). We found that they qualitatively matched each other more than the computed may have made it seem. We imagine that trying different fitting procedures (especially making use of methods developed in the context of statistical physics, such as finite scaling (Goldenfeld) (005)), as well as using more independent random seeds, will enable more accurate and robust approximations of 7.
15


Reviews:
[{"review_id": "zt3EyLuSpR", "reviewer": "Reviewer_cSyR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper tries to tackle an interesting and important question on the understanding of LTH ticket properties. I also appreciate that the paper focused on trying to explain experiments from prior work and utilized realistic experiments that are of interest to machine learning practitioners. \n\nWeaknesses:\nWhile I appreciate the high-level conceptual connection between RG and IMP, I am not fully convinced that the connection is really a precise one, or that it provides strong quantitative \"explainability\" for LTH phenomenon (that is, RG can be used to explain things that a simpler hypothesis can not). On the other hand, I appreciate that there is some preliminary evidence for this in the paper. Perhaps there are two primary ways in which the paper can be strengthened:\n(1) Either a more extensive set of experiments (also of other predictions from RG) that would demonstrate that RG is really a good explanation for the phenomenon.\n(2) Ruling out some simpler hypotheses, that don't invoke RG, that can explain the phenomenon.\n\nStated otherwise, I don't find the experimental support thus far to be a precision \"test\" that RG is a useful theory for describing the observations. For example:\n--Exponential scaling of the M_i(t). Is there something about constraints from IMP and dynamics of weights during gradient descent that would roughly lead to exponential scaling for such a macroscopic / coarse-grained variable?\n--\"Universality\" in Table 2. This experiment suggests that where the bulk of the pruning is done (which blocks) is set by the architecture / topology. Perhaps a more \"precise\" test of RG here would be if it could be used to tell us *why* topology largely categorizes the universality class?\n--I'm not sure how Figure 2 shows universality, except for the cases where there is scaling collapse.\n--In Sec. 3.2, in discussion on how the smallest ResNet has the weakest transferability: a simpler explanation could be that a smaller model is not expressive enough to learn a good representation useful for more expressive architectures. Maybe this is correlated with aspects of RG, but my point is that there is a simpler mechanism behind it.\n\nOverall, I think this paper has potential and has preliminary evidence in a promising direction, but I think more support would be necessary to convince a reader that the full machinery of RG is crucial for explainability of LTH phenomenon.\n\nQuestion: pg. 8, paragraph 4. Where it says \"...which does not match the source tickets' structure\", should it read \"target\" tickets'?\n", "summary_of_the_paper": "This paper seeks to explain empirical observations in the literature on the Lottery Ticket Hypothesis (LTH) based on ideas from renormalization group (RG) theory in physics. The authors focus on the particular case of iterative magnitude pruning (IMP) as the pruning scheme and view the flow on the space of model parameters during IMP as analogous to RG flow in the space of couplings of a Hamiltonian. The primary experimental evidence for the connection comes from two considerations. (1) Computing the percentage of all non-zero parameters that are in a particular residual block (of a ResNet model) at a particular time. For ResNet-50 models considered, this leads to four scalar, dynamical quantities. These four quantities are found to grow or shrink exponentially with time, suggesting they are eigenfunctions of the IMP map. For three different types of learning & dataset (ImageNet / simCLR / MoCo) and one architecture (ResNet-50), the authors find the same eigenvalue across the three settings, per block. From this, the authors conclude that this universality (shared exponent) across different settings is evidence for the analogy to RG. (2) Transferability of lottery tickets. The authors hypothesize that the relevance / irrelevance / marginality  of a particular block ought to be matched by the source and target architecture when transferring tickets. More specifically, there are three conclusions from the literature (#1-3 in Sec 3.2) that are explained (for example, why the smallest model has the weakest transferability). ", "summary_of_the_review": "This paper takes a bold step in connecting a theoretical framework from physics to a set of important empirical observations in machine learning (LTH). It presents preliminary evidence that this connection could be useful, but I think more support is needed to show that the machinery of RG is crucial and predictive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TnVdRouAwf8", "reviewer": "Reviewer_i8Gz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n - The paper is easy to follow.\n- The motivation of trying to explain the transferrable of winning tickets using the renormalization group seems interesting.\n- The paper attempts to build a theoretical basis for understanding the iterative magnitude pruning, which is widely used in works related to the lottery ticket hypothesis.\n\n### Weakness\nAuthors claim the paper provides experimental support of the renormalization group theory in large scale lottery ticket experiments. However, I have some questions about the experiments.\n- For the experiments on the Elastic Lottery Ticket Hypothesis, the datasets used to train different networks (Table 3) seems not introduced in the paper. Do we still have the same conclusion for the networks trained on different datasets, e.g., CIFAR10 and ImageNet? \n- The authors provide some experimental results/conclusions and try to explain them using the renormalization group. However, this explanation doesn't answer the question proposed at the beginning of the work - \"whether a given ticket can be transferred to a given task.\" I wonder if authors could provide experiments in a way that the theory helps us find a winning ticket that can be transferred to some tasks where previous studies cannot find. Such experiments can help readers understand how to use the theory in practice for searching a winning ticket for a specific task. \n\nAnother strong claim in the paper is that the authors mention that the work provides new tools for classifying combinations of DNN architectures, activation functions, and optimizers. Similar to the previous question, I don't quite understand how to leverage the introduced theory of renormalization group in practice to achieve that.", "summary_of_the_paper": "The work aims to provide a deeper understanding of a corollary in the lottery ticket hypothesis: how to know whether a winning ticket found in one task can be transferred to another task. The paper tries to combine the widely used pruning method in the lottery ticket hypothesis (iterative magnitude pruning) with a concept from statistical physics, which is the renormalization group. The authors show iterative magnitude pruning can be considered as a renormalization group operator. Experiments are conducted using the renormalization group to examine whether pruned models are in the same universality classes and extend the renormalization group to the Elastic Lottery Ticket Hypothesis.", "summary_of_the_review": "While trying to obtain a prior for which tasks of a given winning ticket can be transferrable using the theory from statistical physics seems interesting, the paper doesn't provide enough experimental results to show how to use such an explanation to improve iterative magnitude pruning or determine the best architecture that can be transferred for different tasks. The work is more like working in the progress report, and more results can help strengthen the work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Gp30hBZhiF0", "reviewer": "Reviewer_ud5a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n+ This paper has a very clear motivation and tries to answer a very important question about the Lottery Ticket Hypothesis.\n\n+ It provides a theoretical basis for understanding the success of IMP and the universality of winning tickets.\n\n+ To show the universality in the IMP flow, extensive experiments are conducted based on various important methods in different settings: supervised ImageNet training, and self-supervised simCLR and MoCo training.\n\n+ The insights and finding from this paper have potential to inspire a future explorations between IMP and RG.\n\n\nConcerns and questions:\n\n- Authors claim that RG theory can provide a way to predict which combinations of task, optimizer, activation function, and architecture can have winning tickets transferred between them. Is there any experimental evidences related to pruning to support such claims?\n\n- The experiments and analysis are mainly about RestNet based backbone. How about other commonly used backbone, e.g. VGG, DenseNet, etc?", "summary_of_the_paper": "Although Lottery Ticket Hypothesis has suggested an exciting corollary about the winning tickets, it is still unclear why winning ticket universality exists, or any way of knowing a priori. In this paper, authors make use of renormalization group theory to perform detailed understanding of this hypothesis. Authors find that the principle method iterative magnitude pruning (IMP) is directly related to the renormalization group (RG).  \n\nBased on the contributions of  RG leading to a first principled understanding of the universality in behavior near phase transitions, viewing the IMP from an RG perspective may lead to new insight on the universality of winning tickets. Such insight is build upon the experimental support of the theory in large scale lottery ticket experiments.", "summary_of_the_review": "Overall, the paper has a good quality with a clear motivation, it tries to answer a very important question about the Lottery Ticket Hypothesis. Authors' findings about the relationships between principle method iterative magnitude pruning and the renormalization group may lead to new insight on the universality of winning tickets. Such insight is important and can inspire a considerable potential for future collaboration between IMP and RG. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2YPAh6hlVW2", "reviewer": "Reviewer_iuDr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written. The ideas of bringing renormalization group theory to explain the lottery ticket hypothesis theory is very interesting. However, I have some confusions need to be clarified. \n\n1). The analogy between the IMP and the Ising model, i.e. equation (1) and (2) is a bit far-fetched to me. The two equations appear alike, however, I donot quite see the intuition behind it. For example, one can argue that the magnetization (order) disappeared when the temperate increases in spin system, however, the error (disorder) on the other hand decreased when the density increases in DNN, almost two opposite directions.\n\n2). In the section 2.2 where the RG and IMP is proved connected, specifically in equation 12, I donot see how the amplitude of the parameters plays in a role in defining the RG operator. One of the key factors in IMP is that the selection of parameters is based on the amplitude. If the iterative pruning is done randomly without considering the amplitude, is the RG frame still work? In addition, the operator (equation 12) seems only consider per layer operation, but the IMP often performed per model. Could the author clarify the difference and how that could impact the operator?\n\n3). Another main concern is the IMP flow results on ResNet50 (Fig. 2). The normalized initializations scale the initial weights according to the channel dimensions, the consequence of which is smaller weight amplitude on the later ResBlocks. During the IMP, the percentage of pruning elements in each ResBlock will naturally follow the amplitude ratio. Since the eigenfunction of the IMP operator is defined based on the pruning rate in each ResBlock, the result IMP flow in Figure 2 should just follow the amplitude difference between blocks. Not sure how this result could have supported the theory. Besides, equation 13 does not include amplitude term as well.\n", "summary_of_the_paper": "This paper tries to find a theoretical explanation of the transferability of lottery ticket used in similar tasks. Observing the similarity between the universality in renormalization group and the lottery ticket hypothesis, the author proposes that the iterative magnitude pruning, which is used to find the winning tickets, could be a renormalization group scheme. The authors also provide some evidence on their theory on vision model of ResNet families.", "summary_of_the_review": "In general, I think this paper is well written and proposed an interesting theory, however, the argument and proof are not very convincing to me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper observes the similarity between the universality in renormalization group and the lottery ticket hypothesis and proposes that the iterative magnitude pruning, which is used to find the winning tickets, could be a renormalization group scheme. The authors also provide some evidence on their theory on vision model of ResNet families. While it is interesting to try a theoretical explanation of the transferability of lottery ticket used in similar tasks using the theory from statistical physics, the paper does not provide enough experimental results to show how to use such an explanation to improve iterative magnitude pruning or determine the best architecture that can be transferred for different tasks. Therefore the work is more like working in the progress report and not ready for publication yet.
Decision: Reject