Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Although Lottery Ticket Hypothesis has suggested an exciting corollary about the winning tickets, it is still unclear why winning ticket universality exists, or any way of knowing a priori. In this paper, authors make use of renormalization group theory to perform detailed understanding of this hypothesis. Authors find that the principle method iterative magnitude pruning (IMP) is directly related to the renormalization group (RG).  

Based on the contributions of  RG leading to a first principled understanding of the universality in behavior near phase transitions, viewing the IMP from an RG perspective may lead to new insight on the universality of winning tickets. Such insight is build upon the experimental support of the theory in large scale lottery ticket experiments.
main_review: Strength:

+ This paper has a very clear motivation and tries to answer a very important question about the Lottery Ticket Hypothesis.

+ It provides a theoretical basis for understanding the success of IMP and the universality of winning tickets.

+ To show the universality in the IMP flow, extensive experiments are conducted based on various important methods in different settings: supervised ImageNet training, and self-supervised simCLR and MoCo training.

+ The insights and finding from this paper have potential to inspire a future explorations between IMP and RG.


Concerns and questions:

- Authors claim that RG theory can provide a way to predict which combinations of task, optimizer, activation function, and architecture can have winning tickets transferred between them. Is there any experimental evidences related to pruning to support such claims?

- The experiments and analysis are mainly about RestNet based backbone. How about other commonly used backbone, e.g. VGG, DenseNet, etc?
summary_of_the_review: Overall, the paper has a good quality with a clear motivation, it tries to answer a very important question about the Lottery Ticket Hypothesis. Authors' findings about the relationships between principle method iterative magnitude pruning and the renormalization group may lead to new insight on the universality of winning tickets. Such insight is important and can inspire a considerable potential for future collaboration between IMP and RG.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper seeks to explain empirical observations in the literature on the Lottery Ticket Hypothesis (LTH) based on ideas from renormalization group (RG) theory in physics. The authors focus on the particular case of iterative magnitude pruning (IMP) as the pruning scheme and view the flow on the space of model parameters during IMP as analogous to RG flow in the space of couplings of a Hamiltonian. The primary experimental evidence for the connection comes from two considerations. (1) Computing the percentage of all non-zero parameters that are in a particular residual block (of a ResNet model) at a particular time. For ResNet-50 models considered, this leads to four scalar, dynamical quantities. These four quantities are found to grow or shrink exponentially with time, suggesting they are eigenfunctions of the IMP map. For three different types of learning & dataset (ImageNet / simCLR / MoCo) and one architecture (ResNet-50), the authors find the same eigenvalue across the three settings, per block. From this, the authors conclude that this universality (shared exponent) across different settings is evidence for the analogy to RG. (2) Transferability of lottery tickets. The authors hypothesize that the relevance / irrelevance / marginality  of a particular block ought to be matched by the source and target architecture when transferring tickets. More specifically, there are three conclusions from the literature (#1-3 in Sec 3.2) that are explained (for example, why the smallest model has the weakest transferability). 
main_review: Strengths:
The paper tries to tackle an interesting and important question on the understanding of LTH ticket properties. I also appreciate that the paper focused on trying to explain experiments from prior work and utilized realistic experiments that are of interest to machine learning practitioners. 

Weaknesses:
While I appreciate the high-level conceptual connection between RG and IMP, I am not fully convinced that the connection is really a precise one, or that it provides strong quantitative "explainability" for LTH phenomenon (that is, RG can be used to explain things that a simpler hypothesis can not). On the other hand, I appreciate that there is some preliminary evidence for this in the paper. Perhaps there are two primary ways in which the paper can be strengthened:
(1) Either a more extensive set of experiments (also of other predictions from RG) that would demonstrate that RG is really a good explanation for the phenomenon.
(2) Ruling out some simpler hypotheses, that don't invoke RG, that can explain the phenomenon.

Stated otherwise, I don't find the experimental support thus far to be a precision "test" that RG is a useful theory for describing the observations. For example:
--Exponential scaling of the M_i(t). Is there something about constraints from IMP and dynamics of weights during gradient descent that would roughly lead to exponential scaling for such a macroscopic / coarse-grained variable?
--"Universality" in Table 2. This experiment suggests that where the bulk of the pruning is done (which blocks) is set by the architecture / topology. Perhaps a more "precise" test of RG here would be if it could be used to tell us *why* topology largely categorizes the universality class?
--I'm not sure how Figure 2 shows universality, except for the cases where there is scaling collapse.
--In Sec. 3.2, in discussion on how the smallest ResNet has the weakest transferability: a simpler explanation could be that a smaller model is not expressive enough to learn a good representation useful for more expressive architectures. Maybe this is correlated with aspects of RG, but my point is that there is a simpler mechanism behind it.

Overall, I think this paper has potential and has preliminary evidence in a promising direction, but I think more support would be necessary to convince a reader that the full machinery of RG is crucial for explainability of LTH phenomenon.

Question: pg. 8, paragraph 4. Where it says "...which does not match the source tickets' structure", should it read "target" tickets'?

summary_of_the_review: This paper takes a bold step in connecting a theoretical framework from physics to a set of important empirical observations in machine learning (LTH). It presents preliminary evidence that this connection could be useful, but I think more support is needed to show that the machinery of RG is crucial and predictive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to find a theoretical explanation of the transferability of lottery ticket used in similar tasks. Observing the similarity between the universality in renormalization group and the lottery ticket hypothesis, the author proposes that the iterative magnitude pruning, which is used to find the winning tickets, could be a renormalization group scheme. The authors also provide some evidence on their theory on vision model of ResNet families.
main_review: The paper is well written. The ideas of bringing renormalization group theory to explain the lottery ticket hypothesis theory is very interesting. However, I have some confusions need to be clarified. 

1). The analogy between the IMP and the Ising model, i.e. equation (1) and (2) is a bit far-fetched to me. The two equations appear alike, however, I donot quite see the intuition behind it. For example, one can argue that the magnetization (order) disappeared when the temperate increases in spin system, however, the error (disorder) on the other hand decreased when the density increases in DNN, almost two opposite directions.

2). In the section 2.2 where the RG and IMP is proved connected, specifically in equation 12, I donot see how the amplitude of the parameters plays in a role in defining the RG operator. One of the key factors in IMP is that the selection of parameters is based on the amplitude. If the iterative pruning is done randomly without considering the amplitude, is the RG frame still work? In addition, the operator (equation 12) seems only consider per layer operation, but the IMP often performed per model. Could the author clarify the difference and how that could impact the operator?

3). Another main concern is the IMP flow results on ResNet50 (Fig. 2). The normalized initializations scale the initial weights according to the channel dimensions, the consequence of which is smaller weight amplitude on the later ResBlocks. During the IMP, the percentage of pruning elements in each ResBlock will naturally follow the amplitude ratio. Since the eigenfunction of the IMP operator is defined based on the pruning rate in each ResBlock, the result IMP flow in Figure 2 should just follow the amplitude difference between blocks. Not sure how this result could have supported the theory. Besides, equation 13 does not include amplitude term as well.

summary_of_the_review: In general, I think this paper is well written and proposed an interesting theory, however, the argument and proof are not very convincing to me.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work aims to provide a deeper understanding of a corollary in the lottery ticket hypothesis: how to know whether a winning ticket found in one task can be transferred to another task. The paper tries to combine the widely used pruning method in the lottery ticket hypothesis (iterative magnitude pruning) with a concept from statistical physics, which is the renormalization group. The authors show iterative magnitude pruning can be considered as a renormalization group operator. Experiments are conducted using the renormalization group to examine whether pruned models are in the same universality classes and extend the renormalization group to the Elastic Lottery Ticket Hypothesis.
main_review: ### Strength
 - The paper is easy to follow.
- The motivation of trying to explain the transferrable of winning tickets using the renormalization group seems interesting.
- The paper attempts to build a theoretical basis for understanding the iterative magnitude pruning, which is widely used in works related to the lottery ticket hypothesis.

### Weakness
Authors claim the paper provides experimental support of the renormalization group theory in large scale lottery ticket experiments. However, I have some questions about the experiments.
- For the experiments on the Elastic Lottery Ticket Hypothesis, the datasets used to train different networks (Table 3) seems not introduced in the paper. Do we still have the same conclusion for the networks trained on different datasets, e.g., CIFAR10 and ImageNet? 
- The authors provide some experimental results/conclusions and try to explain them using the renormalization group. However, this explanation doesn't answer the question proposed at the beginning of the work - "whether a given ticket can be transferred to a given task." I wonder if authors could provide experiments in a way that the theory helps us find a winning ticket that can be transferred to some tasks where previous studies cannot find. Such experiments can help readers understand how to use the theory in practice for searching a winning ticket for a specific task. 

Another strong claim in the paper is that the authors mention that the work provides new tools for classifying combinations of DNN architectures, activation functions, and optimizers. Similar to the previous question, I don't quite understand how to leverage the introduced theory of renormalization group in practice to achieve that.
summary_of_the_review: While trying to obtain a prior for which tasks of a given winning ticket can be transferrable using the theory from statistical physics seems interesting, the paper doesn't provide enough experimental results to show how to use such an explanation to improve iterative magnitude pruning or determine the best architecture that can be transferred for different tasks. The work is more like working in the progress report, and more results can help strengthen the work.
review_title: null

Meta-review: This paper observes the similarity between the universality in renormalization group and the lottery ticket hypothesis and proposes that the iterative magnitude pruning, which is used to find the winning tickets, could be a renormalization group scheme. The authors also provide some evidence on their theory on vision model of ResNet families. While it is interesting to try a theoretical explanation of the transferability of lottery ticket used in similar tasks using the theory from statistical physics, the paper does not provide enough experimental results to show how to use such an explanation to improve iterative magnitude pruning or determine the best architecture that can be transferred for different tasks. Therefore the work is more like working in the progress report and not ready for publication yet.
Decision: Reject