Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a new Lottery Tickets procedure by introducing a layerwise importance score to find the lottery tickets. Additionally, this work demonstrates several intriguing observations: 1) from a fixed initialization, there are different lottery tickets that significantly differ in their structure. 2) these tickets have some common connections, which survive the LTH procedure even when the other weights are reinitialized.
strength_and_weaknesses: ### Strength:

1. This paper is well-written and easy to follow.
2. Both quantitative and qualitative results are reported.
3. Experiments are conducted across different datasets and architectures.

### Weakness:

1. Only small-scale datasets are considered in the experiments.
2. I'm a little bit confused about the main contribution of this work. Using a layerwise importance score can improve the performance of identified lottery tickets or some observations that suggest lottery tickets are not unique. With different mini-batch sequences during training, we may also obtain significantly different lottery tickets from the same initialization.
3. Using layerwise importance score, can the LTH procedure identifies the winning ticket with higher sparsity?
clarity,_quality,_novelty_and_reproducibility: None
summary_of_the_review: None
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the popular lottery ticket hypothesis (LTH) framework, which discovers sparse, trainable subnetworks via iterative magnitude pruning (IMP). Typically, IMP is performing in a global manner, in which weight magnitudes are compared with all other weights within a network when determining which weights should be pruned/discarded. Alternatively, the authors argue that comparing weights on a per-layer basis is more appropriate, as importances/magnitudes of weights oftentimes differ on a layer-to-layer basis throughout the network. Instead, they propose an alternative procedure that evaluates the importance of weights within each layer. In particular, this method (1) adopts the same weight magnitude and (2) normalizes this magnitude in a per-layer fashion using a few different techniques. This technique is evaluated for 4 different types of normalization on computer vision experiments. The authors then go on to show that different layer-based pruning techniques can be used to find winning tickets with different structures. 
strength_and_weaknesses: Strength:
* The LTH problem considered is popular and the focus on an intuitive/simple issue with typical approaches to LTH (i.e., global pruning leads and layer collapse). I believe this is an interesting direction that the authors motivate well. 
* I believe the observation that many different structures of lottery tickets can be discovered within a network has already been made (see e.g., https://arxiv.org/abs/1912.05671), but the authors do provide some extra analysis that goes beyond previous work that is interesting/useful. I really enjoy this extra analysis, and I think the authors could emphasize this more!

Weakness:
* Because the paper is purely empirical, I believe that large-scale experiments on ImageNet are very necessary in this case. This is because, for the LTH, it is widely recognized that many observations made on LTH for smaller-scale experiments do not hold at scale (see https://arxiv.org/abs/1810.05270 and https://arxiv.org/abs/1902.09574). Thus, in this work, performing the larger-scale experiments (e.g., ImageNet) is necessary. 
* Accuracy improvements of the proposed procedure are small and there is not a single procedure that performs best. The proposed methods are outperformed by magnitude pruning (i.e., the normal LTH procedure) in many of the considered cases. 
* Several pruning works consider pruning in a layer-normalized fashion (see https://proceedings.mlsys.org/paper/2020/hash/d2ddea18f00665ce8623e36bd4e3c7c5-Abstract.html and references, just search “layer wise vs. global pruning” in the paper). None of these techniques are compared against within the analysis. I believe comparing against a few of the SOTA layer-based pruning techniques would be useful. 

Small Stuff:
* Figure 1 is difficult to parse. It might be better to use a different subplot for each layer, then plot the sparsity as a function of the pruning iteration. 
* L1/L2 normalization of per-layer importance are basically identical (even in the results). It might be better to just use one or the other and state that they perform very similarly. 
* In the introduction, your posing of LTH makes it seem like it solves the problem of the train-prune-finetune loop. It might be good to state that LTH still requires this loop (i.e., the network needs to be pre-trained to discover the winning ticket still). 
* “Using importance metrics makes this bias both more explicit and stronger, dependent on which metric is used.”: I am not quite sure what this means here or why this would be the case. It might be nice to explain a bit more or maybe provide a citation. 
* “The only computational overhead incurred in our method is the calculation of importance scores, which is negligible.”: Whether this computational cost is negligible depends on the importance score you choose/implement, right?

Questions:
* Is there any motivation for the importance metrics that you consider, or are you simply trying a bunch of different approaches and trying to determine the best strategy empirically? Are there other approaches beyond these defined importances that could be useful?
clarity,_quality,_novelty_and_reproducibility: Clarity/quality/reproducibility are good.

I feel that the paper somewhat lacks in novelty. I make this statement because:
* Layer collapse and the issues of global IMP for LTH have already been observed
* Layer-wise pruning techniques have also been proposed and explored in depth 
* The main proposed techniques for layer-wise LTH are L1/L2 normalization (minmax/softmax perform worse in nearly all cases), which is believe is fair to say is straightforward.

Despite my opinion on novelty, I believe novelty does exist in the authors' analysis of different winning tickets achieved with their methods. The novelty could be improved by 1) considering/deriving a wider set of potential layer-based importance metrics or 2) emphasizing and extending the analysis of the properties/overlap of tickets achieved via these different layer based techniques and normal IMP (e.g., can we use these different tickets to produce better ensembles or something similar?) 
summary_of_the_review: To begin, I want to emphasize that my current review reflects my initial impression of the paper. I am fully open to discussion with authors/other reviewers, and my final score will be mostly determined by this discussion.

I believe the problem the authors study and the proposed approach are interesting. However, I think the paper--in its current form--falls short of providing enough valuable insight to the problem of improving upon global IMP for LTH. Currently, the main weaknesses of this paper in my opinion are:
* Lack of large-scale experiments. This is very necessary for LTH as mentioned above. 
* Lack of novelty in layer-wise importance schemes. The work could be greatly improved by exploring more schemes (possibly inspired by work in the pruning literature mentioned above).
* Improvements over normal LTH procedures are not consistent. I am not convinced the layer-wise importance approach truly provides a clear benefit.

Nonetheless, the paper is well-written and motivated, and I strongly encourage the authors to continue working in solving the problems I outline above. I especially enjoy the analysis of lottery ticket overlap at the end of the paper. I believe by furthering extending this analysis, thinking more deeply about other layer-wise techniques that could be used/compared-to, and expanding the empirical scope to more domains (e.g., ImageNet, transformer tasks, or other larger-scale cases) the authors will greatly improve the paper and provide a valuable contribution to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to modify the LTH procedure by using a score that measures the layerwise importance in order to select the weights to be pruned. The authors propose a few possible scores and evaluate their performance on a set of experiments. The experiments show that in most of the cases the proposed method does not improve with respect to the standard LTH that uses magnitude in order to select the weights to be pruned. The authors also perform a comparison of lottery tickets obtained with the different scores showing that hey have a noticeable amount of overlapping connections and such connections seems to be more stable.
strength_and_weaknesses: The paper is well written and easy to follow. However, I think that the novel contribution is limited (a similar idea of layerwise importance of the weights is already present in Tanaka et al. 2020). In addition, the results of the experiments do not show a significant gain using the proposed method and I haven't found the comparison of different tickets particularly insightful.
clarity,_quality,_novelty_and_reproducibility: As said in the previous section, the paper is well written and easy to follow. The experiments are well described and the authors provide all the details to reproduce them. The novelty of the proposed method seems quite limited (more comments above).
summary_of_the_review: Even though the paper is well written and it provides a solid experimental evaluation of the proposed method, I believe that the novel theoretical contribution is very limited and the proposed method does not exhibit a significant gain with respect to the state of the art.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The Lottery Ticket Hypothesis (LTH) proposes that it is possible to extract small subnetworks at initialization that can be trained to full accuracy, through algorithms that typically involve pruning connections with the lowest global weight magnitude. However, as the authors of this paper argue this approach may not consider the context of the connections within a layer, as the weight distributions in different layers can vary. In this paper, the authors explore ways to recover this context by considering the importance of connections based on the weights of other connections within the same layer, rather than simply their absolute magnitude. They also generalize the LTH to use these importance metrics rather than weight magnitudes. Through experimentation on various architectures and datasets, the authors found that applying different importance metrics can result in distinct, high-performing "lottery tickets" with little overlap in connections.

The reviewers found that the work had a number of strengths. They noted that the research had both quantitative and qualitative results, and conducted experiments on a variety of datasets and architectures. Furthermore, they commented that the work was well written and easy to follow.

However, the reviewers identified several weaknesses in the work. Primarily, the lack of a novel contribution, as well as the experiments being limited to small-scale datasets. Furthermore, the proposed method does not show a significant gain compared to the state of the art LTH algorithms. Lastly, it is uncertain whether using a layerwise importance score can significantly help to identify sparser "winning tickets".

Overall, some reviewers engaged post the rebuttal session, however, the feedback provided by the authors was not convincing enough (in terms of timeline of implementation) to lift some of these criticisms. In particular, reviewer 8dwk provided extensive and constructive feedback to the authors, suggesting that they should focus more on analysis and theory in their paper and include comparison and contrast with related works. The reviewer also suggested that the authors should run experiments on a larger scale dataset, such as ImageNet, if possible. The authors responded to the feedback by explaining the technical differences between their work and the related works, and their plans to expand the research by refining the analysis and potentially bringing back a proof regarding Layerwise importance to the paper. However, (and I am in agreement with this) the changes requested, although possible, go beyond the scope of this round of reviews. 

The authors are encouraged to take into the account all the feedback provided, and perhaps resubmit to a future venue.

justification_for_why_not_higher_score: 
Given the lack of a significant novel contribution, limited experiments on small-scale datasets, and absence of significant gains compared to the state of the art, I believe that this paper is not ready for publication at this stage.
Decision: Reject