Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a method for adapting lottery tickets (LT) found in one network architecture to a related architecture of different size. The motivation for this work is to study the connection between LTs of different architectures, and to reduce the cost of performing iterative magnitude pruning (IMP). If LTs can transfer between architectures, then it may be possible to find lottery ticket only once and adapting the size of the model as needed, without having to perform IMP for each model architecture. They restrict to adaptation to models of the same family (only differing in depth), and show that the adapted lottery tickets outperform pruning-at-initialization and RigL.

Networks are viewed at the level of "units", which are layers (+normalization), or residual blocks for ResNets. The adaptation method is to replicate nearby units of a smaller source model for each additional layer, or drop nearby units of a larger source model for each extra layer. They perform ablation studies on various choices for the replication/dropping of units.
main_review: Originality:
- This work appears to be very original, as it is the first work I have seen for adapting one lottery ticket to multiple architectures. 

Quality:
- My major issue with this work is the claim that lottery tickets (LT) can be stretched or shrunk to different architectures. This is a bold claim and the current experiments leave room for doubt. To claim that the new network is a LT, we should see that 1) reinitialization under the LT mask, 2) LT weights with random mask, and 3) random weights with random mask all perform worse than the LT (while keeping the source blocks the same). However, the "Interpolation 1234 - Random" experiment in Figure 3 shows that it is not significantly different from "Interpolation 1234", raising serious doubts about whether the approach generates "lottery tickets" or just a way to stretch or shrink existing models. No other experiments are conducted to establish that the new network is indeed a LT, which also raises questions.

- It’s unclear how much of the gain is due to simply having access to the LT on the original source model. For pruning-at-initialization baselines, what if you copy the same weights as LTH for existing blocks, and only perform pruning for the new layers using pruning-at-initialization techniques? This would be a more appropriate baseline to compare to. The current experiments disadvantages other baselines by leveraging a prior LT model only for ETTs. 

- Are the baselines (unpruned, IMP) initialized in the same way as the elastic LTH? I.e. Weights are repeated for certain blocks? Or is it done with random initializations? If the latter, I would be curious to see what happens when you replicate blocks in the same way at initialization. How much of the benefit of ELTH comes from replicating initial weights (if any)?

- The claims that ETTs may transfer to general pruned solutions seems unfounded (also unclear what "general pruned solutions" mean here, so I may be misreading). To make the claim that LTs derived from different model architectures are connected, and that they stay in the same basin, the authors could look at linear mode connectivity between IMP solution on a target model and the ETT solutions. 

Clarity:
- The paper is generally well written. My only comment is that the takeaway of section 4.4 confuses me. I hope the authors can elaborate further on what the implication of their linear mode connectivity result is.

Significance:
- The paper raises a question that would be significant to the community. However, the results so far do not make a compelling case for it. Nonetheless, I would be convinced otherwise if my earlier concerns were sufficiently addressed. 


=================================================================
Updates post rebuttal:
The authors have provided additional experiments and clarifications that addressed my earlier concerns. Although the performance difference between ELT and the appropriate baselines suggested in the review is small, it seems consistent, and would be interesting to the community given the general novelty of this work. Assuming that these new results will be added to the paper accordingly, I will increase my score from 4 to 6.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper focused on exploring the transferability of a subnetwork obtained from the given dense network. Following the Lottery Ticket Hypothesis (LTH), a new hypothesis called Elastic Lottery Ticket Hypothesis (E-LTH) was proposed along with a corresponding validation method––Elastic Ticket Transformations (ETT). Extensive experiments on different benchmarks are provided to prove the effectiveness of ETT.
main_review: This paper is drafted with a strong motivation resulting in an interesting research topic --- the transferability of a winning ticket. The previous surprising findings in LTH require expensive computational costs to benefit real-world network training. This paper tries to alleviate this disadvantage by transferring the obtained winning ticket to other network structures, showing a good potential for practical application and theoretical exploration.

The experiments are comprehensive. In addition to the comparisons between networks in the same family based on several strong baselines, this work also provides other experimental analyses, such as the comparisons with dynamic sparse training and the analysis of linear mode connectivity. 

Some experimental settings are unclear. As mentioned in L257-L258, the sparsity ratio will be different. It is better to provide detailed comparisons and how the “best possible extent” (L259) was made to compensate for the difference.

Some potential missing baselines. I am curious about the extreme situation of both “stretching” and “squeezing.” How about only stretching the last unit to finish transferring from lower to deeper network, or even stretching randomly picked unit.

The value of “squeezing” network: “stretching” network will benefit the winning ticket finding for a larger network based on a smaller one. However, based on the results (e.g., ResNet56 to ResNet32 in Fig. 4), the structural changes in winning tickets will also cause a performance drop for the converse scenario. It is better to elaborate on the potential benefits of this situation. 
limitations_and_societal_impact: Suffice.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper transfer the winning ticket found for one model to other models, the author claims it can reduce the pruning cost for a large-scale model. The motivation is clear.  



main_review: However, I still have some concerns about this manuscript.

The readability of Fig3 4 5 6 is poor.

The author does not present a principal method to grow the depth and increase the width, and the depth and width are critical dimensions for a large-scale model.
 The SOTA dynamic method should include the lastest method(e.g. DWN[1] DFP[2] STR[3] ...)

Besides, this paper only focuses on unstructured sparsity. In fact, the unstructured sparsity is hard to have speed gain emphasized by[4, 5], The authors are suggested conducting experiments on channel/filter pruning[6, 7] or fine-grained structured sparsity [4, 8, 9, 10].

typo: line 358 "onlu".

Overall, the author proposes a framework to reduce **the IMP cost** via transferring "winner ticket" to other neural models. I go through the whole paper. I cannot find any training time report in the experimental evaluation. Therefore, it is not convincing to support the claim with the experiment of this manuscript.    

[1] Discovering Neural Wirings

[2] Dynamic Model Pruning with Feedback

[3] soft threshold weight reparameterization for learnable sparsity

[4] Accelerating Sparse Deep Neural Networks

[5] PCNN: pattern-based fine-grained regular pruning towards optimizing CNN accelerators

[6] Neural Pruning via Growing Regularization

[7] Rethinking the Value of Network Pruning

[8] Learning N: M fine-grained structured sparse neural networks from scratch

[9] 1xN Block Pattern for Network Sparsity

[10] PCNN: pattern-based fine-grained regular pruning towards optimizing CNN accelerators
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper investigates whether lottery tickets can transfer across architectures of different depths, following up on prior work which showed transfer across datasets for the same architecture. Reviewers all found the clarity and originality to be strong and the topic is of significant interest. There were some concerns regarding the soundness of the claims presented, but these were resolved through additional experiments performed during the rebuttal period. I would strongly encourage the authors to include these experiments in the final version of the paper. Overall, I think this paper presents a valuable contribution and I recommend acceptance. 
Decision: Accept (Poster)