Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes the so called Dual Lottery Ticket Hypothesis which claims that every ticket in the ticket pool, i.e., every random sparse subnetwork in a randomly initialized dense neural network, can be transformed into a winning ticket with admirable trainability. This "transformation" proposed in the notation of Random Sparse Network Transformation (RST), in practice, comes in with the form of a squared L2 norm regularization on the masked weights, making them still involved during training while extruding the information contained in them and transferring into the unmasked ones. The empirical experiments show good empirical performance of RST compared to the vanilla LTH, pruning-at-initialization methods and LTH variants such as EB-LTH.
main_review: Two main concerns I have on this work are the way it is posed and the writing issue.

For how this work is posed, as the authors mentioned in multiple places in the write-up, DLTH is about a proper "transformation" of any ticket in the lottery pool into the winning ticket. Correspondingly, the authors proposed the RST as the transformation above to substantiate DLTH. However, according to the formulation in eqn. (3) and (4), Random Sparse Network Training seems to a better name for RST as far as I can see because it looks more like a general (potentially a uniformly better one) training method for sparse neural networks from scratch. For example, we could for sure apply RST to the winning tickets found by IMP directly. Will combining IMP + RST give us a better performance than RST + any tickets? This experiment will give us a clue of which one of the following is more true (or both not true):

1) a winning ticket achieves the best performance regardless of the training method and any transformed ticket with RST can not achieve better; OR
2) the winning tickets are still better subnetworks and can also be improved in terms of accuracy with RST?

If the first case is more true, then the contribution of this work would be much more interesting to me. If the second is more true, however, I think this work is not posed well. It is not about transforming any ticket into a winning one, but about proposing a general better training method for sparse neural networks. Let alone that the core regularization technique in RST is not new.

Another concern of mine is the writing. Firstly, the introduction and discussion of literature before Section 4 seem to be too long to me. Secondly, Section 3 is so disconnected with Section 4 and Section 5. Even the notations are not consistent in Section 3 and 5. The authors made efforts to explicitly define the transformation of weight and mask but lacked discussion about them in Section 4 and 5, like how RST is related to them, making RST just a training technique with regularization, totally losing connection to Section 3. I think this issue is also related to my first concern about the posing of this work. My suggestion is to shorten the contents before Section 3 and add richer discussion in Section 3, 4 and 5 to make them more logically consistent and connected.
summary_of_the_review: My evaluation for this work is that it is below the acceptance threshold, but I tend to give a score 4 (but I don't have option). Major reason is the posing of the DLTH claim and the contribution. I also think the writing could be fundamentally improved. I still think the technique and the idea are interesting and encourage the authors to try the next venue with enough improvements.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper claims that "Any randomly picked subnetwork in a dense randomly initialized neural network can be transformed into a trainable condition, where the subnetwork can be trained in isolation and achieves better at least comparable performance to LTH and other strong baselines."

In order to demonstrate how such a transformation may be achieved the paper introduces RST which gradually penalizes the magnitude (L2) of the non-selected parameters during training (upon which they are zeroed out explicitly) in order to arrive at the randomly selected sparse sub-network. 


main_review: Major Pros:
That randomly selected sub-networks, can attain same performance as dense and comparable sparsity IMP derived LT has reached significant attention. The main idea presented here at tackling how a random network may be 'transformed' via RST is interesting.


Major Cons:
1. The main claim: " *any* randomly picked subnetwork .... " is incorrect. Examples where such a claim will fail are easy to construct: a choice of subnetwork which is disconnected, or lost its residual connection(s) or is woefully bottlenecked ...

2. The results with respect to cycle dependency are very noisy raising suspicion with respect to their statistical significance and/or the representativeness of the regime in which they were conducted. Why are they noisy (e.g. figure 5).
Specifically, for high sparsity levels --- performance in iterative magnitude pruning depends strongly on number of iterations (I assume this is what is meant by cycles?).

Style and clarity (not affecting score):
Throughout the paper much is left to the reader to infer burdening the clarity of the paper and its decipherability:
- At the onset of the main claim: what is a " trainable condition "
- Information Extrusion mentioned but not defined (as early as the abstract), only to be exemplified via implementation in section 5.
- 'Cycles' are not defined
- "As mentioned in implementation section, the extra training cost can be calculated..." the reader is referred to an equation in the Discussion of DLTH v. LTH section of the optimization cost , but no such equation is to be found in that reference. 



*Updated review in light of author responses [Nov 26, 2021]:*

I have reviewed the updated manuscript following several iterations with the authors:  https://anonymous.4open.science/r/DLTH_Updated-3020/2022_ICLR_DLTH_Updated.pdf.

The updated manuscript changes the erroneous original claims ("any network can be transformed to trainable condition") and precisely modifies and scopes them ("random ticket wins"). In particular, giving room for a key novelty in the paper in the form of RST.

Issues with respect to the regime in which the conclusions are made remain, including an insufficient analysis of the sensitivity to iteration and duration of extrusion. These issues *should be expanded further in the final manuscript by additional experiments* in the regime where sensitivity is more pronounced (e.g. reaching higher sparsity levels where error materially increases --- and high sensitivity to pruning details is typically exhibited). Some concerns with respect to the generalizability of these results also remain.

However, with the updated discussions and revisions as well as the correction of the core claims, the paper rises to the level of acceptance --- assuming addressing the above issues, which the authors demonstrate willingness to do.
I thus accordingly update my score and thank the authors for their efforts in addressing concerns and for the open and constructive discussion and modifications. 

summary_of_the_review: The paper's main claim as presented seems to be false --- it is expected not to hold true that *any* randomly picked subnetwork is transformable to a network as performant as the original dense network if trained. 
For the particular method presented (RST) dependency on cycles leaves utility / insight unclear.

*Updated review in light of author responses [Nov 26, 2021]:* see above update. claims corrected, RST novel contribution, experiments for high sparsity in order to address high-sensitivity regime expected.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new pruning method that resembles distillation - keeping pruned connections within the pruned network and using them when generating the models' output. The method works better than the currently popular topic - LTH, and works better than some pruning-at-initialization methods such as GraSP. 
main_review: Pros: 

- The method is well-motivated. The sparse training techniques are now more and more important as the model sizes are increasing.
- The paper is overall clearly written, and the concept is easy to understand.
- The experiment results show the superiority of the proposed method compared to other methods like vanilla LTH. 

Cons:

- I am not sure if this work can be seen as a complement/supplement/extension to the LTH. One key point in LTH is "trained in isolation", which means that the sparse sub-networks should be extracted out. In this work, however, the subnetwork emerges late, and all the connections in models are alive during the training. Also, there is also no such phase as re-training. Based on the comments above, I think this paper is more like a pruning paper aiming at compressing models and training models at the same time. 
- If the above comments are correct, then I think the authors should provide a comparison between existing compressing methods. More specifically, pruning-after-training methods seem to be the correct way to look at. 
- Some notions should be explicitly defined and some need correction. For example, the \bar{\theta^*} was never pre-defined. Also, the second condition in Equation (4) may never be satisfied. Changing "=" to be "\lt" would be better. 

Minor: Some citation formats should have parentheses. Consider using \citep.

 


summary_of_the_review: My final score will be mostly based upon discussion with authors regarding my points outlined above. Currently, I think the paper is somehow off-topic, but I am open to hearing from the authors and willing to change my view if I had made mistakes.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the Lotter Ticket Hypothesis (LTH) and proposes a Dual Lottery Ticket Hypothesis (DLTH). DLTH describes that any ticket in a given lottery pool can be transformed into a winning ticket. The paper uses a regularization-based method for this transformation. The experiments have been conducted on CIFAR10/100 with ResNets, which indicates a consistent empirical result with DLTH. 
main_review: The idea of exploring a transferable LTH is interesting, which is complementary to the original LTH. The empirical study is clean and shows promising results on some datasets. In addition, the paper also compares DLTH with other techniques such as Pruning at Initialization (PI). The general writing is good. 

However, I have a few concerns about the current version:
1. The empirical study is only on small backbones and datasets, how about larger ones (ImageNet, etc.)?
2. The paper only introduces the regularization method to transform the RST to winning tickets. It is better to have a deep dive into what kind of approaches could work. 
3. The paper ignores some related but important works, such as [a,b,c]. It would be better to compare with them, e.g., RigL. 
    a. Rigging the lottery: Making all tickets winners.
    b. The Elastic Lottery Ticket Hypothesis.
    c. Drawing early-bird tickets: Toward more efficient training of deep networks.
summary_of_the_review: Overall, this is an interesting and good paper. If the authors can address my concerns in the rebuttal phase. I will consider making the recommendation for the paper. 

-----------------------------------------------------------------------------------------
Most of my concerns have been addressed by the authors. I change my score and recommendation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends the Lottery Ticket Hypothesis(LTH) to a more challenging case and proposes the concept of Dual LTH: Any randomly selected subnetwork of a randomly initialized dense network can be transformed into a winning ticket. And the authors propose Random Sparse Network Transformation(RST) to accomplish the transformation process. Experiment results on CIFAR-10/00 and part of ImageNet with ResNet-18/56 demonstrate the effectiveness of the RST and validate the Dual LTH.
main_review: Strength:

1. The concept of the Dual Lottery Ticket Hypothesis is interesting and worth studying.
2. The construction of the paper is clear and easy to follow. 

Weakness:

1. As formulated in the abstract, the DLTH demonstrates that any ticket in a given lottery pool can be transformed into a winning ticket. And the definition of winning tickets is a subnetwork, which can be trained from scratch and match the performance of the dense counterpart. But on CIFAR-10/100 and ImageNet with a 50-98% pruning ratio, the performance of all the subnetworks still remains a unignorable gap against the dense model. Personally, I think it is more meaningful to investigate the proposed method in the setting when subnetworks identified by both LTH and DLTH are winning tickets.

2. As suggested in [1], the connections of neurons in individual layers in initial tickets can be totally rearranged without performance drop, and only the layer-wise sparsity level matters. The RST method follows the same layer-wise sparsity ratio derived by GraSP. Is the proposed RST still effective when the layer-wise sparsity level is also randomly arranged?

3. The related work about sparse network training needs to be re-organized, which only tells several works without any logic.

[1] Su et al. Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot, NeurIPS 2020

Minor:

1. The caption of Figure 3 covers part of the y label.
summary_of_the_review: Overall, it's interesting to investigate the proposed Dual Lottery Ticket Hypothesis. However current results are a little bit insufficient to validate the DLTH. Specifically, no winning tickets have been identified in current experiments and whether DLTH can be extended to the setting when layer-wise sparsity level can also to randomly arranged.
review_title: null

Meta-review: I recommend this paper for acceptance but I do so with significant reservations. Since this metareview will be public for all time, I direct this metareview to future readers of this paper so that they can weigh its merits and drawbacks in a clear-minded way.

This paper proposes a "dual lottery ticket hypothesis." For those unfamiliar, the original lottery ticket hypothesis (Frankle & Carbin, ICLR 2019) states approximately that any randomly initialized neural network contains a subnetwork that can be trained in isolation to full accuracy in the same number of steps as the original network. That is, $\forall$ neural networks, $\exists$ a subnetwork such that $Accuracy(Train($subnetwork$)) \geq Accuracy(Train($network$))$ for a standard, fixed training procedure $Train$. (For the sake of posterity, note that this claim was supported on small-scale neural networks but there is not evidence that it holds in general; only that it holds on the state of networks *early* in training. See *Linear Mode Connectivity and the Lottery Ticket Hypothesis* by Frankle et al. 2020.) To support this claim, Frankle & Carbin develop a procedure that finds such subnetworks, demonstrating that they exist in certain settings.

As far as I understand, the dual lottery ticket hypothesis states that, $\forall$ subnetworks of a neural network, $\exists$ a setting of the weights such that $Accuracy(Train($subnetwork$)) \geq Accuracy(Train($network$))$. Like the original lottery ticket paper, this paper shows that such subnetworks exist: it trains the subnetwork with an L2 penalty on all of the weights except those of the subnetwork, allowing them to gradually fade away and leaving a new setting of the weights for the subnetwork that then allows it to train in isolation to full accuracy (like those subnetworks found in the original lottery ticket hypothesis paper).

The reason that I have reservations about this approach is that the subnetwork found by the dual lottery ticket hypothesis procedure contains fully trained weights. This is novel but - to me - much less surprising and interesting: a randomly sparse subnetwork can be set with trained weights such that, after all of the other weights are fully pruned away, it can recover full accuracy. On the one hand, this is almost reminiscent of a standard pruning procedure where the network is both trained and pruned until a sparse subnetwork reaches full accuracy, with the dense network needed for much or all of training. On the other hand, the impressive part is that this can be done with a *randomly selected* sparse network rather than one chosen by a pruning heuristic. To me, that is the most interesting part of the paper. (And, for those readers wondering why specifically this paper is distinct from standard pruning, this is it.)

I wonder about the significance of this finding given that the subnetwork is set by training (not by random initialization or a tiny amount of training as in work on the lottery ticket hypothesis), but it's a novel idea and I think future scholars and future research should be the judge of that significance, not me or the reviewers. The novelty alone merits publication, and we will have to wait and see about the significance. Thus, I weigh in favor of acceptance, although with reservations.
Decision: Accept (Poster)