Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method to effectively find structurally sparse winning tickets. It consists of some post-processing techniques that can be added to each round of standard iterative magnitude pruning (IMP) methods. Starting from unstructured sparse sub-networks, the method uses a "re-filling and re-grouping" manner to enforce the formation of structural sparsity.
main_review: # Strength:

The structural sparsity and unstructured sparsity are not new. However, in previous works about the lottery ticket hypothesis (LTH), the winning tickets are typically unstructured. Therefore, the idea of finding structural winning tickets is meaningful, and the existence of structural winning tickets fills in a crucial piece of the puzzle for the LTH. In this work, the authors propose a method to find structural winning tickets. Extensive experiments on various datasets and network structures show the method is effective and efficient.

# Weakness:

1. In some experiments (such as Figs. 3 and 5), the authors report time-saving when using a specific GPU and batch size. The time saving might vary when other GPUs or batch sizes are used. Therefore, I believe it would be better to also include the number of FLOPs in the results.
2. IMP-Refill and IMP-Refill+ generate different results. Why only IMP-Refill is included in Fig. 8?
3. In Figs. 3, 4, and 5, it seems different methods stop at different percentages of sparsity. Why don't they all prune to the same percentage?
summary_of_the_review: This work demonstrate that lottery tickets can not only be unstructured but also be structural, which makes it a novel and interesting work. There are minor issues in experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an exciting new finding – by properly reorganizing elements during the IMP process, it’s possible to achieve structurally sparse winning tickets at high sparsity levels that can be easily hardware accelerated.
main_review: The authors explored hardware-friendly structural sparsity (including channel-wise and group-wise patterns) to find lottery tickets. They propose new refilling techniques based on the classical iterative unstructured pruning process to create channel-wise structural sparsity by growing back the pruned elements within the most important channels and abandoning the rest. 

On top of that, they introduce another regrouping algorithm based on hypergraph partitioning (Rumi et al., 2020) to establish group-wise structural patterns which are more amenable to pruning due to the shape flexibility of grouped blocks. 

The authors reported many experiments on CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet, using various backbones like ResNets, VGGs, and MobileNets. Overall, the structural winning tickets are shown to achieve 53.75% ∼ 64.93% GPU running time savings at 45% ∼ 80% channel- or group-wise sparsity. 

One should note that the sparsity level discussed in this work is not as the usual high as in unstructured tickets (often >95%), but the hardware acceleration is way more apparent. The ablation studies and visualization are thorough. 

I have just a few clarifications or curiosity questions:
- It seems the acceleration effect on the structural ticket (especially after regrouping) requires some specific hardware accelerator and efficient implementations such as GEMM. It is then unclear whether the comparison with vanilla LTH is fair or not: could unstructured tickets also benefit from those? How much performance gain of the reported numbers comes from hardware specialization (Rumi 2020)?
- Whether the same techniques can be applied to lottery tickets found not from random initialization, e.g., how about the pre-trained model lottery tickets as in (Chen et al., 2020b)? Will the refilling/regrouping rely on random weight initialization anyhow?
- How much is the extra overhead of refilling/regrouping compared to the IMP cost?
summary_of_the_review: The paper is in a good shape that displays the first set of positive results bridging the gap between the lottery ticket hypothesis and practical accelerations on real-world hardware. Algorithm details, as well as results, are sufficient and convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: LTH (lottery ticket hypothesis) was mainly proposed for unstructured pruning. This paper shows it can also be validated on structured pruning, for the first time. The key for them to achieve so is the newly proposed post-processing techniques, refilling(+) and regrouping. They show by these techniques, structural winning tickets can be found, with up to 6.67x on hardware platforms.
main_review: Strengths

●	Up to date, LTH has not been shown valid on structured pruning (I mean, filter pruning or channel pruning). This paper is meant to bridge this gap, which could be a significant step forward.

●	They propose several post-processing techniques, which deliver practical speedup. Potentially, they can be very useful.

Weaknesses

1.	One contribution this paper keeps emphasizing is that they find structural winning tickets for the first time. However, some important concepts, baseline results are either not established or (potentially) flawed. I do not think the presented results can faithfully support their claimed contribution.

First, one important property of LTH is that, if the subnet is trained using randomly re-initialized weights instead of the winning tickets, the subnet cannot achieve full accuracy. This paper does not have this kind of comparison at all. For example, Fig. 3, for channel sparsity (refill and refill+, not regroup), what are the accuracies when you train the pruned network from scratch (I mean, fully randomly reinit all the weights in the pruned network)? Without this comparison, we cannot tell if the initial weights can be identified as “winning tickets” in the first place.

Second, the presented results are (potentially) flawed. Many baseline accuracies are far below the average reported by other papers. E.g., in Fig. 3, ResNet50 on TinyImageNet, others typically can get ~67% accuracy (see [1*]), while this paper only got a little bit over 53% at sparsity 0. More than 10% gap.

Similar issue: Fig 3, ResNet50 on ImageNet. The typical top-1 accuracy is around 76.1% (see torchvision models), while this paper reports 75.2%. Nearly 1 point top-1 accuracy drop on ImageNet is quite big. Fig 4, their WRN-32-2 on CIFAR100 only achieves 72%, while even WRN-16-2 (much smaller than WRN-32-2) can achieve 73.26% (see [2*]). Fig 5, their VGG16 on CIFAR100 only archives less than 72%, while VGG13 can achieve over 74% (see [2*]).

I thus deeply doubt the validity of these experiments -- Note, the accuracy of dense models is important in validating LTH because it is the baseline accuracy that subnets are going to compare against. Using a lower baseline accuracy, of course it would be easy to achieve high sparsity “without performance drop”. 

-- I think this is also why in many plots, the accuracy of pruned models can strangely surpass their dense counterparts even at a large sparsity.

2.	They do not really achieve what they claimed.

In pruning literature, structured pruning typically narrowly means filter or channel pruning,  while this paper actually also refers to a more fine-grained sparsity structure (see the “Group-wise Structural Sparse Mask” in Fig. 2). This actually make it much less interesting, because this kind of sparsity still needs customized implementation (“a more sophisticated GEMM-based efficient implementation (Rumi et al., 2020) is needed to accelerate better our refilled / regrouped structural patterns”). How to get practical speedup is actually unknown from this paper (they defer many important details to (Rumi et al., 2020)). 

Then, only look at the channel sparsity. Fig. 3, ResNe50 on ImageNet, Refill and Refill+ start to drop accuracy even at a marginal sparsity. This means they do not find winning tickets. They also admitted this in the paper: “There is almost no difference between the performance of IMP-Refill and IMP-Refill+, and both can not find channel-wise structural winning tickets”. 

What attracted me at the first sight of this paper is that they validate LTH on filter/channel pruning. Unfortunately, they do not really achieve what they claimed. 

3.	From the methodology perspective, the proposed techniques are “post-processing”. Most structured pruning methods are independent from unstructured pruning. While here, they blend them together with “post-processing” instead of proposing a really independent technique to find winning tickets for filter/channel sparsity. I would say, it doesn't look like an elegant neat scheme (although “elegant” may not be really important).

4.	Minor issue. Presentation: Fig. 3, radar plots. It is pretty hard to read the radar plots. In Particular, some dots are overlapping and pretty small. They are barely recognizable in print. They can be presented in a better way.

[1*] https://openaccess.thecvf.com/content_CVPR_2020/papers/Yuan_Revisiting_Knowledge_Distillation_via_Label_Smoothing_Regularization_CVPR_2020_paper.pdf

[2*] https://arxiv.org/pdf/1910.10699.pdf, ICLR 2020

summary_of_the_review: Important baseline results are missing. Experiments are potentially flawed. They claimed what is not really realized in the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work demonstrates the existence of structural winning tickets for the first time by leveraging post-processing techniques.
main_review: It is well known that lottery tickets so far are only successfully found with unstructured pruning. This paper challenges this commonsense, and its positive finding could be of broad interest and impact. Instead of directly adopting structural pruning, the authors propose the postprocessing techniques to accompany IMP, namely, the refilling/regrouping algorithm to form channel-wise and group-wise structural sparsity, respectively. These algorithms are simple and intuitive, yet novel and convincing.  The paper for the first time achieves high sparsity (up to 80%) with structure in the lottery ticket literature and shows practical GPU hardware runtime savings at around 50-60%.

I suggest the authors to tone down their claims. There definitely exist similar works in relevant problem domains, such as sparse training with structures: for example, “Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch” ICLR 2021, should be discussed and compared. 
Besides the scientific interest, I am not sure if I understand from a practical point of view, why we really need particularly pursue a structurally sparse lottery ticket, in contrast to directly using a structurally pruned NN. 
Also, how this work compares to using NAS to search for channel numbers?
I don't understand for ImageNet, why the current results can show "our picking rule (i.e., channel weights l1 norm) provides a great estimation for channel importance"
No theoretical insight on why the proposed algorithm can work is provided. But for an empirical pilot study, this is fair and acceptable.

summary_of_the_review: The contribution of this paper is solid. I hope the authors could clarify the weak points.
review_title: null

Meta-review: ### Summary

This work demonstrates that it is possible to identify lottery tickets with some manner of structural sparsity. The work finds success through refilling (perhaps better termed infilling) and regrouping, two techniques that have found previous homes in other parts of the literature. 

### Discussion

#### Strengths

- Tackles an interesting problem. 

- At face value, I believe the paper achieves its claimed goal, though not with full clarity as written.

#### Weaknesses

- There is room to clarify the atypical form of structure here for readers. The suggested title change would appropriately set expectations. However, refinements in the text as well would be welcomed.

- As I discuss below, the claims should be settled with respect to the strongest baselines.

- Random reinitialization should play a primary role in the presentation of the text. As the authors note, the original lottery ticket paper did not require besting the performance of random reinitialization. However, the random reinitialization results are central to the main figures of the original paper and demonstrate that the result is not merely happenstance. This paper should follow that practice.

### Recommendation

I recommend Reject and I do not do so lightly, given the scores. The work here is promising because finding a path to better-performing lottery tickets remains an open challenge. However, Reviewer YWUh has voiced reasonable concerns about the evaluation methodology. 

I've read the detailed authors' responses and agree with the authors that the presented results may depend on choices in the hyperparameters and training strategy. Having said that, it is critically important for the paper to include in the primary text the strongest baselines available, despite this dependence. On such baselines, the results are worse than originally reported and hence, the primary claims must either be revised to be these results or revised to include these results, with the primary claims providing a range of results. Though the authors offer to make revisions for the camera-ready, the required revisions here are substantial enough to require additional review, which is out of the scope of the current process.

An additional oversight in this methodology -- at least as reported -- is that the primary target of the lottery ticket hypothesis is sparse training, not sparse inference. Evaluating solely the inference performance -- rather than training performance, which includes both the forward and backward pass -- is, therefore, inconsistent with the purpose of lottery tickets. This methodological error will too need to be repaired in the final version of this work.
Decision: Reject