Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes the details on how the authors prune FBNetV3 (which is already a compact network architecture) models with existing methods to make them more compact. The resulting models have better FLOP-accuracy tradeoffs than the original FBNetV3.
main_review: Strengths
- This paper is the among the first if not the first to demonstrate that FBNetv3 (a state-of-the-art compact network architecture searched with a sophisticated NAS method) can be pruned to achieve better computation-accuracy performance, than searching for the architecture at the same complexity. It also shows that pruning the larger FBNetv3 model gets to a higher accuracy more quickly than running architecture search.
- The experimental results are good and they show that at different FLOP points, pruned FBNetv3 models always perform better than their original searched counterparts. 

Weaknesses
- The paper has limited novelty and does not present any ideas on its own in terms of the pruning method. It merely uses some existing pruning methods on the FBNetv3 architecture in a straightforward manner.
- It fails to make experimental comparison to NAS methods that jointly search and prune, such as the cited paper `(Wang et al., 2020)`. 
- Only FBNetv3 is evaluated in the paper. How about other compact NAS architectures? It seems like the authors tried with other architectures but find that they do not work well with pruning. It could be that the claims in the paper are only narrowly applicable to FBNetv3.

summary_of_the_review: This paper has some good empirical contributions but those are all it has to offer. For a paper like this, I wish to see a stronger version of the paper where pruning is generally applicable to all sorts of compact NAS architectures.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: Paper focuses on pruning 1x1 convolutions in FBNet with existing technique (magnitude pruning). Authors tend to answer 2 questions: a) how a larger pruned model compare to smaller model trained from scratch; b) is pruning an efficient way to get NAS model derivatives.  
main_review: Paper is primary focused on empirical results. Applied technique, magnitude pruning, is a well known techniques. 

Two research questions that paper tries to answer are interesting, but have been answered multiple times. 

It is known that pruning a larger model to the level of smaller will give a more accurate model. Researchers have been showing this by pruning Resnets from 101 to 50 to 34 to 18 etc. 

Pruning is more efficient to get a model derivative than running a new NAS targeted to a smaller model, also models like EfficientNet provide a scaling rule that doesn't require to run NAS again for a different computational budget.

For an overview of recent work in pruning please refer to [R1]. MobileNet and EfficientNet are popular model to benchmark a pruning technique. For this kind of study, it is highly recommended to apply multiple pruning techniques to get technique agnostic insights.

R1) Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, John V. Guttag, What is the State of Neural Network Pruning? MLSys 2020
summary_of_the_review: Paper presents an empirical study on a well studied problem. The setup, method and results are not significant and are well known to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper applies conventional pruning-and-finetuning techniques to further compress the networks searched by NAS. The experiments and evaluations are based on the family of FBNetV3. The authors show that by pruning large FBNetV3 model to small one, the accuracy of pruned model may be slightly better than the original target (small) model, achieving better tradeoff between computational complexity and accuracy. Also, the authors show that pruning is more training-efficient than searching by NAS to achieve a compact FBNetV3 model. 
main_review: Pros: 
1) The paper is generally well-written and easy to follow. 
2) The authors provide valuable evaluations to show that pruning can be useful even for optimized network architectures, i.e. those searched by NAS.

Cons:
1) The technical contribution and novelty are limited. This paper only applied previous pruning methods to the network obtained by NAS and made a series of evaluations and comparisons. Technically, this paper didn't make novel contributions. And considering the experimental evaluations, the conclusions are not surprising to me, i.e. they didn't indicate or inspire new scientific problems. So generally, I think the contributions are quite limited. 

2) There are many typos in Table 1. The gains shown in Table 1 seem incorrect. It is strange that the shown numbers are not equal to (accuracy of pruned models - accuracy of baselines). And in the last line of Table 1, the baseline accuracy seems incorrect, i.e. it should be 82.5. 

3) Some details are not clear. For example, in Fig. 1, what are the types of baseline models and corresponding seed networks? For Table 2, what are the selected seed networks? Are the selected seed networks the same in Table 2? 

4) The evaluation is only based on FBNetV3 architecture. I just doubt the generality of the conclusions. 
summary_of_the_review: I made my recommendations mainly considering the limited contribution and limited practical significance made by this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an experimental evaluation of simple pruning techniques applied to modern architectures that are designed to be inherently resource-efficient. It is shown that pruning large models in the FBNetV3 family achieves better accuracy-FLOPS trade-offs than smaller models in the FBNetV3 family. It is also shown that pruning large models is faster than performing NAS to find the smaller models directly.
main_review: The paper is very well written and clear in most aspects. Applying pruning techniques to recent architectures, especially architectures that are designed to be resource-efficient, is a task that is rarely addressed in the literature. The two main contributions as stated in the paper summary above are novel.

My main concern is about the practical relevance of the presented work. The paper is mainly an experimental contribution that shows the potential of reducing the number of FLOPS. However, reducing the number of FLOPS appears to be mostly of theoretical relevance. It is also mentioned in the paper (end of Section 4.3) that FLOPS are not well correlated with latency. I wonder if FLOPS can then be seen as a good proxy for computational footprint and power consumption as mentioned in Section 3.3. From what I know, sparsity in matrices gives a decent practical improvement only at high levels of sparsity. Special sparse-matrix structures typically store positions and values of non-zero entries and, therefore, for small amounts of sparsity the memory footprint even increases when using sparse-matrix formats. This is also supported by Section 4.3 where it is reported that sparsities below 40% do not provide a benefit in terms of latency. Consequently, I do not see a real benefit of the achieved sparsity levels in Table 1 which are often below 40%. In this view, I believe that comparing a small dense matrix with larger sparse matrices having the same number of non-zero entries is unfair. In summary, I doubt that the proposed method of pruning a larger model is "practically advantageous" as promised in the introduction.

The paper only shows experiments where pruning works reasonably well. I am missing experiments where we can see that the pruning breaks down, i.e., how far can we go until the accuracy degrades drastically on some specific model. This is important to study the impact on pruning for architectures that are inherently resource-efficient and to understand how "overparameterized" these models really are.

The second main contribution states that pruning a larger model is faster than finding a smaller model directly by means of NAS. Although the reduction in training time compared with NAS is substantial (approx. 4x), the process still seems to take around 2000--3500 GPU hours. I am not sure whether this amount of GPU hours can be considered as "can be obtained inexpensively with limited resources that are generally available to a machine learning practitioner" as mentioned in the introduction. What hardware was used for these experiments and for the latency experiments.

Minor:
It appears that F_{1x1} in Eq. (1) is the number of FLOPS of *all* 1x1 layers of the whole network. Please emphasize the "all".

summary_of_the_review: The paper is largely experimental and, in my opinion, shows improvements in FLOPS that are not practically relevant. However, the paper promises a few times that the proposed methods provide practical benefits. I am not sure, what to take away of the paper: in my opinion there are hardly any practical benefits and there is no clear takeaway message from the theoretically relevant decrease in FLOPS.
review_title: null

Meta-review: This paper presents an empirical study which shows that pruning FBNets with larger capacity results in a model with higher accuracy than one searched via neural architecture search. The below are pros and cons of the paper mentioned by the reviewers:

Pros
- The observation that optimized architectures such as FBNets can benefit from pruning is interesting.
- The paper is well-written and easy to follow.

Cons
- It is trivially known that training larger model and then pruning it will yield a better performing model, than training a smaller model from scratch.  
- The authors do not propose a novel pruning technique for optimized CNN architectures, and use existing pruning techniques for all experiments. 
- The experimental validation is only done with FBNets on ImageNet, and it does not show when pruning starts to break down. 

All reviewers unanimously voted for rejection, especially since the main “findings” of this paper that compact architectures can be further pruned down for improved accuracy/efficiency tradeoff, and that pruning a larger compact model results in models that outperform smaller models trained from scratch, have been already shown in many of the previous works on neural pruning. In fact, compact networks such as MobileNets and EfficientNets are the standard architectures for measuring the effectiveness of pruning techniques, and thus the contribution of this work reduces down to showing that the same results can be obtained with FBNets. This could be of interest to some practitioners, but is definitely not sufficient to warrant publication.
Decision: Reject