Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied the problem of structured Pruning-at-initialization (PAI) on CNNs. It first introduced synaptic expectation (SynExp) as a proxy metric for accuracy. Then, this paper formulated an optimization problem to maximize SynExp for determining the layer-wise pruning ratios, which are subject to model size/FLOPs constraints. The authors observed empirically and prove theoretically that the accuracy of CNN models pruned by PAI depends on the layer-wise density. Experiments are conducted on several CNN architectures like ResNet, MobileNetV2, and EfficientNet, on CIFAR-10 and ImageNet datasets to verify their statement and proposed method.
strength_and_weaknesses: ### Strengths:

   - This paper is generally well-written, and the proposed method is clearly introduced.

   - This paper formulated structured PAI as an optimization problem via maximizing SynExp under model size/FLOPs constraints, which seems interesting.

   - Multiple CNN architectures are used, including ResNet, MobileNetV2, and EfficientNet, on CIFAR-10 and ImageNet datasets.

### Weaknesses:

   - The comparisons in the experimental section are insufficient. This paper stated SynFlow [1] is the current state-of-the-art approach, and this paper only compared to the baseline model and SynFlow without any other PAI methods on each individual architecture. However, SynFlow was published in 2020 which was two years ago. There must be some new PAI papers in the same field. It’s better to compare with them in this paper. This is necessary to examine the advancement and advantages of the proposed method over other competitive approaches.

[1] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05467, 2020.

   - In Figure 2, it seems the layer-wise densities of the proposed method are close to SynFlow and have no significant advantage over it. The results in Table 1 also reflect this.
	
   - In the experimental part, since ResNet-50 is a more typical architecture than ResNet-34, it’s better for the authors also to provide results on ResNet-50, so that we can have an intuitive and straightforward comparison with other strong pruning methods.

clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this paper are acceptable. Novelty is also enough. It is encouraged for the authors to provide the source code for reproducibility.
summary_of_the_review: Overall, this paper has merits in writing, formulation, and method originality. However, the experiments and comparisons in this paper are insufficient. The results also seem not competitive.

My final rating will be based on the authors’ responses to my concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper firstly proposes a metric called SynExp that is an expectation of the SynFlow values over the weights and mask distributions. According to the authors, SynExp is a constant and independent of the pruning granularity so the pruning granularity may not have an influence on the trained models' accuracy. Later on, the authors propose two techniques, PreCrop and PreConfig, that prune the model structurally and adjust the models' parameters, respectively. PreCrop solves a convex optimization problem that results in an optimal structure, while PreCrop optimizes the models' structures in place. Experiments show the empirical performance of the proposed techniques. 
strength_and_weaknesses: Strengths:
- The presentation is generally clear and easy to follow. 
- The methods are conceptually simple yet seem to be effective. 

Weakness:
- Some claims are not well supported. (See below)
clarity,_quality,_novelty_and_reproducibility: Questions:
- Equation (4) says that the expectation on the scores is the same with fixed values of $\{p_l\}$ and is independent of the pruning granularity. The authors then use this fact to claim that the trained CNN is expected to have similar accuracy. Say one score is associated with one trained model accuracy, then how could the mean of the scores be associated with individual models' accuracy? If the variance of the scores is small then it is somewhat arguably, however the authors do not show this point which makes me feel less convinced. 
- Then, the proposed optimization problem in (5), though somewhat empirically effective, is not theoretically justified. Problem (5) is agnostic to the initial weights of neural networks, while SynFlow (and other PAI methods) knows the initial weights, which means they do have additional information on the model training process. The proposed algorithm, however, does not have this information but achieves superior performance without twisting fundamentally the pruning algorithm (i.e., it is still using SynFlow as a scoring function after all). So, it is somehow surprising to me that the proposed algorithm can outperform SynFlow. What is the standard deviation in Table 1? Is it possible that the performance advantage is not statistically significant?
- Following on the previous question, we can come up with an extreme distribution of initial weights to downgrade the performance. For example, the performance is likely to drop if we put $0$ in those un-pruned weights. Solving problem (5) only results in a structure that is agnostic to certain initial weights, 
- It would be better to use a smaller line size in Figure 5 as the two lines are overlapping a lot. 

summary_of_the_review: I am open to change my score - but currently I feel there are some points make me confused in this paper, so I am giving a 5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed SynExp as a proxy for accuracy and format pruning at initialization as an optimization problem to maximize SynExp. Based on this, they proposed PreCrop for channel pruning and PreConfig for channel optimization, both at the initialization. Experimental results show the proposed method can further improve existing models like EfficientNet and MobileNetV2 in terms of accuracy vs. model size trade-off. 
strength_and_weaknesses: Strength:
1. Firstly, the paper is generally well-written and easy to follow. 
2. The proposed heuristic to estimate the network's performance outperforms existing ones on the ablation benchmark.
3. Compared to the existing PAI method SynFlow, the proposed method has a better FLOPs/params vs. accuracy trade-off at an even coarser granularity. 

Weakness:
1. Apart from PAI methods, the proposed method uses a very similar setting as zero-shot neural architecture search (NAS) (e.g., [a, b, c]) when only the channel dimension is searched (zero-shot NAS also needs to estimate the performance of a model architecture before training and compare different architectures to pick out a good one). The paper should reference such literature and compare the proposed method with zero-shot NAS, especially the performance estimation function.  We can keep the search space the same (i.e., only the channel dimension) and use the model performance estimator from zero-shot NAS literature for comparison to see if the proposed method is better. 
2. Although the proposed method aims for a training-free setting, it would be better to also include a comparison with post-training pruning results to see how large the gap is.  
3. In the paper, it says, "The SynExp Invariance Theorem shows that the pruning granularity of PAI methods should not affect the accuracy of the pruned model". However, in real cases, the granularity could affect the performance of pruning by a lot (e.g., fine-grained pruning can achieve a larger learning rate). How do we explain the contradiction?

[a] Zhang and Jia, GradSign: Model Performance Inference with Theoretical Insights
[b] Lin et al., Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
[c] Mellor et al., Neural architecture search without training 

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility are good. 
In terms of novelty, the proposed method is quite similar to existing PAI and zero-shot NAS methods, with only the difference in the model performance estimator (i.e., SynExp). However, there is no evidence to show SynExp can outperform existing estimators (only compared with SynFlow, which has been outperformed by recent work like [a]).

[a] Zhang and Jia, GradSign: Model Performance Inference with Theoretical Insights
summary_of_the_review: Overall, the proposed method aims to solve an important problem. However, it lacks a comprehensive comparison to existing PAI/zero-shot NAS work, which makes the contribution less convincing. The core novelty also highly depends on whether SynExp is actually better than existing ones.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied Pruning-at-initialization (PAI). The authors show that the accuracy of CNN models pruned by a PAI method depends on the layer-wise density. They further propose a structured PAI method, PreCrop, to prune CNNs in the channel dimension.
strength_and_weaknesses: +PAI saves training costs compared to standard pruning methods.

+They theoretically show that PAI only depends on the layer-wise density (in theorem 1). 

+Emperical results seem good across different pruning settings, such as weight level, filter level, and channel level.

-The advantage of the proposed method is not always obvious compared to SynFlow.

-The baseline accuracy of MobileNet-V2 and Efficientnet-B0 is much lower than the reported accuracy in their original papers. Authors should double-check their implementations. The results on top of lower baselines are less meaningful.

-Soft channel pruning methods, such as SCP [1], use no or small fine-tuning epochs. It would be better if the authors could compare the proposed method with soft channel pruning methods since their overall budget is similar.

[1] Kang, Minsoo, and Bohyung Han. "Operation-aware soft channel pruning using differentiable masks." International Conference on Machine Learning. PMLR, 2020.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and easy to follow. The idea is novel to some extent. They presented enough details for reproducibility, but the baseline accuracy of MobileNet-V2 and Efficientnet-B0 has some problems.
summary_of_the_review: Overall speaking, the paper provides some insights into PAI, especially the results of theorem 1. However, the low baseline accuracy of MobileNet-V2 and Efficientnet-B0 makes the empirical results less meaningful. In addition, soft channel pruning methods can achieve good performance giving a similar training budget to PAI.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper proposes a method for pruning neural networks at initialization at the channel granularity. 

**Strengths:**
* The paper makes an interesting observation: for the accuracy of existing PAI methods, it is possible to prune at a larger granularity (specifically the channel granularity) without further hurting accuracy. The paper shows this theoretically and empirically. The downside is that this is within a range of the relatively poor accuracy that existing PAI methods get, but it's a nice improvement to at least get that performance in a way that is easy to accelerate.

**Weaknesses:** I work in the pruning-at-initialization space, and I echo several of the reviewer's comments on this paper:
* "SynExp is invariant to weight shuffling and reinitialization" - This statement reflects a fundamental misunderstanding of the literature on PAI. This is implied to be a benefit of the method, but it is instead a flaw. Su et al. and Frankle et al. both show that this is an indication of the poor performance of existing methods like SNIP, GraSP, and SynFlow and the potential inherent difficulty of pruning at initialization. It is desirable to be sensitive to weight shuffling and reinitialization, since that means the method is choosing specific connections and weights that are important. Specific to channel pruning (the focus of this paper), Li et al. ("Rethinking the Value of Network Pruning") show that a variety of then-SOTA structured pruning methods are also invariant to initialization even when pruning after training, so it may be difficult or impossible to do this for any channel pruning, even after initialization.
* One of the reviewers observes that MobileNet and EfficientNet have strangely low baseline accuracies. This is known to affect the ordering of pruning methods (Blalock et al., "What is the state of neural network pruning?"), and this needs to be rectified for the evaluation to carry weight.
* This paper fails to grapple with one of the main challenges with current PAI methods: none of them preserve accuracy at very high sparsities; they all entail an inherent tradeoff between sparsity/FLOPs and accuracy. (This is in contrast to methods for pruning after training, which typically retain accuracy at high sparsities, meaning they are drop-in replacements for existing networks.) The challenge with this tradeoff is that there is no clear way of knowing what a _good_ tradeoff is. Given that this paper does structured pruning, there is actually a good baseline to try: narrower and shallower versions of the same networks (e.g., ResNet-14 or narrower versions of ResNet-20). These are smaller dense networks that may or may not do better than those networks produced by SynExp. If SynExp were to find better dense networks than those produced by simply training a smaller network, that would be a nice win. As it stands, though, there is no way to evaluate whether the tradeoffs found by this paper are worthwhile other than that they are better than a flawed alternative method on a FLOP-basis (on an accuracy basis, the lines are too thick and the y-axis ranges too wide to tell).

**Overall:** I argue in favor of rejection due to the weaknesses noted by the reviewers and others noted above that are common in the PAI literature. At the end of the day, existing PAI methods are quite poor (in terms of performance and in terms of how little aspects like reinitialization, shuffling, and - now - granularity matter). Progress in PAI should address those weaknesses rather than embrace them. The main advance of this paper is to show that PAI results are invariant to the granularity of pruning, which itself is another concerning aspect of the state of affairs when it comes to pruning at initialization.
justification_for_why_not_higher_score: The current literature on pruning-at-initialization is flawed, and this paper reflects many of those flaws.
Decision: Reject