Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new pruning algorithm Pruning with Output Error Minimization (POEM), which focus on the problem of "picking which neurons for pruning", and "how to reconstruct the optimal value for unpruned weights".

The paper's main claim is using the value after activation, as the targets and output, to minimize their difference before and after pruning. 
For the proposed method, pruned channel selection and reconstruction are based on this optimization criterion.

The experiments show that using POEM's channel selection and reconstruction, DNNs can have smaller reconstruction error and better classification accuracy.
strength_and_weaknesses: Strength:
- This paper start from an important observation for the problem of how to select the channels for pruning, and how to adjust the values of unpruned weights after pruning.
- The proposed formulation looks solid and reasonable: the author considers both computation complexity and algorithm accuracy in the formulation and algorithm derivation.
- From the experiments, the proposed algorithm show its effectiveness in reducing the output activation L2 error, and also show that ImageNet classification accuracy is better than compared methods, with or without finetuning.

Weaknesses
- On the pruning criterion, although it makes more sense to use the layer's output after activation function than before activation function. However, why not directly use the final output as the error can be accumulated to the final layer and minimizing each layer's output will be sub-optimal. In addition, directly optimizing the final metric: accuracy or loss should be a better choice. E.g., in LeGR[1], the pruning metric is directly set as the accuracy.
- In the experiments, pruning before finetuning is significantly worse than after finetuning (e.g., table 3). It means that finetuning is very important to pruning methods. However, in the experiments, fine-tuning is only performed for 25 epochs, and the effect of different fine-tuning hyperparameters are not comprehensively studied. I wonder if better fine-tuning is conducted (e.g., better lr schedule and longer training epochs, using knowledge distillation etc), the benefits of better channel selection and weights reconstruction is less or more.
- In [2], experiments show that smaller network trained from scratch with similar or even better accuracy than channel pruning. While in this paper, table 4, training scratch has significantly worse accuracy than any pruning methods. Why the experiments in this paper cannot repro what was observed in [2]?

[1] Chin, Ting-Wu, et al. "Towards efficient model compression via learned global ranking." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Liu, Zhuang, et al. "Rethinking the value of network pruning." arXiv preprint arXiv:1810.05270 (2018).
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and easy to understand.

Some experiments and observations in this paper leads to different conclusion from existing paper. Making it's not clear if the "training from scratch" baselines is well set. (more details see `Strength And Weaknesses`)
summary_of_the_review: The paper proposed a pruning channel selection method and show that it is better than some other channel selection methods.
The experiments comparison is not clear on two issues: 1) it's not clear how is the advantage of proposed method if better fine-tuning is conducted; 2) training from scratch baseline in this paper gives different conclusion to an existing reference.

=================================================

Thanks for the authors' response and additional experiments results. From the new results I think the gap between different pruning methods is relatively smaller.
I think it will be a good point that the proposed method is more important on datasets with insufficient samples per classes. However, in the current version of this paper, this point was not proposed as the main contribution. I would suggest the authors to dive deep on this point and show more results in terms of this observation.
Thus I keep the previous score for the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a layer-wise pruning method that conducts pruning and performs reconstruction to minimize the output error caused by pruning. In reconstruction they minimize the output error of the activation function, while the previous methods minimize the error of the value before applying the activation function. Their experimental results show improvement over existing methods.
strength_and_weaknesses: Strength:
1- The paper motivates minimizing the output error of the activation function by illustrating that it is more important to reduce the error for the elements in non-flat zones. In flat zones of these functions, the errors are suppressed. 

2- In neuron selection they use a greedy approach to pick the best neuron next.

3- Since the regression should be performed for each column separately they showed that this can be done by  parallelizing this for each j to obtain the solution efficiently

Weakness:
1- This approach looks similar to the distillation methods where the goal is to minimize the distance between the output distribution of the teacher and student model where the knowledge transfer is on feature level. The paper needs to clarify how different they are.

2- Table 1 shows the layer wise analysis of the channel selection and reconstruction methods for Conv1-1. Even though this results sounds interesting I am wondering how applying this approach in one layer would affect other layers? Also for the result in table 2 does the proposed approach apply all at once (independently) on all layers or it is an iterative approach where one is conditional on the previous pruning. 

3- In both scenarios above I am wondering how the channel selection approach would affect the final result. It is true that it is shown in table 1 proposed channel selection and reconstruction is best (for one layer) but apparently the reconstruction approach improves the result independent of the channel selection. So I am wondering if you use the proposed reconstruction method but use (L1, L2, Lasso, …) for the channel selection for all the layers then what table 2 looks like? This experiment would show the importance of proposed channel selection.

4- The paper mentions that POEM is superior to the previous methods in maintaining the accuracy of the pruned model since it performs reconstruction only for the positive elements, while the previous methods perform reconstruction for all elements including negative ones. I am not sure if this is a correct statement given that this method is data dependent.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and is easy to follow. The proposed approach is considered the output layer after applying activation. A similar approach has been used in model distillation. It is worth extending the related work and mentioning how they are different. 
summary_of_the_review: The paper proposed to minimize the output error after applying the activation function. This approach is very similar to knowledge transfer at the feature level. The paper needs to clarify the similarity and mention how different they are. Also the experimental results need to be improved to show the impact of the channel selection over the existing method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a layer-wise pruning method by minimizing the reconstruction errors of nonlinear outputs. Unlike the previous methods that compute the errors before the nonlinear function, the proposed method does it after the nonlinearity and focuses on reducing the errors in the non-saturating regions. The authors formulate their idea as a weighted least squares method and also introduce an efficient way to solve it. The proposed approach is validated with relatively shallow networks.
strength_and_weaknesses: Strength
- The idea that considers the reconstruction errors of post-activation values is simple yet well-motivated.
- The authors validate the proposed layer-wise optimization under various experimental setups (e.g., at fixed compression ratios, combined with recent compression ratio optimization methods). Furthermore, the results before as well as after finetuning are presented to show the effectiveness of the method.
- The paper is easy to follow and fairly clearly written. I think most members of the community would be able to easily dissect it without requiring substantial prior knowledge.

Weaknesses 
- I am not sure whether good reconstruction of the features leads to (or is actually related to) good final accuracy after finetuning. One may think that because finetuning significantly recovers the performance, how well the features are reconstructed may have less impact. I would recommend the authors to show lower MSEs in Table 1 result in better accuracies after finetuning. Furthermore, I was wondering if the result trend in Table 1 could be also observable in higher layers and/or other networks.
- I am not sure whether omitting f’(Y) when deriving (9) from (8) is okay. I think a key element of the proposed method is applying the derivative of nonlinearity to a typical LS problem of (1), changing it into the weighted LS of (5). However, the omission of f’(Y) in (9) looks similar to the typical LS setup of (1) and may be unreasonable according to the authors’ claim in Section 3.1. Could the authors provide why this approximation is valid?
- The main idea of this paper looks very closer to that of [Jiang, IJCAI’18]: “Taking the widely used activation function ReLU r(x) = max(0,x) as an example, the Euclidean distance can be large between two negative input values, but would become 0 after activation. Thus, nonlinear reconstruction error (NRE), which computes the Euclidean distance between nonlinear activation values of the unpruned model and those of the pruned model, can be a more reasonable metric than LRE when performing layer-wise pruning.” What are the differences and merits of the proposed approach over [Jiang, IJCAI’18]?
    - Jiang et al., Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error, IJCAI'18
- I think the experiments are conducted on relatively shallow networks (i.e., VGG-16, ResNet-18). Recently, many pruning algorithms have been tested on ResNet-50 and/or ResNet-101 [Luo et al., ICCV’17; Liu et al., ICCV’19; He et al, CVPR’19; Li et al., CVPR’22]. I would recommend the authors to present additional results with deeper depth (at least, ResNet-50) to demonstrate the scalability and effectiveness of the proposed method. 
    - Luo et al., ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression, ICCV’17
    - Liu et al., MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning, ICCV'19
    - He et al., Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration, CVPR’19
    - Li et al., Revisiting Random Channel Pruning for Neural Network Compression, CVPR’22

clarity,_quality,_novelty_and_reproducibility: - I think this paper should be properly compared with [Jiang, IJCAI’18] as described above.
- I think additional experiments are necessary to show the scalability and effectiveness of the method (see above).
- The method is introduced mainly for fully connected layers, but it would be better to (briefly) describe its extension to convolutional layers.
summary_of_the_review: I think the main idea is simple and intuitive, but the empirical results are quite weak and some claims should be better supported. I thus find it difficult to argue for acceptance of the work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose Pruning with Output Error Minimization (POEM) technique to improve model accuracy after structured pruning. In order to include activation function in the objective function to be optimized, the authors suggest the weighted least squares method while the authors claim the previous methods minimize the error of the value before applying the activation function. One example is ReLU function when negative inputs are converted into zeros such that such highly non-linear functions need to be included during selecting neurons to be pruned. Throughout various CNN models, POEM significantly outperforms the previous methods when fune-tuning is not employed. Even after fine-tuning, POEM usually yields better top-1 and top-5 accuracy.
strength_and_weaknesses: *Strength
- The need to include activation function is clear (especially as shown in Figure 1)
- Simplified reconstruction strategy is intuitive and computationally reasonable.
- WLS can be combined with various pruning criteria and pruning ratio decision methods.

*Weakness
- Even though model accuracy without fine-tuning is impressive by POEM, after fine-tuning, the impact of POEM seems to be marginal.
- Experiments are limited to old models. What about the chances to apply POEM to Transformers? Is the method specific to vision tasks?
- Limitations of the proposed techniques are not clearly described.
- There are numerous structured pruning methods recently published. Comparison with those methods are not provided (such as SNIP?)
clarity,_quality,_novelty_and_reproducibility: - For post-training quantization, similar approaches have already been introduced. For example, BRECQ and QDrop employ block-level objective function that can include not only activation function but also even neighboring linear layers. Then, an optimizer is employed (instead of analytical solution introduced in this manuscript) to optimize the objective function. Can structured pruning also follow such an approach (i.e., block-level reconstruction based on an optimizer)?
- Why is the proposed method useful for structured pruning? What about applying POEM principles to quantization or fine-grained pruning? What is the motivation to choose structured pruning while POEM itself can be applied to different model compression schemes?
- Can the authors provide differences and advantages compared to recently published structured pruning techniques?
summary_of_the_review: POEM idea itself is quite interesting and intuitive while this reviewer believes that the underlying principles (that can include activation functions in the objective function) have already been considered in other model compression techniques (especially for post-training quantization, e.g., BRECQ). Also, this reviewer is wondering why structured pruning has been investigated in the manuscript while POEM can be extended to other model compression ideas (experimental results are not very impressive after fine-tuning).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a layer-wise pruning method by minimizing the reconstruction errors of nonlinear outputs. Unlike the previous methods that compute the errors before the nonlinear function, the proposed method does it after the nonlinearity and focuses on reducing the errors in the non-saturating regions. The authors formulate their idea as a weighted least squares method and also introduce an efficient way to solve it. The proposed approach is validated with relatively shallow networks.

All the reviewers appreciated the main idea behind the paper, finding it simple and intuitive. On the other hand, they all thought it was falling slightly below the bar for ICLR acceptance. They expressed concerns about the quality of the empirical investigation, as well as coverage of related work, which can be addressed in a revision.
justification_for_why_not_higher_score: See meta-review.
Decision: Reject