Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The main contribution of the paper is a new model compression method that simultaneously 1) preserves the original model's per-example decisions, 2) maintains the network's structure, 3) determines the per-layer compression levels automatically, and 4) eliminates the need for retraining the compressed model. The authors demonstrate the efficacy of their method through extensive experiments on CIFAR-10 and ImageNet datasets.
strengths_and_weaknesses: I generally like the approach, and the goal of the paper, and find the analyses theoretically sound. My only hesitation is their claim that preserving the model's per-example decisions would boost fairness and adversarial robustness. If we look at the "average" correlation between the compressed and original model, we will still miss errors in underrepresented data samples. So I am not sure how preserving the correlation brings an advantage over preserving the accuracy of the model in terms of fairness. The authors also mention adversarial robustness. Again, I could not make the connection between preserving the model correlation and adversarial robustness. Can the authors explain the relation and make it more clear in the paper? 

I find the paragraph between lines 47-57 a bit confusing. What does "an independently trained VGG16 model" refer to? 
questions: I listed some questions in the previous section. My biggest concern is that authors use "preserving the model correlation" as one of the main criteria in model compression by claiming that this implies improved adversarial robustness, fairness, and sub-class accuracy. This relation is never made clear in the paper and I don't see why this is true. 
limitations: The authors addressed the limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a pruning method that uses interpolative decomposition of matrices to prune neurons/channels of a model. Generalization bound for the pruned model is provided for the two-layer network case. The method shows good pre-finetuning, post-finetuning quality metrics and good correlations with the un-pruned models.
strengths_and_weaknesses: Strengths:

1. The paper proposes a novel approach to use interpolative decomposition to compress learned linear/convolutional layers and propagate compression corrections to other layers. The idea of using interpolative decomposition is new and solid;

2. The authors provided theoretical analysis on generalization bounds of a compressed two-layer network;

3. The experiments are done carefully and multiple metrics are used to compare relative performance of different compression methods;

4. The authors provided theoretical results on dependence on the number of samples for the two-layer network case. 

Weaknesses:

For deeper networks, the paper didn't offer discussion on dependence on the number of samples.


questions: To successfully approximate the pre-compression layer, it seems that a good pruning data set is crucial. Would it be possible to show some empirical results on the dependence on the number of pruning sample?
limitations: NA
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: This paper proposes a pruning approach based on interpolative decomposition computed by rank revealing QR. The proposed method can automatically determine the layer size. The authors also provide theoretical analysis for the proposed methods.
strengths_and_weaknesses: Strengths:
1. Using interpolative decomposition is new to compression and it is interesting to maintain the pretrained model information with ID.
2. Pruning from a pre-trained model without retraining is practically useful. 
3. The motivation of this paper is good and the theoretical guarantee is significant.

Weaknesses:
1. The experiments are too weak. VGG is an extremely dated model. Results and comparisons for ResNet-50 and MobileNet on ImageNet are required. If authors could provide results on vision transformers, it could be more convincing.
2. Even though the theoretical parts are promising, the practical results are weak. It is not sure whether the proposed post-training pruning method really works. Based on Table 2,  the performance is even inferior to AMC.
3. The metric 'correlation' used in this paper is not a general metric to evaluate the model performance. Better correlation shows nothing.
4. This paper lacks several SOTA references, e.g., [R1-R4].

[R1] Scop: Scientific control for reliable neural network pruning. NeurIPS 2020.

[R2] ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations. ICLR 2021.

[R3] Provable filter pruning for efficient neural networks, ICLR 2020.

[R4] Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition, NeurIPS 2021. 
questions: 1. There is no explanation about ID+LRank. LRank decomposes layers into a low-rank format using an alternating Lagrangian algorithm, learning to compression. How does it combine with the proposed ID-based pruning approach? 
2. What is the performance gap between the pruned model and the pre-trained model? All the results miss baselines.
3. How does this approach perform for SOTA models on ImageNet?
limitations: To make the proposed approach convincing, I highly recommend the authors to provide more general experimental results for fair comparison, e.g., ResNet-50, MobileNet and ViT on ImageNet. I also recommend the authors to not recycle to another top-tier conference without significant revision or improvement.
review_title: null

Meta-review: metareview: The submission proposes an interpolative decomposition scheme for neural network compression to reduce FLOPs and number of parameters at a reduced cost in accuracy/faithfulness to the original model. The authors provide theoretical evidence, albeit in the two layer case, and empirical evidence on a set of architectures and datasets of the soundness of their claims. While the most negative review (12sE) contained several inaccuracies that misrepresented the submission, discussions with reviewers have nonetheless helped clarified several points in their paper to the point that the majority of reviewers were satisfied with the submission. 
Therefore, I recommend this paper for acceptance.
confidence: Less certain
Decision: Accept