Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for post-training quantization and pruning by primarily building on the Optimal Brain Surgeon (OBS) framework. It achieves this by creating an efficient version of the Hessian-based approach in OBS and applies it to quantization and pruning separately. The quantization approach is called Optimal Brain Quantizer (OBQ) and their pruning approach is ExactOBS. They evaluate their methods in multiple modes, including unstructured sparsity, block sparsity, quantization, and mixed compression. They find their method performs better or equal in most of these modes, with the added benefit in some cases of being layer-wise and more parallelizable. 
strengths_and_weaknesses: === Strengths ===

The authors include a strong motivation for their work especially from the perspective that modern hardware supports quantization and block sparsity.

It seems that in general although exact methods are usually unnecessary in machine learning, this may not be true in the context of post-training quantization and limited calibration data. This provides again a high-level motivation for pursuing an exact OBS solution.

The authors include a variety of evaluation tasks, including classification, detection, and question answering.

Paper written in an organized and clear way.

=== Weaknesses ===

In the end, what matters with compression is the accuracy vs. real-world performance curves. It would be useful to show these comparing against other methods to the extent that they are supported.

questions: As correctly predicted by the authors, it is not  intuitive that one-by-one quantization of weights would give a strong benefit. How much are the (not-yet-quantized) weights being adjusted after the quantization of other weights?

There is a significant body of work on post-training quantization methods that use various amounts of calibration data, including those that use no data at all. These include HAQ, OCS, ZeroQ, QAT, etc. Are these not directly compared against because they are not solely weight quantization? Or do the baselines included claim to outperform them?

The quantized experiment mentioned using a consistent pre-trained model for comparison. Is this true with all the experiments?
limitations: No significant limitations than otherwise mentioned.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the important problem of post-training compression in deep nets, namely pruning and quantization. The paper adopts the Optimal Brian Surgeon framework for the per-layer squared loss objective, and presents efficient and exact solvers for the greedy and sequential Hessian-based pruning algorithm, allowing it to scale to modern deep nets. The authors also propose a quantization variant, Optimal Brain Quantizer, which leverages the same exact and efficient solver and enables accurate post-training quantization. The authors validate the efficacy of their methods across various tasks, networks, and compression strategies, showcasing real hardware benefits. 
strengths_and_weaknesses: Strengths:
- The paper is well written and very easy to follow. Everything was well organized
- The proposed method is simple and scalable, allowing the authors to implement the exact OBS solver at the scale of modern deep nets
- The idea of extending OBS for quantization seems novel to me, and is very interesting
- The experimental results demonstrate that the proposed method is better/competitive with existing post-training methods

Slight Weaknesses:
- The overall novelty of the method is hard to assess. Please correct me if I'm wrong, but my current understanding is that the main contribution is having an efficient and exact OBS solver, but the implementation itself is built on existing/simple ideas. For instance, the theoretical result (Lemma 1) seems to be a direct application of the Woodbury-Sherman-Morrison formula [1] (if so, it should be cited in the text). I like how the paper is written, but I feel some improvements can be made to better highlight the technical contributions, and distinguish it from existing methods/ideas.

- A comprehensive comparison that includes both the setup cost (time to compress), runtime cost (inference time), and task performance (accuracy) is missing in the main text. Such comparisons are vital to improving the quality of the manuscript, and help the reader better asses the performance of the proposed method compared to prior art.

[1] Woodbury, Max A. Inverting modified matrices. Statistical Research Group, 1950.

questions: Please see Weaknesses above

limitations: The authors should mention the limitations of their method, ideally in the last section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a compression framework for post-training quantization and pruning. The paper takes a layer-by-layer compression method to quantize and sparsify model weights and shows that this can be done under a unified setup without retraining a network from scratch. The evaluation on ResNet and BERT models show that the proposed method lead to 2 points drop in accuracy when composing quantization and pruning while obtaining 12x model size reduction. 
strengths_and_weaknesses: Strengths:

- The paper explores the combination of sparsification and quantization from an interesting angle: the optimal brain surgeon framework, and introduces two techniques to induce sparsity and quantization in a layer-by-layer manner. 
- Promising results on combining quantization and sparsification using layer-wise compression techniques without extensive retraining. 
 
Weaknesses:

- The problem definition of the work is a bit confusing. Usually, the problem of compression has been defined as an optimization problem where the optimization objective is to reduce some metrics such as model size or latency while satisfying certain accuracy constraints (e.g., little or no accuracy loss). However, the paper defines the problem as a layer-wise compression problem, which raises a few concerns. First, it includes the solution space of compression as part of its problem definition, e.g., saying that the compression has to be layer-wise, but layer-by-layer compression is only one out of many ways to perform compression, and there is no clear advantage on why one has to use layer-by-layer compression. Therefore, including it in the problem definition seems to artificially reduce the solution space (e.g., in the evaluation, the paper does not compare its method with many existing compression methods that also combine sparsification and quantization). Second, the goal of the problem is to find a "compressed version of a layer weight that performs similarly to the original weights", but this appears to be an indirect optimization target where the paper does not show whether it has a direct correspondence to the metrics people care in practice (e.g., model size, latency, compression cost). The paper might need to provide more justification on the rationale behind defining the problem this way. 
 
- The paper introduces the challenge of compression from a rather indirect angle. There have been numerous work on both pruning and quantization in recent years that target compressing a model. However, instead of talking about the challenges of state-of-the-art approaches, the paper starts with the challenges in the OBS (optimal brain surgeon) framework, which was published almost 30 years ago, and its variations, but none of those appear to have been established as the state-of-the-art for model compression. In contrast, there are actually quite a few more recent work that also performs quantization and pruning together in a unified manner, such as https://arxiv.org/abs/2101.09671 and https://arxiv.org/abs/1811.01907. Therefore, it would be better to more directly compare and position itself with respect to those existing works. Btw, the paper does not cite nor compare those unified compression frameworks. 
 
- The difference between the paper's proposed post-training compression and existing compression methods is not very clear. Notably, popular and state-of-the-art compression methods do not require full retraining, such as quantize-aware training.  Instead, only a small portion of the training data (e.g., <10%) is used to calibrate the quantization errors, as in https://arxiv.org/pdf/1702.03044.pdf. Similarly, for pruning, it is rarely the case to train a sparse model from scratch and again only a small number of iterations are performed to calibrate the model to recover the accuracy, as in https://arxiv.org/pdf/1709.05027.pdf. Since the proposed method still need training data (a subset of training samples) and refinement/calibration training, it is unclear how fundamentally this method has an advantage over existing compression techniques.  


---------------
Post-rebuttal comments

The authors addressed my concerns. I raised my score from 4 to 6 to reflect my post-rebuttal perspective. 
questions: Can you provide a more direct comparison of the proposed unified method with the existing method that also combines sparsification and quantization without layer-by-layer compression? For example, the sparsification can be weight magnitude-based pruning or movement pruning [1], and quantization can be quantize-aware training [2]. 


[1] Sanh et al. "Movement Pruning: Adaptive Sparsity by Fine-Tuning"
[2] Jacob et al. "Quantization and training of neural networks for efficient integer-arithmetic-only inference."
limitations: Yes.
review_title: null

Meta-review: metareview: The reviewers have reached a consensus in favor of accepting this paper, and I agree with this consensus. This is a technically solid paper that makes a good contribution to the field of post-training compression. The issues brought up in the reviews were adequately addressed by the author response, and I expect the final version of the paper will clarify some of the confusion regarding comparison with retraining.
confidence: Certain
Decision: Accept