Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a zero-shot and training-free quantization scheme for large-scale language-based Transformer models.

Specifically, they investigate the challenges of NLP Transformers and propose group-wise or token-wise quantization for weights and activations, respectively.

Also, the consider an affordable layer-by-layer knowledge distillation algorithm (LKD) without having to access the original training data, which is often not available.

In addition, they incorporate above contribution to build a highly optimized quantization system, which can perform real-device speedups.
strengths_and_weaknesses: Strengths

- I am glad to see this post training quantization method can achieve real device speedups, i.e., not fake quantization with dequantization involved for mimicing the behaviour on general GPU devices.
- The authors are working on large models, which often require a lot of GPU resources to train or retrain. So post training quantization (and data free quantization) is expected. Such a motivation makes a lot of sense to me.

Weaknesses

- The author put a lot of benchmark results while I am somehow get lost how many accuracy drops (a range there will be better) are expected when applying your proposed quantization techniques?
- Will the proposed techniques also be generalized to small Transformer models? How about CNN models, will there are any differences among them when applying the quantization scheme?
- How do you handle the softmax functions in the Transformer models? I assume that it will need more specific step as its quantization will lead to more accuracy degradation as I herad form recent publications.
questions: - How about vision transformers (ViTs), will these family of models also suffer form the same problem and therefore can use the proposed techniques?

- The distillation may require even larger models? Which one are you considering? What if we use smaller models to distill the larger models when applying quantization?

- Any opensource idea for the quantization itself? A miminalist codebase that provide soly quantitzation function will make it more convincing.

- A literature review of current staus of quantization on large-scale language models are expected, e.g., a roadmap of the past few years and which point does this paper fit in.

I am open to further boost the score given good answers/rebuttals.
limitations: No negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The work uses zero-shot row and group-wise quantization that is optimized with fused kernels to perform inference on BERT and large transformers with more than 6B parameters. Layer-by-layer knowledge distillation and post-training quantization is applied to reduce the quantization model degradation.

strengths_and_weaknesses: Strengths:
- a broad range of evaluations that go beyond BERT
- fused functions and open-source software. This takes a lot of effort and is a highly valuable contribution

Weaknesses:
- speedups benchmarks seem unrealistic. Raw GEMM comparisons within CUTLASS are often favorable when comparing Int8 vs FP16 for fused or infused operations, but comparison with cuBLAS usually yields small speedups.
- comparison to row/token quantization and row/token+column quantization are missing (Wu et al., 2020)[1]. This is the state-of-the-art for zero-shot quantization, it should be included for comparison with current methods.
- the paper does not explain the quantization with equations, it is difficult to follow which quantization scheme is actually used. It appears to be absolute maximum or symmetric quantization. But it could also be a form of truncated min-max quantization. A discussion on symmetric vs asymmetric quantization is missing. An evaluation with asymmetric quantization methods is missing (it might address the outliers found in the token statistics)
- It seems evaluation on BERT favored the author's method due to extensive hyperparameter search. The best result from the hyperparameter search was reported and no multiple seeds were run. This information is only found in the appendix but should be in the main paper. In my experience, at least 5 seeds are required for reliable results and it is best to use the same hyperparameters for both baseline methods and newly developed methods. Otherwise, results are often dominated by noise. 

I am happy to increase my score by up to 2 if these weaknesses are addressed and other reviewers do not find other major issues with this work.

[1] Integer quantization for deep learning inference: Principles and empirical evaluation.
questions: - how many seeds do you use for your BERT experiments? For reliable results at least 5 seeds are needed from my experience.
- I benchmarked raw Int8 vs FP16 and I cannot replicate the speedups that you get. What GEMMs are you using? The best comparison is with a recent cuBLAS version (gold standard) vs Int8 CUTLASS (which is very slow for me compared to cuBLAS, but it fast compared to FP16 CUTLASS GEMM)

limitations: The limitations are not addressed. While current GPUs only support Int8/Int4 Tensor cores, a discussion on FP8 transformers is missing.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This manuscript proposes a new post-training quantization framework for large Transformer-based language models (e.g., GPT-3). Specifically, ZeroQuant consists of hardware-constraint group-wise weight quantization and *kernel fusion* based token-wise activation quantization, low-cost layer-by-layer distillation (LGD), and optimized transformer kernels backend support. The proposed framework successfully quantizes the large language models into INT8 or INT4 representation with negligible performance degradation, which results in remarkable compression performance and real-time speedup.
strengths_and_weaknesses: ### Strength

 1. This manuscript is well-motivated, targeting a relevant problem of modern large Transformer-based language model post-training quantization and deployment. The idea of combining group-wise weight quantization under hardware constraints and token-wise activation quantization with the optimized kernel is pretty reasonable. It has its practical value from the modern industrial aspect.
 2. The idea of LKD is simple but effective. The experiments also show its low-cost benefits, and the results of LKD w/o original training data are interesting to me.
 3. This manuscript looks solid with extensive experiments across many benchmarks and advanced large language models like BERT and GPT-3. The real-time speedups on A100 devices are reported.
 4. The overall presentation quality is good.

### Weakness

1. Group-wise weight quantization (GQ) was first proposed by Q-BERT [1], and ZeroQunt directly adopts it but with the hardware constraint, which weakens the technical novelty. As shown in Tab. 9, GQ is the core component that boosts performance. I wonder what the result of the TQ-only is, which is now missing in the ablation study? Do the benefits mostly come from GQ? Please complete the ablation study.
2. The proposed quantization-optimized Transformer kernels are built upon the CUTLASS library [2]. I suggest the authors include more illustrations of the scheduling framework and show some more in-depth study of the benefits brought by the proposed kernel optimization strategy. Besides, since the proposed deployment flow is targeted at GEMM, it will be a limitation for mobile devices which generally have no GPUs but CPUs.
3. I suggest the authors include some visualization results of the quantized models to support the claim that the proposed method can deal with the dynamic range of full-precision activations and weights.

[1] Shen, Sheng, et al. "Q-Bert: Hessian based ultra low precision quantization of bert." *Proceedings of the AAAI Conference on Artificial Intelligence*. 2020.

[2] NVIDIA blog. CUTLASS: Fast Linear Algebra in CUDA C++. https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/, December 2017.

---
#### Post Rebuttal:

I appreciate the detailed rebuttal from the authors. I have carefully read the rebuttal and other reviews, and I have no further concerns except for a conservative view about the technical novelty. As such, I decide to raise my rating to Accept and I believe this paper is a solid paper that will be of interest to the community and industry. I hope the authors could incorporate the valuable evaluations and the reviewers' suggestions into the final version. 


questions: 1. In Tab. 3, why are the results of Q-BERT and MP-PTQ [3] not compared? I think it's better to conduct a fair comparison between these two methods.
2. I wonder whether the authors are willing to release the backend scheduling codes since I think it will be of value to the community and industry.

[3] Bondarenko, Yelysei, Markus Nagel, and Tijmen Blankevoort. "Understanding and Overcoming the Challenges of Efficient Transformer Quantization." *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. 2021.
limitations: See Weakness.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies an important task to quantize very large transformer models leveraging three tricks, namely hardware friendly WA quantization scheme, layer-wise distillation, and an optimized backend. Results are fairly SOTA covering mainstream large nets on comprehensive datasets, with clear ablations.
strengths_and_weaknesses: Strength
+ This is a very solid paper in experimental fronts with all software-hardware perspectives highly optimized to push the frontier further.
+ The evaluation is very thorough.
+ The method scales well across datasets, and scenarios considering dataset access, etc.
+ The paper is very well written.

Weekness
- Though it works great, the deployed methods are known methods and tricks, with very limited new inspiring factors to the literature. 
questions: - For Fig. 2, what does Dequante mean?
- Eqn. 1 LKD, though mentioned but has other losses been tried?
- Table 2, the W4/8 A16 LKD improvement margins are observed smaller than other cases, any specific potential reasons?

Minor
- Though this is an NLP-oriented paper, how will the method scales potentially to ViTs?
limitations: These have been addressed well.
review_title: null

Meta-review: metareview: This paper explores post-training quantization on large transformer-based models, using techniques such as group-wise weight quantization, token-wise activation quantization and layer-by-layer distillation. The paper also provides a system backend support that demonstrate speedup on commercial GPU devices.

Overall, this is a solid paper. The quantization methods used in the paper are not exactly novel. Not only the group-wise and token-wise quantization as pointed out by the reviewers, but also the layer-by-layer distillation using high precision models have been reported recently (e.g. BRECQ [ICLR2021] and AdaQuant[ICML2021]). However, it is appreciated that the authors could take these techniques further to 1) implement and optimize GPU kernels to demonstrate real speedup and 2) evaluate the techniques on large scale models. With the open-source code (as promised), I think the research community and industry can all benefit from this work.

In addition, the paper is well written and easy to follow. The methods are evaluated thoroughly. During the rebuttal period, the authors provide very detailed response to address the questions and concerns raised by the reviewers. Therefore, this paper is recommended for acceptance. 

confidence: Certain
Decision: Accept