Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper tries to make improvements on the problem of quantizing transformers, wherein the key problem is the presence of outliers. The paper presents an analysis of the problem and tries to track the origin of outliers and find that the scaling parameter (gamma) in layer norm acts as an outlier amplifier. The paper then presents a method named Gamma Migration, which moves computations associated with the gamma parameter to subsequent layers. This method improves quantization performance against a simple minmax baseline. The key gains in the paper come from token-wise clipping, in which the authors propose coarse to fine grained pipeline to clip the more aggressive outliers.
strengths_and_weaknesses: The strengths of the paper are:

1. The empirical results are considerably strong. The 6-bit PTQ results are the first such high-quality results on GLUE.
2. Given the importance of pre-trained transformer LMs such as BERT, Roberta, the proposed methods can help system developers and deployers. 
3. Each of the methods (although highly applied) is well motivated, with good intermediate quantifications to illustrate their utility.

The paper has the following weaknesses:

1. In general, the paper writing can certainly be improved, e.g. a number of acronyms such as PTQ, QAT are never introduced to the reader. 
2. Limited novelty of the work. The ablation study points out that token clipping is the most important component, and that gamma migration is of limited importance. This does not gel well with the fact that the analysis on the origin of the outliers is presented as a significant work.
questions: 1. The ablation study points to the limited efficacy of gamma migration. Have you conducted an analysis of why is this the case? Given that gamma amplification is presented as the origin of the outlier problem, a conclusive analysis of its impact will be useful.
2. Have you tried the proposed methods (Gamma migration and Token-wise clipping) on GPT-2? 
3. Do you have any comments on whether the work is generic to transformers or is highly applied on the text domain? For example, does the analysis hold on vision transformers or multimodal models?
4. The tasks tackled are only classification tasks, where word-similarity post quantization can give a good estimate of downstream performance. Have you evaluated any generation tasks on BART?
limitations: The authors haven't adequately addressed the limitations and the pain points on reproducing this work. The negative societal impact is irrelevant for this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The papers pursues a better solution to deal with outliers in special tokens that are considered as the critical bottleneck for the quantization accuracy. They identify that it is the \gamma variable in LayerNorm which amplifies outliers and propose an outlier suppression framework, consisting of Gamma Migration and Token-wise Clipping to overcome the quantization bottleneck. Extensive results show that the proposed framework outperforms existing quantization algorithms and for the time time fully recovers full-precision BERT results with a  6-bit post training quantization model and 4-bit model produced by quantization-aware training.
strengths_and_weaknesses: ### Strengths 
- The proposed approach is well motivated by the observation of the layernorm outliers and the experiments are extensive and empirically strong on a wide range of tasks including sentence classification, question answering, and summarization.
- The analysis and visualization are very informative and well presented

### Weaknesses
- One figure is missing from the description
- Some results could be better justified. e.g., It seems that Gamma Migration and Token-Wise Clipping complement each other, but why? How to understand the joint effect of the two techniques and how do they compare to approaches like QBERT or Q8BERT?
questions: - Missing citation: [Compression of Generative Pre-trained Language Models via Quantization](https://arxiv.org/pdf/2203.10705.pdf(https://arxiv.org/pdf/2203.10705.pdf), they developed a dynamic scaling approach and it would be nice to understand how are the observations in two papers are related with each other.
- Figure 2 is cited in texts multiple times but is not present in the paper.

limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This manuscript focuses on Transformer quantization for Bert. It first analyzes the performance regression from outliers during model quantization, specifically in LayerNorm. Following this, the authors propose a Gamma Migration approach to change how we consider the parameter Gamma in LayerNorm, during quantization. In addition, they propose a coarse-to-fine algorithm to find the outliers during token-wise clipping.
strengths_and_weaknesses: Strengths
* The paper is well organized and very easy to follow
* The references are enough
* The technical novelty is enough (see summary)

Weakness
* Although the authors conducted extensive experiments, there are still some key aspects in results missing. See below

questions: * What is the total number of parameters of the models in the experiments? With a fixed size training set, the degradation caused by quantization should be highly related to the number of parameters. Please consider adding these numbers to the tables. If possible, please also conduct experiments on different model sizes to see if the proposed approach is beneficial to Transformer models in GLUE in general.
* For PTQ, what would be the performance of 4-bit quantization? I understand that 4-4-4 PTQ with any quantization paradigm might resulting in garbage results. However, it would interesting to see something like 4-8-8 or 4-6-6. The rationale is that, in real-world scenarios like on-device models, memory usage could be the bottleneck, while the inference speed with 8bit or 6bit are fast enough. In this case, 4bit weight quantization would be very helpful in reducing memory usage.

limitations: The authors mentioned that "In Discussions we leave some topics as future work." However, I would suggest to explicitly summarize them either in a separate discussion section or change conclusions to something like "conclusions and discussions of limitations". Also, I don't think the discussions in the results section covered "the limitations and potential negative societal impact of their work" well enough. Please revise according to NeurIPS requirements.
review_title: null

Meta-review: metareview: This paper proposes an outlier suppression method to improve transformer quantization. The method is derived based on careful analysis and thorough experiments demonstrate the efficacy of it. All reviewers agreed that this is a good paper. I recommend acceptance. 
confidence: Certain
Decision: Accept