Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The largest language models in practice become resource intensive for inference which interferes with accessibility for researchers or e.g. edge device use cases. This paper offers a new type of weight quantization to int8 data type which enables lower overheads in inference for accessibility with reduced GPU utilization. Although int8 quantization is not new for language models, prior work has been interested in smaller models (e.g. on order of BERT scale). 

One of the findings of this paper was that when such practices were attempted on larger scaled models (they looked at up to 13B parameters), there was an emergence of increased density of extreme outliers found within the weight values which impacted performance of prior quantization schemes. Their solution was to perform what they refer to as vector quantization, which scales the int8 quantization derivation unique to each vector after decomposing between those with prevalent outliers (which are retained as FP16) and those without (which are converted to int8), and after performing matrix multiplication the decomposed vectors are recombined into a common FP16 matrix. Their benchmarks demonstrated that the practice was able to duplicate performance of FP32 at up to 13B parameters, outperforming prior quantization schemes for high parameterization scales.
strengths_and_weaknesses: Strengths

1. Prior language model quantizations required adaptions during training, this approach can be performed in inference independent of training regime.

2. Prior language model quantizations experienced performance degradations a higher tiers of parameterization. This paper demonstrated through benchmarking that inference could be quantized without material impact to performance, demonstrated at scales up to 13B parameters.

3. These methods could make large language models much more accessible to research and industry by way of lowering GPU overhead in inference.


Weaknesses

1. Authors note that the method wasn’t applied to attention layers. Paper should have provided better detail of why this method was insufficient, especially considering that the intended use case is for large language models.

2. The paper introduced some interesting emergent phenomenon at larger parameterization scale (of outlier weight values) but did not attempt to draw context towards other phenomenon found in overparameterization. 

3. I found the reference to what author’s call “outlier feature dimensions” lacked clarity of definition.
questions: 1. Did the researchers consider or attempt to identify whether the emergence of extreme outliers in layers correlated with emergence of sparsity of weights? It seems possible that prior work noting sparsity of deeper models (e.g. Gissen et al, ICLR 2020) may have neglected to inspect for presence of extreme outliers, or for this work vice versa. Such alignment of sparsity and outliers emergence may suggest that it is a shared phenomenon.

2. The scaling methods of the paper appeared of most interest to negating potential for representation overflow in the quantization. Did author’s consider whether scaling based on a maximum value might result in a disproportionate degree of representation underflow? After all if scaling is based on extreme outliers, that suggests that a resulting absolute maximum scaled distribution will likely be “squished” into the lower registers of the data type. (Such distribution width is what is addressed when dividing by a standard deviation in z-score normalization, and wouldn’t be addressed by paper’s noted zero point extension.)

3. I am wondering if scaling derivations might be redundant with similar scaling performed as a part of batch normalization, and if so have authors considered a variant with hard coded scaling assumptions assuming presence of batch norm?

4. Could the authors please clarify what is meant by “feature dimensions h” (247) and how it is derived in comparison to outlier feature dimensions h_i? (I didn’t understand your example of “dimension i=888 for h=1024” (236))

5. I would have been interested in additional detail for context of the noted parameterization thresholds. While I recognize that this is a quantization process for inference, I expect the presence of outlier values arises during training, and various phenomenon from overparameterization (like double descent) are found to arise by parameterization in context of scale of training tokens. (See e.g. Belkin et al 2019 "Reconciling modern machine-learning practice and the classical bias–variance trade-off"). I would be curious if future work may yet identify that the benefit of this form of quantization, as well as the emergence of “outlier feature dimensions”, were found to be impacted by variations of such conditions experienced during training like proximity to the interpolation threshold (which is a function of both parameterization and training sample counts).

6. Does the method only work on low rank tensors as per the demonstration or can it be adapted to diverse configurations?
limitations: Ideally benchmarking could be extended to larger parameterizations exceeding 13B, the authors noted the possibility of additional emergence of feature dimensions. Since this is a method for inference as opposed to training authors may find that such experiments may not be hugely out of reach from a resource utilization standpoint.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a memory-efficient matrix multiplication method, mainly for multi-layer perceptron (MLP) and attention projection.
The method consists mainly of two parts.
One is vector-wise quantization to allow quantization with vector-wise independent scaling.
The other is mixed-precision matrix decomposition to adopt the outliers that often occur in larger Transformer models.
This method can reduce the required memory for inference by half while retaining full precision performance. 
This paper first loads a 16/32-bit standard checkpoint and then converts it to Int8 for inference.
This method requires no post-quantization training.
This paper also reveals that existing quantization methods perform poorly at scale due to emergent outlier feature dimensions. 
Experimental results on LM performance indicate that the result of the proposed method achieved no degradation from the original-size models. 
These results indicate more accessibility to larger models (The authors state that GPT-J and T5-11B can work on a single free cloud GPU, GPT-NeoX-20B on a single gaming-grade GPU, and OPT-30B on a single data-center-grade GPU. 


strengths_and_weaknesses: Strengths:
* The existence of outlier features is already discussed in the previous study, but this paper seems to be the first paper that clearly indicates the reason for poor performance on existing quantization methods due to emergent outlier feature dimensions. 
* The proposed method seems novel and unique.


Weaknesses:
* There are several missing definitions for notations in equations, and seems some errors (See Questions). 
* The Absmax quantization and Zeropoint quantization seem not novel enough as main contribution of this paper since a similar approach has already been used in the literature (e.g., see https://intellabs.github.io/distiller/algo_quantization.html for example.
* The remaining part of the proposed method is Mixed-precision Matrix Decomposition. I feel that this technique is straightforward.

questions: * The definition of O, that is, $O = \\{i|i ∈ \\mathbb{Z}, 0 ≤ i ≤ h\\}$, in Section 2.2 is somewhat strange. Please check it again, and if this is not wrong, then could you show how to compute this set by using some actual examples.

* $ X^{h}\_{f16}$ and $ X^{h}\_{i8} $  in Eq. 8 should be in bold face.

 
limitations: This paper has limitations and potential negative social impact sections.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper explores how we can apply the uniform quantization method to large pre-trained language models. They utilize vector-wise quantization method and decomposition techniques for over 1B models and show the validation ppl of the 13B language model can be not degraded by uniform quantization method. For the 13B model, the decomposition method for outliers can improve the model performance regardless of quantization schemes.
strengths_and_weaknesses: <<Strengths>>
- Since there have been a few studies on large transformer models (over 1B parameters), the results on this paper can be a good reference for many researchers. We can learn the existence of many outliers in big transformer models (Table 2) and the performance degradations in conventional uniform quantization methods (Table 1).
- I think scaling to the 13B model is a big strength of this paper, and the consistent trend to 13B is also a good point of this paper. 

<<Weaknesses>
- Although we can learn many things from Table 1, I have a concern on the experiments in the aspect of the limited evaluation. Many researchers have given attention to the powerful generation capability of Transformer decoders. But, this paper only measured the perplexities, the probabilistic results of language models. I think it seems to be not sufficient to note that there is no performance degradation on generation tasks. How about zero/few-shot evaluation results? With various kinds of tasks, the contribution of this paper would be strengthened.
- I have also another concern on the acceleration of this method, the vector-wise quantization and decomposition method. There is a short explanation about acceleration in Appendix A and Table 3. But I think it is not sufficient to prove there is no acceleration overhead. There is no description on the environmental setup of inference measurement. I think this paper should include some arguments about this issue in detail.


questions: None
limitations: To support their assertions, I think this paper can be refined in two aspects: 1) additional experimental results on performance of generation models, and 2) additional arguments in the acceleration environment and implementation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to quantize GPT style models using INT8 quantization. The paper finds that existing quantization methods perform poorly on quantizing GPT models. The authors identify that this was because of the outliers in feature maps, leading to large quantization errors. To address this issue, the authors propose to use finer-grained vector-wise quantization. The evaluation shows that the proposed method can quantize GPT style models without performance degradation to up to 13B parameters. 
strengths_and_weaknesses: Strengths:

- The paper studies a timely problem and proposes practically useful techniques for quantizing large-scale GPT models.

- The paper evaluates the effectiveness of their approach on some very recent large-scale models such as billion-scale GPT-NeoX, OPT-30B, and demonstrates that they can be deployed on a single GPU with Int8 quantization. 

Weaknesses:

- The paper misses quite some related work on quantization for transformer models, making it difficult to judge the novelty of the proposed method. For example, the outliers in feature maps have already been previously identified such as in "Understanding and Overcoming the Challenges of Efficient Transformer Quantization" by Bondarenko et al., https://arxiv.org/pdf/2109.12948.pdf. Therefore, the findings of emergent outliers on GPT models are not that new. Furthermore, the paper claims that its method does not require quantize-aware training, but again, the post-training quantization method has also been proposed by https://arxiv.org/pdf/2109.12948.pdf. Again, no references or discussions on this prior work. Finally, mixed-precision quantization and finer-grained quantization such as group-wise quantization have been studied in prior work such as https://www.stat.berkeley.edu/~mmahoney/pubs/NeurIPS-2020-hawq-v2.pdf, http://proceedings.mlr.press/v139/yao21a/yao21a.pdf, https://arxiv.org/pdf/1909.05840.pdf, https://www.ijcai.org/proceedings/2021/0472.pdf. Many of these works are either not cited or not properly discussed in the related work. Also, given that many of these techniques have been proposed and studied by prior works, the main contribution of the paper is limited to applying to billion-scale GPT models. 

- The inference latency impact of the proposed method is not very clear. One of the main motivations for having INT8 quantization is that INT8 arithmetic computation is supported on commodity hardware such as Nvidia GPUs. Therefore, it appears to be important to either evaluate the inference latency of the proposed method or at least provide an explanation of the implication of the proposed method to the hardware in comparison to alternative quantization methods, e.g., those used together with QAT. 


----------------
Post-rebuttal comments
The authors addressed my concerns. I increased my score from 5 to 6 to reflect my perspective after the rebuttal. 
questions: Please cite and describe the relationship between this work and prior works on quantization, like those mentioned in the weaknesses section. 

Please provide inference latency measurements or describe the hardware implication of the proposed method. 
limitations: Yes
review_title: null

Meta-review: metareview: This paper provides a solid contribution to compressed ML by showing how to apply 8-bit arithmetic to large-scale transformers. All the reviewers had a borderline or positive impression of the paper, with agreement among the reviewers that the results would be interesting to 
the community especially inasmuch as they represent an empirical exploration of compression on much larger scale models than previous studies. The result about the phase transition is particularly interesting, and may merit future work. While a couple of the reviewers assigned borderline scores, the weaknesses pointed out in the reviews mostly coalesced around questions of novelty and comparison to the literature: while I agree that novelty is important, I think that identifying a simple way to combine and adapt some subset of existing methods to work at new scales _is_ itself a novel contribution. (The authors should, of course, make sure to cite the additional papers mentioned by the reviewers as proposed in their author response.) And the author response does a good job of contextualizing the reviewers' concerns: there is no unaddressed weakness that would amount to a strong justification for rejection. At the same time, all reviewers seem to agree on the strengths of the paper, and this sort of result will be interesting and useful to the community. As such, I recommend acceptance.
confidence: Certain
Decision: Accept