Token-Scaled Logit Distillation for Ternary Weight Generative Language Models
Authors: Minsoo Kim
Sihwa Lee
Janghwan Lee
Sukjin Hong
Du-Seong Chang
Wonyong Sung
Jungwook Choi
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Language Model
Quantization
QAT
Knowledge Distillation
Causal Attention
Language Modeling
TL;DR: Investigate the unexplored challenges of Ternary QAT on GLMs and propose an innovative KD method to overcome them

Abstract: Generative Language Models (GLMs) have shown impressive performance in tasks such as text generation, understanding, and reasoning. However, the large model size poses challenges for practical deployment. To solve this problem, Quantization-Aware Training (QAT) has become increasingly popular. However, current QAT methods for generative models have resulted in a noticeable loss of accuracy. To counteract this issue, we propose a novel knowledge distillation method specifically designed for GLMs. Our method, called token-scaled logit distillation, prevents overfitting and provides superior learning from the teacher model and ground truth. This research marks the first evaluation of ternary weight quantization-aware training of large-scale GLMs with less than 1.0 degradation in perplexity and achieves enhanced accuracy in tasks like common-sense QA and arithmetic reasoning  as well as natural language understanding. Our code is available at https://github.com/aiha-lab/TSLD.

Token-Scaled Logit Distillation for Ternary Weight Generative Language Models
Minsoo Kim!
Sihwa Lee!
Janghwan Lee'
Sukjin Hong 1.2
Du-Seong Chang"
Wonyong Sung² Jungwook Choi'
Hanyang University, Seoul, Republic of Korea
KT, Seoul, Republic of Korea
SSoul National University, Seoul, Republic of Korea (minsoo2333, macto94, hwanii0288, choij}0hanyang. kc.kr {sukjin. hong, dschang)@kt. com, wysungdanu. ac.kr
Abstract
Generative Language Models (GLMs) have shown impressive performance tasks such as text generation, understanding, and reasoning. However, the large model size poses challenges for practical deployment. To solve this problem, MuantizationAAwar Training (QAT) has become increasingly popular. However, current QAT methods for generative models have resulted in a noticeable loss of accuracy. To counteract this issue, propose novel knowledge distillation method specifically designed for GLMs. Our method, called token-scaled logit distillation, prevents overfitting and provides superior learning from the teacher model and ground truth. This research marks the first evaluation of ternary weight quantization-awan training of large-scale GL.Ms with less than 1.0 degradation in perplexity and achieves enhanced accuracy tasks like common-sense QA and arithmetic reasoning well natural language understanding
Introduction
Generative language models (GLMs) have made impressive strides in text generation, understanding, and reasoning, attracting significant attention in the field [177]. However, deploying GLMs remains challenge due to their enormous model sizes. There is rising interest in practical GLMs with less than billion parameters. Their capability can be improved through instruction fine-tuning MM For instance, Alpaca [12] showed that fine-tuned billion-paramett parameter model can match the text generation performance of a 175 billion parameter GLM, highlighting the potential of smaller, more manageable models.
Since practical GLMs still contain billions of parameters, there is extensive research into model compression techniques for their efficient deployment. One such method is post-training quantization (PTQ), which simplifies the process by reducing bit-precision fine-tuning pre-trained GLMs 13177 This approach has gained traction due to its straightforward and fast processing time. However, it's been observed that these techniques cause a significant decrease in accuracy when the parameter count drops below 10 billion or when the bit-precision falls under bits. As a result, there's a clear need for more reliable quantization approach for GLMs with sub-4bit precision.
or 4 bits without the need for
In response, we propose an alternative method, quantization-awarr training (QAT), address the issues PTQ poses for fine-tuned GLMs. QAT is prevalent quantization technique that counteracts
"Corresooning Author
'Our code is available at httpp://gihhbb.co//aaaaaa///////
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

accuracy loss and attains a high compression rate for efficient deployment [18]. Notably, successful fine-tuning of sub-4bit natural language understanding models has been achieved through layer-to- layer (1.2I.) knowledge distillation (KD), a method used to offset errors resulting from aggressive quantization, such as binary or ternary weights [T9] However, applying QAT to GL.M has limited success. While 24 introduced a token-level contrastive loss and [25] offered initial insights into the challenges of quantizing GLMs, both studies encountered substantial increases in perplexity in language modeling. Furthermore, no existing studies apply QAT to GLMs with billions of parameters, primarily due to the expensive nature of training with KD.
This study delves into the fundamental challenges of applying QAT to fine fine-tuned GLMs. We identify two main issues. First, the structure of a self attention map in masked self-attentio causes cumulative quantization errors across tokens. which conventional L2L KD struggles to compensate for. Second, the feacher-forcing mechanism 26 used fine-tuning Transformer decoder necessitates ground-truth loss (GT Loss) factor largely overlooked in previous QAT methods but including GT Loss risks overfitting. Our investigation reveals that logit distillation can overcome the limitations of L.21. KD in token prediction recovery by reforming intermediate representations. Additionally, we found that applying token-wise logit scaling can significantly mitigate the risk of overfitting.
Drawing from our findings, we introduce a novel KD technique known as Token-Scaled Logit Distillation (TSLD), designed to enhance QAT for ternary quantization inference. We evaluate TSLD across a range of GLMs originating from GPT-2 (2). OPT [4] and LLaMA [S] of various sizes, including 7 billion models for the first time. The results show that TSLD achieves comparable, not superior, performance in language modeling on ternary and 4-bit inference. When TSLD is applied to reasoning tasks, it surprisingly prevents overfitting to achieve task accuracy that is at least on par, if not better. These remarkable outcomes underline the potential of our proposed TSLD method in facilitating the deployment of ultra-low precision GLMs.
2 Related Work
Fine-tuning for Generative Language Model. GLMs are renowned for their unparalleled text generation, comprehension, and reasoning capabilities [177] Studies reveal that their performance can be enhanced through instruction fine-tuning methods like Prefix- Tuning [[]] or using natural language instructions and examples
[8]. Fascinatingly, instruction-tuned models, including smaller ones, can outperform larger counterparts in specific tasks 99 R 12). However, the vast parameter count in these models, compared to popular models such as BERT 27 can restrict their practical utility. To mitigate this, we suggest investigating efficient, lightweight techniques for GLMs encompassing up to billion parameters.
Quantization for Generative Language Model. Quantization, a method that minimizes the inference cost of large models by utilizing a limited number of bits to represent weights. has been recently applied to GLMs [3. This process has considerably cut down GPU memory usage and execution time. Two main types of quantization exist, quantization-awarc training (QAT) and post-training quantization (PTQ), which differ in their requirement for re-training or fine-tuning. Although QAT has been effectively used in Transformer encoder models 22 25 its application GLMs poses challenges (24), with observed performance declines when applied to decoder-only models like GLMs 255. This paper assesses the imbalance of quantization errors based on attention traits and language model generation patterns. We demonstrate that QAT can be conducted without substantial performance loss across various tasks, even models exceeding one billion parameters.
Knowledge Distillation for Language Model Compression. Knowledge distillation (KD) is a prevalent transfer learning framework that imparts knowledge from a larger "teacher" model to a smaller "student" model, and it is effectively used to curb accuracy decline in models compressed through quantization-awan
training (QAT) 22 I 23 In encoder models, KD trains the quantized model (student) using the full prreiiiii model's 's (teacher's) s) intermediate representation as the objective in QAT. Despite its effectiveness, this method requires more memory due to the intermediate representation from the teacher model and has been less explored in decoder models such as GLM 25 24 This paper introduces a novel, less memory-intensive KD method applicable to models with up to 7 billion parameters. We offer a thorough analysis of the teacher's information transfer in the decoder model and suggest QAT-based KD method that retains minimal performance degradation, even when applying ternary weights to various GLMs.

---

Background and Challenges
3.1 Transformer Layer
Generative language models [ are built with Transformer layers 29 A standard Transformer layer includes two main sub-modules: Multi-Head Attention (MHA) and Feed-Forward Network (FFN). Input to the 1-th Transformer layer is X₁ E Rexd where n and d are the sequence length and hidden state size, respectively. Let Nи be the number of attention heads and dp d/N н. W W. W E TRdxda are the weight parameters projecting X into Query (Q), = X, W R.) Key ht It ), and Value (V, = X,W)), respectively. The attention score (A,) is computed with the dot product of the projected Query and Key (A, K, The normalized version of this result then passed through the softmax function and multiplied by the Value to get the output as head,- softmax(A,, (A, / /Vd,)VA, VA. Then, the output of the Multi-Head Attention (MHA) is defined as
h
(K,, X,W,
follows:
MHA(X;) =
Concat(heead, head W
(1)
FFN consists of two fully-connected layers with weight parameters W and W
FFN(Y;) = GeLU(Y, w + + +
(2)
Therefore, the operations at the 1-th Transformer layer can be defined as:
Y, = X MHA(LayerNor((X)) X Y₁ +FNN(LayeNoom())) Yi)).
(3)
3.2 QAT with KD for Transformer Decoders
QAT emulates inference- tirrennce-tmmeeeeeo quantization during training to learn parameters robust to the quantization error. In particular, ternary quantization represents all the weight paramcters (W?, W WV, W W W2) into ternary values t E {+1,0, -13 along with a scale fac- tor for sub- 2bit inference at deployment. In this work, we follow the approach of TWN [30] that analytically estimates the optimal a and to minimize ||w - atlls, where w = vec W) and & the number of elements of the weight parameters.
Due to aggressive bit-reduction, ternary quantization causes significant accuracy loss. KD can help compensate for accuracy degradation, where the original full-precision model works as teacher to guide the training of the quantized model as a student. In case of Transformer models, prior works [99 21 22 31) applied KD on every Transformer layer's output activation X₁ well as attention scores A with mean squared error (MSE) loss, denoted as L121:
L+1 Σ
MSE(X X + Σ MSE(A (A)),
LL21
where superscripts S and T represent the student and teacher models, respectively.
The final output logits of the student (Z") and the teacher (Z") are used to compute the cross-entropy loss. Given that N is the total number of the tokens in input sequences, and V is the vocabulary size the language model can recognize and generate. Using the softmax function, we can convert each model's 4mh. token prediction logit output into probability distributions, which are then utilized for the loss for logit distillation( (Taiaatiiooo(naa)
P
7
Llogit
1
p
(pg )
Σ/=1 ch.s
N
The overall loss for KD generally computed as LKD LIa5. + Liogit, without GT Loss as noted
in previous studies 19 22 31. Yet, some methods utilize only Ltogit [24] Our study underscores
the necessity of integrating Liggtt and the GT Loss for an effective application of QAT in Transformer decoders.
3

---

Encoder
Correct
the
force
Bineay or 1)
Token
label
(-B0K)
X
X2
o
Loos 4
Correct Incomrett
Torca
the with
Self-Attention
Decader
Head
Head
Cerrect Incorrect
forre
Token
output
Yds
Y1
Y/
Enjouder layers
layers
Masked
Value
Output
Output
Self-Altention
Qur
input
Xas
X,
X2
X
X3
Token Error by Quantization (Qu.)
Mask
Like
nilk
May
the force be
Token Error Quantization (Q..)
Attention
Encoder
Decoder
Figure 1: (a) Illustration of attention mechanism in the encoder (top) and decoder models (bottom). (b) Left: performing NLU task [32 by encoder model. Right: performing language modeling task by decoder model with teacher-forcing (input token is independent the previously generated token)
3.3 Quantization Challenges GLMs
In this section, we compare the computations Transformer encoders and decoders to deepen our understanding of the fresh challenges that surface within the realm of GLMs.
Cumulative Quantization Errors in Causal Attention. Causal attention, which integrates masking into self-attention to avoid referencing future tokens, is vital for causal language modeling tasks. To comprehend the quantization characteristies of GLMs, we contrast the computation procedures of the self-attention map. For a Transformer encoder, the quantization error reflected on the self-attention map due the process of projecting Query, Key, and Value evenly spread across tokens because of the token- token-parallel nature of computing inherent in Transformer encoders. However, the mask in the causal attention accumulates quantization errors from each token, creating uneven errors in the output computation.
Fig. Taa) contrasts the attention mechanism of the encoder and decoder model under quantization. The encoder's s self-attention illustrated at the top of the figure, decomposed into self-attention and quantization error (Qerr) components. The combined attention probabilities utilized in weighted sum with the Value, where tokens and (in a blue and red box respectively) are affected by an identical number of attention probabilities with quantization error. Conversely, the decoder's causal attention, shown at the bottom, uses only the attention probabilities of the current token and its preceding ones. For instance, the Value for token in the bottom of Fig. [T] (in blue box) uses only two attention probabilities affected by quantization error, while token (in red box) includes those from all preceding tokens. This illustration highlights that causal attention inherently leads to a disproportionate accumulation of quantization errors in the latter tokens. Thus, we need decoder QAT strategy that addresses this imbalance within the causal attention module. In Section 4.1 we assess the limitations of current KD methods in managing cumulative quantization errors and introduce an enhanced logit distillation error compensation mechanism.
Necessity of Ground Truth Loss. fine-tuning, encoder models, often used in Natural Language Understanding (NLU), and decoder models, common in Natural Language Generation (NLG), employ distinct mechanisms for receiving GT Loss, as shown in Fig T(b). Encoder models for NLU tasks use single special token to compute cross-entropy loss with limited number of classes 277, depicted on the left of Fig lib). On the other hand, decoder models in NLG tasks predict each subsequent token, transforming each token's 's representation into a logit vector with class size equivalent to the vocabulary size, often exceeding 50k [I], shown on the right of Fig. bb.. This process allows decoder models obtain GT Loss for each input token, providing detailed token-level prediction information. Given these differences, there is a compelling need to consider the necessity of GT Loss the decoder model's QAT a token-wise manner. However, previous QAT [24 on the decoder models neglects the consideration of GT Loss due to perceived degradation in performance when GT Loss utilized. Accordingly, Section offers an in-depth analysis of the interplay between KD and GT Loss during the QAT.

---

Attention Heatmap
Tolion Attention Hoatmap
1000
12.5
Lagant
10.0
600
-ayer
2.5
50
100
200 Tokon Indox
Tokln Incex
Token Index
(b)
1000
2000
Loge Usit
200
Legit Distil
1000
Atemon Horking Hato
Actooaonn Achoaton Mindee
20
100
S0
200
400
12.
Tokion Index
3000
A.A.A Lad
20
00
100
120
140
Logit Distil
1000
0.0
200
400
600
500
Head Naarierr
Layor Number
Output Cherrnel Inows
(c)
Figure 2: Comparison of weight quantization error on attention map through MSE loss in (a) encoder (BERT-base, RTE task) and (b) decoder (GPT-2, PTB task) model. (c) Left: min-max dynamic range per layer (Logit Distill vs L2L Distill). Middle: attention ranking ratio comparison (FP vs Logit Distill vs L2L Distill). Right: per layer token-wise logit distance and MSE distance
Method
4.1 Logit Distillation for Cumulative Quantization Error
Motivation. The inherent nature of causal attention, where each token representation builds upon the representation of the previous tokens, presents previously unseen challenges when applying quantization to decoder models. For a clearer understanding the decoder model, we conduct a comparative analysis with the encoder model to examine the impact of quantization error the model. In Fig. 2 (a), the quantization error of the encoder self-attention map exhibits a widespread presence of errors due to the absence of masking in self-attention, and the per-token quantization errors along the layers also show irregular patterns depending on the token index. However, in Fig. 2 (b), the heat map of the decoder model reveals an increasing brightness of quantization toward the later tokens. When examining the token index, the phenomenon of quantization errors accumulating toward the later tokens becomes even more pronounced. This previously unconsidered phenomenon of token quantization eттor accumulation in the decoder model is a crucial feature to consider in GLM QAT, Reflecting on this feature, we analyze the effectiveness of prior KD methods for language modeling and explore suitable KD approaches for the decoder model. Analysis on cumulative quantization error for wider variety of GLMs can be found in Appendix A.33
move
Comparison of KD Methods for Decoder QAT. Building on a deeper comprehension of the decoder model, we evaluate the efficiency of current KD methods for QAT in decoders and propose an enhanced KD approach informed by our decoder model analysis. We analyze how two different KD methods, Layer-to-Layee distillation (1.2I. KD) and logit distillation (Logit KD), tackle systematic outliers in QAT [15), using the min-max dynamic range per token and per channel of each layer's intermediate output. As shown in Fig. 2[c) left, both KD methods demonstrate distinct strategies in addressing the teacher model's systematic outliers. While L2L distillation guides the QAT process to mirror the outliers of the teacher model, Logit KD deviates from this pattern, generating new outliers not seen in the teacher model. These outliers consistently emerge in specific channel indices where the teacher model's ' outliers are present. Additionally, to compare the relative token attending order within each QAT model's 's self-attention map, we employ ranking ratio comparison method [22]. This technique conveys the average relative importance of a single token within each attention map. As depicted in Fig. 2 middle, the 1.21. KD method closely mirrors the teacher model's ranking changes. However, the Logit KD method exhibits substantial variation in this ranking shift within certain head range.
5

---

Logt aT TSLD
Low Confidence Region
High Confidence Region
Training Loss
15.0
Per Token CE Loss
10
Per Token CE Loss
Evaluation Loss
6.5
Logit at TSLD
7.5
4.0
5.0
25
Epoch Numbar
Per Token Confidence
Per Taken Entrogy
0.06
Logit
TSLD
Logit
TSLD
Per Token Scale
10-
0.04
0.02
000
0.00
08
30 40
10
20
30
40
Takan Index
Token Indiax
Figure 3: (a) Training/evaluationn loss curve with different QAT-KD methods. (b) token- wise predic- tion statistics scatter plot (left: cross entropy loss and token confidence, right: cross entropy loss and token entropy). (c) Impact TSLD: per-token scale and cross-entropy loss (left: per-token scale, Right: per-token cross-entropy loss). Analysis utilizes OPT-125m for PTB task language modeling. See Appendix A.2 for further analysis on other GLMs
Logit Distillation for Tosen-wise Prediction Recovery. We further analyze the QAT model's token logit distributions. Since token representations evolve along the layers to form the next token's probability 333, we assess each layer's logit distribution and the logit distance from the teacher model. As depicted in Fig- (()) L.2L KD creates token representation that closely mirrors the teacher model both logit distribution and mean-squared error (MSE) distance during mid-layer stages but fails to match the final logit distribution. Conversely, Logit KD, despite diverging from the teacher model's logit distribution in the middle layers, accurately reproduces the final logit distribution. These observations highlight Logit KD's distinct mechanism for token-wiss prediction recovery, managing cumulative quantization error decoder models. In intermediate layers, Logit KD varies the attention values across channels shown in Fig((c), leading to a diverging token representation from the FP model, with this middle stage adjustment acting to counteract accumulated quantization errors in later tokens. Consequently, Logit KD aligns the final logit distribution for each token, crucial for the accuracy of causal language modeling. Therefore, Logit KD, aligning with the characteristics of natural choice for QAT. The subsequent section will delve into
the decoder model, stands previously unexamined issues encountered by Logit KD in decoder QAT.
4.2 Token-Scaled Logit Distillation for Avoiding Overfitting with GT Loss
This section tackles the overfitting problem arising from the combination of Logit KD and GT Loss during QAT. We also investigate probabilistic behavior displayed by the decoder model in language modeling tasks. The study by 24] highlights instances where employing GT Loss and Logit KD adversely impacts the performance of decoder QAT. To understand this issue better, we conduct tests using Logit KD both independently and combine with ground truth loss. As depicted in Fig. (a), overfitting is observed the QAT when both ground truth loss and Logit KD are applied.
Motivation.
Understanding Causes of Overfitting, To better understand the causes of overfitting, analyze the logit output for each token that the teacher model generates during language modeling. From the logit information by the teacher model, we derive the probability distribution (P? softmax )) for ith token prediction. Based on this distribution, we further calculate cross-entropy (-Bn,i (...)...)) confidence score (ma )) and entropy (-E, P.f log(P?)) )) for each token prediction. These

---

metrics unveil a confidence disparity in language modeling trend uniformly observed across decoder models of varying scales. The cross-entropy loss of token prediction logits plotted against the probability confidence score, as illustrated in Fig. F (b), distinctly demarcates the Low Confidence Region (blue box) with low probability confidence and high cross-entropy loss from the High Confidence Region (yellow box) with high probability confidence and low cross-entropy loss. This observation implies a potential overlap between high-confidence Logit KD and the role of GT Loss cross-entropy, suggesting that redundant information from high-confidence Logit KD might mirror the effects of ground truth loss, thereby contributing to the overfitting observed.
Token-Scaled Logit Distillation (TSLD). Based on investigations into the probabilistic relation of token predictions and overfitting in QAT, we propose an adaptive KD method that adjusts Logit KD based on token confidence. This approach utilizes the phenomenon of confidence disparity in token predictions from the teacher model. Our method, called Token-Scaled Logit Distillation (TSLD), de-emphasizes Logit KD for high-confidence tokens to prevent overfitting in QAT while emphasiz- ing Logit KI) for low-confidence tokens possessing with a high entropy probability distributions. Specifically, low-scaled Logit KD (high confidence, low entropy) effectively reduces the overlap with GT Loss, leading to an improvement in overfitting. On the other hand, high-scaled logit KD (low confidence, high entropy) emphasizes the distillation of more informative token prediction distribution from the teacher model, which has rich soft label information.
//
CE =
Σ U log( lo.("" scale, 1=1
(6)
ΣK=1 CCEL/T
LTSLD
scale, x
pT m, m,
log(ps.
(7)
The implementation of TSLD is straightforward. By considering the relationship between token confidence and token prediction cross entropy loss in Fig we can determine the Re token scale values (scale,,) based on the cross entropy loss of the teacher model (CE)) ) using softmax function as shown in Eq. 6 Note that yn.i is a ground truth label where yn,i = 1 if token is the true next token at position 7. and is the temperature parameter for softmax function. The scale of the each token (scale,) is then applied in the logit distillation by multiplying it with the cross entropy loss between the student and teacher models as shown in Eq. 7 As depicted in Fig. (ec) left, the scale values for token-specific Logit KD are determined adaptively based on the per-token cross-entropy loss of the teacher model. In the right graph of Fig-B(c) we can observe that tokens with higher cross entropy loss in the teacher model correspond to higher scale values in the per token scale graph, compared to Logit KD method applying the same scale value (1/N) across all tokens as shown in the left graph of Fig. Cic..
TSLD brings about two significant effects by applying different scales based on confidence disparity, with negligible computational overhead. As shown in Fig. 3a)), TSLD de-emphasizes Logit KD for tigh-confidence tokens, thereby preventing overfitting. Conversely, for now-confidence tokens possessing a high entropy probability distribution, TSLD emphasizes the Logit KD. This action allows the student model to more closely mimic the teacher model's cross-entropy loss seen in the right graph of Fig. 3(c.. A detailed analysis of the TSL. method's computational cost can be found in Appendix A.1
Experiments
5.1 Experimental Settings
In this section, we evaluate the effectiveness of TSLD in the QAT of various sizes of decoder models with sub-4bit quantization. We'v ve set up comparative experiments to demonstrate the proficiency of our TSLD method against existing PTQ and other QAT KD methods. Our findings illustrate that TSLD substantially enhances both the language modeling performance (measured by Perplexity or PPL) and the accuracy in tasks related to reasoning (common-sense QA and arithmetic) and natural language understanding.
Task and Models. We evaluate our proposed method for language modeling (PTB common-
sense QA tasks (PIQA [3), OpenbookQA (36), ARC_eas _aasy (37), ARC_ chcallene 37 and arithmetic
7

---

Quantization
Optimization Method
GPT-2
OPT
Precision
Method
0.1B
0.3B
0.8B
1.5B
0,1B
1.3B
2.7B
6.7B
FP16 baseline
20.91
18.21
15.20
14.26
18.17
13.75
11.43
10.21
PTQ
OPTQ
14
22.41
19.35
17.26
15.86
19.75
14.30
11.82
11.73
Logit 24
20.98
18.54
16.79
15,42
17.60
13.73
11.82
11.20
W4A16
QAT
Logit-GT
21.51
18.58
15.49
14.89
19.63
15.03
12.58
11.78
TSLD
19.95
17.53
15.32
14,50
17.45
13,90
11.59
11,00
L.2++Logit 25
23.79
21.21
17.80
15.82
20.47
17.62
14.67
11.75
Logit 24
22.84
19.87
16.46
15.27
18.86
14.80
12.26
11.33
W2A16
QAT
Logit+GT
23.80
20.20
17.77
16.52
21.62
16,41
13.20
12.41
TSLD
21.74
18.57
16.14
15.02
18.58
14.60
11.97
11.17
Table 1: Perplexity comparison in GPT-2 and OPT series across various model sizes (0. to 6.7B)
on the PTB dataset with QAT-KD (tensor-wise) and PTQ (channel-wise) quantization methods
PIQA
OpenhookQA
ARC_ean
ScC_cehllenne
GSMMKK
QAT KD Method
(C)
PPL (L)
ACC (f)
PPL (1)
ACC (1)
PPL (1)
ACC (1)
PPL (L)
ACC (*)
PPL (L)
OPT-2.7B FP16
76.71
10.91
49.60
26.16
66.12
7.41
37.20
8.96
20.39
2.07
Logit (24)
74.32
11.69
45.40
29.41
58.92
9.05
31.91
12.38
20.02
2.03
GT+Logit
74.97
12.10
46.20
31.08
58.84
8.66
32,16
12.04
19.56
2.12
TSLD
75.62
11.35
46.81
28.93
59.39
8.12
33.45
11.05
20.24
2.03
Model
GPT-Neo-1..
OPT-6.7B
LLaMA-7B
QAT KD
PTB (PPL)
GSM8K (ACC/PPL)
PTB (PPL)
GSMBK (ACC/PPL)
FP16
17.62 (L)
22.52 (1)
1.89 (1)
8.76
30.25 (1) 1.47
Logit
24
21.01
21.08
1.93
12.22
25.47
1.52
19.27
24.49
26.23
1.52
Table 2: Top: Results for the OPT-2.7B model fine-tuned on common-senss QA and arithmetic reasoning task using various QAT-KD (tensor-wise) methods. Bottom: QAT-KD (channel- -wise) results on language modeling task and arithmetic reasoning task across various GLM models.
reasoning based text-generation task (GSM8K (38 Additionally, our assessment extends to Nat-
ural Language Understanding (NLU) task (GLUE (39),, ensuring comprehensive analysis. Our
benchmark models encompass widely used GLMs, such as GPT-2 (2), OPT [4) GPT-Neo 40 and
LLaMA
41 with various sizes ranging from 0. to 7B parameters.
Fine-Tuning Settings, In fine-tuning the language modeling task, we employ chunk-based pre- processing method: all training datasets are concatenated, then split into shorter chunks defined by imput sequence length. For reasoning task fine-tuning, we utilize a sentence-basee based approach, concatenating each dataset's question and answer parts form new sentences, individually used as the fine-tuning dataset. Detailed hyper-paramett parameter settings and other specifics are in Appendix C.2 Experiments are conducted on an A100-40GB GPU. Our QAT experiments start with models that have undergone task-specific fine-tuning. During quantization, the KD process employs the FP fine-tuned model as the teacher model, while the quantized model acts as the student.
Implementation Settings. We devise a QAT-KD framework that leverages pipeline parallelism with PyTorch Pipe API enabling the training of models with capacities exceeding 1.3 billion. We apply weight quantization to the matrix multiplication layers in each decoder layer of the GLM. We conduct experiments on L2L KD (25 but encounter out-of-memory problems for models with more than 1.3B parameters in A 400-- GPU. This issue is believed to arise due to the requirement for both the teacher and student models store the outputs generated by all of their respective intermediate layers during the knowledge distillation process. GPU memory consumption comparisons for each QAT-KD method can be found in the Appendix
5.2
Evaluation on Language Modeling Task
Table D joutlines the performance comparison of TSLD with leading PTQ and QAT methods [4 24 for language modeling of PTB dataset. For 4-bit weight quantization, OPTQ sees a notable performance drop in GPT-2 and OPT models up to 6.7 billion parameters, aligning with the original paper's 's observations [14] However, QAT methods show lower perplexity due to weight parameters fine-tuned for robust reduced- precisson inference. QuantGPT (24, which exclusively uses Logit KD achieves impressive perplexity, whereas Logit+GT KD sees degradation. Conversely, TSLD offers

---

CoLA
MRPC
SST-2
RTE
QAT KD Method
Precision
(CC(1)
PPL (1)
ACC (T)
PPL
ACC (1)
PPL
ACC (1)
PPL (U)
OPT-1.3B FP16
61,03
1.34
81.92
2,58
94.26
2,00
76.53
3,94
OPTQ
14
54.61
1.36
80,14
2,43
95.07
2.02
56.32
3,96
AWQ
13
13.63
1.45
66.42
3.49
94.26
2.02
54.51
4.72
W4A16
Logit
24
50.76 x2.35
1.36
81.94
+1.48
2.62
93.57 10.23
2.09
75.23 +0.83
4,34
GT+Logit
54.07
20:34
1.34
83,17
$0.51
2.60
93.34 20.22
2.11
75.31 +1.00
4.09
TSLD
56.33
$0.98
1.34
83.33
21.22
2.52
94.05 #0.19
2.04
75.97 #0.31
4.05
Logit 24
48.72 62
1.37
81.62
40.62
2.79
93.08 +0.35
2.11
74.15
#1.36
4.72
W2A16
GT+Logit
6138
1.34
82
20.99
2.65
92.77
#0.28
73.79 a1.16
TSLD
54.47 s1.47
1.34
82.20
20.94
2.63
93,92
2.06
75.31
#0.54
4.36
Table 3: Results for the OPT-1.3B model fine-tuned on GLUE [39]
using different QAT-KD methods with five random seed tests. Channel-wise quantization is applied in both PTQ and QAT-KD.
the lowest perplexity, underlining token-wise scaling method's effectiveness in incorporating GT knowledge. Remarkably, TSLD's performance boost allows QAT models match full-precision performance across various capacity ranges all decoder models.
For 2-bit weight quantization, L2L KD sees significant accuracy degradation, and Logit+GT KD suffers from overfitting. TSL. outperforms Logit KD [24] across all model sizes, maintaining PPI. degradation of no more than 1.0 from the baseline. We also tested the general applicability of TL.S on popular open-sourced GLM models (GPT-Neo-1. 1.3B [40], LLaMA-7B (5)) in language modeling PTB task. Table below indicates that TSLD consistently surpassed the competitor, Logit KD Notably, 2-bit TSLD utilizes simple ternary weight quantization, which is hardware-friendly.
[24]
5.3 Evaluation on Reasoning Task
We assess the effectiveness of our proposed method in commonsense QA (PIQA, OpenbookQA, ARC_easy, ARC_ (eC__hhllengg) and reasoning- text-generation task (GSM8K) employing the LM Evaluation Harness framework from EleutherAl [42.. Given the capacity requirements for reasoning tasks, we use OPT-2.7B/6. 7B and LLaMA-7B models as baselines, rather than smaller models. Presee]s performance comparison of 2-bit weight quantization with different KD methods. In commonsense QA tasks, TSLD consistently showcased the lowest perplexity and, consequently, the highest accuracy, drawing the 2-bit quantization results even closer to FP performance as shown in Table@ttop
Considering the GSM8K task, Table 2 reveals that TSLD outperformed Logit KD in terms of perplexity and accuracy with OPT-2.7B76 7.B and LLaMA models. Notably, while QuantGPT (Logit KD) achieves comparable better perplexity, its reasoning task accuracy is lower, potentially due to insufficient GT information. Conversely, TSLD achieves excellent reasoning accuracy while maintaining competitive perplexity, underscoring TSLD's ability to balance language modeling and reasoning performance through its token-wing wise scaling to avoid overfitting. The generated text sample results of the GSM8K task are provided in the Appendix D
5.4 Evaluation on Language Understanding Task
We fine-tune the decoder model for Natural Language Understanding (NLU) tasks using language modeling approach as illustrated in Fig. bb). In our experiments outlined in Table we compare the performance of the latest PTQ methods (AWQ [3). OPTQ [14) and QAT-KD methods with OPT-1.3B model. For 4-bit quantization, the PTQ technique shows a noticeable degradation in performance compared to QAT results, excluding SST-2 task. TSLD achieves the lowest perplexity and the highest accuracy across all the experiments except SST-2, where its accuracy is in-par with the full-precision case. These findings demonstrate that TSLD can robustify the performance of 4-bit quantized GLMs for various NLU tasks, while 4-bit PTQ may suffer from performance degradation.
In ternary quantization, TSLD consistently outperforms the alternative QAT-KD methods for all the cases, demonstrating its superior performance bridging the accuracy gap with the full-precision cases. Interestingly, ternary TSLD even achieved similar or superior accuracy compared to 4-bit PTQ in many tasks (e-g., CoLA, MRPC, SST-2), highlighting its benefits on both accuracy and memory savings.

---

Model config.
6.7B
13B
175B
Input channel
4096
4096
16384
5120
5120
20480
12288
12288
49152
Output channel
4096
16384
4096
5120
20480
5120
12288
49152
12288
FP32 baseline
0.067
0.201
0.194
0.100
0.285
0.287
0.391
1,472
1.522
8-bit
0.039
0.068
0.068
0.043
0.095
0.092
0.121
0.407
0.395
(x speedup)
x x1.71
x2.93
x x2.83
x2.32
x3.01
x3.13
x3.23
x3.61
x3.85
4-bit 0.030
0.057
0.057
0.041
0.076
0.075
0.096
0.326
0.318
(x speedup)
x2.23
x3.52
x3.40
x2.45
x3.75
x3.82
x4.07
X4.51
x4.78
2-hit 0.025
0.053
0.053 x3.65
0.039 x2.57
0.072
0.064
0.077
0.221
0.232
(x speedup)
x2.68
x3.77
x3.95
x4.48
x5.07
x6.66
x6.56
Table 4: Kernel execution time (msec)
5.5 Ablation Study
Rediced-Precision Kernels Execution Time. We developed custom CUDA kernels to enhance inference speed with applied (2-.4-,--bit quantization. Like OPTQ, packed the weights to minimize the model size and load overhead. Our kernel eliminates the need for weight unpacking during the model forward pass, resulting in speedup shown in Table 4 We tested our kernel mainly on models larger than 6.7B, where weight load overhead is notably high. The reported times the average execution time for 10,000 kernel runs single A100-80GB GPU. For the FP32 baseline, we used Py TyTorch's nn.Linear. As shown in Table our 2-bit kernel for the 175B model can potentially accelerate a single matrix multiplication operation by an average of approximately 6.1 times compared to FP32.
GPT-2
OPT
GPT-Neo
LLaMA
Precision
Granularity
0.1B
0.3B
0.8B
0.1B
1.3B
1.3B
7B
FP16 baseline
20.91
18.21
15.20
18.17
13.75
17.62
8.76
Tensor-wise
21.74
18.57
16.14
18.58
14.60
30.60
12.31
W2A16
Channel-wise
21.30
18.48
15.97
18.42
14.46
19.27
11.60
Table 5: Comparison of tensor-wise and channel-wise quantization across various GLMs (GPT-2, OPT, GPT-Neo and LLaMA). The TSLD KD method is employed in this experiments
Quantization Granularity Impact. To account for output channel weight variations, channel-wise quantization is used [55 [4]. By integrating our QAT-KD approach with channel-wise quantization, we can achieve further performance enhancement. An interesting observation emerges: the gains from channel-wise quantization vary by the type of GLM. As illustrated in Table[5] for the GPT-2 and OPT series, PPL performance increase due to channel-wise quantization is less than 1. However, for GPT-Neo and LLaMA, the performance enhancement effect resulting from channel-wise quantization significantly pronounced. This variation in performance gains suggests distinct channel-wise weight distributions across different GLM models. A detailed analysis of the weight distribution for each GLM is addressed in the Appendix A.4
Conclusion
We introduce token-scaled logit distillation, a new approach for Quantization-Awan Training of Generative Language Models. This method effectively reduces overfitting and enhances learning from the teacher model and ground truth. Importantly, this research is the first to evaluate ternary quantization-awe training on large-scale GLMS, achieving less than 1.0 perplexity degradation and preserving commonsense QA and arithmetic reasoning task accuracy.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & Communications Technology Planning Evaluation (IITP) grangts funded by the Korea government (MSIT) (2020-0-01333, Artificial Intelli- gence Graduate School Program Hanyang University, 2023-RS-2023-0022 2SS2202--00233114 14, artificial intelligence semiconductor support program to nurture the best talents), the National Research Foundation of Korea (NRF) grant, funded by the Korea government (MSIT) (No. 302IRIA2CI0135 2023-00260527), and the Artificial Intelligence Industrial Convergence Cluster Development Project, funded by the Ministry of Science and ICT (MSIT, Korea) and Gwangju Metropolitan City.
No. RS-
References
[1] Alec Radford, Karthik Narasimhan, Tim Salimans, lya Sutskever, et al. Improving language understanding by generative pre-training, 2018.
[2] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[4] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01063 2022.
[5] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 2022.
[7] OpenAI. Gpt-4 technical report. arXiv preprint urXiv:2303.08774. 2023.
[8] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are sero-shot learners. In International Conference on Learning Representations, 2022. URL httt::t///ppnneeeee //ppereeie/ net/forrumidd-gtzrrCoom
[9] Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani. Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-fiettnne language models, 2022.
[10] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprint arXiv:2212. 10560, 2022.
[1I] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 4582-4597 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v12200 1/2021 aacl-long.353. URL
.rg.errg//go 2021 1022..a0 long- 353
[12] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An foslloctionnfollooiing llama model. https: / gi thub. www.totaaaac.oooooo..o conn/tatsullae wtwllaalaaaaaaa sttsssaaorrrrroo _alpacalpaaaapppepp 2023.
11

---

[13] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv. 2023.
[14] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers, In The Eleventh International Conference on Learning Representations, 2023. URL https://opperrevier www. ditprrmmndddtccteffxx.
[15] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. L.m.int8(): 8- bit matrix multiplication for transformers
scale. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 30318-30332. Curran Associates, Inc.. 2022. URL https::/pproceee// http://eeeeaeeeeeeeeeeee.c//pepp httpe//papprfreleeeeeeeeeeeeeeee http:pppww..eeeaappe//////////// httppppppp/////////eeee [aa45550000005444000000006e----0
newr
[16] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023.
[17] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yux- iong He. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Scale Transformers. arXiv, 2022.
[18] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, 2018.
[19] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-aware ultra-low bit bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 509-521, 2020.
[20] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang. Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint 1rX1v 2020.
[21] Jing Jin, Cai Liang, Tiancheng Wu, Liqin Zou, and Zhiliang Gan. Kdlsq-bert: A quantized bert combining knowledge distillation with learned step size quantization, 2021.
[22] Minsoo Kim, Sihwa Lee, Suk-Jin Hong, Du-Seong Chang, and Jungwook Choi. Understanding and improving knowledge distillation for quantization aware training of large transformer en- coders. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Pro- cessing. pages 3-25,, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL BRLLhttp:: /tttt::/acclaaaat http:s/://caaahhllgggggo org/ 2022. e022lemnlpppp....mmmmmmmmmm22m.p emnlp- maain. 450
[23] Minsoo Kim, Kyuhong Shim, Seongmin Park, Wonyong Sung, and Jungwook Choi. Teacher intervention: Improving convergence of quantization aware training for ultra-low precision transformers. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 916-929, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL https:/ htap:p:s//cllannoooog...oo org 2023 macc-mainn.. 64
[24) Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4821-4836, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.18653/v1/20222.ce aww.:o2....111111.333333
UtLhtttps htattp:://aclaanhholgge.. org/ 2022.0222222222 acl- .oog.3311 331
[25] Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quantization for language models: Latency speedup. composability, and failure cases. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedingg of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages PMLR, 23-29 Jul 2023. URL http:://ppoeeeeee.eeooo mlr. www.pprsssvvvee 2rees/v202// ww23k. html www.seesssr.rreeeeeee.e.oeeeeeee
12

---

[26] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-280, 1989. doi: 10.1162/neco.1988 1 1.2.270.
[27] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 11118863/111111111 1423.
URL htttpe///aclaatttoooooooo/////// org/
[28]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1 2015.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
[30] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv: 1612.01664 2016.
[31] Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, and Yuxiong He. Extreme compression for pre-trained transformers made simple and efficient. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, URL
2022. URL https: //tp//reveeeeeeee////ee net aet/forruniddxxxeehhccc00
[32] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 12471, 2018.
[33] Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL /020 2022. ewnlp-main.......
[34] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)311-3330, 1993. URL https:://cclatthooooyg org/ J93-2004
[35] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
[36] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018
[37] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 1XXiv:1803.0545774 2018.
[38] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
[39] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353 353-355. 2018.
[40] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow March 2021. URL hhttps //doi. org 00..0.5881/zeoeooe 5297715
13

---

[41] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, liyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. LJama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307 09288, 2023.
org/ abs 2307 09288
[42] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell. Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, September 2021. URL lattps://dll org/ 5281/ 537 1371628
[43] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023. URL buttps://operreeee wtwp://wwwweet/tooddddddd-daeaee AwOrrPUUU
[44] Dan Hendrycks and Kevin Gimpel. Gaussian enror linear units (gelus), 2020.
[45] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint 1rXiv:1803.08337 2018.
[46] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023.
[47] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104 00400984, 2021.
[48] Noam Shazeer. Glu variants improve transformer, 2020.
[49] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0002 2020.
[50] Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization, 2022.
14

---

A Supplementary Analysis
A.1
Computation Requirements of TSLD
GPT2-0.3B 512
GPT2-0.8B 512
OPT-1.3B 256
OPT-1.3B 1024
Training Method
QAT-KD Method
Speed (iter/sec)
Memory (MiB)
Speed
Memory (MiB)
Speed (iter/sec)
Memory (MiB)
Speed (iter/sec)
Memory
(iter/sec)
(MiB)
QAT
GT
2.03
19663
1.03
36317
5.15
10051
2.83
15167
Logit
1.57
22622
0.81
40989
4.44
11589
2.27
17529
Goggo
1.56
22622
0.81
40989
4.44
11589
2.27
17529
QAT-KD
Log+L
1.51
31462
OOM
0OM
4,28
12143
2.12
25315
TSLD
1.57
22622
0.81
40989
4.43
11589
2.26
17529
Table A1: QAT memory consumption and training speed study for KD method. The results are
reported on the PTB dataset the Ternary QAT-KD of GPT-2 and OPT series models with input sequence lengths ranging from 256 to 1024
The TSLD method integrates token-wise cross-entropy loss with Logit KD, involving two operations as detailed in Eq 7 Specifically, the term E Yn.i log( PT.)) computes the cross-entropy loss from teacher logits( 2, This result, processed through a softmax function, derives the scaling value for each token. Multiplied element-wise with Logit KD term, Σ PT, log(P.,;) it yields boken-wise scaled Logit KD. In fact, TSLD leverages the teacher logits that are pre-computed in Logit KD, circumventing extra memory usage. Furthermore, the associated computations have complexity of O(N), making TSLD's ' overhead negligible for training.
To evaluate TSLD's s efficiency. we detail training speeds and GPU memory consumption for various QAT-KD methods using GPT-2 models in Table Compared Logit-based methods, TSLD maintains speed without extra memory consumption. In contrast, L2L KD stores intermediate activations from both the teacher and student models for KD, resulting significantly increased memory requirements, evident from T left. As model size grows, as evidenced in scenarios utilizing GPT2-Large, memory requirements rise, leading to "Out Memory" errors on an A100- 40GB GPU. These findings highlight efficacy of TSLD, enhancing QAT-KD performance with memory comparable to Logit KD, while L2L KD demands significantly more. Even when the sequence length is extended from 256 to 1024, Table AT right SToLs, maintains the same GPU memory consumption and training speed as the Logit KD method.
A.2 Token Confidence Disparity Analysis
Our analysis of confidence disparity in token predictions, detailed in Section 4.2 specific GLM model. In fact, this observed trend is consistently present across various GLM models. As shown in Fig Al we can distinetly observe the emergence of the Low Confidence Region (blue box) and the High Confidence Region (yellow box) consistently across models: OPT-6.7B(cft), and LLaMA-2-7B(rigge
extends beyond a
Additionally, as shown in Fig AT right, we plot the token prediction's s statistics with varying input sequence lengths of 128 and 5T2. Regardless of the sequence length, the demarcation of confidence disparity remains consistent. This observation demonstrates that the TSLD methodology. grounded in the the probabilistic dynamics of token prediction, can be universally applied across various GLMs.
ALLMA-7B(middae
A.3 Cumulative Quantization Error Analysis with LLM
In this section, we aim to expand our analysis of the cumulative quantization error discussed in Section to GLMs larger than 6B parameters. By implementing 2-bit ternary quantization [30] on the OPT-6.7B and LLaMA-7B models, we assess the attention map quantization error in comparison to the FP model through MSE loss. These errors are visualized using a heatmap plot (Fig. A2 top). and the average attention map loss per token was plotted against each layer (Fig. A2 below). For the OPT-6.7B model, quantization error is measured for the 5th and 15th layers. Regarding the LLaMA-7B model, quantization errors are depicted for input sequence lengths of 128 and 512.
For the OPT-6.7B model at its Sth layer and the I.L.MA7B model with a sequence length of 128, note accumulation of quantization emror towards the latter tokens, discussed in Section 4.1
15

---

OPT-8.7B
LLaMA-7B
LLaM/-2-7B
15.0
14
14
51.
Takan CE Loss
12
12
17
10.0
10
10
7.5
8
6
4
2.5
2
0.0
0-
0.00
0.25
0.50
0.75
1.00
0.25
0.75
1.00
Per Taken Confidence
Per Token Confidence
Token Confidence
12
12
Por Token CE Loss
12.5
10
10
10.0
6
5.0
2
512
128
0
2
B
0
2
Per Token Entropy
Per Token Entropy
Per Token Entropy
Low Confidence Region
High Confidence Region
Figure Al: Scatter plots representing probabilistic relations of token prediction. The top plots show CE loss versus confidence for each token prediction, while the bottom plots plot CE loss with entropy. From left to right: OPT-6. 7B, LLaMA-7B, and LLaMA-2-7B. For LLaMA-2, results for two input sequence lengths (128, 512) plotted. Input dataset is wikitext-2
OPT-8 76
LLaMA-78 Layer)
Length
E12
483
140
Tokoon Indox
3504
200
53
200
3000
302
400
400
120
LayerO
Layer13
Token Attontion Loss
LayerO
Token Attention Loss
Layero
Attention
0.4
Layer17
Layer16
Layar4
Layer2
0.10
Layer3
0.2
Token
0.05
0.5
Token Index
Tokan India
Token Index
Token Indix
Figure A2: Top: Heat map of 2-bit ternary weight quantization error on attention map MSE loss. (a) OPT-6.7B's 5th and 15th layer attention loss. (b) LLaMA-7B attention loss for sequence lengths 128 512. Below: Average per-token attention MSE loss across layers from each attention map loss heatmap
However, as we delve deeper into the layers of OPT-6.7B or introduce longer input sequences to LLaMA-7B, this phenomenon becomes less pronounced. We speculate that this attenuation might arise from the intricate interplay quantization errors as the depth of GLM increases, and the evolving attention patterns associated with varying sequence lengths influencing accumulation of quantization errors. thorough exploration of cumulative quantization errors for larger GLMS will be reserved for future research.
16

---

QPT-677
LLaMA-7B
GPT-Xieo-11
1.5
0.2
Wiigh Daagge
1.0
0.1
0.0
0.6
-1.0
-1.6
1030 20000
2000
4200
0
1000
2000
5000
4000
500
1000
1600
2000
Ourput Channel Index
Out Channel Index
Cutpur Channel Index
(8)
(b)
(c)
Figure A3: Min-Max range of Linear module weights for three types of GLMs per output channel. In the graphs, each color represents the min-max range an each module weight. (a) OPT-6.7B (b) LLaMA-7B (c) GPT-Neo-1.3B.
A.4 Comparison of GLM Weight Distribution
Pre-trained GLMs show a wide variety of weight distributions 443 We examine the Min-Max range of weights for cach linear module across the output channel in various GLM models (OPT, LLaMA, and GPT-Neo) as visualized in Fig. A3 This analysis aims to elucidate the performance disparities observed in Section 5.5 due to quantization granularity (tensor-wise and channel-wise). For the OPT model, we observe that each module exhibits a consistent channel-wise min-max range, which is notably narrow, spanning from -0.2 to 0.2. In contrast, both LLaMA and GPT-Neo showcase much more diverse min-max range across output-channels for each module, with the range itself being significantly broader, approximately from -2 to 2. This diversity in the output channel-specifie min- max range clarifies the performance differences between tensor- wise and channel-wise approaches, as highlighted in Ta Specifically, OPT, which has limited output channel diversity, showed minimal performance differences between tensor-wise and channel-wise methods. Conversely, models like GPT-Neo and LLaMA, characterized by extensive channel diversity, exhibit significantly enhanced performance with channel-wise quantization. These findings suggest that determining the appropriate quantization granularity in QAT, with the aim of minimizing quantization error, necessitates comprehensive understanding of the channel-wise weight distribution of the target GLM.
B
Supplementary Experimental Results
B.1 8-bit Activation Quantization
Table A2 showcases experimental results applying both ternary weight quantization and 8-bit activa- tion quantization (W2A8). We apply min-max quantization for activation quantization in the same way as in 19 24
25 22 taking into account the asymmetric distribution of certain activation
parts. Specifically, asymmetric min-max quantization is implemented in the multiplication of the Query and Key in self-attention and the input activation the FC2 linear layer 3
In W2A8, in line with observations from Section 5.2 L2L KD exhibits substantial accuracy degra- dation than Logit KD. Although Logit + GT performs less optimally than Logit KD due to the previously mentioned overfitting impact, method outperforms the others across all model sizes, thereby underscoring the effectiveness of the TSLD method.
B.2 Clipping Value Exploration in 4-bit Weight Quantization
When adopting QAT method like QuantGPT, which determines the clipping value with a learnable scale factor, it's crucial to initialize the scale factor appropriately to match the weight distribution of the quantized model. the case of the OPT model, a much narrower distribution observed compared to GPT-2, as illustrated in Fig. A4 a). If we set the clipping value (7=1.0) in the same way as GPT we can observe that over 40% weight elements are detrimentally clipped, as shown in Fig. A4 b). To alleviate the destructive clipping phenomenon in 4-bit quantization, we conduct an
'e use activation quantization code in the following repository http:://ithub..omm/nnua http://hwab.com//nnnneonoooottte htaae////ee//msssteeeeaaaaar//
17

---

Optimization
GPT
OPT
Precision
Method
0.1B
0.3B
0.6B
0.IB
1.3B
2.7B
6.7B
FP32 bascline
20.91
18.21
15.20
14.26
18.17
13.75
11.43
10.21
Logi 25
24.88
21.61
20.50
Logit 24].
23.14
20.13
16.59
15.34
19.21
15.28
12.87
11.70
W2A8
Logit+GT.
24.37
20.78
18.01
16.87
21.59
16.58
13,49
12.81
TSLD
22.01
18.83
16.26
15.23
18.92
14.95
12.14
11.43
Table A2: Impact of activation quantization (Ternary weight, 8-bit Activation Quantization results) in
QAT-KD (tensor-wise)
15-
0.4
7
W4A32 PPL())
10
0.8
18.66
Cloppnng
1.0 (QuantGPT)
18.01
82-
GPT-188
1.2
17.58
1.4
17.71
DMT
2.78 1y 1.21
-0.5 0.0
0.5
k2 10
(n)
(b)
Figure A4: (a) Weight distribution GPT-221.5B/OPT--.77 (4th layer, FC-1) (b) We measure the ratio representing how many weight elements were clipped in the FFN-2 layers weight quantization. Upon applying QAT with the original QuantGPT recipe ( = 1.0), we observe that over 40% of values were clipped in OPT-2.7B, significantly higher rate compared to GPT-2-1.5B. (c) initialization exploration PPL results in OPT-0.1B with PTB dataset.
experiment exploring the initial value of the scale in QuantGPT. (y scale determines clipping value in QuantGPT. Detailed quantization implementations
QuantGPT are further elaborated in [C.2) Through exploration of y initialization, we are able to reduce the proportion of clipped weights as in (gg()))b by increasing the initial value of the 7 scale, and consequently, achieving performance improvement as shown in Fig. A4fc). Through exploring 7 scale hyper-parameters tailored to the OPT weight distribution, we manage to fairly compare multiple KD methodologies in 4-bit OPT QAT without the adverse effects of excessive quantization clipping. These experimental results suggest that, when initializing the clip value in the Icarnable clipping QAT method, one should consider the weight distribution characteristics of the target GLM.
B.3 Results of Decoder-Style BERT QAT
In Section 4.1 we discuss the cumulative quantization eπrоr due to the structural feature of the GLM's masked self-attention, and compare the effectiveness of Logit KD and L2L KD in the QAT. In this experiment, we compare distillation methods in the Encoder model (BERT-base (27)) where, due to the absence of masking, the quantization error is evenly distributed among all tokens. According to [22. L2L KD is crucial in the Encoder model QAT KD, and having more layers to distill has proven beneficial for QAT performance.
As explained in Section 3.3 in the Natural Language Understanding tasks of the encoder model, we calculate the cross entropy loss us- ing the representation of single special token (class token, [CLS]) as logits. Drawing from the fact that the decoder model's language mod- eling fine- tine-tuning uses cross entropy loss of all token representations, we attempt use every token's 's final representation outputs as logits in the encoder model and measure cross entropy loss with the teacher model's final token repre- sentation logits and use this loss as Logit KD (we call this KD method "Logit All Token").
Task
RTE
STS-B
MRPC
CoLA
Full precision
73.28
89.24
87.77
58.04
Logit [CLS] Token (Logii KD)
55.59
86.46
82.43
38.60
Logit All Tokens
70.54
87.46
87.03
Logit [CI.S] Token L.21.
(L.2L KD)
Table A3: QAT-KD (tensor-wise) performance with multiple KD options on selected GLUE (39 tasks with BERT-base [27 model.
18

---

As can be seen in Table A3 the Logit All Token method, which utilizes all final token representa- tions as logits, considerably more beneficial for performance than utilizing a single special token's representation Logit (Logit- [CI.S] Token). However, when compared with Logit [CL.S] Token + 1.21. KDD, we found that employing I.2L. KD yields superior performance in the QAT of the encoder model.
This additional experiment reveals that in an encoder model QAT KD where the quantization error is distributed among all tokens, L2L KD, which forces the student model's each layer output to closely mimic that of the teacher model, the most effective distillation method in QAT. This understanding extends our comprehension of how to adjust QAT KD methodologies to accommodate the structural nature of each model.
C Experimental Details
C.1 Model Description
Configuration
GPT
OPT
GPT-Neo 1.3B
LLaMA
0.1B
0.3B
0.6B
1.5B
0.1B
1.3B
2.7B
6.7B
7B
Il of Layers # Hidden Dim
12
36
48
12
24
32
24
768
1024
1280
1600
768
2048
2560
4096
2048
4096
# Head
12
20
25
12
32
32
32
16
Learning Rate (FP)
le.4
le.4
le-4
le.4
le-4
le-4
5e-5
5e-5
le.4
Se.5
Epoch (FP)
3
3
3
3
3
3
3
3
3
Learning Rate (QAT)
1e-4
le-4
le-4
1e:4
1e-4
1e-4
1e-4
5e-5
1e-4
7e-5
Epoch (QAT)
90
60
30
30
90
30
30
10
30
Table A4: Configuration of each pre-trained decoder model with various sizes and
selection for fine-tuning FP and QAT-KD, All experiments consistently batch size of 4, and sequence length of 512 in language modeling fine-tuning
In our experiments, we conduct task specific fine-tuning for various pre-trained GLMs (GPT-2 2
OPT 4. GPT-Neo [40]. and ILaMA (5)) at various sizes (O.1B 7B). The GPT-2 pre-trained model has a vocabulary size (V) of 50257 and employs the GeLU activation function [44). The OPT pre-trained model features a vocabulary size (V) 50272 and uses the ReLU activation function [45). On the other hand, the GPT-Neo pre-trained model has the same vocabulary size (V) OPT and utilizes the new GeLU activation function [46]. It also incorporates Rotary Positional Embedding (RoPE) [47] for positional embeddings. As for the LLaMA pre- trainrained models, they have vocabulary size (V) of 32000 and utilize the SwiGLU activation function [48]. These models also employ RoPE for positional embeddings. For detailed configuration information for each model size, please refer to
Table A4
C.2 Quantization Details
Quantization Aware Training with KD. In order use KD in QAT, we need initialize teacher and student models respectively. The teacher model undergoes task-specific fine-tuning in full precision (FP) based on pre-trained model. The student model then initialized from the teacher model, after which quantization is applied. The hyper-parameter settings of FP fine-tuning and QAT-KI), across various model types and sizes, can be found in Table A4 Furthermore, our experimental implementation utilizes the Huggingface language modeling code basc"
Post Training Quantization. We conduct our experiments of post training quantization with OPTQ and AWQ [3), using the code from original paper respectively [16] We utilize a calibration dataset comprising 128 randomly selected 2048 token segments from the PTB (34) dataset for OPTQ and
Pile 49) dataset for AWQ. To ensure a fair comparison with QAT, we adopt per-channel quantization as our quantization granularity.
https://gihuu..oom/SSS ab/gptt
https///ithub..om/miitta https//ggthbb.coommiiimaaabbblll
19

---

QuantGPT Implementation. In this paper, we primarily draw comparisons with QuantGPT, state-of-the-art methodology above prior works regarding decoder QAT. This approach introduces two main contributions: a module-dependent scaling method and token-level contrastive distillation.
For 4-bit QAT-KI) experiments, we adopt the QuantGPT 24 quantization method (module-dependent dynamic scaling). QuantGPT considers the quantization scale factor as a learnable parameter and optimizes it through QAT. Following the dynamic scaling method of QuanIGPT, we determine the clipping value a for quantization by multiplying the average weight magnitude [wll with a learnable
n scale factor 7, where || |l denotes & norm: = 7. [wlll In this case, the initial value for the 7 is set to I, and the learning rate for 7 is 0.0002.
Upon implementing token-level contrastive distillation, we observe issues of robustness in replicating the token-level contrastive distillation KD method, where incorrect choices in negative sampling could lead to performance degradation X Therefore, To ensure fair comparison, we exclude the contrastive loss from our implementation of Logit KD
AsylC Fine-Tuning for Arithmetic Reasoning Task. In arithmetic reasoning task (GSM8K) fine-tuning, We employ the ALPACA style fine-tuning method [12. proposed for cotruction-followingg demonstration fine-tuning. This fine-tuning method fundamentally employs language modeling approach, as demonstrated in Fig. [b), predicting the next word in a sequence. However, the ALPACA- style fine-tuning process has distinctive characteristic: it transforms the datasets into a format that comprises instruction-respone pairs, as illustrated in Table [A5] We apply this ALPACA-style fine-tuning method to large pre-trained GLMs exceeding 2 billion parameters (OPT-2.7B/6.7BB LLaMA-7B).
D Examples of Arithmetic Reasoning Text Generation
In this section, we examine the QAT KD method on arithmetic reasoning task through a comparison of generation results from the QAT model. The GSM8K dataset serves as benchmark for measuring arithmetic reasoning abilities, and models are expected generate text responses regr-segrysyyvely based on the questions provided. This task requires not only correct mathematical calculations to produce the right answer, but also logical problem-solving process, and the final answer is correct if both the logic and calculations are accurate. In evaluating GSM8K, we employed a greedy decoding strategy for the text generation process.
In Table A5 we can observe that the answers generated by the QuantGPT QAT model appear to make sense at first glance (corresponding to low PPL results in T) but upon closer examination, becomes evident that the necessary problem-solving process and computations are incorrect. Particularly in Question the model writes that it should perform multiplication in the solution process, but actually executes division, leading to an incorrect intermediate result. From there, continues to develop calculations align, incorrect methods are used to derive the intermediate results, eventually leading to wrong answer. In Table A6 it can be seen that in Question 3, the model skips necessary intermediate steps in the problem-solving process, resulting in an incorrect answer. In Question 4, while the solution process is correct, there are errors in the calculation leading an incorrect answer. In contrast, we can confirm that the generation results using our method accurately list the problem-solving process necessary solve the question, just like the ground truth. Moreover, the calculation results are accurate, thereby resulting in the correct answer.
entirely wrong solution. In Question 2, while the solution process and
Such sample comparisons allow to see the limitations of using Logit KD alone in reasoning tasks. As shown in Table A5 and A6 when Logit KD is used solely for QAT, as the low PPL result in Table 2 shows, it appears to well recover the natural text generation ability of the teacher model. However, in the results of the reasoning tasks that require a logical progression of solutions and accurate calculations, deficiencies such as errors in the solution process and calculation errors observed. With our method that effectively combines GT Loss and Logit KD, we achieve the natural text generation capabilities of the teacher model while simultaneously enhancing logical reasoning and accurate calculation abilities through the ground truth during the QAT. Consequently, our method exhibits superior accuracy in the GSM8K benchmark.
Thii issue has been acknowledged in the revised QuantGPT paper
50
20

---

Question Below is an instruction describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. The zookeeper feeds all the apes in the zoo. He orders all the bananas from local farm every 2 months. If the monkeys need 200 bananas, the gorillas need 400 bananas, and the baboons need 100 bananas every month, how many bananas does he need order to last for months? Response:
Ground Truth:
The total number of bananas the prime apes need every month is 200+440-+100 200000000000000000--000000000000
Therefore months, total number hananas they need 700x= <7700"==140- 1400. #*## 1400
QuantGPT::
The needs 200 bananas/monkey monkeys +200+2-400+4000 bananas for the gorillas. It needs 400 bananas/gorilla x gorillas = +400~411600~1600 bananas for the baboons.
It needs 1600 bananas/baboon 100 baboons =1600/1000166v11 bananas for the baboons.
It needs 16 bananas/baboon x 4 baboons = <116#4644=64 bananas for the baboons.
It needs 64 bananas/baboon x 4 bahoons =64*4=256x2566 bananas for the 200.
It needs 256 bananas/2 months <256/2=128> 128 bananas to last for months. #UU## 128
Ours:
The zoo needs to buy 200 bananas/month, 2 months x200+2=400~4000 bananas. The gorillas need 400 bananas/month x 2 months = <<000==800~~800 bananas.
The baboons need bananas/month x months =100=2=200=2000 bananas.
The total number of bananas needed is 400 bananas 800 bananas 200 bananas
400-800+200-1400- 1400 bananas. ##### 1400
Question Below is an instruction describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. The employees Google went on a day trip. buses were rented that have the capacity of holding 60 emplayees. 6 minibusses that can hold 30 employees, and 10 minivans that can hold 15 employees. How many employees can join the day trip? Response:
Ground Truth:
60 x 4 = 26042440:44 employees can ride on the bus.
30 -300==1000~~ employees ride the minibusses.
15 10
150 employees can ride on the minivans.
Therefore 240 180 150 -240--88+1 employees can join the day trip. #**# 570
QuantGPT:
The capacity of each bus is 60 employees /4 buses = +60/4=15515 employees.
emplooyee...
capacity each minibus is employees / minibus <30/6=5>5 employees.
The capacity of each minivan is 15 employees / 10 minivans = -15/10=1.5w1. employees.
The total capacity the buses and minibus 15 employees employees 1.5 employees <<15+5+1.=20022
employees.
The total number of employees who can join the day trip 20 employees + 60 employees <20+60=80~80 employees. #### 80
Ours:
4 buses can hold 60 4 <60 employees.
6 minibusses can hold 30 x 100 employees.
10 minivans hold 15 x 10 = 150 employees. Therefore, 240 + 180 150 =
employces can join day trip. #8##
Table A5: Samples of arithmetic reasoning outputs generated by OPT-6. 7 ternary weight quantized with different methods. Italics part refers the instruction formatting component question.
21

---

Question 3: Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request. On particular week, tow truck pulled ten cars for each of the first three days and then four lewer each of the remaining days of the week. Calculate the total number of cars towed that week. Response:
Ground Truth:
On the first three days, towing ten day. the truck pulled 3*10 <3* 10=30=30 It pulled four fewer cars each day, which is 10-4 = x10-4=6>6 cars on the remaining days.
If it pulled ten for three days and four less the remaining days, then pulled for 7-3 4 days, four each day.
For the four days, the car pulled 6*4= a6*4=24=24 cars.
total number cars pulled that week 24+30 <24+30=54m54 cars.
#### 54
QuantGPT:
On the first three days, the truck towed a total 3*10 =3=10=30 cars.
On the remaining days of the week, it towed total of 10-4 = <<10-==6w6 cars.
total number cars towed that week 30+6 <330+6=36x36 cars.
#### 36
Ours:
On the first three days, the truck pulled 10#3 = ==003=30=300000 cars.
On the remaining days, it pulled 10-4 <110-4=6w6 each day.
The total number of cars pulled on the remaining days is 6*4 = =6*4-24224 In total, the truck pulled 30+24 <30+24=54x54 cars that week.
#0## 54
Question 4: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. Nick, Richard, Jason and DJ cach have paintball guns. DJ has guns, Nick has 10 guns, RJ has gun and Richard has 5 guns. If they were to share their guns equally, how many guns would each of them have? Response:
Ground Truth:
The total number of guns is 8+10+1+5 ==8+10+1+5=24- 24 guns.
They would each 24/4=<24/4=6> guns.
###H# 6
QuantGPT:
DJ guns, Nick guns, RJ has gun and Richard has guns.
For total of 8+10++++ 1+5 <8+ 18+10++ 14+100++5:22 1+5=22> 22 guns.
they were to share their guns equally, each of them would have 22/4 <22/4=5>> guns. ##HH
Ours:
DJ has guns, Nick has guns, RJ has gun Richard has guns.
Total they have 8+10+1+5 <8+10+1+5=24p 24 guns.
If they were to share them equally, each of them would get 24/4 <24/4=6> 6 guns
Table A6: Samples of arithmetic reasoning outputs generated by OPT-6. 7 ternary weight quantized with different methods. Italics part refers to the instruction formatting component in question.
22


Reviews:
[{"review_id": "mrclc8xetP", "reviewer": "Reviewer_EGP4", "summary": "The article introduced a new approach for LLM Quantization Aware Training (QAT).", "questions": "1. It's not clear to me whether table 1 shows the training or the test perplexity for the PTB dataset. \n\n2. Why only results from W2A32 are shown in table 2? Can you please also include W4A32 ones?\n", "limitations": "Need more evaluation results to justify the contributions.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and the proposed idea is described clearly.\n", "weaknesses": "I have some concerns about the evaluation in the paper. I\n\nFirst, the number of tasks used for accuracy/perplexity comparison between float and the proposed method is too small. The authors only evaluate the proposed method on two small datasets. I would encourage the authors to include more datasets in the evaluation, such as those Q&A tasks and SuperGlue tests datasets listed in the GPT-3 paper. This would help to provide more evidence to support the claims made in the paper: the proposed method counteract the issues of accuracy loss. \n\nSecond, the authors only report accuracy and perplexity results for the int2 data type. It would be great for the readers to see the accuracy and perplexity numbers for all data types (int8, int4, and int2).  The authors do provide latency benchmarks for the int8 data type, but they do not report any quality results. I would encourage the authors to report accuracy and perplexity results for all data types so that the readers can have a better understanding of the trade-offs between accuracy, latency, and memory usage.\n\nHere are some specific suggestions for how the authors could improve the evaluation:\n\n1. Include more tasks in the evaluation, such as those Q&A tasks and SuperGlue tests datasets listed in the GPT-3 paper.\n2. Report accuracy and perplexity results for all data types (int8, int4, and int2).\n3. Conduct evaluation to assess the quality of instruction tuning dataets.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback. Responses to your comments are provided below.\n\n**Comment 1) Expanding TSLD Evaluation Across NLU/CSQA Tasks and Models**\n>Weakness 1) “the number of tasks used for accuracy/perplexity comparison between float and the proposed method is too small. The authors only evaluate the proposed method on two small datasets. I would encourage the authors to include more datasets in the evaluation.”\nWeakness 3) “Include more tasks in the evaluation”\nQuestion 2) “Can you please also include W4A32 ones?”\nLimitation) “Need more evaluation results to justify the contributions.”\n\n**Answer 1)**\n\nThank you for the constructive feedback on our paper's evaluation approach. To address the concern about the limited datasets we initially used, we've expanded our evaluation to include 4 additional NLU tasks and 4 CSQA tasks. Furthermore, to ensure the robustness and generalizability of our method, we extended our comparative experiments to other GLM models, GPT-Neo-1.3B and LLaMA-7B. (We added experimental results as best we could, time permitting.)\n\n[R1] Lin et al. “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration”, Arxiv 2023.\n\nThe experimental results are detailed in Tables R1-R5 of the attached PDF. The lessons derived from these experimental results are as follows:\n\n- W4A16 OPT-1.3B on NLU and CSQA (Tables R1,R3): 4-bit TSLD consistently exhibits significantly more robust performance compared to OPTQ and/or AWQ, the latest SOTA PTQ [9][R1] techniques (we used the code published by the authors of corresponding papers). In particular, TSLD achieved the lowest perplexity and the highest accuracy across all the experiments except SST-2, where its accuracy is in-par with the full-precision case. On the other hand, 4-bit PTQ experienced significant accuracy degradation, especially in NLU tasks like CoLA, MRPC, and RTE. These findings demonstrate that TSLD can robustify the performance of 4-bit quantized GLMs for various natural language tasks, while 4-bit PTQ may suffer some catastrophic failures.\n- W2A16 OPT-1.3B on NLU and CSQA (Tables R1,R2,R3): ternary TSLD consistently outperforms the alternative QAT methods for all the cases tested, demonstrating its superior performance in bridging the accuracy gap with the full-precision cases. Interestingly, ternary TSLD even achieved similar or superior accuracy compared to 4-bit PTQ in many tasks (e.g., CoLA, MRPC, SST-2, PIQA, OpenbookQA), highlighting its benefits on both accuracy and memory savings (W4A16-761.9MB -> W2A16-473.2MB). \n(Note: 2-bit PTQ is not successful - e.g., the perplexity of 2-bit OPT-1.3B on Wikitext-2 is degraded from 16.41(FP16) to 35.26(W2A16) even if both AWQ and GPTQ are applied, cf. Table 7 of [R1]. Our observation concurs: see Table R2.)\n- TSLD on GPT-Neo-1.3B and LLaMA-7B (Table R4): We tested the general applicability of TLSD on popular open-sourced GLM models (GPT-Neo-1.3B, LLaMA-7B) using the PTB and GSM8K datasets. Our findings indicate that TSLD consistently surpassed the competitor, Logit KD [20, QuantGPT]. Particularly in the GSM8K dataset for LLaMA-7B, TSLD equaled the perplexity while attaining higher accuracy, demonstrating its capability to mitigate overfitting issues. However, we observed that ternary weight quantization posed greater challenges on LLaMA-7B, a similar phenomenon also noted in [R1] (see Table 8 for PPL increase of 3-bit OPT vs LLaMA). This difficulty seems to correlate with the broader weight range found in LLaMA-7B. As a result, we are exploring methods to tune the diverse weight distributions of GLMs amenable for quantization, as an interesting future research.\n- W4A16 OPT-2.7B on GSM8K (Table R5): We also added the experimental results of various 4-bit weight quantization techniques on OPT-2.7B for GSM8K task, which was initially missed in Table 2 of our paper. Note that TSLD consistently achieved the lowest perplexity and highest accuracy, and all QAT-KD results outperform the accuracy of 4-bit PTQ.\n\n**Comment 2) Experimental results on various data types**\n> Weakness 2) “It would be great for the readers to see the accuracy and perplexity numbers for all data types (int8, int4, and int2).”\nWeakness 4) “Report accuracy and perplexity results for all data types (int8, int4, and int2).”\n\n**Answer 2)** Since the primary focuses of our paper is sub-4-bit QAT, we emphasized ternary and INT4 weight quantization as detailed in Tables 1-3 and R1-R5. Note that INT4 TSLD achieved accuracy and perplexity that matched or surpassed full-precision cases. But we acknowledge the reviewer's suggestion to include results for a complete set of data-types and will incorporate this in the final version.\n\n**Comment 3)**\n> Question 1) ”It's not clear to me whether table 1 shows the training or the test perplexity for the PTB dataset.”\n\n**Answer 3)** Table 1 reported the “evaluation perplexity,” the perplexity measured on the evaluation set of the PTB dataset. \n"}, {"review_id": "z5bPyFi8dz", "reviewer": "Reviewer_kT5b", "summary": "This research paper focuses on Generative Language Models (GLMs) and the challenges of their deployment due to their large model sizes. The authors introduce a novel approach to tackle the challenges faced by Quantization-Aware Training (QAT) in fine-tuned GLMs.\n\n**Problem**: GLMs, despite their impressive performance in text generation, understanding, and reasoning, face deployment challenges due to their large model sizes. Quantization techniques like Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) have been used for model compression but result in significant decreases in accuracy.\n\n**Approach**: The authors propose a new approach, Token-Scaled Logit Distillation (TSLD), to enhance QAT for sub-4bit QAT inference. TSLD is a knowledge distillation technique specifically designed for GLMs. It counters the issues of quantization errors and overfitting, prevalent in existing QAT methods. The method is based on token-scaled logit distillation, which prevents overfitting and allows better learning from the teacher model and ground truth.\n\n**Results**: When TSLD was applied to a variety of GLMs, including a 6.7 billion model for the first time, it achieved comparable, if not superior, performance in text generation on ternary and 4-bit inference. On an arithmetic reasoning task, TSLD even achieved task accuracy at least on par, if not better, with other methods.\n\nExperiments were performed on two tasks: language modeling (using the PTB dataset) and a reasoning-based text-generation task (using the GSM8K dataset). The models were trained using the HuggingFace fine-tuning repository, with varying learning rates and sequence lengths depending on the model's size. They also implemented a QAT-based KD framework using PyTorch's Pipe API for pipeline parallelism.\n\nResults showed that TSLD achieved the lowest Perplexity (PPL), signifying it performed the best in both language modeling and reasoning tasks, outperforming other methods such as Post-Training Quantization (PTQ), QAT KD, and others. TSLD's performance allowed QAT models to match full-precision performance across different model capacities.\n\nRegarding sub-4 weight quantization, the researchers found that L2L KD couldn't run on larger GLMs due to memory constraints. TSLD, however, outperformed other QAT approaches across all model sizes, maintaining PPL degradation of no more than 1.0 from the baseline.\n\nFor the reasoning task, TSLD was shown to effectively balance language modeling and reasoning performance, achieving excellent reasoning accuracy while maintaining competitive perplexity. Ablation study also suggested that sub-4bit quantization with TSLD considerably speeds up the inference of large models while maintaining competitive model quality.\n\n**Conclusion**: The successful results underline the potential of the proposed TSLD method in facilitating the deployment of ultra-low precision GLMs. The method allows for the practical deployment of large-scale GLMs with less than 1.0 degradation in perplexity and no loss of accuracy in a reasoning task.", "questions": "1. About the TSLD methods details \"Token-Scaled Logit Distillation (TSLD). Based on investigations into the probabilistic relation of token predictions and overfitting in QAT, we propose an adaptive KD method that adjusts Logit KD based on token confidence. This approach utilizes the phenomenon of confidence disparity in token predictions from the teacher model. Our method, called TSLD, de-emphasizes Logit KD for high-confidence tokens to prevent overfitting in QAT while emphasizing logit distillation for low-confidence tokens with high entropy probability distributions.\"\nI am eager to know, how much additional computing are required in the TSLD process, such as incorporating the token confidence into the distillation process?\n\n2. Without specific designed hardware, how much advantage could TSLD powered Ternary Weight quantization have, over the 4-bit quantization of LLMs? With what kind of designed hardware, how much advantage could TSLD powered Ternary Weight quantization have, over the 4-bit quantization of LLMs? The clarity regarding this point will illustrate the practical significance of this work.", "limitations": "**Hardware Restrictions**: The TSLD method uses simple ternary weight quantization, which is designed to be hardware-friendly. However, this means that the effectiveness of TSLD may be largely dependent on the specific hardware it's being implemented on. Some hardware configurations might not support ternary or lower precision computations efficiently, which could limit the usefulness or performance of TSLD in such environments.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Innovative Method**: The paper introduces the Token-wise Scaling for Low-bit Distillation (TSLD) method for Quantization-Aware Training (QAT), a novel approach that seems to substantially improve the performance of low-precision models. And this a very early work about Ternary quantization and sub-4 quantization of GLMs.\n\n**Detailed Implementation information**: The paper provides clear and comprehensive details on the experimental setup, including task and model selection, training settings, and implementation settings. This thoroughness makes the results more credible and the experiments reproducible by other researchers. \n\n**Clean and accurate formulation**: the formulation process is clean and accurate, to the best of my knowledge.\n\n**Practical Results**: The paper demonstrates practical advantages of TSLD, such as matching full-precision model performance across various model sizes and maintaining low perplexity degradation with sub-4 weight quantization. These findings suggest that the TSLD method is not only theoretically promising but also practically beneficial, paving the way for low-precision inference in real-world applications.", "weaknesses": "Lack of Broader Evaluation: The paper focuses primarily on GLMs like GPT-2 and OPT for the experiments. This limited scope might not fully illustrate how the TSLD method performs with other types of GLM models. Specifically, if the evaluation tasks can be more various and abundant, it would be very helpful\n\nSpecificity of Tasks: While the paper does test the proposed method on different tasks, it mainly focuses on the PTB language modeling task and GSM8K reasoning task. An evaluation of the method's effectiveness on a broader array of tasks, or in specific real-world applications, could have added to the robustness of the study. (My concern comes from: perplexity is very insensitive, and arithmetic reasoning dataset GSM8K is not large. So \"less than 1.0 perplexity degradation and preserving arithmetic reasoning task accuracy\" is not that convincing.  it is really recommended to have more evaluation on other tasks) ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the insightful feedback provided by the reviewer. In response to the raised questions, our answers are as follows.\n\n**Comments 1) Expanding TSLD Evaluation (Weakness)**: To address the concern about the limited datasets we initially used, we've expanded our evaluation to include 4 additional NLU tasks and 4 CSQA tasks. Furthermore, to expand the number of baselines of our experiments, we extended our comparative experiments to other GLM models, GPT-Neo-1.3B and LLaMA-7B. \n\nThe experimental results are detailed in Tables R1-R5 of the attached PDF. The lessons derived from these experimental results are as follows:\n\n- **W4A16 OPT-1.3B on NLU and CSQA (Tables R1,R3)**: 4-bit TSLD consistently exhibits significantly more robust performance compared to OPTQ and/or AWQ, the latest SOTA PTQ [9][R1] techniques (we used the code published by the authors of corresponding papers). In particular, TSLD achieved the lowest perplexity and the highest accuracy across all the experiments except SST-2, where its accuracy is in-par with the full-precision case. \n\n- **W2A16 OPT-1.3B on NLU and CSQA (Tables R1,R2,R3)**: ternary TSLD consistently outperforms the alternative QAT methods for all the cases tested, demonstrating its superior performance in bridging the accuracy gap with the full-precision cases. Interestingly, ternary TSLD even achieved similar or superior accuracy compared to 4-bit PTQ in many tasks, highlighting its benefits on both accuracy and memory savings (W4A16-761.9MB -> W2A16-473.2MB).\n\n- **TSLD on GPT-Neo-1.3B and LLaMA-7B (Table R4)**: We tested the general applicability of TLSD on popular open-sourced GLM models (GPT-Neo-1.3B, LLaMA-7B) using the PTB and GSM8K datasets. Our findings indicate that TSLD consistently surpassed the competitor, Logit KD [20, QuantGPT]. Particularly in the GSM8K dataset for LLaMA-7B, TSLD equaled the perplexity while attaining higher accuracy, demonstrating its capability to mitigate overfitting issues. However, we observed that ternary weight quantization posed greater challenges on LLaMA-7B, a similar phenomenon also noted in [R1]. This difficulty seems to correlate with the broader weight range found in LLaMA-7B. As a result, we are exploring methods to tune the diverse weight distributions of GLMs amenable for quantization, as an interesting future research.\n\n\n**Comments 2) Analysis of Computation Requirements of TSLD (Question 1)**\n\nThe TSLD method, which incorporates token cross-entropy loss into logit distillation, requires two distinctive operations. \n\n$L_{TSLD} = L_{GT} + \\sum_{n=1}^{N} \\left( \\text{softmax} \\left( \\sum_{i=1}^{V} y_{n,i} \\log(P_{n,i}^{T}) \\right) \\circ \\sum_{i=1}^{V} P_{n,i}^{T} \\log(P_{n,i}^{S}) \\right)$\n\nSpecifically, given that \\( N \\) represents the entire sequence length and \\( V \\) is the vocabulary size, the term $ \\sum_{i=1}^{V} y_{n,i} \\log(P_{n,i}^{T}) $ computes the cross-entropy loss using the teacher logits. This result is then passed through a softmax function to derive the scaling value for each token. This scale value is then element-wise multiplied with   $\\sum_{i=1}^{V} P_{n,i}^{T} \\log(P_{n,i}^{S})$, which is token-wise logit distillation. (note that typical logit distillation would multiply the token-specific logit distillation value by ${1/N}$).\n\nSince TSLD leverages the final logit values of the teacher model used in logit distillation for dynamic scale values, no additional memory consumption is required. The computations for the teacher model's cross-entropy loss and the softmax computation for scale values have a complexity of ${O(N)}$, making the overhead of TSLD negligible in terms of training latency.\n\nTo assess the efficiency of TSLD, we report the training speeds, and GPU memory consumption, for various QAT-KD methods, as shown in Table R7. When comparing the TSLD method to other Logit distillation-based methods, we found that there's no additional memory consumption, and the training speed either slightly decreases or remains unaffected. Such outcomes underscore the advantages of the memory-efficient TSLD; it elevates QAT-KD performance while consuming memory similar to logit distillation, which uses significantly less memory compared to the memory-intensive L2L distillation.\n\n**Comment 3) Specific Hardware for Ternary Weight Quantization (Question 2, Limitations)**\n\n**Answer 3)** Ternary quantization, representing weights as {+1, 0, -1}, is an appealing method for accelerating matrix multiplication by transforming multiplication into simpler additions. This approach enables a \"multiplication-less\" implementation, reducing the computational complexity of matrix multiplication, a universal operation for deep learning models. The weights determine whether corresponding activations are added (w=1), subtracted (w=-1), or ignored (w=0), bypassing the need for complex hardware logic for multiplication. Examples of ternary weight quantization applications include:\n- Custom Accelerator: [R2] utilized ternary weight quantization in FPGA to streamline computational logic for custom acceleration, showcasing superior performance compared to equivalent GPU implementations.\n- CPU Implementation: [R3] showed how ternary weight quantization can be applied in CPUs, not just custom accelerators, achieving a 2 to 4x speedup in inference running on off-the-shelf CPUs using sub-word SIMD instructions.\n- In-memory Computing: [R4] was a pioneer in incorporating ternary weight quantization into in-memory computing. By representing weights as {-1, 0, 1}, this method significantly simplified the in-memory processing implementation, enhancing the performance of DNN inference.\n\n[R2] Nurvitadhi et al, “Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks?” FPGA 2017   \n​​[R3] Choi et al, “​​TernGEMM: GEneral Matrix Multiply Library with Ternary Weights for Fast DNN Inference” SIPS 2021   \n[R4] Jain et al, “TiM-DNN: Ternary in-Memory accelerator for Deep Neural Networks” VLSI 2020\n"}, {"review_id": "03dRBBG6nV", "reviewer": "Reviewer_aiZD", "summary": "This paper proposed the token-scaled logit distillation method to counteract accuracy loss caused by using Quantization-Aware Training（QAT） methods for generative language models . (1) The author proposed the logit distillation method to overcome the limitation in token prediction recovery by modifying the intermediate representation. (2) the author proposed an adaptive KD method that adjusts Logit KD based on token confidence to prevent the overfitting problem caused by the combination of Logit KD and ground truth loss in QAT.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、The idea of adjusting Logit KD based on token confidence to prevent the overfitting problem is novel.\n\n2、The method used in this paper is generalized and applicable to models of different sizes and different types of text generation tasks, including language modeling and reasoning.\n\n3、The experiments involves multiple LLMs with various model scale (0.1B,2.7B,6.7B), which makes the main experiments solid and convincing.", "weaknesses": "1、The FP 32 baseline used in the comparison experiment is not explained. And the experiment only uses one baseline, the proof is not sufficient.\n\n2、 In addition, the experiment selected one dataset for each tasks for validation, and the number of datasets is small. We recommended to add SVAMP and CSQA datasets for inference tasks, and SGD datasets for text generation tasks.\n\n3、The experiments include only main results (comparing with baselines) and the ablation studies. It would be better to conduct more detailed analyses or qualitative analysis to verify the proposed methods (e.g. whether the issues proposed in the motivation are fixed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the insightful feedback provided by the reviewers. In response to the raised questions, our answers are as follows.\n\n**Comment 1) Clarification of Baseline Settings and Expanding TSLD Evaluation Across Tasks and Models (Weakness 1, 2)**\n\n**Answer 1)**\n\nFP 32 baseline settings: ​​​​ Our primary focus lies in recovering the performance of a fine-tuned model using QAT after fine-tuning a pre-trained model for a specific task. Therefore, we first fine-tune the open-sourced pre-trained model for the target task to establish an FP baseline. \n\n\nExpanding TSLD Evaluation: To address the concern about the limited datasets we initially used, we've expanded our evaluation to include 4 additional NLU tasks and 4 CSQA tasks. Furthermore, to expand the number of baselines of our experiments, we extended our comparative experiments to other GLM models, GPT-Neo-1.3B and LLaMA-7B. \n\nThe experimental results are detailed in Tables R1-R5 of the attached PDF. The lessons derived from these experimental results are as follows:\n\n- **W4A16 OPT-1.3B on NLU and CSQA (Tables R1,R3)**: 4-bit TSLD consistently exhibits significantly more robust performance compared to OPTQ and/or AWQ, the latest SOTA PTQ [9][R1] techniques (we used the code published by the authors of corresponding papers). In particular, TSLD achieved the lowest perplexity and the highest accuracy across all the experiments except SST-2, where its accuracy is in-par with the full-precision case. \n\n- **W2A16 OPT-1.3B on NLU and CSQA (Tables R1,R2,R3)**: ternary TSLD consistently outperforms the alternative QAT methods for all the cases tested, demonstrating its superior performance in bridging the accuracy gap with the full-precision cases. Interestingly, ternary TSLD even achieved similar or superior accuracy compared to 4-bit PTQ in many tasks, highlighting its benefits on both accuracy and memory savings (W4A16-761.9MB -> W2A16-473.2MB).\n\n- **TSLD on GPT-Neo-1.3B and LLaMA-7B (Table R4)**: We tested the general applicability of TLSD on popular open-sourced GLM models (GPT-Neo-1.3B, LLaMA-7B) using the PTB and GSM8K datasets. Our findings indicate that TSLD consistently surpassed the competitor, Logit KD [20, QuantGPT]. Particularly in the GSM8K dataset for LLaMA-7B, TSLD equaled the perplexity while attaining higher accuracy, demonstrating its capability to mitigate overfitting issues. However, we observed that ternary weight quantization posed greater challenges on LLaMA-7B, a similar phenomenon also noted in [R1]. This difficulty seems to correlate with the broader weight range found in LLaMA-7B. As a result, we are exploring methods to tune the diverse weight distributions of GLMs amenable for quantization, as an interesting future research.\n\n**Comment 2) Detailed and Qualitative Analysis of TSLD Method (Weakness 3)**\n\n**Answer 2)** We aim to provide a detailed analysis connecting each challenge with our analysis and the proposed method.\n\nStarting with cumulative quantization error, we've unveiled 1) distinct error characteristics in Transformer decoder quantization not addressed by previous L2L KD, and 2) Logit KD's unique error compensation mechanism, which imposes flexibility in adjusting magnitude across middle layers to align the final logit, thus enabling accurate text generation.\n\nWe've prepared Fig. R1 (attached pdf) to clarify our findings:\n- Fig. R1(a): Transformer decoders show cumulative quantization errors at attention maps from token zero to N-1, resulting in notable accuracy loss. This increasing error magnitude is universal, and Fig. R1(a) aligns with Fig.2(b) from our paper.\n\n- Fig. R1(b) and (c): Applying L2L KD (SOTA Transformer encoder quantization [33],[13],[14]) failed to overcome these errors, causing significant performance degradation (Table 1) and memory inefficiency (Table R7). L2L KD reduced attention map errors (Fig. R1(c)) but not the final logit distance (Fig. R1(b)), which most affects text generation accuracy. Despite prior success, L2L KD proved ineffective for Transformer decoders.\n\n- Fig. R1(b) and (d): Recognizing the final logit's importance, we used Logit KD to target finetuning solely on it (without layer-wise regularization). Although attention map error reduction was modest (Fig. R1(d)), logit distance was sharply reduced thanks to Logit KD (Fig. R1(b)), enhancing Transformer decoder performance (Table 1).\n\n- Fig. R1(e): We discovered Logit KD's success lies in allowing intermediate tokens to vary their attention values across channels (Fig. R1(e)-Left), diverging from full-precision (increasing attention map error as in Fig. R1(d)) in the middle layers while aligning the last layer output with the full-precision model (Fig. R1(e)-Right). This focus on the final logit instead of layer-wise regularization leads to more robust text generation.\n\nConcerning overfitting issues with GT loss and the efficacy of TSLD in addressing them, We've added Fig.R1(f) to furnish this evidence. Specifically, we broke down TSLD into two components, TSLD_down and TSLD_up, and conducted an ablation study to analyze the method.\n- GT loss causes significant overfitting and Logit KD alone cannot rectify. TSLD_down addresses overfitting from GT loss by reducing the influence of high-confidence token predictions. By implementing this scale in logit distillation, we significantly reduced the overfitting that was previously detected in both the GT-only and GT+Logit cases.\n- TSLD_up places additional emphasis on distillation for low-confidence token predictions. By adding TSLD_up to GT+TSLD_down, we observed a noticeable decrease in evaluation loss. This underlines the vital role of distillation in enriching low-confidence predictions with soft-label information, thus improving QAT performance.\n- The benefits of TSLD_down and _up stem from their ability to reduce logit distance more effectively than Logit KD, a key factor in successful text generation (Fig.R1(b)).\n"}, {"review_id": "VAnsotuJ9Y", "reviewer": "Reviewer_QKmL", "summary": "This paper proposes to improve the quantization of generative language models via an adaptively scaled logit distillation method.\n\nWith an intuitive illustration in Figure 1 and validation results with attention heatmap discrepancy, the paper identified a cumulative quantization error introduced by the masked attention, i.e., later tokens attend to more tokens before, leading to a larger quantization error. \n\nFurthermore, the paper investigated the characteristics of Layer-to-Layer (L2L) distillation (aligning output activations and attention heatmaps) and the logit distillation from teacher models. Results show these two methods differ from the inner representations and min-max dynamic ranges. This result motivates a proposal of Token-Scaled Logit Distillation, where tokens with higher confidence of teacher predictions are down-weighted in the distillation to avoid overfitting.\n\nExperimental results on GPT-2 and OPT-series show that the proposal effectively maintains the generation quality (lower perplexity compared with baseline methods) and reasoning ability of the original full precision model (better accuracy on GSM7k).\n\n", "questions": "- In Figure 2(b) Left, why is the bottom left corner (where tokens attend fewer tokens ?) of the heatmap also bright?\n- In Figure 2(b) Right, what is the reason for deeper layers exhibiting lower attention MSE error in the decoder?\n- Original QuantGPT incorporates Token-level Contrastive Distillation and Module-dependent Dynamic Scaling, did you solely utilize logit distillation?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "\n-  Exploring the quantization techniques for billion-level language models is a vital problem.\n\n- The finding of cumulative quantization errors of mask attention in Transformer models is interesting.\n", "weaknesses": "The paper can benefit from addressing the following weaknesses:\n\n- Paper Organization: The current paper lacks clarity in its organization, making it challenging to follow the logical flow between sections. Improving the coherence and connection between sections would greatly enhance the reading experience.\n\n- Integration of Challenges: The paper identifies two challenges in quantizing LLMs, namely the cumulative quantization errors of masked attention and the necessity of ground-truth loss. However, it would be more effective to investigate these challenges independently. Additionally, the proposed method seems less relevant to the first finding, which, in my opinion, is the most crucial problem in the quantization process.\n\n- Lack of Clear Motivation: The motivation behind the proposed dynamic weighting method, aimed at addressing the potential overfitting problem, is not adequately explained. The evidence presented in Figure 3(b) is insufficient to support the claim that \"(high confidence) logit distillation might mirror the effects of ground truth loss, thereby contributing to the observed overfitting.\" It would be beneficial to explore whether using ground truth alone leads to overfitting, as well as whether removing distillation of high-confidence tokens can mitigate the overfitting problem. Additionally, justifying the up-weighting of low-confidence tokens, which exhibit high cross-entropy loss and could potentially represent incorrect predictions from the teacher models, is crucial to convince readers of the proposal's validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the thorough and thoughtful reviews provided by the reviewer. Below are our responses to the raised questions.\n\n**Comment 1) Paper organization (Weakness 1)**\n\n**Answer 1)** We concur with the reviewer's feedback regarding the shortcomings in the logical flow, structure, and coherence of our main contents. Acknowledging reviewer's remarks, we will enhance the paper's organization, in particular by revising Sec.3.3 and Sec. 4, to clarify our key motivations and claims.\n\n\n**Comment 2) Integration of challenges: Connection between cumulative quantization errors and Logit KD (Weakness 2)**\n\n**Answer 2)** We thank the reviewer for highlighting ambiguities between our observation of cumulative quantization errors and our Logit-KD proposal. In summary, we've unveiled 1) distinct error characteristics in Transformer decoder quantization not addressed by previous L2L KD, and 2) Logit KD's unique error compensation mechanism, which imposes flexibility in adjusting magnitude across middle layers to align the final logit, thus enabling accurate text generation.\n \nWe've prepared Fig. R1 (attached pdf) to clarify our findings:\n- Fig. R1(a): Transformer decoders show cumulative quantization errors at attention maps from token zero to N-1, resulting in notable accuracy loss. This increasing error magnitude is universal, and Fig. R1(a) aligns with Fig.2(b) from our paper.\n\n- Fig. R1(b) and (c): Applying L2L KD (state-of-the-art Transformer encoder quantization [33],[13],[14]) failed to overcome these errors, causing significant performance degradation (Table 1) and memory inefficiency (Table R6). L2L KD reduced attention map errors (Fig. R1(c)) but not the final logit distance (Fig. R1(b)), which most affects text generation accuracy. Despite prior success, L2L KD proved ineffective for Transformer decoders.\n\n- Fig. R1(b) and (d): Recognizing the final logit's importance, we used Logit KD to target finetuning solely on it (without layer-wise regularization). Although attention map error reduction was modest (Fig. R1(d)), logit distance was sharply reduced thanks to Logit KD (Fig. R1(b)), enhancing Transformer decoder performance (Table 1).\n\n- Fig. R1(e): We discovered Logit KD's success lies in allowing intermediate tokens to vary their attention values across channels (Fig. R1(e)-Left), diverging from full-precision (increasing attention map error as in Fig. R1(d)) in the middle layers while aligning the last layer output with the full-precision model (Fig. R1(e)-Right). This focus on the final logit instead of layer-wise regularization leads to more robust text generation.\n\n**Comment 3) Integration of clear motivation: Empirical evidence of overfitting issues with ground truth (GT) loss effectiveness of TSLD (Weakness 3)**\n\n**Answer 3)** We appreciate the reviewer's request for empirical evidence concerning overfitting issues with GT loss and the efficacy of TSLD in addressing them. We've added Fig.R1(f) to furnish this evidence. Specifically, in response to the reviewer's comment, we broke down TSLD into two components, TSLD_down and TSLD_up, and conducted an ablation study to analyze the method.\n- GT loss causes significant overfitting and Logit KD alone cannot rectify. TSLD_down addresses overfitting from GT loss by reducing the influence of high-confidence token predictions. By implementing this scale in logit distillation, we significantly reduced the overfitting that was previously detected in both the GT-only and GT+Logit cases.\n- TSLD_up places additional emphasis on distillation for low-confidence token predictions. By adding TSLD_up to GT+TSLD_down, we observed a noticeable decrease in evaluation loss. This underlines the vital role of distillation in enriching low-confidence predictions with soft label information, thus improving QAT performance.\n- The benefits of TSLD_down and _up stem from their ability to reduce logit distance more effectively than Logit KD, a key factor in successful text generation (Fig.R1(b)).\n\n\n**Comment 4) Questions regarding Fig. 2(b) (Question 1, 2)**\n\n**Answer 4)** We've plotted an additional attention map loss in Fig.R2(a) to address the reviewer's inquiry, with the following details: \n- We measured the masked attention error for each token within the attention map by normalizing the error for each token row by the number of tokens attended to. This normalized error is represented as a heatmap on the left, with averaged errors per token plotted on the right by token index. \n- Though the trend of increasing error magnitude is consistent, variations can be seen in the heatmap patterns between Fig.2(b) and Fig. R1(a). Likewise, while the cumulative error trend is universal, the error magnitude patterns across layers may differ; the 8th layer shows the lowest magnitude in Fig.2(b), while the 0th layer shows the smallest magnitude in Fig. R1(a).\n\n\n**Comment 5) Difference in quantization settings between our Logit KD and QuantGPT [20] (Question 3)**\n\nWe appreciate the opportunity to clarify the difference in quantization settings between our Logit KD and QuantGPT[20], specifically regarding token-level contrastive distillation and module-dependent dynamic scaling (PACT+).\n\nQuantGPT's token-level contrastive distillation uses random negative sample tokens throughout the sequence, a selection that can potentially cause performance degradation and issues with robustness. This issue has been acknowledged in the revised QuantGPT paper (https://arxiv.org/pdf/2203.10705.pdf). To ensure a fair comparison, we omitted the contrastive loss from our implementation of Logit KD.\n\nWe also experimentally found that PACT+ does not provide meaningful gain compared to our TWN-based ternary quantizer. Table R5 shows a comparison between PACT+ and ours; no significant performance differences were noted across PTB and Wikitext-2 tasks, with the TWN method actually performing slightly better. Thus, we chose the TWN-based ternary quantizer for our comparative experiments.\n"}, {"review_id": "PAKC9DvcF2", "reviewer": "Reviewer_cozw", "summary": "The work proposes modifications for Quantization-Aware Training (QAT) that utilize knowledge distillation and takes into account token-scaled logit distillation for preventing overfitting. They show the motivation for their method through a deep analysis and present the empirical merits of this method via language modeling and mathematical reasoning tasks.", "questions": "Please respond to the concerns stated under the Weaknesses section.", "limitations": "I don't see any limitations section, please review and add this. One limitation can be for example the extra training time required by this method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed Token-Scaled Logit Distillation (TSLD) method as well as the analysis that comes before  is novel and interesting. I like the fact that the authors detected a problem with standard KD and proposed a nice way to mitigate it which replaces the uniform weighting over the tokens. I’m not sure though about the time complexity it takes to compute this loss, as well as the quadratic nature of this complexity given larger sequences, due to the nested summation. I’d suggest discussing this in a future revision.\n\nGenerally the paper is well written and motivated. The authors present a series of analysis that showcase the importance of utilizing their method during quantized training. ", "weaknesses": "Minor comments:\n* L17: state that ALPACA was fine-tuned from the LLaMA 7B model.\n* The notations in Section 3.2 do not follow standard mathematical rules (e.g., matrix is denoted by two letters, a vector letter  is capitalized, etc.) and prior works. Please check previous works describing the transformer modules and change the equations accordingly.\n* L119: Masked self-attention is the wrong wording here. You probably meant causal attention.\n* Section 3.3 is written in a vague form and too informally. Consider adding a mathematical proof for the error accumulation in causal models. \n\nMajor comments\n* What about long-range transformer models? How will the suggested method influence them?\n* In L50 you state one example for a task (arithmetic reasoning). What about other tasks? There is a missing analysis/discussion why would this task specifically benefit from the proposed method, while other tasks don’t.\n* I know that the paper was submitted before the QLORA was published, but in case this paper is accepted, I would expect the authors to add a comparison or at least a discussion about the merits of the proposed method against QLORA and other quantized low-rank adapter techniques. I do not consider this point in the final score calculation of this paper.\n* An inaccurate point in Section 3.3 second part (Necessity of Ground Truth Loss) is the fact that decoder-only models use the entire vocabulary for prediction, unlike encoder models that use only the number of classes. This is also not correct, since one may consider and optimize a few tokens instead of the entire vocabulary, and thus there is a slight difference between encoders and decoders in that case for classification tasks. \n* Empirical results over two datasets is not enough for me, and I wonder how the performance will change given more language modeling and other NLU tasks.\n* There is an empirical evaluation regarding inference speed, but I wonder how fast is training time. This is crucial, since pre-training for example is costly anyway.\n\nMissing references\n* Jin, Jing, et al. \"KDLSQ-BERT: A quantized bert combining knowledge distillation with learned step size quantization.\" arXiv preprint arXiv:2101.05938 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewer for your valuable and constructive comments. To address the queries posed, our explanations are provided below.\n\n**Comment 1) Expanding TSLD Evaluation Across NLU/CSQA Tasks\n(Major Comments 2,5)**\n\n**Answer 1)** To validate the effectiveness of TSLD, we broadened our comparative experiments to encompass four CSQA datasets and additional NLU datasets, as suggested by the reviewer. As displayed in Tables R1 and R2, TSLD consistently exhibited the lowest perplexity and therefore the highest accuracy. This success is most pronounced in NLU's CoLA and RTE tasks, shown in Table R1, where TSLD notably outperformed traditional QAT-KD methods in terms of accuracy. Moreover, in all four CSQA tasks outlined in Table R2, TSLD's performance was superior, bringing the 2-bit quantization results closer to full precision (FP) performance. These experimental results further underscore the effectiveness of our TSLD approach.\n\n**Comment 2) Computational complexity of TSLD: Training time of long range Transformers (Strength, Major comments 1 and 6)**\n\n**Answer 2)** Thank you for pointing out these essential aspects of computational complexity and the application to long-range transformers. We'll address the training time cost of our proposed method as follows:\n\n- **Training time of QAT with KD for Transformer decoders (Table R7):** QAT's tuning of weight parameters is costly, especially with larger model sizes. However, TSLD does not incur additional overhead compared to QAT-KD. Specifically, the previous KD method, L2L-KD, necessitate additional memory requirements to store intermediate activation data. However, our proposed TSLD requires only the teacher's last logit, avoiding this memory overhead. As seen in Table R7, this memory savings enables QAT for larger models, unlike L2L, which struggles with out-of-memory problems for models of 0.8B or larger.\n\n- **Computational complexity of TSLD wit long sequence (Table R7,R8):** TSLD's computational complexity consists of two main operations in addition to basic Logit-KD, both of O(N) complexity, so it doesn't increase quadratically with larger sequences. This results in almost no additional training speed relative to standard Logit-KD (Table R7). We also considered the reviewer's focus on long-range transformers, examining training costs related to input sequence lengths. Table R8 shows the training costs for various QAT KD methods as sequence length varies. Remarkably, TSLD's complexity only modestly increases with sequence length, causing a negligible rise in training time without extra memory use.\n\n\n**Comment 3) Comparison with QLoRA (Major comment 3)**\n\n**Answer 3)** Thank you for bringing attention to interesting aspects. QLoRA enhances the accessibility of LLM by enabling parameter-efficient fine-tuning, even at lower precision levels. However, its approach differs fundamentally from QAT in that it doesn't directly incorporate the quantization error during training; i.e., it requires FP16 adaptors to be processed separately during inference, causing non-trivial latency overhead. To this end, merging QLoRA with QAT to conduct QAT in a more parameter-efficient manner would present an intriguing direction for future work.\n\n\n**Comment 4) Discussion about class sizes between Transformer encoders and decoders. (Major comment 4)**\n\n**Answer 4)** Thank you for highlighting these details. We concur that the number of classes can change depending on the task, but our main emphasis was on the distinction in how encoder and decoder models typically handle GT loss. While encoders, aiming for sequence comprehension, use a limited number of tokens to compute GT loss, decoders, working on causal language modeling, involve all input tokens in this calculation. We noticed that this makes the GT loss more significant in the decoder model in many cases. We will revise our description to make this point clearer.\n\n\n**Comments 5) Comments on writing & limitation (Minor comments 1-4, Limitation)**\n\n**Answer 5)** Thank you for examining the complexities of our paper. We will address the points you've raised, including issues related to the LLaMA-7B model, equation notation, causal attention, and writing style in section 3.3. Additionally, we will consider the weaknesses you've highlighted and update the limitation section in the revised version.\n"}]

Author rebuttal:
We sincerely thank the reviewers for their invaluable feedback. To effectively respond to your inquiries, we have attached a PDF containing new experimental results and figures. Detailed responses to each question raised by the reviewers can be found in the respective sections, with references to the data included in this PDF. Once again, we express our gratitude for your time and insights.






Meta-review: The paper proposes token-scaled logit distillation to prevent overfitting and provide superior learning from the teacher model and ground truth. The reviewers agree to accept the paper. The problem itself is important in practice to better utilize LLMs, so the paper would potentially help other LLM researchers. The comments and some author response can also be taken into consideration for the camera-ready version.
Decision: Accept (poster)