Zero-shot Sharpness-Aware Quantization for Pre-trained Language Models
Authors: Miaoxi Zhu
Qihuang Zhong
Li Shen
Liang Ding
Juhua Liu
Bo Du
Dacheng Tao
Venue: EMNLP 2023 Main (2023)
Keywords: zero-shot quantization
minimax optimization
pre-trained language model
TL;DR: We propose a novel zero-shot sharpness-aware quantization (ZSAQ) framework for the zero-shot quantization of various PLMs.

Abstract: Quantization is a promising approach for reducing memory overhead and accelerating inference, especially in large pre-trained language model (PLM) scenarios. While having no access to original training data due to security and privacy concerns has emerged the demand for zero-shot quantization. Most of the cutting-edge zero-shot quantization methods primarily 1) apply to computer vision tasks, and 2) neglect of overfitting problem in the generative adversarial learning process, leading to sub-optimal performance. Motivated by this, we propose a novel zero-shot sharpness-aware quantization (ZSAQ) framework for the zero-shot quantization of various PLMs. The key algorithm in solving ZSAQ is the SAM-SGA optimization, which aims to improve the quantization accuracy and model generalization via optimizing a minimax problem. We theoretically prove the convergence rate for the minimax optimization problem and this result can be applied to other nonconvex-PL minimax optimization frameworks. Extensive experiments on 11 tasks demonstrate that our method brings consistent and significant performance gains on both discriminative and generative PLMs, i.e., up to +6.98 average score. Furthermore, we empirically validate that our method can effectively improve the model generalization.

Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models
Miaoxi Zhu'; Qihuang Zhong', Li Shen', Liang Ding', Juhua Liu" Bo Du 1 Dacheng Tao
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence
and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China
2 JD Explore Academy, China 3 The University of Sydney, Australia
{zhumx, hww.gniioogggiuuaaggg Liujuhua, dubo yewhu. edu. cn, (mathshenli, maapsshenn,,aandiing Liangding....... liam, dacheng. tao comm
Abstract
2023). Hence, it is crucial and green to lighten the PLMs and reduce the memory footprint (Schwartz et al., 2020).
Quantization is a promising approach for reduc- ing memory overhead and accelcrating infer- ence, especially in large pre-trained language model (PLM) scenarios. While having no ac- cess original training data due security and privacy concerns has emerged the demand for zero-shot quantization. Most of the cutting- edge zero-shot quantization methods primar- ily apply to computer vision tasks, and @ neglect of overfitting problem in the genera- tive adversarial learning process, leading to sub-optimal performance. Motivated by this, we propose novel zero-shot sharpness-aware quantization (ZSAQ) framework for the zero- shot quantization of various PLMs. The key algorithm in solving ZSAQ is the SAM-SGA optimization, which aims to improve the quan- tization accuracy and model generalization via optimizing minimax problem. We theoreti- cally prove the convergence rate for the mini- max optimization problem and this result can be applied other nonconvex-Pl PL minimax op- timization frameworks. Extensive experiments ll tasks demonstrate that method brings consistent and significant performance gains on both discriminative and generative PI.Ms, ie., up to +6.98 average score. Furthermore, we empirically validate that our method can effectively improve the model generalization.
To achieve this goal, various model compression methods have been developed, including knowl- edge distillation (Liu et al., 2019), pruning (Liu et al., 2018) and etc. Among these methods, quan- tization has attracted great attention (especially in the large language model scenarios) (Yao et al., 2022: Frantar al., 2022), owing to its impressive ability reduce memory footprint and accelerate inference while maintaining the network structure (Bai et al., 2021a). However, quantization-aww awarc training (QAT) usually requires retraining using the original training data to mitigate performance degradation. While in some application scenarios, access to the original training data is oftentimes not available due to the protection of privacy and consideration of data security.
In response this problem, post-training quan- tization (PTQ) (Bai et al., 2021a) is proposed to address the training-data access, as gener- ally requires no retraining. But most of those methods would lead to an accuracy drop in low- precision quantization since they are training- and primarily rely on the analysis of weight dis- tributions. More recently, zero-shot quantization (ZSQ) (Nagel et al., 2019; Zhuang et al., 2022; Zhang et al., 2021) shows promising results in the computer vision community. Specifically, ZSQ per- forms the quantization process with the synthetic data generated from generator. To alleviate the side effect of inaccurate synthetic data, Choi et al. (2020) further improve the ZSQ by adaptively tun- ing the generator with the supervision of adversar- ial learning. However, it is non-trivial to adopt such an adversarial-based ZSQ in the NLP field, as backpropagation on discrete words is reason- able, i.e., it seems "unlikely" to pass the gradient through the text to the generator; @ minimizing the difference between the teacher and (quantized) stu-
1 Introduction
Pre-trained language models (PLMs), such as BERT (Devlin et al., 2018) and GPT-3 (Brown et al., 2020), have achieved great success in a vari- ety of NLP tasks (Raffel et al., 2020; Zhong et al., 2022b, 2023a; Liu et al., 2023b). However, with the scaling of model size, the inference of larger PLMS, e.g., OPT (Zhang et al., 2022) and LLaMA (Tou- vron et al., 2023), becomes more computation- ally expensive and memory-intensive (Shen al..
Miaoxi Zhu and Qihuang Zhong contribute equally this work.
Corresponding Authors: Juhua Liu (e-mail: liu- juhua@ whn.cc..c.. Bo Du (c-mail: dwwo@whu.chee www.cdu.cn)

---

dent models would be prone to over-fitting problem, leading to poor model generalization.
minimax optimization problem (ZSAQ). (3) Ex- tensive experiments show that our SAM-SGA can bring consistent and significant performance gains, up to +6.98 average score, on both discriminative and generative PLMs.
To address the above issues, we propose a novel Zero-shot Sharpness- Aware Quantization (namely ZSAQ) framework to improve the performance and generalization of the quantized model. Firstly, re- garding the problem I, design simple feature adaptation module convert the output token rep- resentations of the generator into the target quan- tized model's representation space, thus avoiding the gradient propagation on the discrete words. Sec- ondly, for the problem @, we are inspired by the sharpness-aware minimization (SAM) (Foret et al., 2020) and propose an alternating SAM-SGA algo- rithm to boost the ZSAQ and improve model gener- alization. Specifically, the one hand, SAM-SGA uses SAM to robustly minimize the divergence of output distributions between teacher and quantized student models. On the other hand, it uses stochas- tic gradient ascent (SGA) to encourage the gen- erator model to maximize the divergence at each iteration. By optimizing such minimax problem, we can not only train the well-performed with gen- erative adversarial learning but also improve the generalization of quantized models.
2 Related Works
Compression method for language models. Com- pression is efficient in reducing computation cost, memory overhead, and energy consumption as well as shortens inference time for large neural net- work models, which is in great need for pre-trained language models to be deployed in the mobile or resource device. Quantization works by transmitting the models with lower-bit parame- ters. QAT works by minimizing the rounding error on the training dataset. FullyQT (Prato et al., 2020) shows the fully quantized Transformer can avoid any accuracy loss in translation quality. I-BERT (Kim et al., 2021) eliminates floating point calcu- lation in BERT inference and achieves similar ac- curacy to the full-precision baseline. BinaryBERT (Bai et al., 2021b) achieves good performance by ternary weight splitting which avoids the complex and irregular loss landscape. There are some other techniques, low-rank factorization (Tahaei et al., 2021; Edalati et al., 2021: CHen et al., 2020; Reid et al., 2021), factorizing the weight matrices which are usually low-rank into several smaller matrices by means of singular value decomposition. Parame- ter sharing (Rothe et al., 2020; Takase and Kiyono, 2021; Reid et al., 2021), reduces memory over- head by reusing the same parameters in multiple computations. Pruning (Mishra al., 2021; Guo et al., 2020; Chen et al., 2020; Xia et al., 2022: He et al., 2022; Liu et al., 2023a), deletes some parameters which are "useless" " in network under the same model performance.
Theoretically, we provide rigorous analysis for the SAM-SGA algorithm solving the minimax optimization (ZSAQ). We show that SAM-SGA achieves (I/VTT) convergence rate. Furthermore, our theoretical results are not limited to the single case but are applicable wide range of min- imax optimization problems in both nonconvex- strongly-concan and nonconvex-Pe -LL conditions. Empirically, we adopt our ZSAQ framework to quantize both the discriminative and generative PLMs. and evaluate 8 language understanding tasks and 3 language modeling tasks. Extensive experi- ments demonstrate the effectiveness and versatility of our approach. More encourugingly, our ZSAQ brings +6.98 average performance gains in the low- precision quantization, compared to the baselines. Extensive analyses show that our framework has the potential to expand to more large language mod- els and prove that SAM-SGA indeed brings better model generalization.
Zero-shot quantization. In contrast to QAT, PTQ relies less on the training data and does not require end-to-end training (Nagel et al., 2019; Nahshan et al., 2021; Nagel et al., 2020; Li et al., 2020; Zhao et al., 2019; Hubara et al., 2020), which can be zero-shot. Bai et al. (2021a) propose module- wise quantization error minimization and perform close to QAT. SmoothQuant (Xiao et al., 2022) transmits the activation quantization to weights to smooth the outliers in activation and makes a large language model implemented efficiently. Tao et al. (2022) proposes an adaptive quantiza- tion for different modules and performs compara-
In summary, contributions are three-fold: (1) We propose novel method, zero-shot sharpness- aware quantization (ZSAQ), which realizes zero- shot quantization without much accuracy loss. (2) We provide a thcoretical convergence guarantee for the SAM-SGA algorithm which aims to solve the

---

bly with the full-precision generative pre-trained language models. GPTQ (Frantar et al.. 2022), one-shot weight quantization method with approx- imate second-orde information, can efficiently quantize generative pre-trained transformers with large amounts of parameters to 3 or 4 bits with neg ligible accuracy degradation. Nahshan et al. (2021) figures out that aggressive quantization can lead the steep curvature of the loss landscape, while they propose method combining layer-by-layer quantization and multivariate quadratic optimiza- tion that can improve accuracy over current PTQ. NuQmm (Park et al., 2022) uses a non-unifom quantization method which allows for a trade-off between accuracy and compression ratio. Another way to achieve zero-shot is by means of the gener- ative adversarial technique. (Cai et al., 2020; Liu al., 2021b; Choi al., 2022) has shown promis- ing performance on small-scale computer vision tasks, while has not been applicable to the NLP realm. To the best of our knowledge, we are the first to adopt the adversarial learning approach for quantizing both the discriminative and generative PLMs in zero-shot manner and provide its rigor- ous convergence analysis.
MSE loss
Tokens
Generator
Figure I: An overview of our ZSAQ framework.
of the pre-trained model. So the objective for Q is to minimize the discrepancy between ((G(2)) and ((G(z)), denoted as D(P((())), ((G()))). While for the generator G, it tries to maximize the divergence, that the quantized model Q be trained confuse even the most strict generator. We try to balance the trade-off between the genera- tor G and the quantized model Q so that we can get a well rell-performed quantized model without access to the original training data. The structure of our method is demonstrated in the Figure 1. And the objective function of our method is shown below:
3 Methodology
min max DPP
(2)
this section, we will present our zero-shot sharpness-aware quantization method.
where D represents the divergence MSE. ZSAQ: Zero-shot sharpness-aware quantization. As shown in Liu et al. (2021a), there a sharper loss landscape in the low-precision model than in the full-precision model. To avoid overfitting prob- lems caused by the training process, we should consider the generalization ability of our quantized models. Inspired by Foret et al. (2020), we pursue flatter landscape during the training process of the quantized model which can guarantee a better generalization performance. So we can adjust our objective function as:
Network quantization. For real-valued vector which represents the parameter in the pre-trained full-preed pell-precision model, quantize it into x by the signed symmetric quantization:
x = 8 clamp - (-
(I)
where 8 E R- denotes scale factor and b E Z rep- resents the bit-width of the quantized version. So the parameters can be projected to the integer grid:
{-2h-1 - + 1,..,,,, O...,, 2h-1 1). And the default experimental setting is presented in Sec. 5. Generative adversarial learning. In our paper we achieve zero-shot by generative adversarial learn- ing. Specifically, the generator G takes i.i.d. token samples z as input to produce synthetic data. Then the generated sentence segments will be fed into the pre-trained model P and the quantized model Q. Intuitively denoted as (2)())) and (2)) respectively. During the training process of the quantized model, it aims to simulate the output distribution
min
E (iiinmaa)AB....
+ +S(P:Q(4)))
where we denote the sharpness function as:
S(P: Q(+))
(4)
get the output distributions,
max E. B.-TD(P(G(2) 2( +z)(2)))
1 EZ- D(P((())) ((4

---

The sharpness function searches for the sharpest point among a neighbor centered at w with ra- dius of 3, which will represent the worst-case data. Then our optimization process not only minimizes the divergence but also the sharpness of the diver- gence, i.c., getting quantized model with both lower divergence and a flatter landscape. Solving the inner maximization problem to ob-
Algorithm 1 SAM-SGA for ZSAQ
Input: generator Go. quantized model ( pre- trained model P. learning rate ne and Tu and the neighborhood size for the perturbation
1: Initialize: 2(wo) is quantized from P by Eq.(1)
2: for 1=0,
3: //training quantized model by SAM
tain the optimal:
4: Compute the optimal perturbation e*(w) by Eq.(5);
c((w) V.B DiPIGa)))))))) D(P(IID(P(((())) ((
(5)
Calculate the gradient and update w via Eq.(7);
Remark. Here use á as the parameter in the sharpness function (4) instead of w+e, for the reason that e may be so small that there will be no difference adding the perturbation. And this issue is also illustrated in Liu et al. (2021a).
Quantize W++1 to 4:41 by Eq -Q);;);
7: //training generator by SGA
8: Update 0:11 by Eq.(
9: end for Output: a uniformly drawn from "i.,,..:::
Algorithm. For the objective function, we train the generator to maximize the discrepancy between the pre-trained model and the quantized model:
Eg ((P( (z))) with general representation f(ū, 0) [f {)]), where ç follows iid. Then our detailed objective function (3) can be turned into:
D(-TD()()()),, Q(O;; (())))
(6)
Then we train the quantized model to minimize both the discrepancy value and the sharpness of the discrepancy, i.e., the addition of D(P, Q) and S(P, Q). on the synthetic data produced by the generator G. While the addition comes out to be:
min
(max min 0
l( 0) + pohare ( (i,
(8)
where fsharp (w. 0) A 5SAM (Lo, 0) ن(ا, 0) and fSAM (io, 0) - maxiiellass f(e + c, 0). We can
D(-( 2(4 (0))(())))
solve the maximization problem that e (w)) (af(6,0)
(V=f((,0)) So that we can write fSAM (w. 0) as f(o + (
where the optimal (w)) is obtained from Eq. (5).
We obtain our final algorithm by applying stochastic gradient descent on the minimization objective (7), which turns to be SAM, and ap- plying stochastic gradient ascent the maximiza- tion objective (6). Generator G is parameterized by 0, quantized model Q initially quantized from the pre-trained model P and the quantized parameter is denoted as w, while the parameters before quan- tization is denoted as w. We alternatively update the parameters W and 0 respectively. The pseudo code is shown in the Algorithm
Next, we restate the update rule our Algorithm in our new denotation that:
Wh+1 = w. - (w
(w: + 3 She (.) Wr+1 = Quantizzaiinn (aantization((:+1))
(w). O₂)
θt+1 = G, + nogo (wr, 0.)
where and ge are the approximated gradients calculated by Iw (w, 0) = M ΣM Gw (v))) and (w 0) = 1 I Go(w, tively. And G f (Gw, Go) is the unbiasd gradient of the function f. i.e., [[[[(w, 0; ff(w, 8). Remark. Actually function f maps Bd x R to IR. The first parameter can be either a floating point or an integer. It is well-defined for us to represent f( -) or -.,
respec-
4 Optimization
In this part, we will analyse the convergence of algorithm to solve this minimax optimization problem, which can provide theoretical guarantee for our method.
Before illustrating the theoretical results, first introduce some necessary assumptions and defini- tions that are used in the analysis.
First in order to make our theoretical conver- gence result applicable to a broader variety of such problems, we replace the detailed loss function

---

Assumption 4.1. f(w,0 0) is differential and 1- Lipschitz smooth:
Remark. Due the space limitation, the proofs and strongly-concave case are placed in Appendix A. Observing the above results, we evaluate the averaged gradients at the quantized parameter and we can draw the conclusion that our averaged output of the quantized models can converge to an e-stationary point during (1)/c²) iterations under the neglect of quantization error dA?.
[wf(wi, Vul(wy, 0)l s 11 w2 [Vo[(w, Vol(w, 02)1 S 1||0; Ozll:
v0
Assumption 4.2. f(w.-) satisfies PL condition on every given w, i.e. there exists ş > 0 such that [ww,000 0)].
[ee[ 0) 2 2u|maxe [ww,
5
Experiment
Assumption 4.3. Parameter is restricted in convex and bounded set with a diameter of D.
5.1 Setup
Assumption 4.4. The approximated gradient is unbiased and has a bounded variance, i.e.,
Tasks and Models. In this section, we evaluate our SAM-SGA method on both discriminative (i.e. BERT-style) and generative (i.e., GPT-style) lan- guage models. For discriminative models, we fol- lowed many prior works (Zhong et al., 2023c,d) and tested both BERT and BERTharge (Devlin et al., 2018) on GLUE benchmark (Wang et al., 201 8); and for generative models, we tested the OPT- family (Zhang et al., 2022) (mainly OPT-350m) on the GLUE benchmark and several language modeling tasks, iee., WikiText2 (Merity et al.) and Penn Treebank (PTB) (Mikolov and Zweig, 2012) and WikiText103 (Merity et al.). For evaluation, we report the performance with Accuracy ("Ace.") metric for most tasks, except the Pearson correla- tion ("Pear:") for STS-B, the Matthew correlation ("Mec.") for CoLA, the perplexity (PPL) score for language modeling tasks. We report the averaged results over random seeds to avoid stochasticity, The details of all tasks are shown in Appendix A.1.
E[G 0, E Vf(w,)) 0); [[G(w.0,5) V c
Remark. These assumptions are common in the context of minimax optimization problems and can be realized in empirical practice.
Lemma 4.1. (Zhu et al., 2020) The quantization error is bounded by the grain, where d denotes the dimension of parameter w:
|| - w|ll S Vda.
Remark. The quantization error can be naturally derived from Eq. (1) and the grain A is decided by default quantization setting, which will not be discussed in our paper.
Definition 4.1. (ww) maxo f(w, 0); Φ" min (w)). And we define an e-stationary point when E| (2) S..
Implementation Details. Following many previ- ous studies (Tao et al., 2022; Bai et al., 2021b), we first fine-tune a full-precision model using the pre- trained checkpoint from huggingface' for each task, and the fine-tuned model the full-precision teacher model. Then, we apply the cutting-edge PTQ methods, i.e., ZeroQuant (Yao al., 2022) for BERT models and GPTQ (Frantar et al., 2022) for OPT models, to quantize the fine-tuned model and use the quantized network to initialize the student model. For the generator model, we directly use the OPT-350m (Zhang et al., 2022) (without any further tuning) model?. i.e., the generators are not trained on any end-task data. All experiments are conducted on a single NVIDIA A100 (40G) GPU
Remark. By defining the function Φ we can sim- plify our minimax optimization with two opposite variables into a minimization optimization problem with single variable. When the gradient of (w) diminishes to zero, we consider our algorithm con- verges.
Theorem 4.1. Under Assumption 4.1.4.2,4.3,444 and restrictions 3
<
Tle
21 110
= 64k2na. nuo Nw S
M(E
we have the convergence bound for our problem:
min 128K-1
1321 la
T
E
(6))
T
htttps://huggingffe
wttp//aaa.....c.......cc co/nodels
1=0
"Notably, we can use any generative L.MSs as the generator models. In this work, we use the OPT models as they are open- sourced and cover multiple model scales.
(WW))
Φh )m
2
+ (d)
MT

---

#Bits
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
GLUE
Method
(W-A)
Mec.
Acc.
Acc.
Acc.
Pear:
Acc.
Acc.
Avg.
(1)
Full-precision
W32A32
60.82
83.12
85.05
90.52
89.91
65.34
92.43
88.84
82.00
Baseline
W8A8
60.07
83.16
84.31
90.41
89.86
64.98
92.66
88.82
81.78
QAT-GT
W8A8
62.26
83.58
85.29
90.57
89.87
66.79
92.66
88.85
82.48
+0.70
QAT-Rand
W8A8
60.06
83.37
84.81
90.52
89.92
65.14
93.00
88,98
81.98
+0.20
SAM-SGA
W8A8
63.70
82.93
85.05
90.52
89,93
65.3
92.08
88.96
82.31
+0.81
Baseline
W4A8
52.13
77.24
82.60
85.10
86.34
50.90
89.11
84.41
75.98
QAT-GT
W4A8
57.30
78.22
83.09
86.95
86.41
55.86
90.71
87.34
78.24
+2.26
Qan-Rand
W4A8
53.37
76.98
77.94
86.61
86.23
48.38
90.48
86.48
75.81
-0.17
SAM-SGA
W4A8
53.01
76.64
83.58
86.12
86.16
64.26
89.68
86.94
78.30
+2.32
Baseline
W2A8
0.00
35,47
68.38
49.13
67.29
47.29
77.52
58.43
50.44
QAT-GT
W2A8
0.00
39.86
69.61
50.59
67.55
47.29
77.64
69.41
52.74
+2.30
QAT-Rand
W2A8
0.00
39.26
31.62
49.52
67.93
47.29
76.83
63.11
46.95
-3.49
SAM-SGA
W2A8
10.16
36.84
68.38
53.41
89.95
57.76
76.26
66.61
57.42
+6.98
Table I: Results of BERT on the development set of GLUE (Wang et al., 2018) benchmark. "#Bits (W-A)" denotes the bit-width for weights of Transformer layers and activations.
#Bits
CoLA
MRPC
RTE
#Bits
PTB
WikiText108
WikiTexx2
Method
Method
(W-A)
Moc. (1)
Acc.
Acc.
(W-A)
PPI. (L)
PPL. (1)
PL).
Full-precision
W32A32
64.47
86.03
65.71
Full-precision
W32A32
28.58
13.38
19.21
Baseline
WBA8
64.80
86.76
62.45
Baseline
WBA8
56.02
48.16
55.4
QAT-GT
WBA8
66.61
87.75
65.34
QAT-GT
WBA8
50.68
42.15
46.29
QAT-Rand
WBA8
66.86
86.52
65.15
QAT
W8A8
54.59
48.69
60.46
SAM-SGA
WBAS
66.08
$7.25
66.06
SAM-SGA
WBAS
46.59
46.36
$4.08
Baseline
W4A8
49.11
74.26
47.29
Basetine
W444
60.25
56.16
62,89
QAT-GT
75.73
56.68
QAT-GT
QAT-Rand
W4A8
45.63
70.34
47,29
QAT-Rand
W444
55.32
63.63
SAM-SGA
W448
52.97
77.70
52.71
SAM-SGA
W444
51.19
54.32
61.22
Table 2: Results of BERT large parts of tasks of GLUE (Wang et al., 2018) benchmark.
Table 3: Results of OPT-350m on the dev set of four language modeling benchmarks.
and the detailed hyper-parameters can be found in Appendix A.2.
QAT-Rand: follow Yao al. (2022) and use the random data (using the random integer number generate token ids) to perform the QAT process.
Compared Methods.
For reference, compare SAM-SGA method with the following cutting-
edge counterparts:
Notably, for each method, carefully grid-search the best learning rates from (5e-6, 1c-5, 2c-5, 5e- 5), and report the best results that are chosen from the best single run among those learning rates.
Full-precision:
we report the results of full- precision fine-tuned model, i.e., teacher model in our framework.
5.2 Main Results
Baseline: we adopt the powerful PTQ meth- ods i.e., ZeroQuant (Yao et al., 2022) for BERT models and GPTQ (Frantar et al., 2022) for OPT models, to quantize the fine-tuned model, and report the results as the baseline.
We report the results of BERT BERTharge and OPT-350m in Table 1, 2 and 3, respectively. Notably, we use WxAy to represent the x-bit weight quantization and y-bit activation quantiza- tion. From the results, we can observe that:
after obtaining the teacher and dent models, we use the original training data to guide the quantization-awar aware training pro- cess student model.
QAT-GT:
SAM-SGA outperforms the other counterparts in most settings. As shown in Table 1, our SAM- SGA outperforms the baseline methods among all quantization settings by large margin, i.e.,

---

CoLA
MNLI
MRPC
QNLI
QQP
RTR
SST2
STSB
GLUE
Method
Mcc.
Acc.
Acc.
Acc.
Pear.
Acc.
Acc.
Acc.
Avg.
(1)
Baseline
52.13
77.24
82.60
85.10
86.34
50.90
89.11
84.41
75.98
SAM-SGA (Ours)
53.01
76.64
83.58
86.12
86.16
64.26
89.68
86,94
78.30
+2.32
-Fo SOA
45.30
78.06
'84.3
"85.53
86.05
55.23
"89.36
86.59
76.30
"2200
-w/o SAM
50.74
76.29
82.60
85.74
85.93
63.53
89.44
86.46
77.59
-0.71
-w/ SGA&SAM
43.70
76.02
82.80
85,33
85.56
56.04
89.48
86.11
75.63
-2.67
Table 4: Ablation study of important components of SAM-SGA. Here, we use the BERT- base models quantified into W4A8 bit-width.
up to +6.98 average score. Specifically, QAT-GT also achieves remarkable performance gains, ow- ing to the supervised information from the orig- inal training data. When using the random data ("QAT-Rand"), the performance gains brought by the vanilla QAT process will be much slighter. En- couragingly, can be found that ZSAQ performs better in the lower precision settings, indicating the potential of validation methods in extreme com- pression scenarios. These results prove the effec- tiveness and significance of our data-free method.
(i] BCRE--ba WAA
(ni UCRT-3as WEAR
HBFC
04.3
$5.00
253-
54
Parfeman (hi)
02.5
15.27
00.
61.21
035
64.
1200
153M
1255
16:2
125M
Genernter Mac
Madel
Figure Ablation study of different generator models. We use OPT-family models with different sizes as the generators.
SAM-SGA brings the consistent performance gains among both model sizes. We further adopt our SAM-SGA to the larger BERT model. Due to the space limitation, we only report the results of parts of GLUE benchmarks in Table 2, and provide the full results in Table 7 of Appendix A.3. It can be found the similar phenomenon in the BERT- base experiments. That is, our SAM-SGA brings the consistent performance gains on both models.
Impact of important components of SAM-SGA.
There are several important components in SAM-SGA framework, i.e. "SGA": tuning the gen- erator model with the stochastic gradient ascent, "SAM": training the quantized model with SAM optimizer. Here, we conduct the ablation study to verify whether these components are necessary. Specifically, taking the BERT-base model as an example, compare our full SAM-SGA method with the following methods: i) "-w/o SGA": we remove the SGA process, i.e., keeping the gener- ator model fixed; ii) "-w/o SAM": we replace the SAM optimizer with the vanilla Adam optimizer; 3) "-wlo SGA&SAM": fixed generator model and base optimizer are used.
SAM-SGA works well in both BERT-style and GPT-style models. In addition to the discrimi- native PLMs, we also verify the effectiveness of our SAM-SGA on the generative models. The re- sults of language modeling tasks are in Table 3, and can seen that, with the help of SAM-SGA, OPT-350m achieves much better quantization per- formance against the baselines. Moreover, we also evaluate the OPT-350m on the GLUE benchmark. The contrastive results are listed in Table of Ap- pendix A.3, illustrating that our method outper- forms the other methods, which further validates the effectivvnnss of our methods for generative style models.
As shown in Table 4, we can find that removing any component will cause the performance degra- dation, indicating the necessarily of these compo- nents. More specifically, without the SGA, our SAM-SGA would perform much worse. One possi- ble reason is that the quantized model might over-fit the synthetic data, without the constraint of adver- sarial training.
5.3 Ablation Study
this part, we 1) first evaluate the impact of im- portant components of our SAM-SGA, and 2) then investigate the effect of different generator models.
Effect of different generator models. Here, we examine whether our SAM-SGA can still work well using different generator models. We con-

---

Wisits
GLUE
QNLI - RTE
QNLI MRPC
Method
(W-A)
Avg.
(t)
(%4
Areaney (S}
71
Full-precision
W32A32
82.00
56
+1.81
+0.98
PTQ-vanilla
W8A8
79.81
-2.19
PTQ-PEG
W8A8
81.06
-0.94
PTQ-Adaround
W8A8
82.02
+0.02
SAM-SGA (Ours)
W8A8
82.59
+0.59
Basdline
w/l SAM
OUIS
Baseling
MeO SAM
Ours
(2) BERT-baae, W4A8
PTQ-vanilla
W4A8
46.93
QNLI RTE
QNLI
MRRC
PTQ-PEG
-42.54
52
PTQ-Adaroo I'Q-Adaround
W4A8
77.49
.4.51
Arennwy (%)
SAM-SGA (Ours)
W4A8
78.30
-3.70
50
+3.97
+1.25
48
Table 5: Comparisons with other zero-shot quantization methods on the development set of GLUE (Wang et al.. 201 18) benchmark. Full results are shown in Table 9.
46
Baseline
HHO SAN
CUT
Bascline
we SAM
OurS
(b) BERT-ERREEE
W8AS
Figure 3: Analysis of task generalization. The model fine-tuned on the QNLI task and transferred to four different tasks. We can see that our method consistently brings better generalization compared to the baseline.
WikiTex12
Method
(W-A)
PPL (1)
PPL (1)
Full precison
W32A32
24.88
15.52
Bascline
W8A8
36.75
24.05
SAM-SGA
W8A8
32,66
20.96
A ())
J. 1,409
4 3.09
Adaround (Nagel et al., 2020). Specifically, taking the BERT-base as an example, we show the con- trastive results in the W8A8 and W4A8 settings in Table 5. As seen, our method SAM-SGA outper- forms the other zero-shot quantization methods by clear margin, especially in the low-bit setting.
Baseline
W4A4
44.46
27.54
SAM-SGA
W4A4
36.62
22.88
A ())
4.7.84
4 4.66
Table 6: Results of OPT-1.3b on two language modeling benchmarks. We can find that our SAM-SGA can still work well in the billion-level model scenarios.
Scaling to Billion-level large models. Some readers may concern that whether our method can be used to quantify the larger language models. Here, we attempt to expand our method to the quantization of billion-level models. Specifically, due the limited amount of computing resources, have tried our best extend SAM-SGA to a larger model setting, and lastly choose the OPT- 1.3b for experiments. For reference, we compared our method with the baseline GPTQ m
duct the contrastive experiments by varying the generator model size from 120M to 1.3B, and il- lustrate the results in Figure 2. As seen, compared the baseline, our SAM-SGA consistently brings improvements across all model sizes, basically in- dicating that the performance of SAM-SGA is not very sensitive to the generator model size. More specifically, the case of OPT-350m performs best, and we thereby use this setting in our experiment.
As shown in Table 6, compared to the baseline. our SAM-SGA brings the significant and consis- tent performance improvements among all settings, iee, up +7.84 gains. These results prove that our method also works well the larger language model scenarios.
5.4 Analysis and Discussion
Here, we conduct extensive analyses discuss: I) whether our SAM-SGA outperforms the other zero-shot quantization counterparts; 2) whether our SAM-SGA can expand to more larger language model scenarios, and 3) whether it gains better model generalization.
Does SAM-SGA Bring Better Generalization? One of main contributions is that introduce the SAM optimizer to improve the generalization of quantized model. Here, we verify from two perspectives: measuring the cross-task zero-shot
Comparisons with other zero-shot quantiza- tion methods. To better assess the strengths of our method, we further compare it with more zero-shot quantization methods, i.e., PTQ- vanilla (vanilla post-training quantization), PTQ- PEG (Bondarenko et al., 2021) and PTQ-
Notably, we empirically found that the performance of GPTQ is very sensitive to the number of input sumples (default 1), especially in the large model scenarios. Here, we set this number to 10 to ensure the effectiveness of GPTQ.

---

performance, and ii) visualizing the loss landscapes of quantization models.
130
Basclin SAM
Bassine
90
Wio SAM
Outs
munl spp..
Task Generalization. As stated in many previ- ous studies (Zhong et al., 2023b; Ding et al., 2022),
20-
70
the performance of out-of-domain (OOD) data is widely used to verify the model generalization. Hence, we follow Xu et al. (2021); Zhong et al. (2022a) and evaluate the performance of quantized model on several OOD data. In practice, we first quantize BERT-base models (fine-tuned on QNLI task) with different methods, including "Bascline", "Ours" and its variant "-w/o SAM" (without using SAM). Then, we inference the quantifized models on other tasks, i.e., MRPC and RTE. The results are illustrated in Figure We observe that "Ours" con- sistently outperforms the other counterparts. To be more specific, compared with baseline, our SAM- SGA brings +2.0 average improvement score on all tasks, indicating that our SAM-SGA boosts the performance of PLMs OOD data.
50-
0.5
0.0
0.5
1.0
0.5
0.0
0.5
1.0
Figure 4: ID visualization of loss landscapes of OPT- 350m quantized by different methods. PTB (Mikolov and Zweig, 2012) is used for evaluation.
provide theoretical analysis for the minimax opti- mization algorithm under both nonconvex-strongll concave and nonconvex-PL conditions, which guar- antees convergence rate of ((1/VT) for our ZSAQ method. Our experiments show outperform- ing quantization results, up to +6.98 average score, and validate improved generalization ability.
Limitations
Our work has several potential limitations. First, given the limited computational budget, we only validate our SAM-SGA method on the Large and Base model sizes. It will make our work more con- vincing if scaling the experiments up the much larger model sizes, e.g., LLaMA-650 (Touvron et al., 2023) and OPT-66b. On the other hand, al- though our method brings much performance gains compared to the other PTQ method, it would lead to more computation overheads. How to accelerate the process has not been explored in this work.
Visualization of Landscape. To have close look, we visualize the loss landscapes of differ- ent quantized OPT-350m models fine-tuned on the PTB task. In practice, follow He al. (2021); Zhong al. (2022a) to plot the ID loss curve by lin- ear interpolation between the model weights before (denoted as 06) and after (denoted as f1) tuning, i.e., "0, + a (0 - 0o)",, where a is scalar parameter that is ranged from -1 to 1. The ID visualization re- sults are illustrated in Figure 4, and we find that our optimal setting "Ours" shows flatter and optimal property. These results prove that our SAM-SGA can smooth the loss landscape and improve the generalization of PLMs effectively.
Ethics Statement
We take ethical considerations very seriously, and strictly adhere to the EMNLP Ethics Policy. This paper focuses on zero-shot quantization for current open-sourced pretrained language models, but not capturing the privacy knowledge. Both the mod- els and evaluation datasets used in this paper are publicly available and have been widely adopted by researchers. Therefore, we believe that this re- search will not pose ethical issues.
A Note on More Analyses and Discussions
Notably, in addition to the above results and stud- ies, we further conduct more in-depth and system- atic analyses and discussions in Appendix, due to space limitations. Specifically, we provide 1) parameter analysis of 17, and 2) visualization of the loss curve of adversarial training stages in Ap- pendix A.3. Please refer to the Appendix for more details.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 62225113 13 and 62076186. The numerical calcula- tions this paper have been done on the super- computing system in the Supercomputing Center of Wuhan University.
6 Conclusion
In this paper, we propose a novel zero-shot quanti- zation framework ZSAQ, which effectively realizes zero-shot quantization without much accuracy drop and shows promising generalization ability. We

---

References
Liang Ding, Longyue Wang, Shuming Shi, Dacheng Tao, and Zhaopeng Tu. 2022. Redistributing low- frequency words: Making the most of monolingual data in non-autoregressive translation. In ACL.
Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang. Irwin King, and Michael R Lyu. 2021a. Towards efficient post- training quantization of pre-trame pre-trained language models. arXiv preprint arXiv: 15082.
Bill Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang. Qun Liu, Michael R Lyu, and Irwin King. 2021b. Binarybert: Pushing the limit of bert quanti- zation. In ACL.
Ali Edalati, Marzieh Tahaci, Ahmad Rashid, Vahid Par- tovi Nia, James J Clark, and Mehdi Rezagholizadeh. 2021. Kronecker decomposition for gpt compression. arXiv preprint arXiv:2110.08155.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2021. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and
Behnam Neyshabur. 2020. Sharpness-aware mini- mization for efficiently improving generalization. In International Conference on Learning Representa- tions.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 2210::2210.......
Yaohui Cai. Zhewei Yao. Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Zeroq: A novel zero shot quantization framework. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paruphrasing.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task I: Semantic textual similarity and cross-lingual focused evaluation. arXiv preprint arXiv:1708.0m 00055.
Demi Guo, Alexander M Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff prun- ing.
Ruidan He, Linlin Liu, Hai Ye. Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In ACL.
Patrick CHen, Hsiang-Fi Fu Yu, Inderjit S Dhillon, and Cho-Jui Hsieh. 2020. Data-aware low-rank compres- sion for large nlp models.
Tianlong Chen. Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural informa- tion processing systems, 33:15834-15846
Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022. Sparseadapter: An easy approach for improving the parameter-efficiency of adapters. In Findings of EMNLP.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. 2020. Improving post training neural quantization: Layer-wise calibration and inte- ger programming.
Kanghyun Choi, Hye Yoon Lee, Deokki Hong, Joon- sang Yu, Noseong Park, Youngsok Kim, and Jinho Lee. 2022. It's all in the teacher: Zero-shot quantiza- tion brought closer to the teacher. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8311-8321.
Sehoon Kim, Amir Gholami. Zhewei Yao, Michael W
Mahoney, and Kurt Keutzer. 2021. 1-bert: Integer- only bert quantization. In International conference on machine learning, pages 5506-5518. PMLR.
Yoojin Choi, Jihwan Choi, Mostafa El- El-KKamy, and Jungwon Lee. 2020. Data-free network quantiza- tion with adversarial knowledge distillation. In Pro- ceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 710-711.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2020. Breeq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Reprexentations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv: 1810.000004
Tianyi Lin, Chi Jin, and Michael Jordan, 2020. On gra- dient descent ascent for nonconvex-comeave minimax problems. International Conference on Machine Learning, pages 6083-6093, PMLR.

---

Chuang Liu, Xueqi Ma, Yibing Zhan, Liang Ding, Dapeng Tao, Bo Du Wenbin Hu, and Danilo Mandic. 2023a. Comprehensive graph gradual prun- ing for sparse training in graph neural networks. IEEE Transactions on Neural Networks and Learning Systems.
Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Ja- son D Lee, and Meisam Razaviyayn. 2019. Solving class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems, 32.
Gunho Park, Baeseong Park, Se Jung Kwon, Byeong- wook Kim, Youngjoo Lee, and Dongsoo Lee. 2022. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.0980
Jing Liu, Jianfei Cai, and Bohan Zhuang. 2021a. Sharpness-aware quantization for deep neural net- works, arXiv preprint arXiv:2111. 12.2233..
Juhua Liu, Qihuang Zhong. Liang Ding, Hua Jin, Bo Du, and Dacheng Tao. 2023b. Unified instance and knowledge alignment pretraining for aspect-based sentiment analysis. IEEE/ACM Transactions on Au-- dio, Speech, and Language Processing.
Gabriele Prato, Ella Charlaix, and Mchdi Reza- gholizadeh. 2020. Fully quantized transformer for machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1-14.
Liyuan Liu, Xiang Ren, Jingbo Shang, Xiaotao Gu, Jian Peng. and Jiawei Han. 2018. Efficient contextu- alized representation: Language model pruning for sequence labeling. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 1215-1225.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. JMLR.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint 10X.v.2904.0948..
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP.
Yuang Liu, Wei Zhang, and Jun Wang. 2021b. Zero- shot adversarial quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Put- lern Recognition, pages 1512-1521.
Machel Reid, Edison Marrese- and Yutaka Mat- suo. 2021. Subformer: Exploring weight sharing for parameter efficiency in generative transformers. Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 4081-4090.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representa-
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Associ- ation for Computational Linguistics, 8:264 280.
tions.
Tomas Mikolov and Geoffrey Zweig. 2012. Context de- pendent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 234-239 IEEE
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu. and Paulius Micikevicius. 2021. Accelerat- ing sparse deep neural networks. arXiv preprint arXiv: 104.0270....37..
Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, and Dacheng Tao. 2023. On efficient training of large-scale deep learning models: A literature review. arXiv preprint arXivv2304.0333
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for pestntraining quan- tization. In International Conference on Machine Learning, pages 7197-7206. PMLR.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Confer- ence Computer Vision, pages 1325-1334.
Marzich S Tahaci, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. 2021. Kro- neckerbert: Learning kronecker decomposition for pre-trac pre-trained language models via knowledge distilla- tion. arXiv preprint arXiv:2109.06243.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Exgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. 2021. Loss aware post-training quan- tization. Machine Learning.
Sho Takase and Shun Kiyono. 2021. Lessons pa- rameter sharing across layers in transformers. arXiv preprint arXiv:2104.0602..

---

Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre-trained language mod- els via quantization. In ACL.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa. and Zhiru Zhang. 2019. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552. PMLR.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Hua Jin, and Dacheng Tao. 2023a. Knowledge graph augmented network towards multiview representation learning for aspect- sentiment analysis. IEEE Transactions Knowledge and Data Engineering.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP.
Qihuang Zhong, Liang Ding. Juhua Liu, Bo Du, and Dacheng Tao. 2023b. Self-evolution learning for dis- criminative language model pretraining. In Findings of ACL.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics.
Qihuang Zhong, Liang Ding, Juhua Liu, Xuebo Liu, Min Zhang, Bo Du, and Dacheng Tao. 2023c. Re- visiting token dropping strategy in efficient bert pre- training. ACL.
Qihuang Zhong, Liang Ding, Keqin Peng. Jubua Liu, Bo Du, Li Shen, Yibing Zhan, and Dacheng Tao. 2023d. Bag of tricks for effective language model pretraining and downstream adaptation: A case study on glue. arXiv.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A cover-coverage challenge corpus for sen- tence understanding through inference. In NAACL.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate mod- els. In ACL.
Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhua Liu, Bo Du, and Dacheng Tao. 2022a. Improving Sharpness-award aware minimization with fisher mask for better generalization on language models. Find- ings of EMNLP.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large lan- guage models. arXiv e-prints, pages arXiv-2211.
Qihuang Zhong, Liang Ding, Yibing Zhan, Yu Qiao, Yonggang Wen, Li Shen, Juhua Liu, Baosheng Yu, Bo Du, Yixin Chen, et al. 2022b. Toward efficient language model pretraining and downstream adapta- tion via self-evolution: A case study superglue. arXiv.
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. 2021. Raise a child in large language model: To- wards effective and generalizable fine-tuning. In EMNLP.
Chen Zhu, Zheng Xu, Ali Shafahi, Manli Shu, Amin Ghiasi, and Tom Goldstein. 2020. Towards accurate quantization and pruning via data-free knowledge transfer. arxiv preprint 2010:2201..07334.
Junchi Yang. Antonio Orvieto, Aurelien Lucchi, and Niao He. 2022. Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference Artificial Intelligence and Statistics, pages 5485-5517. PMLR.
Yuanxin Zhuang, Lingjuan Lyu, Chuan Shi, Carl Yang. and Lichao Sun. 2022. Data-free adversarial knowl- edge distillation for graph neural networks. arXiv preprint arXiv:2205.0381.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Ad- vances in Neural Information Processing Systems.
Susan Zhang, Stephen Roller, Naman Goyal. Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li. Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint 200502205.
Yiman Zhang, Hanting Chen, Xinghao Chen, Yip- ing Deng, Chunjing Xu, and Yunhe Wang. 2021. Data-free knowledge distillation for image super- resolution. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7852-7861.

---

A Appendix
MRPC Microsoft Research Paraphrase Cor- pus (Dolan and Brockett, 2005) is task to predict whether two sentences semantically equivalent. A.2 Details of Hlper--paaameter
A.1 Details of Tasks and Datasets
BERT and OPT models are obtained from Hug- gingface", and we follow the instruction (e.g.. fine-tuneing hyper-parameetrs) from Huggingface Transformer Library² to fine-tune these models. For initial post-training quantization of BERT- based models, we use the ZeroQuant (Yao et al., 2022) and set 48 groups for group-wise weight quantization in W8A8 settings, 32 groups for W4A8 and 16 groups for W2A8. While for OPT- based models, we use 128 groups in all settings for GPTQ (Frantar et al., 2022) method.
In this work, we conduct extensive experiments on GLUE benchmark and some widely-used language modeling tasks. Here, we introduce the descrip- tions of the used tasks and datasets in detail:
CoLA Corpus of Linguistic Acceptabil- ity (Warstadt et al., 2019) is a binary single- sentence classification task to determine whether a given sentence is linguistically "acceptable".
For our SAM-SGA method, we use 50 iterations with batch size 32 and sequence length 128 for BERT models, and we use 100 iterations with batch size 4 and sequence length 2048 for OPT models, We grid search the learning rate in range of { (le-6, 5r-6, 1c-5, 2e-5). All the quantized models are trained using single NVIDIA A A100 (40G) GPU.
STS-B Semantic Textual Similarity (Cer al.. 2017) a task to predict how similar two sentences are on a 1-5 scale in terms of semantic meaning. RTE Recognizing Textual Entailment (Giampic- colo et al., 2007), given a premise and hypothesis, is a task to predict whether the premise entails the hypothesis.
A.3 More Results and Analyses
MNLI The Multi-Genre Natural Language In- ference Corpus (Williams al., 2018) is a task to predict whether the premise entails the hypothe- sis, contradicts the hypothesis, or neither, given a premise sentence and a hypothesis sentence.
In this part, we provide more results and analyses to further investigate the effectiveness of our method, Specifically, we first perform the parameter analy- ses on 130 and Tu and then visualize the loss curve of adversarial training stages.
SST-2 The Stanford Sentiment Treebank (Socher et al., 2013) is a binary classification task to predict the sentiment of a given sentence.
Parameter Analyses of fy and Th The factors 1 and Ths are two important hyper-parameters in our Algorithm. In this study, we analyze its in- fluence by evaluating the performance of BERT- base (W4A8) with different 1) spanning (5e-6,1e- 5,2e-5,5e-5) on MNLI and QNLI tasks. Figure 5 illustrates the grid-search results. We find that the results do not exhibit significant fluctuations while demonstrating stability within narrow range. These prove the effectiveness of our method.
QNLI Question Natural Language Inference is binary classification task constructed from SQuAD (Rajpurkar et al., 2016), which aims to predict whether a context sentence contains the answer to question sentence.
QQP The Quora Question Pairs2 dataset is a collection of question pairs. The task is to deter- mine whether a pair of questions are semantically equivalent.
PTB The English Penn Treebank (PTB) (Mikolov and Zweig, 2012) corpus is one of the most known and used corpus for the evaluation of models for sequence labelling, and is also commonly used for character-level and word-level Language Modelling.
Visualization of loss curve of adversarial train- ing stages. Since adversarial training usually suf- fers from instability, some readers may concern about the training stability of our method. To inves- tigate it, we visualize the loss curve of adversarial training stages in Figure 6. We can see the stu- dent loss decreasing and generator loss increasing as time varies and eventually reaching a relatively stable plateau during the final stage. These results indicate that the adversarial training is relatively stable in our SAM-SGA framework.
WikiText2 and WikiText103 Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 (Merity et al.) is over 2 times larger and WikiText- (Merity et al.) is over 110 times larger. The WikiText datasets are also widely-used for language modeling. As they are composed of full articles, the datasets are well suited for models that can take advantage of long term dependencies.
"ht "nttps://huggingfaat http:/hhwggin.aaacccccoo/ooodlld hhttps: thub. tree//main/examplls///ttt
co/nodels

---

Performance BERT-base (W4A8) MNLI
Performance of BERI-base (W4A8) on QNLI
76.5
86.0
Se-6
76.23
75.84
75.9
76.5
5e-6
84.46
83.76
85.0
86.12
Valuue of the purumeter flo
Values the parameter fl8
1e.5
76.49
75.29
1e:5
75.5
84.5
2e:5
76.47
75.95
76.56
74.38
2e-5
84.9
85.7
85.21
83.56
84.0
75.0
83.5
5e-5
75.96
76.64
76.04
75.72
5e-5
85.65
84.57
84.04
82.92
74.5
Se-6
26-5
Se-5
Se-6
16.5
28-5
Values parameter fw
Values parameter n
(a)
(b)
Figure Parameter analyses y/o flu- We report performance of BERT-base (W4A8) on MNLI and QNLI
#Bits
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
GLUE
Method
(W-A)
Mec.
Acc.
Acc.
Acc.
Pear:
Acc.
Ace.
Acc.
Avg.
A (t)
Full-precision
W32A32
64.47
83.12
86.03
92.37
90.85
65.71
93.35
89.22
83.14
Baseline
W8A8
64.80
83.33
86.76
92.27
90.79
62.45
93.46
89.12
82.87
QAT-GT
W8A8
66.61
83.29
87.75
92.54
90.81
65.34
93.23
89.26
83.60
+0.73
QAT-Rand
W8A8
66.86
83.01
86.52
92.42
90.76
65.15
93.00
89.18
83.36
+0.49
SAM-SGA
W8A8
66.08
83,50
87.25
92,55
90.87
66.06
94.04
89.37
83.72
+0.85
Baseline
W4A8
49.11
78.86
74.26
86,86
86.68
47.29
87.73
87.63
74.80
QAT-GT
W4A8
50.85
79.17
75.73
90.44
87.35
56.68
92.32
88.05
77.57
+2,77
QAT-Rand
W4A8
45.63
79.02
70.34
87,26
87.17
47.29
78.84
72.95
-1.85
SAM-SGA
W4A8
52.97
79.19
77.70
90.69
87.20
52.71
90.48
87.72
77.33
+2.53
Table Full comparison results of BERT]aagg jarge on the development set of GLUE (Wang et al., 2018) benchmark.
A.4 Convergence Analysis
E El[g(w, 0)
f(ww, 0)]
Before our proof, we give some notations for brief. W1+1/2 & W, (W,, 0.) like in (Foret al., 2020). So the update rule for can rewritten as: W1+1 = Wi Iw (00
G(w. 0,6, Vf 0)1
M 1=1
1 M
El|G(w, Vf( a²
Lemma A.1. Under Assumption 4.4, we can fur- ther get the evaluation about the approximate func- tion:
Lemma A.2. We give estimation that
E yw (2r 0,)1
E[g(w, 0)] C (
< (439) Vaf (a) (562 2
El|g(w, 0) 72 0²
Proof. have the following decomposition:
E (0
Proof. According to the definition that g(w, M MM G(w, 0, Ei), and & i.i.d. So the expec- tation of the approximate function is the same as G that: Elg(w, 0)] E V fw, i 0) and
gm 012 wn (wi, 0,)112
(10)
E V 0,)
+ 2E (w: 1/2, 0t), Vwf(Wer 0:)

---

#Bits
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
GLUE
Method
(W-A)
Mec.
Acc.
Acc.
Acc.
Pear.
Acc.
Acc.
Avg.
(1)
Full-precision
W32A32
57.51
84.54
83.33
90.30
90.72
69.68
93.23
88.13
82.18
Baseline
W4A8
55.69
84.03
82.60
89.73
90.25
67.15
92.77
87.93
81.27
QAT-GT
W4A8
55.94
83.85
83.58
89.85
90.32
68.23
93.12
88.03
81.62
+0.35
QAT-Rand
W4A8
55.71
83.75
82.60
89.77
90.17
67.51
92.78
88.03
81.29
+0.02
SAM-SGA
W4A8
56.47
83,94
83.33
89,96
90.27
67.87
93.23
88.03
81.64
+0.37
Table 8: Results of OPT-350m on the development set of GLUE (Wang et al., 2018) benchmark, Here, we only report the results in W4A8 settings, and note that the results the other low-bit settings are similar.
#Bits
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
GLUE
Method
(W-A)
Mcc.
Acc.
Acc.
Acc.
Pear:
Acc.
Acc.
Acc.
Avg.
Full-precision
W32A32
60.82
83.12
85.05
90.52
89.91
65.34
92.43
88.84
82.00
PTQ-vanilla
W8A8
58,29
82.08
80.64
88.72
87.47
62.82
90.37
88.10
79.81
PTQ-PEG
W8A8
57.04
83.15
85.29
89.95
89.36
64.62
90.71
88.34
81.06
TTQ-Adaround
W8A8
61,32
83.68
84.07
90.43
89.84
66.06
92.32
88.45
82.02
SAM-SGA
WSAS
63.70
82,93
85.05
90.52
89.93
67.51
992.08
88.96
82.59
PTQ-vanilla
W4A8
1.88
33.91
31.62
50.06
62,97
52.71
48.62
-1.23
35.07
PTQ-PEG
W4A8
0.00
37,91
31.62
50.23
63.18
47.29
73.62
11.83
39.46
PTQ-Adaround
W4A8
55.67
72.59
79.90
87.13
86.49
64.62
91.28
82.21
77.49
SAM-SGA
W4A8
53.01
76.64
83.58
86.12
86.16
64.26
89.68
86.94
78.30
Table Results of comparisons with other zero-shot quantization methods the development set GLUE (Wang et al., 2018) benchmark.
For the cross cross-product term, we evaluate as follows:
And for the first term, we have:
(w 1/2, 0₂) V wwf(wr 0,)11
< 2 Iw (W 0,) ( 0;)12
+ 2E V J 1/2 0.) Vuf 0:)
B(gw( 0:), V Vwf(wir 0.)) E(gw (1/
20²
+ 2/
ن6 1/2 111l
M
2 0²
gw(wet (wr 0²), O1) V (wr
12,3212 +
Vw 0c)l"
M
+ (w Vw 01), 0.) Vwi 01))
(12)
[VV VV 0,), 01), Vol (wl 0:))
Combining the above incqualities and we
+
0,)
get:
V (w..
E E
El|gw (W111/2 0.) $ (4 + 2,31 + 2) /f (i,, (13)
El V.
/ / 0.)1
0.),
0²
+ (5,9112 +2) M
0,)
< (31 | ] V ( 12 821202
2M
(11)

---

Adverurrial trainin BERT-aaic (WAsd) MILI
Taking expectation conditioned on (wi, 0z) and WC get:
11:84
medel
B[ Os]
0250
0.22
Stadent
(a)
(())) Tl (Wr( V / 0z))
Outevtr kib
0.000
2nln B س 1/1 21) )1 3 wi: Oi
Th E V 1| (2xl + nw
8 )da²
0.12
+
0.
32
(15)
Training steps
We again take expectations on both sides of the above inequality so we have:
2.00
1.5
EB 1)]
Loceshtttlatt model
0:30-
Laa- of genrraar ande
[E(D(O))] nh T V V (W/+/22 04)
(b)
km
12 B (WH11 01)12
Th BE ((,))
32
620
8 )dA Two
4.58
+ (2xl +
2225
(16)
Traleing stops
For the second term, we decompose it as follows:
Figure 6: Visualization of loss curve of adversarial train- ing stages.
((V)( (1/(2 0.))
((V))
A.5 Strongly Concave Case
Vuff 0:) (2 Vuf
Lemma A.3. For the descending relationship of the Sunction D, we have:
E (()))))), /Wi 0.))
E| VD Voa. V ( O,11
E E (W++1)
V 140 (ox)) Bلdl
< E ((r))
2 VV)
O:) (p)(a)))
no 15
//
5,l -30--44111444
2 16
2 ( 1 (
N (1
+1)+2klm
2
2
BlV 01) (
olngo²
Vwi 01) + (5
M
(0) 0> - (
3E
Blo?n nw (2kl
8 )da nw
1
M
3IE (w: 0,) V (w.r, 0.)l
2
Proof. We get the conclusion that (w)) is 2nd-
1
1-31
31
E
E
( 2_ /// & " (()))
smooth according to Lemma 4.3 in (Lin et al.. 2020):
2
Blo²
3
(V) 0:)1
((0441)
2M
(17)
< ((+))+ + (V 10t txll ( will
We continue estimating the last term in above inequality (17):
< ( (V (3)) WH+1 wx)) 12xl| WiTH w 12
((V)())) 0:11 Wi11 + 2rldA²
E (Q) 0,)
= T (W))) n (1) Iw @4412, 00) 2kllA
+ +2xln 11 (00+11/2 0,)1
<E| 14 R V -
+
The
[
8 2
(
<DV HE
32
nle
(14)
(18)

---

where the last inequality (i) is due to Young's in- equality.
+ 2(4K - 11) 1) dA
Combining inequality (17) with (18), we can
j=0
get:
where X
1
1
+ 4(4x 1)x2 +
4x
(VV
V (0 0.))
2 + 2). Proof.
1-31 E ( ] V O.) VD
[0+++)))+)) 0ct1ll
Blo
Vie K 01) v
E| 0 (w:11) - (wsr) +10)))000 0 0 (1))
2M
1 -1-5)))
1 2
11 |/)
- (4x - 1) (w1+1) - -0 -w)
+1 4n 2 E 01+1 0" (6:)))
3lo²
Vuf(@r, 0.1131
2M
(21)
(19)
where the inequality is due to Young's inequality. We then continue to decompose the second term as follows according to the update rule and take expectaion:
Finally, we combine inequality (16) with Lemma A.2 and inequality (19):
E [[+((++1)
< {{(((())
E| - H (wx;))
2
(0: + nege (wi, 0,)) 0 (1x
+ Th (1 + 3 [ - Vuf(wir
1
0:)1
2 2
S E| - 0" (1)))1 + ng go (Wr, 0,)11
22l1 (11 + + + V 0
+ +no E(O, 0" (wx). 90 (w), 0x))
+ (5,822 22) 0²
+ Blo2n
< Bl10; 0- (w;))l + n Vel 0.)1
M
M
+ 2n D 0r (.)). (el 0,)) + 180²
+ Nl E (1,))) + (2xl + )Aa
8
M
32
Th
(22)
(1)
Using the property of p-strong concavity of func- tion f on variable 0, we can get:
< [[(()
nh 15
5.31 200-)
2 16
Os 0T+)) (0, (0.- Vol
+1
2
Fl (1+-g)))
31 (011+)+4412(((33-2322222)
2 10 (wx) 0
02
(23)
Vwff 0:) 0:) (5 22)
So the original inequality (22) above can be turned into:
M
Bla'nw nwe
8 dda
+ +(nd +
E El (0) 12
M
n
(20)
<(1 (1- (m))
E E 6; 0" (2.)) Vel (e)
where the last inequality (i) uses the Cauchy- Schwarz inequality that [ 022 2))V + f 0.) (
+ 2n (w: 0.) - O 0" (A:))
M
(24)
Lemma A.4. When we require the learning rate 1
Since f(w,-) is I-smooth and p-strongly con- cave, we have:
satisfy: (2,31 + +1) S 16x21 the evaluation about the error term:
1 and ne = 21 we have
0l(00((++11 - 02+1l2
(6r (w))) O.) (+))) S (Vof( 0r (w+)). (,))),e 0, 8" (wx))
< x D2
(25)
1 > 2
'E (2,))
On the other hand, we can get the following inequality according to optimal condition:
1=0
d 1
+ (
4k-1
Σ x
(Vef(wr, 0" ((::))) 0 0* (w::) & O;
V0 E
710
4k-2
M
(26)
1=0

---

Combining above two inequalities (25)(26), we
property that V - V. (-)) which we refer to Lemma 4.3 in (Lin et al., 2020)).
have:
We then simplify the recurrence relation of 10 (1)+1) - 6 1,+1l We denote the coefficient of E (0,) 0,11 as X, i.g. À - ax + 4(4x
MW 0z) - [W,,, 0- (wx))
21 Vel( 02) - Vel(w,, 0 (+:))l?
1 )m m2 (4/²²² + 2 + 2)
(27)
Then we repeat the recurrence relation and we can get:
Putting pieces (24) and (27) together and on the
occasion that n10- 21 we can get:
E 0| (()))
B10 - 0: 11l
1
020² 0" (3)))]
(28)
S ABI - 1.12
< (1
0
2x
M
+ ([4x--)>>' (22-
4x-1 2 0²
Then we return to the inequality (21), we can get:
H((x-1))<
1x-2 11 M
+ 22(4k - dss
E| (((0((0)1)) 1.11 1?
I
0
S <4k 10010((0+)))) - 0- (1=)))
+ 1(1x - 1)x22) (18212 + 2 2)
1
+ (1 +
4m 2 (1
1B110 0 (2x:))
2x
+ (1 + 1n 2
1
02
+ ((N-D))2
2 10 M
4k-1
a+1)
M
(
1
+ 2(4K 1) dd +1
S (1
(((i) - 0.11
4x
+ (4x - 1) 1)x² 2 BB| wy) 1 - 00,11 + (1k 1)nã0²
S X 110 -0ll
(1x 2)M
(8) 1
1(1k-1)<2 113 > (
S (1 4x
(4)0)) 0,
j=0
+2(1k- 1)) ((1 22 (wr,
1n 0²
+ (4x (1xx-))221255
Σx x
+ (582212 + 2) 0²
(4k - 1)n00²
1n 2
70
M
j=0
M
(1K -2)M
+ 2((4k 1)m?dA2
+ 2(K - 2d2 > x
(68)
1=0
S (1- 4x + +444k= 10gh1)222222100 112 + +2]
< D2
EE1 (4x)) - 00112
+1(x- 1139 Xx X
+ 14(4m-1)x² (2 + 2,311 +,31+ 2))
j=0
4k-1 2 0²
+(4x- (1 2)+
4x-2 110)
M
+ (4K ((K-1))22
1n 2 02
x
110
+ - 1)n?dA2
1n 2
M
(29) where inequality (i) is due to the x-Lipschitz of (-) according to the Lemma 4.3 in (Lin et al., 2020); inequality (ii) follows from the decomposi- tion that W/41 W. W wi+1 + Wi+1 ب, and the update rule of parameter w. together with Lemma 1.A.2. finally inequality (iii) is decom- posed by [ 02)1.2 < 2 f 0.) Vo((), 122 [( < 21 | (wx) - 0,1 1|2
+ 2(4k - 1)k2d=
y=0
(30)
where the last inequality is due to Assumption 4.3.
Theorem A.1. Under Assumption 4.1,4.2,4.3,4.4 and restrictions n (2 + 1) < 1
2/| (I))) second inequality here uses the
1
and no =

---

2, Tag. when f is p-stroegly-conccve the parameter 0, we have:
Then use Lemma A.4, we can get:
T
Σ
E (1)
T+1 1=
E [[D((1)) 1E (((+D(+++1))
576kl D
(T (1+ 96
7(T +1)
0²
1 o²
+ (720k 1/212 + 210k(mw)
+
(96k + 2 M
M
+ (180x212 192kl
768 T
dA
Tw
(31)
[[
< ((1(
nw 15
5,3 22
16
nh 11 +
1
2
Nw 12 + 2
31 + 1kdm (13222 + 2 + 2)]
Proof. We then back to Lemma A.3 and try to make use of Lemma A.4:
2
101k 2/2 (1322 12
1=0
Nu 12 + + 4ndm (1321² + 2,31 + 2) 2
+
E[ 1)]
((4x-1)))))) 1) 2 (5 22 4x 2 0
X
4x-2 10
< (w,)]
M
1=0
15
5,31 580--M 2 16
Tl [1 + 2 31 + 4xlm (4,2222 + 2,31 + 2)]
2
#9 The (1,,)
2 2
+ 2(4k 2dA
da2 2 Σ x
0.11 1 15 + xlngo 02
1=0
M
xlng
Blo nl
8
Blo"n nw
+ (2nl + A
+ (5,322222)
M
+
M
+ 22ll +
dd1
M
nw
(32)
where we use the property that V.()) = Vwf((,0'.)
We then repeat the inequality (32) above from

---

1= T to L O, we have:
(1l 1k
16k (T+1)
1 2 110)
M 3
E| [[((+++)]
The 2 + 21 + 4ktm (4 + 231++ + 2)]
2
$ [E])(wo)]
2(4k - 1)k2 ds 16k
(T+I)
T
?ww 15 5,31 11ll-Anlla1131 (1
3
E [
2 16
(5,39222)
02
Blo
12al
8 ddA((T+)
T
M
M
N
+
Tue (2 [1 3144kl (4
x
S [8))
2
2
t=0
T E (
Nww 15
Th P1 +
1
31 + Akdm (182r² + 2,81 + +2]
5531 44
2
2 16
2
t=0
4(4m-
1)x²
2 nw
(4 + 231 + +2)
12
1 (1
16k
T-1t-1
2
2
3
Σ
(-1 V (1;))?
+
[1
+
31 - 4ndm (432 +289
+ +2]
6
2
1=0 j=0
The [1] + 2
1
+ 2
31 + 4k0m + + 2)]
B|
[w)))l
((Kx - 12 (5,300 + 2) + 4k -
t=
4k 2 v
The | [1 + 30 + Andn 1n + 2.1 + +21++2) 2)]
1
2
7-11-1
ΣΣ> x
2(4K dA² d) 16k
(T+1)
M t= 1=0
3
+ T/ 111 2 31 + Anlm (132p 2,31 +2]
1
+25322-2/ kin
02 Blo
- )dd3(4+1)
8
M
M
17
S [(n()
T-1
2(4m - dm
ΣΣ"
X
The 13
1631 (133
5=
6 16
+55,9922 nlmgo
Blo 12 (T+))
EE (1))))
M
M
(33)
Since the parameters satisfy: (2,31 + 1)
<
8KM 12 D2
+
31 4klm (4822+2811++ 2)
we can evaluate that X = 1 -
1
+
3
2
2(4x
1
1)m²((nn" (4 + 2,31 + 2) <
4x
+
SKM p +
1
31
+ 4klm 1kll (4/32/2 + 2,31 + 2)]
=1 3 Therefore the summation can be
4x
3
2
Tox
Ip
4k-1 0²
Sounded:I=11 ETo - 3 y < 3
16x
16x
((1n 1)k² 2 2 (5,3212 +2) 4
10) M
(T +)
So the above inequality can be turned to:
16 ( + 31+ + 4ntn (2 + 2,31 + 2)]
3
E ((+++))
(4k - 1) 1)k² da (T+1)
< (()())
165,372 12) xh 2
Blo7
8 (dA3(T+H
12kl
Th 15
T B 2
M
M
1lw
5,81-
2 16
T
6
t=0
< [[)
E
(w:)
|| - D2
Nw [ [1+
3l+1kl (132 D2
16x
96 1=0
2 2
3
360
2
240
+ The ( +
1
31
+ 4ndm /w (2 + 2,31 + 2)]
49
K Tuo
4. 3 12
M
(T+1)
49
2
2
15 klm a²
d²
16k T 3
+ 7 M 140
no
(2nl+
dd 2 (T+1) nw
-4(1k- 1)n² N/
2 213212
E ( |
M
on the condition that: K 2 2, 31 < T40
So we bounded
+ nw [ +
1
31 + (4 + 2,31 + 2)]
can get the average sum of EE [1
2 2

---

by:
same way as Lemma A.3:
T
Σ
B (
I [[)((;)]
T+1
1=0
2 '166 5,31 21--222111119--22222222282222111
[[0( - BB (wT+1I]
576kl D2
(T' 1)
The
96
7(T +1)
+ The (1 31) (1)--
2
22
1 0²
2
+ 17200=n100" + 211kkm)
+
(96k + 2 M
V /wr 0,) + (5 LT
M
+ (18 + 192kl
768
2MM
dAA²
Blo T
8
+ (L +
dd
(34)
M
Tl
(36)
So the bound for the algorithm to get an 6- stationary point is
And then using the smoothness of the variables w and respectively, we can get:
2 Do + KD2 0
ka² max
e })
f( 02) f( 0t)
2
- 2 (Vwf 0r) 1 w.r) 2 |
f(Wr) 0811 f((0) 1, 01)
0-+1- 0,) - _ 1 (of(([r-, 01,
A.6 PL Condition
2
First we construct potential function in the same way as (Yang et al., 2022):
Taking expectations respectively can get:
B[f(@1+1, 0:)]
ED [[ 0.) nu EVa VL.
V/- V( 0.) - ((w)) a(o( - f( 0.)l
In Iw W1 0,) 1- -AA
The V W 0,11
where a > 0 is a preset parameter. Then we come to evaluate the descending relationship of the po- tential function Vg.
32
The
Nw E ElAW, 0.)
2- The
E (os
2
The E
01)
V / 01)1
Theorem A.2. Under Assumption 4.1,4.2,4.3,444
2
and restrictions
1 B1
< 16. 7h (2
+
In E gw
/ 0:)1 112--2144 21d
16:
1)2kl 5.. 12
nw 128 and no 64k2nw
T E V
0, _dA² (w)
when f satisfies p-PL condition on parameter O
32
Tw
we have:
497w E| V / (wi, 02)
E 0.)]
32
1232 nw
E [(
E Elggt (.)
12
Mw (1
2
8
((w)) Φ*
N 1+0 @ moon 202
)+0 (nno 1
dAA
(21 +
)dA
< 0
nw
M.
497
1232
(35)
> 0.)
12
32
2
P232
o²
E Vwsl (wo) 01)1
+h (5.39122)
M
Proof. We get the conclusion that ((w) is L- smooth according to Lemma A.5 in (Nouiehed et al., 2019), where L = [ + 4. And we can get the descending relationship of ((s)) in the
8 (21 +
dAA
7/4
(37)

---

[[
(( Vuʃ( 0.)1
497w
n~
tall
1n (13 + + 2))
32
2
(w 0,)1
2
12 1o (4
+ 2 2 12)
2 ( 0.)l + _ V 0.)1
al 100
E
2
2
10 E Voo 0,) V Vo 0:)1
1282 nww
2
(1)+0))5
(5,3
tal
In (1) 0,]1
2
2
1
2
+
a(
110
(5,222 + +2)]|
2 ([ 0:) + 110 E
V o ( 02)l
2
2
M
2
+ ((1+a) (L+ Ha(21 - +
8
8
(10
In
B/
R
2
S
Tl (1 (1+a)
15
531 mm (48
1nndss
2
16
2 ([((0(++1,0)+++
700
Ing
E e ( 0.11
2a((
49/lw
12
12
Tl
+ In + + + 2))
32
2
2
(Pninn044397 +231 2,31+ 2) V w (w., 02 In 2 (533) + 2) 12 ?nods 2
+P no (13212
+ 2331 + [
o²
+((+a)( (1+
Tw
11-331 B1) Ln (189
2
2
M
(38)
2a(( 19
12
nw + bn (48 + 231 +2)
32
Combining the above inequalities we can get the descending relationship of the potential function:
12
2
E
V /,,,0,011
I
LIB 0 (110)) 2
a 10
E
Vellis
2
1 [[]
12
12
(1 + a) (B((((+++)))) - (()))
a
Tw
In
(5,322++))
2
El, (w), 0.)l
In
a
(5,3²1² +2)1
s(++a}|-
nw 15
-
5,1 (39122mm443
2
M
2 16
8
8 + no ddA
+ ((1+a)
(L+
Ha(21+
1
(2,))
31
L (1 (28112)
n
2
Since we have the following property according to Lemma:
0r) (11 ongo2
EI
V.. (w,,
2M
31021 nl
8 - )
[V((2)) (w 0,) S ((,) - 0.11
+ M
+
(L
nw
S K Vo (A,, 0,)
49m
137
1n (432
So we can further the above inequality as fol-
32
2
-p--21110 PB
2²
lows:
EE Vuf (wr, 0011 2-)122(("
Hn
2
M
1[V - E[[
(21 + 8
)dA²
< <- The (1 (1)a){ 2--33-22
15
nw
2
16
+ ( no
In
Vol 0
20(1 49mw
+ 12
m
+ In (4 + 2
2
32
2
- -122 (1 14 + 231 + VV 0,)1
1n no (18222
+ 2,31 + 2) V $ (2)
(5. + 2)) 0²
2
+1
M
12 =
{al 110
In
2
x2 (1 + a)
(1 + マラ2 (
2
2
15
531
(139
2
19mw
+ 12
+ (4 + 231 + 2))++22)22l
2
16
32
1
E
V
(((+a)
(014+300--23333322322
+ +2881
2

---

+1 +
In
El Vol(w), 01)1 + [1 + a +
2
+ a
+ a
1110
+
P
10
(5,321 + 2))
2
2
M
8 +
+ ((1+a)
Ha(l1
+ol no ds
Nw
7w
Then require the parameters satisfy: T6
31 10. (2,1 1) S da &mo S and
6
1/
So the inequality can be further simplified as:
E[V++1] - [[]]]
T (()))
Mw
2
Vo
(of(
0,)1
16
32
M 0²
110
17
+
+
d
64 M
16
Nw
(39)
Telescoping the above inequality we can get:
T-1
T
(Ww)) Φ
010
Poo 022
((nol
1
dA3
M
77
(40)


Reviews:
[{"review_id": "2ubrromFiR", "reviewer": "Reviewer_6MGP", "summary": "", "questions": "- the data on which the generators were trained on should me more clear in the experiments section\n\n- Table 2 (BERT large experiments) and 4 (ablating components of SAM-SGA) need more rigorous experiments; given my experience with RTE and MRPC, these tasks are notoriously random in nature (RTE performance varies by a lot for different seeds while finetuning or while performing in-context learning; MRPC task is highly unbalanced and can get high performance by sending all positive labels)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #6MGP:\n\n*Thank you for your thoughtful and positive comments! We have addressed all your questions and concerns in the following response. You can refer to the detailed responses for further clarification and verification.* \n\n***Q1: \"It is a slight overclaim that the adversarial setup works well for both BERT and generative style models.\"***  \nIt appears that you may be referring to the comparatively lower performance observed in the generative style models. We address this concern in the next questions which you can refer to. \n\n---\n\n***Q2: \"The benefits of adversarial training on perplexity numbers for OPT-350M in Table 3 are inconclusive.\"***  \nWe **expand experiments to more tasks and datasets for OPT-350m**. The contrastive results are listed in the table below, illustrating that our method SAM-SGA outperforms the other methods, which further **validates the effectiveness of our methods for generative style models**.\n\n| Method | #Bits (W-A) | CoLA  | MNLI  | MRPC  | QNLI  | QQP   | RTE   | SST2  | STSB  | Avg. |\n|----------------|:-------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------------------:|:--------------|\n| full-precision |    W32A32   | 57.51 | 84.54 | 83.33 | 90.30 | 90.72 | 69.68 | 93.23 | 88.13 |  82.18  |\n| Baseline       |     W4A8    | 55.69 | 84.03 | 82.60 | 89.73 | 90.25 | 67.15 | 92.77 | 87.93 |  81.27  |\n| QAT-GT         |     W4A8    | 55.94 | 83.85 | 83.58 | 89.85 | 90.32 | 68.23 | 93.12 | 88.03 |  81.62  |\n| QAT-Rand       |     W4A8    | 55.71 | 83.75 | 82.60 | 89.77 | 90.17 | 67.51 | 92.78 | 88.03 |  81.29  |\n| QAT-Rand-Auto. |     W4A8    | 55.37 | 83.69 | 83.09 | 89.96 | 90.27 | 67.51 | 93.12 | 87.67 |  81.34  |\n| **SAM-SGA**        |     W4A8    | 56.47 | 83.94 | 83.33 | 89.96 | 90.27 | 67.87 | 93.23 | 88.03 |  **81.64 (+0.37)**  |\n\nNote that \"QAT-Rand-Auto.\" denotes a more powerful QAT-based method, as suggested by Reviewer #2wRB.\n\n---\n\n\n***Q3: \"Both OPT-350M and OPT-1.3B have a large gap between full-precision and quantized setups.\"***  \nWe have to admit that for generative models, there exists a large gap between full-precision and quantized versions. It's been pointed out that quantization methods for generative style models suffer from homogeneous word embeddings and varied distribution of weights *[Ref 1]*.  \nNonetheless, compared to the other zero-shot quantization methods, our SAM-SGA brings a remarkable performance improvement. This can prove the effectiveness of our method.\n\n*[Ref 1] Compression of Generative Pre-trained Language Models via Quantization*\n\n---\n\n***Q4: \"It is unclear if the generators used are themselves trained on end-task data.\"***  \nActually, we directly use the third-party GPT-style PLMs (**without any further tuning**) as the generators, i.e., the generators **are not trained on any end-task data**.  Owing to their powerful generative abilities, we can easily obtain the generated sentences by prompting the generators with some simple instructions, such as \"Randomly generate some complete and fluent sentences:\". We will clarify it in our revision.\n\n---\n\n***Q5: \"The benefit of adversarial training is more for lower precision end-task results on BERT, rather than solving the access to the pretraining data problem for quantization-aware-training.\"***  \nIndeed, the primary focus of our work is to address the access to task-specific/domain-specific data problems using an adversarial setting in the context of quantization-aware training.  \nWhile it is notable that ZSAQ performs better for lower precision end-task, which indicates the effectiveness of our proposed approach. And we will emphasize the benefits in lower precision end-task results in the revision.\n\n---\n\n***Q6: \"The data on which the generators were trained on should be more clear in the experiments section.\"***  \nAs mentioned in Q4, we **do not train** the generators with any end-task data.\n\n---\n\n***Q7: \"Table 2 (BERT large experiments) and 4 (ablating components of SAM-SGA) need more rigorous experiments.\"***  \n1.  For Table2, following your suggestions, we show the full results of the GLUE benchmark and extend the table below. As seen, our method achieves consistent and significant performance improvements compared to the baseline on BERT-large, confirming the statement of our paper.\n| Method         | #Bits (W-A) |  CoLA |  MNLI |  MRPC |  QNLI |  QQP  |  RTE  |  SST2 |  STSB |  Avg. |\n|----------------|:-----------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----------:|:-----|\n| full-precision |    W32A32   | 64.47 | 83.12 | 86.03 | 92.37 | 90.85 | 65.71 | 93.35 | 89.22 | 83.14 |\n| Baseline       |     W8A8    | 64.80 | 83.33 | 86.76 | 92.27 | 90.79 | 62.45 | 93.46 | 89.12 | 82.87 |\n| QAT-GT         |     W8A8    | 66.61 | 83.29 | 87.75 | 92.54 | 90.81 | 65.34 | 93.23 | 89.26 | 83.60 |\n| QAT-Rand       |     W8A8    | 66.86 | 83.01 | 86.52 | 92.42 | 90.76 | 65.15 | 93.00 | 89.18 | 83.36 |\n| **SAM-SGA**        |     W8A8    | 66.08 | 83.50 | 87.25 | 92.55 | 90.87 | 66.06 | 94.04 | 89.37 | **83.72 (+0.85)** |\n| Baseline       |     W4A8    | 49.11 | 78.86 | 74.26 | 86.86 | 86.68 | 47.29 | 87.73 | 87.63 | 74.80 |\n| QAT-GT         |     W4A8    | 50.85 | 79.17 | 75.73 | 90.44 | 87.35 | 56.68 | 92.32 | 88.05 | 77.57 |\n| QAT-Rand       |     W4A8    | 45.63 | 79.02 | 70.34 | 87.26 | 87.17 | 47.29 | 78.84 | 88.08 | 72.95 |\n| **SAM-SGA**        |     W4A8    | 52.97 | 79.17 | 77.70 | 90.69 | 87.20 | 52.71 | 90.48 | 87.72 | **77.33 (+2.53)** |\n\n2. Similarly, we also report more ablation results in Table 4 as follows. Compared to the full SGA-SAM, the other variants consistently perform worse in most tasks, which can prove the importance and effectiveness of all components.\n| Method         |  CoLA |  MNLI |  MRPC |  QNLI |  QQP  |  RTE  |  SST2 |  STSB |  Avg. |\n|----------------|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:--------:|:-----|\n| Baseline       | 52.13 | 77.24 | 82.60 | 85.10 | 86.34 | 50.90 | 89.11 | 84.41 | 75.98 |\n| SGA-SAM (Ours) | 53.01 | 76.64 | 83.58 | 86.12 | 86.16 | 64.26 | 89.68 | 86.94 | 78.30 |\n| -w/o SGD       | 45.30 | 78.06 | 84.31 | 85.53 | 86.05 | 55.23 | 89.36 | 86.59 | 76.30 ($\\downarrow$2.00) |\n| -w/o SAM       | 50.74 | 76.29 | 82.60 | 85.74 | 85.93 | 63.53 | 89.44 | 86.46 | 77.59 ($\\downarrow$0.71) |\n| -w/o SGA & SAM | 43.70 | 76.02 | 82.80 | 85.33 | 85.56 | 56.04 | 89.48 | 86.11 | 75.63 ($\\downarrow$2.67) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the tackles an important problem of access to the original pretraining data while performing quantization aware training.\n- introduces an adversarial setup training setup which for quantization aware training in conjunction with sharpness aware minimization loss\n- adversarial training setup seems to benefit BERT models at lower precision weight quantization than simple ground truth-based quantization-aware-training\n- paper shows that sharpness-aware-minimization works in an adversarial-setup for language models", "reasons_to_reject": "- the claim that the adversarial setup works well for both BERT and generative style models is a slight overclaim\n- the benefits of adversarial training on perplexity numbers for OPT-350M in Table 3 are inconclusive\n- both OPT-350M and OPT-1.3B have a large gap between full-precision and quantized setups\n- although the paper wants to solve the access to data problem using an adversarial setting, it is unclear if the generators used are themselves trained on end-task data\n- the benefit of adversarial training is more for lower precision end-task results on BERT (as seen for W2A8 in Table 1), rather than solving the access to the pretraining data problem for quantization-aware-training\n", "questions_to_authors": "- the data on which the generators were trained on should me more clear in the experiments section\n\n- Table 2 (BERT large experiments) and 4 (ablating components of SAM-SGA) need more rigorous experiments; given my experience with RTE and MRPC, these tasks are notoriously random in nature (RTE performance varies by a lot for different seeds while finetuning or while performing in-context learning; MRPC task is highly unbalanced and can get high performance by sending all positive labels)", "paper_topic_and_main_contributions": "The paper tackles two issues related to LM quantization: access to original training data for Quantization aware training and loss of generality for quantized models due to sharper loss landscapes. From the earlier works referenced by this paper, it seems that the concept of sharpness aware quantization is more prevalent in computer vision models and underexplored for LMs. The paper also provides an analysis of the sharpness-aware-minimization algorithm in an adversarial min-max optimization setting.\n\nThe solution to the first problem is provided by introducing an adversarial training setup, where a generator generates synthetic data for both the non-quantized teacher model and the quantized student model. The generator's goal is to produce data which maximizes the discrepancy between the teacher and the student model. The goal of the student model (which is being undergoing quantization aware training) is to minimize this discrepancy.\n\nThe solution to the generality problem is tackled by introducing a loss-landscape sharpness aware term in the min-max adversarial loss setup for quantization aware training using a generator.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fZIjpxUAZk", "reviewer": "Reviewer_2wRB", "summary": "", "questions": "Are the adversarial training stages visible on the loss curve? How much effort is required to battle the instability often present in adversarial training? Do the hyperparameters like \\eta require careful tweaking for best performance?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #2wRB:\n\n*Thank you for your careful evaluation and constructive feedback! We have addressed all your questions and concerns in the following response. You can refer to the detailed responses for further clarification and verification.*\n\n\n***Q1: \"Not convincing motivation for zero-shot quantization for generative tasks (like the OPT model on wikitext2).\"***  \nWe respectfully remind you that there exist some quantization scenarios, such as quantizing the financial and medical chatbots, which **rely the task-specific/domain-specific data that are difficult to collect**. Hence, we believe it is critical to explore the zero-shot quantization approach for generative models/tasks. We will clarify it to make our motivation more convincing in the revision.\n\n---\n\n\n***Q2: \"A more suitable baseline in Table 3.\"***  \nFollowing your suggestions, we implement this powerful QAT-based baseline (denoted as QAT-Rand-Auto.) and add its results in the following table. As seen, compared to the QAT-Rand, the added \"QAT-Rand-Auto.\" indeed achieves better performance, proving its effectiveness. \n|     Method     | #Bits (W-A) | PTB   | WikiText103 | WikiText2 |\n|--------------|:-------------:|:-------:|:-------------:|:-----------:|\n|                | (W-A)       | PPL   | PPL         | PPL       |\n| full-precision |    W32A32   | 28.58 |    13.38    |   19.21   |\n| Baseline       |     W8A8    | 56.02 |    48.16    |    55.4   |\n| QAT-GT         |     W8A8    | 50.68 |    42.15    |   46.29   |\n| QAT-Rand       |     W8A8    | 54.59 |    48.69    |   60.46   |\n| \"QAT-Rand-Auto.\" |     W8A8    | 52.97 |    47.38    |   53.38   |\n| **SAM-SGA**        |     W8A8    | 46.59 |    46.56    |   54.08   |\n| Baseline       |     W4A4    | 60.25 |    56.16    |   62.89   |\n| QAT-GT         |     W4A4    | 55.59 |    42.01    |   51.72   |\n| QAT-Rand       |     W4A4    |  59.4 |    55.32    |   63.63   |\n| \"QAT-Rand-Auto.\" |     W4A4    | 57.34 |    55.13    |   61.68   |\n| **SAM-SGA**        |     W4A4    | 51.19 |    54.32    |   61.22   |\n\nHowever, we should note that our **SAM-SGA can still outperform it in many tasks**. And the results in the response of Q2 (Reviewer #6MGP) also show our method brings remarkable performance improvement in other OPT-based tasks. These results prove the superiority of our method. \n\n---\n\n***Q3: \"It is difficult to analyze and get further improvements due to this method involves many components and stages.\"***  \nActually, we have conducted a detailed ablation study in Section 5.3 to investigate the effect of all important components and stages. These results **consistently prove the effectiveness and importance of all components**. \n\nFor example, we have compared SAM-SGA with *w/o SGA*, *w/o SAM*, and *w/o SGA&SAM* respectively to identify the influence of each stage. And our results in Table 4 (more detailed results can be seen in the response of Q7, Reviewer #6MGP) indicate that removing any component will cause performance degradation and our method will perform much worse without SGA, showing the importance of the adversarial training stage.\n\n---\n\n***Q4: \"Are the adversarial training stages visible on the loss curve?\"***  \nDue to the lack of access to submit a PDF file, we can hardly provide the visualization of the loss curve. Instead, we provide the table that documents the variation trend of student_loss and generator_loss respectively. We can see the student_loss decreasing and generator_loss increasing as time varies and eventually reaching a relatively stable plateau during the final stage.\n\n| training loss on CoLA |              |                | 　 | training loss on MNLI |              |                |\n|:---------------------:|:------------:|:--------------:|----|:---------------------:|:------------:|:--------------:|\n|          step         | student_loss | generator_loss |    |          step         | student_loss | generator_loss |\n|           5           |     0.377    |      0.880     |    |           5           | 0.321        | 1.014          |\n|           10          |     0.327    |      1.214     |    |           10          | 0.301        | 1.183          |\n|           15          |     0.341    |      1.449     |    |           15          | 0.220        | 1.407          |\n|           20          |     0.323    |      1.599     |    |           20          | 0.183        | 1.575          |\n|           25          |     0.308    |      1.705     |    |           25          | 0.160        | 1.690          |\n|           30          |     0.295    |      1.784     |    |           30          | 0.149        | 1.764          |\n|           35          |     0.279    |      1.844     |    |           35          | 0.145        | 1.815          |\n|           40          |     0.264    |      1.892     |    |           40          | 0.135        | 1.846          |\n|           45          |     0.249    |      1.927     |    |           45          | 0.129        | 1.867          |\n|           50          |     0.244    |      1.950     | 　 |           50          | 0.118        | 1.878          |\n\n\n---\n\n***Q5: \"How much effort is required to battle the instability often present in adversarial training? Do the hyperparameters like $\\eta$ require careful tweaking for best performance?\"***  \nAlthough adversarial training usually suffers from instability, it is relatively stable in our SAM-SGA framework.\n\nFirst, we theoretically prove the convergence rate under the constraints that $\\eta_{\\theta}\\leq64\\kappa^2\\eta_{\\omega}$ and $\\eta_{\\omega}\\leq\\min\\\\{\\frac{1}{64\\kappa l(2\\beta l+1)^2},\\frac{1}{128\\kappa^2l}\\\\}$. \n\nSecond, we empirically conduct a grid search for generator/student learning rates ($\\eta_{\\theta}$/$\\eta_{\\omega}$) and we find that the results do not exhibit significant fluctuations while demonstrating stability within a narrow range. These prove the effectiveness of our method.\n\n| BERT-base,W4A8,MNLI |       |       |       |       |\n|:-------------------------:|:-------:|:-------:|:-------:|:-------:|\n| $\\eta_{\\theta}$ \\ $\\eta_{\\omega}$ | 5E-06 | 1E-05 | 2E-05 | 5E-05 |\n| 5E-06                   | 76.23 | 75.84 | 75.90 | 76.50 |\n| 1E-05                   | 76.49 | 76.14 | 74.99 | 74.99 |\n| 2E-05                   | 76.47 | 75.95 | 76.56 | 74.38 |\n| 5E-05                   | 75.96 | 76.64 | 76.04 | 75.72 |\n\n| BERT-base,W4A8,QNLI |       |       |       |       |\n|:-------------------------:|:-------:|:-------:|:-------:|:-------:|\n| $\\eta_{\\theta}$ \\ $\\eta_{\\omega}$ | 5E-06 | 1E-05 | 2E-05 | 5E-05 |\n| 5E-06                   | 84.46 | 83.76 | 85.00 | 86.12 |\n| 1E-05                   | 85.45 | 84.68 | 83.01 | 84.70 |\n| 2E-05                   | 84.90 | 85.70 | 85.21 | 83.56 |\n| 5E-05                   | 85.65 | 84.57 | 84.04 | 82.92 |\n\n\n---\n\n***Typos: 447, \"will be much slight\" should be \"will be much slighter\" 466, \"universality our method\" should be \"universality of our method\".***  \nWe apologize for our oversight in paper writing and appreciate your careful reading. We will revise these typos in our paper and polish our writing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method deals with the important problem of reducing weights of transformers to below 8-bit. Theoretical analysis has been applied to gauge the convergence rate of the proposed method.", "reasons_to_reject": "The motivation for zero-shot quantization for generative tasks (like the OPT model on wikitext2) is not convincing. Since generative models can start from virtually any token in the embedding matrix and fill-in the rest with the autoregressive generation, there should not be a lack of fine-tuning dataset for them. As by table 3, QAT-GT is much more advantageous than QAT-Rand and the proposed method for Wikitext, hence one would expect the procedure described above (start from seeds in embedding matrix and autoregressively generate the rest) would server as a more suitable and stronger baseline.\n\nThe method has multiple stages. It is made of three stages: first training with floating point to get a pretrained model, then do PTQ on the pretrained model with range-shaping techniques (SmoothQuant and GPTQ) to get the initial quantized model, then a QAT fine-tuning process is introduced to recover accuracy loss encountered in the PTQ process. As this method involves many components and stages, it is difficult to analyze and get further improvements.\n\n\n", "questions_to_authors": "Are the adversarial training stages visible on the loss curve? How much effort is required to battle the instability often present in adversarial training? Do the hyperparameters like \\eta require careful tweaking for best performance?", "paper_topic_and_main_contributions": "The paper proposes a method to improve the accuracy of quantized transformer models after QAT (which itself is placed after first floating-point training). The method modifies the loss functions of QAT to include a loss landscape-aware term (namely the sharpness-aware) that encourages the learned model to be more robust to epsilon-perturbations. Experiments on BERT and OPT models demonstrate the efficacy of the method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "447, \"will be much slight\" should be \"will be much slighter\"\n466, \"universality our method\" should be \"universality of our method\""}, "extra_text": {}}, {"review_id": "np3zveaSSN", "reviewer": "Reviewer_FTK8", "summary": "", "questions": "The authors can expand the scope of the experiments to involve more tasks and datasets to verify the generality and effect of the ZSAQ method in different domains.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #FTK8:\n\n*Thank you for your careful evaluation and constructive feedback! We have addressed all your questions and concerns in the following response. You can refer to the detailed responses for further clarification and verification.*\n\n\n***Q1: \"Not enough comparisons with other zero-shot quantization methods.\"***  \nFollowing your suggestions, we compare our method with more zero-shot quantization methods, i.e., PTQ-vanilla (vanilla post-training quantization), PTQ-PEG *[Ref 1]* and PTQ-Adaround *[Ref 2]*. Specifically, taking the BERT-base as an example, we show the contrastive results in the W8A8 and W4A8 settings as follows. As seen, our method SAM-SGA **outperforms** the other zero-shot quantization methods **by a clear margin**, especially in the low-bit setting. We will add more contrastive results in our revised paper.\n| Method         | #Bits (W-A) |  CoLA |  MNLI |  MRPC |  QNLI |  QQP  |  RTE  |  SST2 |  STSB |  Avg. |\n|----------------|:-----------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| full-precision |    W32A32   | 60.82 | 83.12 | 85.05 | 90.52 | 89.91 | 65.34 | 92.43 | 88.84 | 82.00 |\n| PTQ-vanilla    |     W8A8    | 58.29 | 82.08 | 80.64 | 88.72 | 87.47 | 62.82 | 90.37 | 88.10 | 79.81 |\n| PTQ-PEG *[Ref 1]*       |     W8A8    | 57.04 | 83.15 | 85.29 | 89.95 | 89.36 | 64.62 | 90.71 | 88.34 | 81.06 |\n| PTQ-Adaround *[Ref 2]*  |     W8A8    | 61.32 | 83.68 | 84.07 | 90.43 | 89.84 | 66.06 | 92.32 | 88.45 | 82.02 |\n| **SAM-SGA (Ours)** |     W8A8    | 63.70 | 82.93 | 85.05 | 90.52 | 89.93 | 67.51 | 92.08 | 88.96 | **82.59** |\n| PTQ-vanilla    |     W4A8    |  1.88 | 33.91 | 31.62 | 50.06 | 62.97 | 52.71 | 48.62 | -1.23 | 35.07 |\n| PTQ-PEG  *[Ref 1]*      |     W4A8    |  0.00 | 37.91 | 31.62 | 50.23 | 63.18 | 47.29 | 73.62 | 11.83 | 39.46 |\n| PTQ-Adaround *[Ref 2]*  |     W4A8    | 55.67 | 72.59 | 79.90 | 87.13 | 86.49 | 64.62 | 91.28 | 82.21 | 77.49 |\n| **SAM-SGA (Ours)** |     W8A8    | 53.01 | 76.64 | 83.58 | 86.12 | 86.16 | 64.26 | 89.68 | 86.94 | **78.30** |\n\n*[Ref 1] Understanding and Overcoming the Challenges of Efficient Transformer Quantization*\n\n*[Ref 2] Up or Down? Adaptive Rounding for Post-Training Quantization*\n\n---\n\n***Q2: \"Expand the scope of the experiments to involve more tasks and datasets to verify the generality and effect of the ZSAQ method in different domains.\"***  \nWe further conduct experiments on more tasks and datasets for OPT-350m and present the results in the following table. It can be seen that our method SAM-SGA outperforms the other methods, which verifies the generality and effectiveness of our approach.\n\n| Method         | #Bits (W-A) | CoLA  | MNLI  | MRPC  | QNLI  | QQP   | RTE   | SST2  | STSB  | Avg. |\n|----------------|:-------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:---------:|\n| full-precision |    W32A32   | 57.51 | 84.54 | 83.33 | 90.30 | 90.72 | 69.68 | 93.23 | 88.13 |  82.18  |\n| Baseline       |     W4A8    | 55.69 | 84.03 | 82.60 | 89.73 | 90.25 | 67.15 | 92.77 | 87.93 |  81.27  |\n| QAT-GT         |     W4A8    | 55.94 | 83.85 | 83.58 | 89.85 | 90.32 | 68.23 | 93.12 | 88.03 |  81.62  |\n| QAT-Rand       |     W4A8    | 55.71 | 83.75 | 82.60 | 89.77 | 90.17 | 67.51 | 92.78 | 88.03 |  81.29  |\n| QAT-Rand-Auto. |     W4A8    | 55.37 | 83.69 | 83.09 | 89.96 | 90.27 | 67.51 | 93.12 | 87.67 |  81.34  |\n| **SAM-SGA**        |     W4A8    | 56.47 | 83.94 | 83.33 | 89.96 | 90.27 | 67.87 | 93.23 | 88.03 |  **81.64 (+0.37)**  |\n\nNote that \"QAT-Rand-Auto.\" denotes a more powerful QAT-based method, as suggested by reviewer #2wRB.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors migrate generative adversarial techniques in computer vision to this domain. For the first time in this domain, the authors employ adversarial learning methods to quantify discriminative and generative PLMS in a zero-shot manner and provide rigorous convergence analysis.\n\n2. The ZSAQ method achieves consistent and significant performance gains on both discriminative and generative language models across different tasks, with an average score improvement of 6.98 units over the baseline method.\n\n3. In the paper, the author proved that SAM-SGA algorithm can improve the generalization ability of the model, and verified it on the model of billion-level large models. I think this is very worthy of reference for training large models under insufficient resources.\n", "reasons_to_reject": "This paper does not fully discuss and compare other existing zero-shot quantization methods, especially for NLP applications. A comprehensive comparative analysis with these methods can better assess the strengths of ZSAQ.", "questions_to_authors": "The authors can expand the scope of the experiments to involve more tasks and datasets to verify the generality and effect of the ZSAQ method in different domains.", "paper_topic_and_main_contributions": "Quantization is a promising approach for reducing memory overhead and accelerating inference. This paper quantizes pre-trained language models without training data, aiming to achieve zero-shot quantization without much loss of accuracy. This is important to protect privacy and ensure data security.\n\nThe authors propose a novel zero-shot sharpness-aware quantization (ZSAQ) framework for the zero-shot quantization of various PLMs. The main method is to improve the quantization accuracy and improve the model generalization ability by optimizing the minimax problem. At the same time, the authors provide a theoretical convergence guarantee for the SAM-SGA algorithm which aims to solve the minimax optimization problem (ZSAQ).  \n\nI think ZSAQ has the potential to scale to larger language models, and the paper demonstrates that SAM-SGA does indeed lead to better model generalization.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jgkHgrNY4M", "reviewer": "Reviewer_vWQy", "summary": "", "questions": "1. The novelty and technique improvement of this submission may be limited. It resorts to Zero-shot Adversarial Quantization (https://arxiv.org/abs/2103.15263) and Sharpness-Aware Minimization variant (https://arxiv.org/abs/2010.01412) to tackle the zero-shot quantization of NLP, and both of them are not the main contributions of this submission.\n2. The results of ZeroQuant (Baseline) in Table 1 and 2 is much lower than that in the original paper, i.e., original W8A8-Avg 84.07 v.s. W8A8-Avg 81.78, which is not mentioned in this submission. The author should provide more information about this. From my perspective, this may be caused by the quantization of multi-head self-attention’s weights and the finetuning, but the author should prove it or provide some other explanations.\n3. The ablation results on the full GLUE should be provided in section 5.3. For example, in Table 4, the ablation results on MRPC indicate that SGA has negative effects, which does not support the authors’ claim “without the SGA, our SAM-SGA would perform much worse” in line 490. Besides, according to Table 4, SGA seems much more important than SAM and the authors do not mention it.\n4. The Task Generalization and Visualization of Landscape experiment do not indicate that the proposed framework leads to better generalization. The better generalization may resort to the SAM which is not the contribution of this submission. Hence, the claim “These results prove that our SAM-SGA can smooth the loss landscape and improve the generalization of PLMs effectively.” should be reconsidered.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #vWQy:\n\n*Thank you for your careful evaluation and constructive feedback! We have addressed all your questions and concerns in the following response. You can refer to the detailed responses for further clarification and verification.*\n\n***Q1: \"Limited novelty and technical improvement.\"***  \nWe would like to clarify that our proposed ZSAQ method is **not a straightforward combination** of the two references you mentioned.  \n\n1) First, it is **nontrivial to apply zero-shot adversarial quantization in the NLP applications**, as backpropagation on discrete words is not reasonable. To the best of our knowledge, we are the (nearly) the first to migrate generative-adversarial quantization techniques in computer vision to the NLP domain, which is also well recognized by Reviewer #FTK8 (commented \"For the first time in this domain\").\n\n2) Second, different from simply adopting the SAM technique, we **provide a rigorous analysis** for our SAM-SGA and theoretically establish a convergence rate of $\\mathcal{O}(1/\\sqrt{T})$ for both nonconvex-strongly-concave and nonconvex-PL cases, shedding light on the method's effectiveness even under challenging conditions. The contribution of theoretical analysis is well recognized by Reviewer #2wRB and #FTK8.\n\n\n---\n\n\n***Q2: \"Explanations for the results that ZeroQuant (Baseline) in Table 1 and 2 is much lower than that in the original paper.\"***  \nSince the settings (e.g., random seed, hyper-parameters) for fine-tuning the full-precision models might be different, the final results of ZeroQuant are different from those of the original paper. However, it should be emphasized that we **implement the ZeroQuant using the official codebase**. We will clarify it in the revision to make the results more convincing.\n\n\n---\n\n\n***Q3: \"The ablation results on the full GLUE should be provided in section 5.3.\"***  \nFollowing your suggestions, we provide all the ablation experiment results of full GLUE datasets in the following table. As seen, compared to the full SGA-SAM, the other variants consistently perform worse in most tasks, which can prove the importance and effectiveness of all components.\n\n| Method         |  CoLA |  MNLI |  MRPC |  QNLI |  QQP  |  RTE  |  SST2 |  STSB |  Avg. |\n|----------------|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:-----|\n| Baseline       | 52.13 | 77.24 | 82.60 | 85.10 | 86.34 | 50.90 | 89.11 | 84.41 | 75.98 |\n| SGA-SAM (Ours) | 53.01 | 76.64 | 83.58 | 86.12 | 86.16 | 64.26 | 89.68 | 86.94 | 78.30 |\n| -w/o SGD       | 45.30 | 78.06 | 84.31 | 85.53 | 86.05 | 55.23 | 89.36 | 86.59 | 76.30 ($\\downarrow$2.00) |\n| -w/o SAM       | 50.74 | 76.29 | 82.60 | 85.74 | 85.93 | 63.53 | 89.44 | 86.46 | 77.59 ($\\downarrow$0.71) |\n| -w/o SGA & SAM | 43.70 | 76.02 | 82.80 | 85.33 | 85.56 | 56.04 | 89.48 | 86.11 | 75.63 ($\\downarrow$2.67) |\n\n---\n\n***Q4: \"The claim “These results prove that our SAM-SGA can smooth the loss landscape and improve the generalization of PLMs effectively.” should be reconsidered.\"***  \nAs stated in the Introduction, minimizing the difference between the teacher and (quantized) student models would be prone to over-fitting problems, leading to poor model generalization. Hence, we are inspired by the SAM and propose to **incorporate it as a key component in our SAM-SGA**. Although the SAM itself is not the contribution of our paper, **using it to tackle the challenge of zero-shot adversarial quantization and providing a theoretical analysis are our contributions**. We will clarify it in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The zero-shot quantization is a practical research problem for textual PLM and even LLM. This work may be one of the first attempts to transfer Zero-shot Adversarial Quantization from CV to NLP. The theoretical convergence analysis and extensive experimental results indicate its effectiveness.\n", "reasons_to_reject": "1. The novelty and technique improvement of this submission may be limited. It resorts to Zero-shot Adversarial Quantization (https://arxiv.org/abs/2103.15263) and Sharpness-Aware Minimization variant (https://arxiv.org/abs/2010.01412) to tackle the zero-shot quantization of NLP, and both of them are not the main contributions of this submission.\n2. The results of ZeroQuant (Baseline) in Table 1 and 2 is much lower than that in the original paper, i.e., original W8A8-Avg 84.07 v.s. W8A8-Avg 81.78, which is not mentioned in this submission. The author should provide more information about this. From my perspective, this may be caused by the quantization of multi-head self-attention’s weights and the finetuning, but the author should prove it or provide some other explanations.\n3. The ablation results on the full GLUE should be provided in section 5.3. For example, in Table 4, the ablation results on MRPC indicate that SGA has negative effects, which does not support the authors’ claim “without the SGA, our SAM-SGA would perform much worse” in line 490. Besides, according to Table 4, SGA seems much more important than SAM and the authors do not mention it.\n4. The Task Generalization and Visualization of Landscape experiment do not indicate that the proposed framework leads to better generalization. The better generalization may resort to the SAM which is not the contribution of this submission. Hence, the claim “These results prove that our SAM-SGA can smooth the loss landscape and improve the generalization of PLMs effectively.” should be reconsidered.", "questions_to_authors": "1. The novelty and technique improvement of this submission may be limited. It resorts to Zero-shot Adversarial Quantization (https://arxiv.org/abs/2103.15263) and Sharpness-Aware Minimization variant (https://arxiv.org/abs/2010.01412) to tackle the zero-shot quantization of NLP, and both of them are not the main contributions of this submission.\n2. The results of ZeroQuant (Baseline) in Table 1 and 2 is much lower than that in the original paper, i.e., original W8A8-Avg 84.07 v.s. W8A8-Avg 81.78, which is not mentioned in this submission. The author should provide more information about this. From my perspective, this may be caused by the quantization of multi-head self-attention’s weights and the finetuning, but the author should prove it or provide some other explanations.\n3. The ablation results on the full GLUE should be provided in section 5.3. For example, in Table 4, the ablation results on MRPC indicate that SGA has negative effects, which does not support the authors’ claim “without the SGA, our SAM-SGA would perform much worse” in line 490. Besides, according to Table 4, SGA seems much more important than SAM and the authors do not mention it.\n4. The Task Generalization and Visualization of Landscape experiment do not indicate that the proposed framework leads to better generalization. The better generalization may resort to the SAM which is not the contribution of this submission. Hence, the claim “These results prove that our SAM-SGA can smooth the loss landscape and improve the generalization of PLMs effectively.” should be reconsidered.", "paper_topic_and_main_contributions": "This work proposes a zero-shot quantization framework for textual PLM, which makes Zero-shot Adversarial Quantization (https://arxiv.org/abs/2103.15263) practical for NLP and boosted the generalization of quantized model with a proposed Sharpness-Aware Minimization variant (https://arxiv.org/abs/2010.01412), namely SAM-SGA. Besides, the authors provide the theoretical convergence analysis for this adversarial optimization problem. The experimental results on 11 tasks with generative and discriminative PLM empirically demonstrates the effectiveness of this framework, especially for low-bit weights.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript addresses critical challenges in the quantization of language models (LMs), focusing on two primary concerns: access to original training data for Quantization Aware Training (QAT) and the loss of generality due to sharper loss landscapes in quantized models. It introduces an adversarial training setup and incorporates sharpness-aware terms in the min-max loss function for QAT. Below is a synthesis of reviewers' perspectives on the paper's strengths and weaknesses.

Strengths:

1. Novelty and Relevance: The paper addresses important problems in LM quantization and introduces new paradigms like adversarial training and sharpness-aware minimization to the domain. These have been prevalent in computer vision but are largely underexplored for LMs.
2. Theoretical Analysis: The paper provides a robust theoretical framework, particularly through its introduction of a sharpness-aware-minimization algorithm in an adversarial min-max optimization setting. This is backed by convergence rate analysis.
3. Empirical Results: The paper demonstrates the efficacy of its method on established models like BERT and OPT, highlighting how the method outperforms the baseline, particularly in terms of model generalization.
4. Transfer of Techniques: The manuscript successfully adapts techniques like adversarial training from computer vision to the domain of NLP, which is commendable.

Weaknesses:
1. Inconclusive Evidence: Certain empirical results, such as the benefits of adversarial training on perplexity numbers for OPT-350M, are not definitively proven.
2. Complexity of the Method: The method involves multiple stages, including floating-point training, post-training quantization (PTQ), and QAT fine-tuning, making it challenging to analyze and extend.
3. Unclear Motivation for Certain Choices: Some reviewers pointed out that the motivation behind zero-shot quantization for generative tasks, and the specific choice of adversarial setup, are not convincingly argued.
Decision: Accept-Main