Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
Authors: Wanyun Cui
Qianle Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
parameter quantization
Primary area: deep_learning_architectures

Abstract: This paper reveals the phenomenon of parameter heterogeneity in large language models (LLMs). We find that a small subset of ``cherry'' parameters exhibit a disproportionately large influence on model performance, while the vast majority of parameters have minimal impact. This heterogeneity is found to be prevalent across different model families, scales, and types. Motivated by this observation, we propose CherryQ, a novel quantization method that unifies the optimization of mixed-precision parameters. CherryQ identifies and preserves the critical cherry parameters in high precision while aggressively quantizing the remaining parameters to low precision. Extensive experiments demonstrate the effectiveness of CherryQ. CherryQ outperforms existing quantization approaches in terms of perplexity and downstream task performance. Notably, our 3-bit quantized Vicuna-1.5 exhibits competitive performance compared to their 16-bit counterparts. These findings highlight the potential of CherryQ for enabling efficient deployment of LLMs by taking advantage of parameter heterogeneity.

Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
Wanyun Cui't Qianle Wang"i
Shanghai University of Finance and Economics
Moo Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics cui. wanyun@sueee edu, cn, wq120000111estu sufe. edu, cn
Abstract
This paper reveals the phenomenon of parameter heterogeneity in large language models (LLMs). We find that a small subset of "cherry" parameters exhibit a disproportionately large influence model performance, while the vast majority of parameters have minimal impact. This heterogeneity is found to be prevalent across different model families, scales, and types. Motivated by this observation, we propose CherryQ, a novel quantization method that unifies the optimization of mixed-precision parameters. CherryQ identifies and preserves the critical cherry pa- rameters in high precision while aggressively quantizing the remaining parameters low precision. Extensive experiments demonstrate the effectiveness CherryQ. Q. CherryQ outperforms existing quantization approaches in terms of perplexity and downstream task performance. Notably, our 3-bit quantized Vicuna-1.5 exhibits competitive performance compared to their 16-bit counterparts.
Introduction
The rapid development of large language models (LLMs) has increased the demand of efficient deployment in various environments 21 23]. However, the parameter size poses significant challenges for GPU memory requirements. Quantization, which reduces the bit-width of model parameters, has emerged solution alleviate memory constraints of LLM deployment [2
18 25 26].
Low-precision parameter representation leads quantization errors. Surprisingly, existing research has shown that I.LMs exhibit high robustness for quantization errors even for low-bit settings. For example, although 4-bit quantization can only represent 16 distinct values, the simplest round-to-ncarest strategy does not significantly degrade performance [17]. This raises the question: what causes LLMs be robust quantization?
We explore the parameters and answer this question via Parameter Heterogeneity, which refers to the significant variation in the influence of quantization different parameters. We reveal that for the vast majority (> 99%) of normal parameters, the effect their quantization to the model minimal and can thus be alleviated or ignored. However, there exists small subset (< 1%) of "cherry" parameters for which the effect are substantial and hard to mitigate
Consider Figure Ta as example. We show scatter plot of the impacts on the model loss when perturbing each individual parameter a parameter matrix from LLaMA2-76 [23]. The derivation
of impacts is detailed in 3 While 99% of the parameters fall within the range of (0, 0.1), I), small
subset of "cherry" parameters exhibits values ranging from (5, 30), which is 50-300 times higher
"Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

30.0
200.0
1%
1%
1%
24.0
150.0
99%
Impact
18.0
2000
4000
0.0
2000
4000
Iippact
0.0
100.0
12.0
50.0
6.0
0.0
0.0
1000
2000
3000
4000
0
1000
2000
3000
4000
1000
2000
3000
4000
Parameter Index
Parameter Index
Parameter Index
(a) LLaMA2 7B
(b) LLaMA2 13B
(c) Mistral 7B
layers 15. self attn. v v_proj
layers 15. self attn y_rojj
layers 15. self attn in. 9- pro
2.8
6.
60.0
04
0.05
1%
Impact
0.0
3.0
2000
4000
99%
Impact
45.0
0.0
2000
99%
Impact
2.1
0.0
2000
4000
99%
400.0
30.0
2.4
200.0
15.0
0.0
0.0
2000
3000
2000
3000
4000
Parametor Index
Parameter Index
Parameter Index
(d) Gemma 7B layers. 25 mlp.down. down_proj
(c) Vicuna- 1.5 7B
Vicuna-1.5 13B
layers. , 15.
layers. 16. self sell_attn_____eeeeeeeeeeeeeeeeee aattn.v_prop
Figure 1: Scatter plot of parameter impacts in different LLMs. We randomly sampled 4096 parameters from the corresponding parameter matrix. Each point represents the impact of an individual parameter. Insets show the zoomed-in y-axis. The heterogeneity is found across different model scales [1a 1b).
different model families [Ic] Id). and both base models and chat models [Ic
than the maximun value of the remaining 99% of parameters. Moreover, this phenomenon not an isolated case. We observed similar patterns across different scales of LLMs (Figure [T][ different families of LLMs, including Mistral [ (Figure Tc and Gemma (2 (Figure ldp, and both base
models and chat models V.5 S Figure T
The consistent presence of this phenomenon
suggests that parameter heterogeneity is an inherent characteristic of LLMs.
Therefore, 99% of normal parameters explain the high robustness of LLMs to quantization errors. However, the small number of cherry parameters still leads to performance degradation under quantization. Consequently, the key to reducing quantization errors lies in addressing the quantization of cherry parameters.
The parameter heterogeneity also explains the previously discovered effectiveness of -ixeedprecissoo quantization strategies 66
12 17 [8]. By preserving small proportion of parameters with high precision, the quantization performance can be effectively improved. Based on the heterogeneity, this strategy alleviates the impact of cherry parameters model performance by maintaining their precision.
Indeed, mixed-precision strategies can effectively address the quantization error issue of cherry parameters. However, the core challenge lies in identifying cherry parameters based on specific metrics. Different metrics of parameter effects include weights 6
13 [8] activations [17 24]
output changes (8) as well as the impact of parameters on model loss used in this paper. Based on the parameter heterogeneity, we argue that effective cherry parameter identification metrics should exhibit high heterogeneity, clearly distinguishing between cherry parameters and normal parameters. Accordingly, we compare three different metrics in Sec and and find that the impact best distinguishes cherry parameters from normal parameters. The experimental results in Sec 6655 verify that choosing metrics with higher discriminative capability indeed leads to better performance.
Based on the above analysis, we design the CherryQ quantization algorithm, which selects cherry parameters based on the impact metric and end-to-end optimizes parameters with mixed precisions. Extensive experiments on various models and benchmarks demonstrate the effectiveness of CherryQ. It consistently yields the lowest perplexity in most settings. Notably, our 3-bit Vicuna- 1.5 model

---

exhibits performance on par with the 16-bit counterpart on Vicuna-bench [5]. Our 2-bit quantization method significantly outperforms the SOTA approaches.
In summary, by systematically revealing parameter heterogeneity in LLMs, we answer the following questions:
1. What causes the high robustness of LLMs quantization? It is due to the 99% of normal parameters in parameter heterogeneity.
2. Why is mixed-precision quantization effective? This strategy addresses the quantization problem of the very few cherry parameters in parameter heterogeneity.
3. How to find the optimal mixed -arammeerr selection strategy? Based on heterogeneity that distinguishes normal parameters from cherry parameters, the impact-based metric demonstrates the highest discriminative capability.
4. How to quantize parameters according to parameter heterogeneity? We propose CherryQ based on these findings. Extensive empirical results verify its effectiveness in 2-, 3-, 4-bit quantization.
2 Related Work
Quantization Strategies for LLMs Various quantization strategies have been proposed in the literature to reduce the precision of weights and activations while maintaining acceptable accuracy. These strategies can be broadly classified into post-training quantization and quantization- training [14]. Post-training quantization methods, such as OBD, OBS, and GPTQ, directly quantize training methods, such as LLM-QAT [8]. incorporate quantization operations into the training process to jointly optimize the quantized model. Some works also explore mixed-precision quantization [12 and adaptive quantization bins 7 to achieve better trade-off between accuracy and efficiency.
the pre-trained model without fine-tuning [[5] 10 99. On the other hand, quantization-awwn
Outliers in Language Model Quantization The idea of modeling parameter outliers in LLM quantization is not new. Exploring outliers mainly includes the perspectives of magnitude [8] and activations [4] [) For example. from the magnitude perspective, QLoRA assumes that parameters follow a Gaussian distribution / and designs information-theonm theoreiiccl optimal quantized bins based on this assumption. [18] keeps outlier parameters in 16- bit precision. From the activation perspective, 17 migrates the outlier amplifier to subsequent modules through an equivalent transformation. Additionally, SM also measures outliers from the perspective of parameter impact [12). To the best of our knowledge, our work is the first to systematically reveal the outliers (heterogeneity) of parameter impact across different models, and show more pronounced imbalance in parameter impacts compared to magnitudes ($ 6.55. Furthermore, we propose a method to unify outlier (cherry) parameter optimization and normal parameter optimization, addressing the optimization challenges of heterogeneous parameters.
3 Quantifying the Impact of Parameters on Model Performance
The impact of parameters model performance is quantified by the increase of the training loss when perturbing the parameter weight, which is widely used in post-training quantization approaches [I5] 10 9. We adopt second-order Taylor approximation of the training loss parameter perturbation. Given a parameter w; and small perturbation A applied to it, such that w, + w; + A, the change in the training loss can be expressed
L L(w; + A) L L(w:)) 9 HH, 2HH +0A2)
(I)
where 9: = E aL represents the expected gradient of the loss with respect to w,, and H., = E Bw:
02
denotes the i-th value of the Hessian matrix of the loss. Since the target model is well-converged model, we can assume that g, Rs 0, simplifying the expression to:
L(w; + +A) L(w,) ~ A
(2)
3

---

Therefore, Has quantify the impact of ienttizzaatonnnnndcce perturbations on the model's 's training loss. Parameters with larger values of H exhibit higher sensitivity to quantization and require careful treatment to maintain model performance. We denote H as the impact of ws.
Efficient Computation Computing H of the diagonal of Hessian matrix for each parameter is computationally expensive, particularly for large-scale models. To overcome this challenge, we propose an efficient approximation using the Fisher Information Matrix (F), Since H the Hessian matrix of negative log-likelihood loss, H is equal to Fisher information matrix [6] For the diagonal of the Hessian matrix, we have:
H Fa Elg"1
4 End-to-End Mixed-Precision Quantization
The insights gained from Figure Shighlihhhl the heterogeneity in model parameters. To mitigate the impact of cherry parameters quantization, propose to preserve their high-precision values during the quantization process. By maintaining the fidelity of these critical parameters, we ensure that the essential information they capture is not compromised.
Optimizing preeddppeeiiionn parameters L.MMs presents unique challenge in the widely adopted Post-Training Quantization (PTQ) framework [14. If we do not allow the updates of the cherry parameters, the quantization will certainly lose the flexibility provided by these critical parameters. This prevents the cherry parameters from reaching their optimum. On the other hand, PTQ struggles to simultaneously optimize
cherry parameters and low-precision normal parameters. This is because the cherry parameter updates during the PTQ process significantly affect the optimal values of the normal parameters. So normal parameters need to be continually updated as the cherry parameter varies. However, in PTQ, once the normal parameters are quantized, they cannot be further updated. This prevents the carly-stage quantized parameters from reaching their optimal values.
To address this challenge, we propose novel approach that end-to-end optimize the mixed -eeec-ppreiiion parameters via backpropagation. Our method leverages a quantization-ave To simultaneously optimize both the cherry parameters and normal parameters, we use two separate backpropagation strategies. The high-precision cherry parameters are updated using standard gradient descent, while the low-precision normal parameters employ the Srraight-Througe Estimator (STE) trick (3 for low-precision gradient descent. This unified backpropagation enables the end-to-end optimization of both cherry parameters and normal parameters, enhancing the overall optimization effect. We show the guantization in Algorithm
training framework.
Algorithm 1 CherryQ
Require: Model parameters W, quantization function Quant(-), threshold T, learning rate 1)
Ensure: Quantized model parameters
1: +- [w; E W Ha > r}
Identify cherry parameters
2: N C
Identify normal parameters
3: for each training batch x do
4: L mo:el( CU Qu)nt(NN)
Compute loss w.r.t. mixed-precistee parameters
5: C+
C 11 1/C
Standard gradient descent for cherry parameters
6: N 1 STE OL
>>
Gradient approximation by STE for normal parameters
7: end for
8: return CU C (uan((NN)
5 Heterogeneity-based Cherry Parameter Selection
Correctly identifying cherry parameters one of the main challenges of CherryQ quantization.
Candidate metrics for parameter influences include weights [13] 18 6 activations [17 24]. and impacts (H.). We propose that an effective metric should reflect heterogeneity, specifically by differentiating the influence of cherry parameters and normal parameters of the model.
To this end, we define the heterogeneity score. In Figure small subset of parameters exhibit signif- icantly higher impacts compared to the maximum of the majority. Inspired by this, the heterogeneity

---

score is defined as the ratio of the mean impact of the top 1% parameters to the maximum impact of the bottom 99% parameters, as shown Equation higher heterogeneity score indicates more significant disparity in parameter importance.
Mean(](u;) top 1%.
Heterogeneity Score(f) Max bottom 999.
where fw;) denotes the parameter influence for parameter and is chosen from parameter weights, activations, and impacts.
Figure 2 presents the heterogeneity scores for different metrics across various LLMs. The impact- based metric consistently shows higher heterogeneity scores compared weights and activations. This indicates that the impact metric better distinguishes between the normal and cherry parameters, thus providing more effective means of identifying cherry parameters. The validity of using heterogeneity scores for cherry parameter selection will be further verified in Sec 6.5 demonstrating that higher heterogeneity scores lead better model performance.
Hoteragenelty Score
80
Impact
Weight
Deterogenelty Score
100
Weight
Impact
Heterageneity Score
80-
Impact
Weight Activation
60
Activation
75
Activation
60
40-
50
40
20
25
20-
0
50
100
150
200
150
250
50
100
150
2:00
Index
(a) LLaMA-2 7B
(b) LLaMA-2 13B
(c) Mistral
Heterogenalty Score
30
Weight Activation
Impact
Heterogeneity Score
Weight Activation
Impact
Heterogeneity Score
Impact Weight
30
60
Activation
20
10
10
Matrix Index
Matrix Index
Matrix Index
(d) Gemma 7B
(c) Vicuna- 1.5 7B
(1) Vicuna- 1.5 13B
Figure 2: Scatter distribution of heterogeneity scores for different parameter matrices in LLMs. Each point represents parameter matrix.
Data Independence We investigate whether impact-based parameter heterogeneity exhibits data independence that is, whether different data samples share the same cherry parameters. Applying identical cherry parameters across different samples only valid when there is data independence. To examine data independence within the same dataset, we randomly selected five sets of 128 WikiText-2 samples. We evaluated the overlap of their cherry parameters in each pair of sample sets. To evaluate cross-dataset data independence. we performed 9 independent sampling trials, each collecting 128 samples from C4 and 128 samples from WikiText-2 evaluate the overlap. For Vicuna, we further added Sharegpt as a data source. Table presents the overlap ratios of the cherry parameters. Despite the fact that cherry parameters constitute only 1/256 of the total parameters, all models demonstrate significant overlap ratios. This finding suggests that the cherry parameters possess an inherent iata-indeeendent nature.
Quantization Experiments
6.1 Implementation Details
Parameter Representation: Based on the observation that cherry parameters occupy a very small proportion, for each row of parameters in each parameter matrix, consider only the top 1/256
5

---

Table Overlap ratio (%) of cherry parameters (top 1/256), where L2: LLaMA-2, V: Vicuna-1.5, M:
Mistral, G: Gemma.
Model
L2-7B
L2-13B
V-7B
V-13B
M-7B
G-7B
Within dataset
84
75
68
63
89
90
Across datasets
68
66
65
60
85
86
parameters with the highest impact cherry parameters and retain their FP16 precisions. For example, the parameter matrix size of LLLMMA-7B is 4096 4096. So we average the impact across all rows for each column and then select the top 16 columns with the highest average impact, resulting in 16 x 4096 parameters as cherry parameters. Furthermore, to recover the complete parameter matrix, an INT16 is required record the indices of these 16 columns. Thus, the storage overhead for the column indices is minimal. For normal parameters, we employ full range symmetric MinMax quantization quantize their weights. And we adopt widely-used parameter grouping strategy. For more details, see Sec C
Quantization Datasets: For the quantization of the base LLMs, follow [9] to C4 [20] as the training data. We selected the first four partitions of C4 and chose data with a length of > 2048
tokens, resulting in total of 50k samples of 2048 tokens. For the chat L.Mss, since Vicuna-1.5 5 is
obtained by supervised fine-tuning based on ShareGPT [5], we also use the ShareGPT dataset for training. We used total of 20k training samples from ShareGPT for QAT and Cherry.
Baselines We compare our method with various quantization methods, including QAT [8]. GPTQ [9]. Lq [12], OmniQuant [21]. and AWQ [17]. For OmniQuant and AWQ, we use their results reported in 220. For SqueezeLLM, we use the results its original paper [2]. For GPTQ, its 4-bit model is obtained from the open-source Due the lack of 3-bit GPTQ model, we quantize the model ourselves via the implementation of Auto-GPTQ Since CherryQ is based on QAT, for fair comparisons, the implementation of QAT is the same CherryQ, except that it does not handle cherry parameters.
6.2 Effect of Base LLM Quantization
In this section, we present the main experimental results demonstrate the effectiveness CherryQ on LLaMA2 [23]. We evaluate CherryQ with both perplexity and downstream tasks, comparing its performance with state-of-the-an quantization methods.
6.2.1 Perplexity Results
We follow 99 21 to evaluate the perplexity of CherryQ on two widely used corpora: C4 and WikiText-2 19) We use the validation split of C4 to avoid data leakage. We show the results of 3-bit quantization using different quantization approaches in Table 2 We show the results of different model scales and different group sizes.
From the results, CherryQ consistently outperforms all other approaches across both model sizes (7B and 13B) and grouping sizes (64 and 128), achieving the lowest perplexity on both the C4 and WikiTexxt2 datasets. Notably, CherryQ's perplexity is significantly closer the full-precision (FP16) baseline compared to other methods, highlighting its ability to preserve model performance after quantization.
Jablercompare different 4-bit quantization methods. Again, Cherry achieves the lowest perplexity scores most settings, demonstrating its effectiveness in higher-bit quantization settings.
6.2.2 Downstream Task Performance
To further validate the effectiveness specific tasks, we evaluated the quantized models on various downstream tasks from the HuggingFace OpenLLM Leaderboard. Table presents the performance comparison of different 3-bit quantization methods for LLMMA2. CherryQ consistently outper-
hhtps:/ttuggingfacccc///hhhhhhh wtt.:faaacccccchhheaea hhupss//iihaa.commm/uuuuqqquuuuu

---

forms other methods across almost all tasks, achieving the highest average score. This showcases CherryQ's ability to maintain the model's generalization capabilities for downstream tasks.
Table 2: Perplexity (U) of 3-bit quantization on LLaMA2 models. gX means the group size X. The
results of OmniQuant and AWQ are from [21]. The results of SqueezeLLM are from [12).
Method
Avg. bit
7B-3bit-gl2
Avg.
7B-3bit-g64
Avg. bit
L3B-3bit-g125
Avg.
13B-3bit-ger 3bit-g64
bit
bit
c4
wiki2
c4
wiki2
c4
wiki2
c4
wiki2
FP16
16
6.97
5.47
16
6,97
5.47
16
6.47
4.88
16
6.47
4.88
QAT
3.13
9.25
6.90
3.25
8.74
7.13
3.13
7.19
5.63
3.25
7.02
5.48
GPTQ
3.15
8.28
6.74
3.30
8.20
6.62
3.15
7.24
5.63
3,30
7.10
5,56
AWQ
3.15
7.84
6.24
3.15
6,94
5.32
OmniQuant
3.15
7.75
6.03
3.15
6.98
5.28
SqueezeLLLM
3.24
7.51
5.96
3.24
6.82
5.23
CherryQ 3.17
7.39
5.93
3.30
7.34
3.17
6.80
5.26
3.29
6.76
5,21
Table 5 extends the comparison 4-bit quantization. CherryQ continues to excel, achieving the highest scores on most individual tasks and the highest average score overall. These results highlight the generalization ability CherryQ across different quantization bits and model sizes.
Table Perplexity (4) of 4-bit quantization on LLaMA2 models.
Method
Avg. bit
7B-4bit-g
128
Avg. bit
13B-4bit-g 128
c4
wiki2
c4
wiki2
FP16
16
6.97
5.47
16
6.47
4.88
QAT
4.13
7.29
5.81
4.13
6.67
5.12
GPTQ
4.15
7.30
5.73
4.15
6.63
4.97
AWQ
4.15
7.13
5.62
4.15
6.56
4.97
OmniQuant
4.15
7.12
5.58
4.15
6.56
4.95
CherryQ
4.17
7.07
5.58
4.16
6.56
4.99
6.3 Effect of Chat LLM Quantization
We conducted experiments Vicuna-1.5 [5]. We apply 3-bit quantization with group size=128 for CherryQ and other baselines.
Evaluation To assess the performance of quantized open- open-ended chat models, we employ a pairwise comparison on the Vieuna-bench [27], which consists of 80 test samples. We compare the responses generated by the quantized models against those generated by the original 16-bit Vicuna-1.55 The evaluation performed using GPT-4, which automatically classifies the quantized model's response as "win". "tie", "lose" relative to the FP16 model's response. To get rid of the ordering effect of the evaluation, we follow [17 to compare the responses with both orders, resulting in 160 trials.
Figure 3 presents the results of the pairwise comparison for each quantized model against its FP16 counterpart. The results demonstrate that CherryQ consistently outperforms other quantization baselines in preserving the performance of chat models. It achieves the highest number of wins and ties against the FP16 models, while minimizing the number of losses.
Notably, 3-bit CherryQ achieves a slightly better win-tie-lose ratio over the FP16 Vicuna model, indicating that the 3-bit quantized model performs par with intuitively CherryQ cannot surpass the target 16 bit model, we think the result suggests that CherryQ maintains almost all its performance even bit, making GPT-4 hard to distinguish the quality of low-bit and FP16 models.
better than the FP16 model. As
6.4 Extreme 2-Bit Quantization
We further explore the extreme case of 2-bit quantization. Although 2-bit quantization greatly reduces memory requirements model storage and inference, existing methods still show a significant performance gap compared to their 16-bit counterparts.
7

---

Table 4: Performance of different 3-bit quantization methods on Huggingface OpenLLM for LLaMA2- 7B and LLaMA2-132 13B.
Method
Hellaswag
Winogrande
ARC
TruthfulQA
GSMSK
MMLU
Average (t)
LLaMA2-78 LaaMA--B--bbtee
FP16
78.6
74.0
53.2
38.8
14.5
46.7
51.0
QAT
75.5
71.6
49.2
37.3
7.3
40.6
46.9
GPTQ
73.9
71.7
48.6
38.8
8.1
39.4
46.8
CherryQ
77.0
71.8
50.6
38.6
10.4
43.9
48.7
LLaMA2-775 -bittg 128
FP16
78.6
74.0
53.2
38.8
14.5
46.7
51.0
QAT
75.4
70.8
48.2
37.7
6.7
39.0
46.3
GPTQ
72.9
70.8
48.6
39.1
5.4
38.2
45.8
CherryQ Q
76.3
72.4
49.7
38.1
8.8
41.6
47.8
LLaMA2- 1------t------- 3bit-g64
FP16
82.1
76.6
59.4
37.4
22.5
55.7
55.6
QAT
80.7
75.1
55.5
39.0
16.8
52.9
53.3
GPTQ
79.2
74.4
56.5
36.0
16.4
52.4
52.5
CherryQ
81.1
76.2
57.3
38.0
18.4
53.5
54.1
LLaMA2-I8 I 3B- 3bit-g 128
FP16
82.1
76.6
59.4
37.4
22.5
55.7
55.6
QAT
80.7
75.5
55.3
38.8
16.0
51.9
53.0
GPTQ
79.1
75.4
54.1
34.9
15.6
50.3
51.6
Cherry Q
81.0
75.4
56.7
38.9
17.8
52.5
53.7
Table 5: Performance comparison of different 4-bit quantization methods for LLaMA2-7B and
LLaMA2-13B models over Huggingface OpenLLM Leaderboard.
Method
Hellaswag
Winogrande
ARC TruthfulQA
GSMSK
MMLU
Average (t)
LLaMA2- MMMAA2-B--4bit--l0
FP16
78.6
74.0
53.2
38.8
14.5
46.7
51.0
QAT
77.5
72.2
52.0
39.0
10.6
43.7
49.2
GPTQ
77.6
72.9
52.0
39.1
11.1
43.8
49.4
Cherry Q
77.8
73.5
51.5
39.5
12.9
44.4
49.9
LLaMA2-1 1 3B- 4bit- g 128
FP16
82.1
76.6
59.4
37.4
22.5
55.7
55.6
QAT
81.9
75.7
57.9
38.9
19.6
54.2
54.7
GPTQ
81.5
76.8
57.4
36.1
20.4
54.6
54.5
Cherry Q
82.0
77.0
58.6
38.8
21.0
54.6
55.3
Implementation Details To achieve high-quality 2-bit quantization, we integrated the scaling-up trick introduced in [17]. Specifically, after identifying cherry and normal parameters, automatically search for the optimal scale of each column of normal parameters that minimizes the output difference after quantization for each layer. The quantization function is formulated Q' Q'(w)
Q(w s)/s. where Q--) represents standard asymmetric quantization on the min-max grid, and s is a constant that scales up the normal parameters and remains fixed during the training process. The cherry parameters are excluded from quantization and retain their 16- bit precision throughout the grid search.
Results Table presents the perplexities of 2-bit quantization LLAMA2 models. Compared to existing methods such as GPTQ, AWQ, and OmniQuant, our proposed CherryQ method demonstrates superior performance across all metrics. Specifically, CherryQ achieves perplexity scores of 9.55

---

Quartized Win
Quantized Lost
QAT
54
35
71
QAT
64
33
63
GPTQ
40
34
86
GPTQ
24
37
99
CherrQQ
61 47
52
ChemyQ
64
36
60
40
80
120
160
80
120
160
Figure 3: Comparison of 3-bit quantized models to FP16 Vicuna-1.. 1.5. (Left) Comparisons to Vicuna- 1.5-7B. (Right) Comparisons Vicuna-1.55 1.5-13B. CherryQ even shows competitive quality compared to the 16-bit counterpart.
Table Perplexity (L) of 2-bit quantization on LLaMA2 models. The results of GPTQ, AWQ and OmniQuant are from (21)
Method
Avg. bit
7B-2bit-g128
Avg. bit
7B-2bit-g64
Avg. hit
13B-2bit-g122
212
Avg. bit
13B-2bit-g64
c4
wiki2
c4
wiki2
c4
wiki2
c4
wiki2
FP16
16
6.97
5,47
16
6.97
5.47
16
6.47
4.88
16
6.47
4.88
GPTQ
2.15
33.70
36.77
2.30
19.40
20.85
2.15
20.97
28.14
2.30
12.48
22.44
AWQ
2.15
> 10
> 10
2.30
> 10
> 10
2.15
>> 10
>> 10
2.30
>> 10*
> 10
OmniQuant
2.15
15.02
11.06
2.30
12.72
9.62
2.15
11.05
8.26
2.30
10.05
7.56
CherryQ 2.19
9.55
8.34
2.34
9.08
7.84
2.19
8.40
7.20
2.33
8.02
6.72
and 8.34 in the 7B-3bit-g128 and 7B-3bit-g64 settings, respectively. These results significantly outperform other methods, validating the effectiveness of CherryQ in 2-bit quantization.
6.5 Comparison of Parameter Selection Criteria
To evaluate the effectiveness of our proposed impact-based parameter selection criterion, we con- ducted experiments comparing it with the criterions of parameter weights and activations. Table presents the perplexity of LLaMA2-7B-3bit and LLaMA2- 133-3bi models, using both criteria for cherry parameter selection.
From the results, is evident that the impact-based criterion consistently outperforms other criterions across all settings. These results demonstrate that our proposed impact- basect-based criterion is a more effective measure identify cherry parameters. The impacts identify and preserve the most critical parameters during the quantization process. These results are consistent with the analysis in Sec regarding the effectiveness of heterogeneity scores selecting cherry parameters.
Table 7: Perplexity (L) of different parameter selection criteria.
Method
LLaMA2-7B
ILaMA2-13B-3hl
c4
wiki2
c4
wiki2
Weight-g64
7.93
6.40
6.91
5.35
Activation-g64
7.37
5.89
6.77
5.22
Impact-g64
7.34
5.87
6.76
5.21
Weight-g122
8.12
6.58
6.94
5.37
Activation-ge 128
7.51
6.03
6.81
5.27
Impact-g128
7.39
5.93
6.80
5.26
LLaMA2-7B-4bin
(LaMA2-13B-442
Weight-g 128
7.19
5.68
6.62
5.05
Activation-g 128
7.09
5.59
6.56
5.00
Impact-g122
7.07
5.58
6.56
4.99

---

7 Conclusion
In this paper, we systematically investigated the phenomenon of parameter heterogeneity in large language models (LLMs). Our experiments on LLaMA2, Mistral, Gemma, and Vicuna models consistently demonstrated that a small subset of parameters, referred "cherry" parameters, play a crucial role in maintaining the model's performance, while the vast majority of parameters can be quantized to ultra-low precision without significant degradation. This finding highlights the potential of the heterogeneous nature of parameter importance.
Motivated by this observation, we proposed a novel CherryQ quantization algorithm, which uses quantization-aware training framework for the end-to-end optimization of both cherry parameters and normal parameters. Extensive experiments demonstrate that CherryQ achieves significantly lower perplexity scores and better downstream performance.
8 Limitations
There some limitations consider. First, the method relies heavily on the accurate identification of cherry parameters, which may vary across different model architectures and training datasets. This dependency could potentially limit the generalization ability of CherryQ new or unseen models. Second, the computational overhead required for the impact-based identification and scaling of parameters, although justified by the performance gains, may pose challenges for extremely large models or those deployed in real-time systems with stringent latency requirements.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088 087744, 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint a30.v: 2013.
[4] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947-7969, 2021.
[5] Wei-Linn Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://imsys. org/blog/2023-03 30-vicuna, 3(5), 2023.
[6] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35:30318-30333 2022.
[7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[8] Tim Dettmers, Ruslan A Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse- quantized representation for near-lossless Ilm weight compression. In The Twelfth International Conference on Learning Representations, 2023. 2023..
[9] Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq; Accurate post-training quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
[10] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference neural networks, pages 293-299, IEEE, 1993.
10

---

[11] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06833 2023.
[12] Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023.
[13] Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, 2021.
[14] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: whitepaper. arXiv preprint arXiv:1806.088 08342, 2018.
[15] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[16] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Reprexentations, 2020.
[17] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.000 00978, 2023.
[18] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LIm-qat: Data-free quantiza- tion aware training for large language models. arXiv preprint urXiv:2305. /7888, 2023.
[19] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint Xrvi /rXiv:6609.00000 609407843, 2016.
[20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
[21] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quan- tization for large language models. In The Twelfth International Conference Learning Representations, 2023.
[22] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0820 08295, 2024.
[23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09238 2023.
[24) Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1648-1665, 2023.
[25] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35: 15:17400 7402-114444 2022.
[26] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[27] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
11

---

A Effect of Chat LLM Quantization on MMLU
We further evaluate the performance CherryQ on the MMLU benchmark by quantizing the Vicunal.5 model. As shown in Table 8 CherryQ outperforms both QAT and GPTQ in terms of average accuracy across almost all categories.
Table 8: Comparison of different 3-bit quantization methods zero-shot MMLU accuracy applied Vicuna-1.5.
Method
Humanities
STEM
Social Sciences
Other
Average
Vicunal.5 Vi
FP16
46.8
39.4
57.9
56.3
49.9
QAT
43.4
37.7
53.0
52.4
46.4
GPTQ
42.7
37.3
53.0
51.0
45.7
CherryQ
43.8
37.2
54.3
53.5
46.9
Vicunal 1.--
1--1--bbtt--
FP16
50.2
43.5
63.0
62.0
54.3
QAT
47.8
40.1
58.6
58.1
50.9
GPTQ
46.1
39.4
57.6
55.2
49.3
CherryQ
49.0
40.6
60.2
58.8
51.9
B
Training Details
For all LLM scales (7B, 13B), and both base models and chat models (LLaMA2, Vicuna-v1.5), we train the models on single node with 8 A 100 80GiB GPUs. We use total batch size of 128, learning rate of 2e-5, weight decay of 0.0, a cosine scheduler with 5% warm-up steps. The final learning rate is 25% of the peak learning rate for 2/3-bit LLMs, 10% for 4-bit LLMs. We train epoch on base models, 2 epochs on chat models.
C Parameter Representation Details
For normal parameters, we employ full range symmetric MinMax quantization to quantize their weights [14]. Specifically, FP16 value is mapped to the range of [-2#-1 2 and symmetri- cally distributed on both sides of the coordinate axis. The quantization of an FP16 tensor FPG to k bits is computed by:
FP16
(5) where 1.1 denotes the round function, S is the quantization scaling factor S Maa([X and
X Intk [Clip(
-2 + 2,, -6) -e) - 0.55
2
a very small positive number (= 0.01 in our setting) ensure that X falls into the target range. Dequantization restores the quantized integer values based on the scaling factor:
Dequant X xlnkk) = S(x + 0.5)
To further improve the quantization accuracy, we adopt widely-used parameter grouping strategy. Specifically, the parameters are divided into groups in order, and each group independently calculates its scaling factor. For example, if we divide parameter matrix W e grx that needs to be quantized with group size of B, we will obtain total of r x (c/B) groups.
D Licenses for Existing Assets
We list the assets used in this paper and their licenses below:
12

---

S
Ilama2
http://hwggiiggnccccccoceeBbbehh llama2
http://ghhtlbcccom/uuuo/////AAug MIT License
arXiv.org perpetual, non-exclusive license 1.0
18
arXiv.org perpetual, non-exclusive license 1.0
20
arXiv.org perpetual, non-exclusive license 1.0
19
arXiv.org perpetual, non-exclusive license 1.0
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have fully explained the motivation and contributions of the paper in the introduction section, and have verified them the experimental section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Please refer to Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 6.1 Section
B and the attached code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We have attached the codes in the submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guuii wwwiiccgguie//eeeeeeee CoodeSubeiiieeeec ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section 6.1 Section B, and the attached code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were completed on nodes with the GPUs of 8*A100 80G. Please refer to Section 6.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttt::://neuips..t httpp///cc/c/cciicccctt.eeeeeee
Answer: [Yes]
Justification:
We carefully read and follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
The paper is purely fundamental research and does not involve social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This paper does not release new models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Section
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have attached codes with documents.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "BhsOMTeYRc", "reviewer": "Reviewer_YuKp", "summary": "This paper presents a new post-training quantization strategy called \"CherryQ\" that identifies \"cherries\" --- meaning parameters that disproportionately impact the model's loss --- stores those in fp16, and quantizes the rest of the model parameters using a standard symmetric min-max quantization (with groups).  It shows that this approach yields better perplexity and downstream performance numbers than existing quantization methods, on several models (Llama-3 7b/13, Vicuna 1.5 7b/13b), several tasks (c4/wiki perplexity, Hellaswag, Winogrande, ARC, TruthfulQA, GSM8K, MMLU, chat win-rate), and at several compression rates (2 bits, 3 bits, 4 bits).\n\nSome more details:\n- To identify cherries, the paper proposes using E[g_i^2] as a measure of the impact of parameter w_i on the loss, where g_i is the gradient of the loss function with respect to that parameter. This is motivated by: (1) approximating the loss around each parameter w_i using a second order Taylor expansion, assuming E[g_i] = 0 (because model has converged), and assuming the second-order derivative of L w.r.t. w_i is equal to E[g_i^2] (based on results about Fisher information matrix equaling the Hessian from [16]).\n- The paper shows that selecting cherries in this way (using the above approximation of the impact of each parameter on the loss) is better than using alternative metrics based on model weights or activations.\n- Once cherries are identified, the non-cherries are quantized, and then the model is fine-tuned by training the cherries in a regular manner (non-quantized gradients and updates), while training the (quantized) non-cherries using the straight-through estimator.", "questions": "- See weaknesses. Are there any obstacles to running the experiment to validate the use of E[g_i^2] as the metric for measuring impact on loss?\n- How \"data-dependent\" are the cherries? Can this be measured?\n- How does CherryQ compare with QuIP/QuIP# on Llama-3?", "limitations": "Limitations are discussed, though could be addressed better with some experiments (see questions above).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper demonstrates that some parameters have *much* larger impact on loss than others (at least according to their way of approximating impact), a surprising result, that nicely motivates using mixed precision quantization.\n- The proposed method perform well across numerous models, tasks, and compression rates relative to baselines.\n- The proposed method is pretty simple and straightforward.", "weaknesses": "- It would have been useful to **directly** confirm that the proposed way of approximating impact on loss is actually a good approximation of the impact of a parameter on the loss. This was only done in indirect ways (e.g., by showing that quantizing these parameters leads to better performance than quantizing using other metrics).  It should be relatively straightforward to do this by selecting a random set of model parameters (both cherries and non-cherries), perturbing them slightly (one-by-one), and passing data through the perturbed and unperturbed model and measuring the loss difference.  One could then show a scatter plot (and measure spearman rank correlation) to visualize how well the approximated impact on loss (E[g_i^2]) aligns with the actual impact on loss.\n- QuIP (https://arxiv.org/abs/2307.13304) and QuIP# (https://arxiv.org/abs/2402.04396) are not compared to, and more recent state-of-the-art models are not benchmarked (e.g., Llama3).\n- No inference speed results are shared. It would be great to see if this method can directly lead to faster inference, especially during decoding (which is memory-bound).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 and Q1**: It would have been useful to **directly** confirm that the proposed way of approximating impact on loss is actually a good approximation of the impact of a parameter on the loss.\n\n**Response**: Following your suggestion, to directly validate the accuracy of our impact approximation, we investigated the effects of different types of parameter perturbations on the model's loss. We compared the following two settings for LLaMA2-7b:\n1. Adding perturbations in the range of (-0.5, 0.5) to all cherry parameters.\n2. Randomly sampling an equal number of normal parameters and adding perturbations in the range of (-0.5, 0.5) to them.\n\nWe repeated this random sampling and perturbation process five times and measured their perplexities. The results are presented in the table below. Perturbations to cherry parameters led to significantly larger increases in perplexity, while normal parameters remained largely insensitive to similar perturbations. This directly validates both the existence of cherry parameters and the effectiveness of our approximation.\n\n|model | c4 | wiki2 | \n|--------|:---:|:---:|\n|LLaMA2-7b | 6.97 | 5.47 |\n|Perturb Normal Parameters|6.98($\\pm$ 0.00) | 5.48($\\pm$ 0.00)|\n|Perturb Cherry Parameters|8.12($\\pm$ 1.78) | 6.12($\\pm$ 0.84)|\n\n**W2 and Q3**: How does CherryQ compare with QuIP/QuIP# on Llama-3?\n\n**Response**: We have compared CherryQ with QuIP and QuIP# on LLaMA2-7b in terms of perplexity, as shown in the table below. We haven't included comparisons over LLaMA-3 as we couldn't find QuIP/QuIP# implementations based on that model.\n\n| Model | Bits | Wiki | C4 |\n|-------|------|------|----|\n| 7b-fp16 | 16 | 5.47 | 6.97 |\n| 7b-QuIP | 4 | 5.94 | 8.01 |\n| 7b-QuIP# | 4 | 5.56 | 7.07 |\n| 7b-CherryQ | 4 | 5.58 | 7.07 |\n| 7b-QuIP | 3 | 6.50 | 8.74 |\n| 7b-QuIP# | 3 | 5.79 | 7.32 |\n| 7b-CherryQ | 3 | 5.93 | 7.39 |\n| 13b-fp16 | 16 | 4.88 | 6.47 |\n| 13b-QuIP | 4 | 5.01 | 6.89 |\n| 13b-QuIP# | 4 | 4.95 | 6.54 |\n| 13b-CherryQ | 4 | 4.99 | 6.56 |\n| 13b-QuIP | 3 | 5.34 | 7.34 |\n| 13b-QuIP# | 3 | 5.10 | 6.72 |\n| 13b-CherryQ | 3 | 5.26 | 6.80 |\n\nCherryQ shows clear improvements over QuIP under the same settings. While QuIP# achieves better results, its main advantage stems from vector quantization, a technique  primarily addressing how to replace the quantization of individual weights with quantization for groups of weights. This technique is orthogonal and complementary to our approach rather than directly comparable.\n\n**W3**: No inference speed results are shared.\n\n**Response**: We have now conducted real-world tests on a single A100 GPU to measure the generation speed and memory consumption of CherryQ when generating 128 tokens. The results for LLaMA2-7b are summarized in the table below. Compared to FP16, CherryQ at lower precision demonstrates significant improvements in both efficiency and memory usage.\n| model | #tokens/s ($\\uparrow$) | peak memory ($\\downarrow$) |\n|---|:---:|:---:|\n|FP16| 46.6 | 25843 (MB) |\n|CherryQ-4bit| 68.0 |4552 (MB)|\n|CherryQ-3bit| 71.4 | 3690 (MB) |\n\n**Q2**: How \"data-dependent\" are the cherries? Can this be measured?\n\n**Response**: To validate the data dependence, we conducted additional experiments in terms of data selection and data scale:\n\n - **Data Selection Independence**: We validated that cherry parameters identified using different data samples are largely consistent. Specifically, we randomly selected **five sets of 128 samples** from WikiText-2 and calculated cherry parameters for each set. We found an **84%** overlap rate on average between any two different cherry parameter sets, indicating that different data selections tend to yield similar cherry parameters.\n - **Data Scale Independence**: We also verified that cherry parameters remain consistent across different data scales. We randomly selected four sample sets from WikiText-2 with **sizes of 128, 256, 512, and 1024**, respectively, and calculated their corresponding cherry parameters. The average overlap rate of cherry parameters between any two sets reached **89%**, demonstrating that varying data scales tend to produce similar cherry parameters.\n\nThese findings demonstrate that cherry parameters can be reliably identified using a relatively small set of samples (as few as 128), enhancing both the generalizability and efficiency of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsSr9nHGXF", "reviewer": "Reviewer_GdR4", "summary": "This paper investigates the phenomenon of parameter heterogeneity in large language models (LLMs) and proposes a novel quantization method called CherryQ. The authors find that a small subset of parameters, referred to as \"cherry\" parameters, have a disproportionately large influence on model performance, while the majority of parameters have minimal impact. Based on this observation, CherryQ selectively preserves the critical cherry parameters in high precision while aggressively quantizing the remaining parameters to low precision. Extensive experiments demonstrate that CherryQ outperforms existing quantization approaches in terms of perplexity and downstream task performance.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The paper addresses an important issue in large language models and provides valuable insights into the parameter heterogeneity phenomenon.\n\n(2) The experimental results are extensive and well-structured, demonstrating the effectiveness of CherryQ in various settings and benchmarks.", "weaknesses": "(1) One major concern is about the implementation. What is *real* improvement in terms of efficiency? The authors are supposed to report the models' throughput and speed compared with baseline models.\n\n(2) The idea of important parameters for quantization is not new and novel.  This paper seems to propose a new term \"Cherry Parameters\" which appears to be a reinvention of an already existing idea. But the authors could argue the difference with previous studies. I am open to adjust ratings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: What is real improvement in terms of efficiency? \n\n**Response**: We have now conducted real-world tests on a single A100 GPU to measure the generation speed and memory consumption of CherryQ when generating 128 tokens. The results for LLaMA2-7b are summarized in the table below. Compared to FP16, CherryQ at lower precision demonstrates significant improvements in both efficiency and memory usage.\n| model | #tokens/s ($\\uparrow$) | peak memory ($\\downarrow$) |\n|---|:---:|:---:|\n|FP16| 46.6 | 25843 (MB) |\n|CherryQ-4bit| 68.0 |4552 (MB)|\n|CherryQ-3bit| 71.4 | 3690 (MB) |\n\n\n**W2**: The idea of important parameters for quantization is not new. \n\n**Response**: We appreciate the reviewer's perspective, but respectfully disagree that our idea is not new. Our paper makes several novel and significant contributions:\n1. **Systematic Revelation of Parameter Heterogeneity**: We are the first to comprehensively demonstrate and analyze the phenomenon of parameter heterogeneity across different LLM scales, families, and types (base and chat models). This fundamental insight provides a unifying explanation for several observed behaviors of LLMs under quantization.\n2. **Mechanistic Explanation**: We provide the explanation for why LLMs are highly robust to quantization. Our work reveals that this robustness stems from the 99% of parameters that have minimal perturbative impact on model performance.\n3. **Foundation for Mixed-Precision**: While mixed-precision strategies exist, our work provides the underpinning for why they are effective. We show that these strategies work by preserving the critical \"cherry\" parameters that have outsized influence on model performance.\n4. **Novel Metric for Parameter Importance**: Our impact-based metric for identifying important parameters demonstrably superior to existing metrics. We provide both theoretical justification (through our heterogeneity score) and empirical evidence for its effectiveness.\n5. **Innovative End-to-End Optimization**: CherryQ introduces a novel approach to simultaneously optimize high-precision cherry parameters and low-precision normal parameters - a challenge that existing methods struggle with. Our 3-bit Vicuna-1.5 model matches the performance of 16-bit models, representing a substantial advancement in LLM quantization.\n\nThese contributions are outlined in the original manuscript (lines 64-72).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iVi6M6WQJN", "reviewer": "Reviewer_P3Ag", "summary": "The paper introduces a quantization technique for LLMs, inspired by an empirical study on parameter importance. The authors identify a small subset of parameters, termed \"Cherry parameters,\" which exhibit significantly higher importance than others. They propose an optimization strategy: retaining full precision for Cherry parameters and their gradients, while quantizing the normal parameters. A straight-through estimator is used to handle the gradients of these quantized parameters during optimization.", "questions": "Please see Weaknesses.", "limitations": "Please see Weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper begins with a weight importance analysis, providing an insightful empirical study as a foundation.\n2. The proposed method is straightforward to implement.", "weaknesses": "Weaknesses:\n\n1. Section 3: The method depends on back-propagation, which could lead to computational and memory overhead challenges in resource-limited scenarios, such as quantizing an extremely large model with only 4 GPUs. However, this issue is acknowledged in Section 8.\n\n2. Section 4: The Fisher information matrix is recalculated at each step. Given that parameters are updated frequently, this could cause the parameter importance to fluctuate significantly, potentially leading to a chain error. This is because gradients influence cherry parameter identification, which in turn affects gradient calculations. The paper does not address how to manage this cyclic error effectively.\n\n3. Section 5: The model for assessing weight importance is based on Optimal Brain Damage and only considers the impact of individual weights when perturbed or quantized. It overlooks the synergistic effects between weights, where quantizing two normal parameters together could have a more significant impact than quantizing one cherry parameter.\n\n4. Section 5: There seems to be a discrepancy in how weight importance is presented: Figure 2 is per weight matrix, whereas Figure 1 is per weight scalar. In the experiments, parameter impact scores are averaged along rows, and cherry parameter identification is performed along columns based on these averages, leading to confusion. If cherry parameter identification is indeed column-based, some cherry parameters may be overshadowed by dominant normal parameters in the same column, leading to potential misidentification.\n\n5. Section 6.2: The improvement in perplexity over OmniQuant is marginal, as shown in Tables 1 and 2. Furthermore, it is weird that OmniQuant is omitted in the QA evaluations in Tables 3 and 4.\n\nI would be inclined to improve the rating if these points are properly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: The method depends on back-propagation, which could lead to computational and memory overhead challenges in resource-limited scenarios. \n\n**Response**: While the quantization indeed requires higher computational resources, this is a one-time operation. Once performed, the quantized model will offer significant reduction in computational requirements for the subsequent inference.\n\n**W2**: Section 4: The Fisher information matrix is recalculated at each step. Given that parameters are updated frequently, this could cause the parameter importance to fluctuate significantly, potentially leading to a chain error. \n\n**Response**: We **do not** need to dynamically update the Fisher information matrix results or re-identify cherry parameters. Instead, we only identify them once during the initialization phase (line 1 Algorithm 1). This is because our objective is to ensure each quantization step brings the model closer to the **initial model**, rather than to the model from the previous step. Consequently, the measurement of parameter impact should always be calculated with respect to the initial model, eliminating the need for dynamic updates.\nTo experimentally validate this, we conducted an additional experiment comparing two settings:\n1. (One-time identification) Calculating the Fisher information matrix and identifying cherry parameters only during initialization.\n2. (Dynamic identification) Recalculating the Fisher information matrix and re-identifying cherry parameters every 40 steps based on the current model state. The perplexities for LLaMA2-7b-4bit-g128 are presented in the table below.\n\n| model| c4 | wiki2|\n|---|:---:|:---:|\n|One-time|7.07|5.58|\n|Dynamic|7.75|6.25|\n\nAs evident from the results, dynamically identifying cherry parameters leads to a noticeable decrease in performance. This aligns with our intuition and theoretical understanding.\n\n**W3**: Section 5: The model for assessing weight importance is based on Optimal Brain Damage and only considers the impact of individual weights. It overlooks the synergistic effects between weights.\n\n**Response**: To enhance computational efficiency, we indeed employed the OBD assumption, measuring parameter impacts independently as in Eq (1). To validate this approach, we compared the average impact of two cases: (1) individual parameters ($avg(|H_{ii}|)$), (2) parameter pairs ($avg(|H_{ij}|)$) for LLaMA2-7B and LLaMA2-13B models, sampling 400k values for each case.\n\n| model | $avg(\\|H_{ii}\\|)$ | $avg(\\|H_{ij}\\|)$ |\n|-------|:-----------------:|:-----------------:|\n| LLaMA2-7b | 0.0113 | 0.0008 |\n| LLaMA2-13b | 0.0091 | 0.0001 |\n\nThis finding indicates that while synergistic effects between weights exist, they are generally much smaller than individual parameter effects.\n\n\n**W4.1**: Section 5: There seems to be a discrepancy in how weight importance is presented: Figure 2 is per weight matrix, whereas Figure 1 is per weight scalar.\n\n**Response**: Figure 1 presents distributions for specific layers, providing more **direct** insights. Figure 2 shows matrix-level distributions, offering a more **global** perspective to ensure that the phenomenon is not biased by individual layers. Both figures demonstrate the existence of parameter heterogeneity.\n\n**W4.2**: In the experiments, parameter impact scores are averaged along rows, and cherry parameter identification is performed along columns based on these averages, leading to confusion. If cherry parameter identification is indeed column-based, some cherry parameters may be overshadowed by dominant normal parameters in the same column, leading to potential misidentification.\n\n**Response**: Parameter impact calculation is **always performed column-wise**. As stated in lines 158-159, \"_we average the impact across all rows for each column._\" This means we calculate the average impact for each column by considering all rows within that column, not by row-wise calculation. \n\nWe also experimented with independently calculating and identifying each cherry parameter, as compared to the column-based approach. The effect is minimal. For instance, with LLaMA2-7B-3bit-g64 on WikiText-2, the separate approach achieved a perplexity of 5.20, only slightly outperform the column-based calculation resulted (5.21). To optimize memory usage and enhance inference efficiency, we opted for the column-based calculation method.\n\nA potential explanation for this phenomenon lies in the low-rank nature of Transformer parameter gradients [1].\n\n[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection\n\n**W5**: The improvement in perplexity over OmniQuant is marginal, as shown in Tables 1 and 2. It is weird that OmniQuant is omitted in the QA evaluations in Tables 3 and 4.\n\n**Response**: We have added a comparison on downstream tasks with OmniQuant, as shown in the table below. We use the setting of LLaMA2-7b-3bit-g64.\n\n| model |  Winogrande | ARC-C | GSM8K |\n|---|:---:|:---:|:---|\n|omniquant|71.7|50.0|7.7|\n|CherryQ|71.8|50.6|10.4|                 \n\nIt's important to note that even small improvements in perplexity can translate to significant enhancements in real-world task performance (Table 1 and 2). The additional downstream task comparisons show that CherryQ achieves better results than OmniQuant across various tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper can be summarized, as reviewer YuKp says, "The paper demonstrates that some parameters have much larger impact on loss than others (at least according to their way of approximating impact), a surprising result, that nicely motivates using mixed precision quantization." And regarding execution: as reviewer GdR4 says, "The experimental results are extensive and well-structured, demonstrating the effectiveness of CherryQ in various settings and benchmarks."

Overall ratings were positive, and there were relatively minor revisions required during the rebuttal process. And what revisions were made, made the paper stronger in a quick and understandable way. For instance, there was a direct validation of impact approximation. Reviewer YuKp: "It would have been useful to directly confirm that the proposed way of approximating impact on loss is actually a good approximation of the impact of a parameter on the loss."
To which the authors responded with an experiment resulting in: "Perturbations to cherry parameters led to significantly larger increases in perplexity, while normal parameters remained largely insensitive to similar perturbations. This directly validates both the existence of cherry parameters and the effectiveness of our approximation." Other concerns about cherry parameter dataset overlap and memory-bound latency were also addressed to GdR4 and YuKp.

There are some limitations though with the paper. P3Ag is still not convinced that |Hii| >> |Hij|. And they also point out that the average QA performance gain over OmniQuant is still relatively small at 1.13%.

Overall, the paper makes a significant contribution to the field of LLM quantization through its novel insights and effective method. The strong author responses and extensive evaluation support its acceptance.
Decision: Accept (poster)