Accuracy is Not All You Need
Authors: Abhinav Dutta
Sanjeev Krishnan
Nipun Kwatra
Ramachandran Ramjee
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM Evaluation
Evaluation of Compressed LLMs
Quantization
Pruning
Sparsification
Primary area: evaluation
TL;DR: We highlight flaws in using accuracy to evaluate LLM compression methods and suggest alternate metrics.

Abstract: When Large Language Models (LLMs) are compressed using techniques such as quantization, the predominant way to demonstrate the validity of such techniques is by measuring the model's accuracy on various benchmarks. If the accuracies of the baseline model and the compressed model are close, it is assumed that there was negligible degradation in quality. However, even when the accuracy of baseline and compressed model are similar, we observe the phenomenon of flips, wherein answers change from correct to incorrect and vice versa in proportion. We conduct a detailed study of metrics across multiple compression techniques, models and datasets, demonstrating that the behavior of compressed models as visible to end-users is often significantly different from the baseline model, even when accuracy is similar. We further evaluate compressed models qualitatively and quantitatively using MT-Bench and show that compressed models exhibiting high flips are worse than baseline models in this free-form generative task. Thus, we argue that accuracy and perplexity are necessary but not sufficient for evaluating compressed models, since these metrics hide large underlying changes that have not been observed by previous work. Hence, compression techniques should also be evaluated using distance metrics. We propose two such distance metrics, KL-Divergence and flips, and show that they are well correlated.

Accuracy is Not All You Need
Abhinav Dutta
Sanjeev Krishnan
Microsoft Research Bangalore, India t b-abdutta@micrroe wtw.otaatamiccoooooooooooooo
Microsoft Research
Bangalore, India
sakrishnan@micrr wwwpaaaawwmioooooo.toooomomoo
Nipun Kwatra
Ramachandran Ramjee
Microsoft Research
Microsoft Research
Bangalore, India
Bangalore, India ramjee@mi crosoft.. com
nipun. www.kktraamiccosott.ccomm
Abstract
When Large Language Models (LLMs) compressed using techniques such quantization, the predominant way to demonstrate the validity of such techniques is by measuring the model's accuracy various benchmarks. If the accuracies of the baseline model and the compressed model are close, is assumed that there negligible degradation in quality. However, even when the accuracies of the baseline and compressed model are similar, we observe the phenomenon of flips, wherein answers change from correct to incorrect and vice versa in proportion. We conduct a detailed study of metrics across multiple compression techniques, models and datasets, demonstrating that the behavior of compressed models as visible to end-users is often significantly different from the baseline model, even when accuracy similar. We further evaluate compressed models both qualitatively and quantitatively using MT-BBench and show that compressed models exhibiting high flips are worse than baseline models in this free-form generative task. Thus, we argue that accuracy and perplexity are necessary but not sufficient for evaluating compressed models, since these metrics hide large underlying changes that have not been observed by previous work. Hence, compression techniques should also be evaluated using distance metrics. We propose two such distance metrics, KL-Divergence and % flips, and show that they are well correlated.
Introduction
The high cost and latency of Large Language Models (LLMs) motivated the design multiple model compression techniques for optimizing LLM efficiency such as quantization (Dettmers et al.) 2022), Key-Value (KV) cache compression (Ge et al. 2023), pruning (Sun et al./ (023) and spar- sification (Ashkboos al. 2024). However, today, there no standardized way of evaluating the effectiveness of these techniques.
The predominant way of establishing the validity of the LLM compression methods today is to report accuracy on selected benchmark tasks such MMLU (Hendrycks al. 2021a). Hellaswag (Zellers) et al. 2019), ARC (Clark et al.) 2018). LAMBADA (Paperno et al. 2016), etc. It is assumed that the compressed model preserves accuracy on such benchmarks, can be used as an equivalent replacement for the baseline model.
In this paper, conduct detailed evaluation of various compression techniques. We find that while the difference in the aggregate accuracy metric across various benchmarks between the baseline and compressed LLM is negligible in most cases 2%), the actual percentage change in the answers
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Baseline Accuracy
Baseline Accuracy
70
70-
16bit
65-
Bitsandbytes WBAB
SmoothQuant
GPTQ W4A16
AWO W4A16
Bitsandbytes W4A4
55
55
45
-2.5
0.0
2.5
5.0
7.5
10.0
12.5
-2.5
0.0
2.5
5.0
7.5
10.0
12.5
Change in %Accuracy
%Flips
Figure 1: All six quantization schemes show negligible difference in accuracy compared to baseline 16-bit model (Llama2-chat 7B, 13B, 70B and Yi-chat 6B, 34B) seven different tasks. However, all schemes, except GPTQ W8A16 (8-bit weight, 16-bit activation), exhibis large number of flips, indicating severe divergence in model behavior.
can be significant (2 5%). In other words, even when the overall accuracy unchanged, a large number of correct answers change incorrect and vice versa in proportion (we call these flips), between the baseline and compressed model. To the best of our knowledge, we believe that we are the first to identify this phenomenon of flips caused due to model compression. Further, we argue that flips serves as an intuitive metric that captures how significantly different the compressed model is from the baseline model, even when both models exhibit similar accuracy on various benchmarks.
Figure shows the change in accuracy and flips % vs baseline 16-bit model, respectively, for six quan- tization schemes seven benchmark tasks (MMLU (Hendrycks al. (202 Hellaswag (Zellers) ct al. 2019). LAMBADA (Paperno et al.] 2016). ARC Easy and Challenge (Clark et al. 2018)) PIQA (Bisk et al. 2019), and Winogrande (Sakaguchi et al. 2019)). We see that all quantization schemes have begligbbl difference in accuracy (0- 2% compared to the 16-bit version. However, except for GPTQ W (8-bit weight, 16-bit activation Frantar et al. (2023) that preserves accuracy with negligible flips, all other quantization schemes exhibit large number of flips (up to 13.6%) indicating significant divergence from the baseline model.
similar behavior of MMLU task accuracy being preserved while flips increase, for two other compression techniques, namely, layer dropping (Gromov et al. 2024) and WANDA weight pruning (Sun et al., 2023). For example, while Gromov et al. (2024) showed that dropping the last few layers of a model did not affect its accuracy standard benchmarks, we find steady, almost linear increase in the number of flips with the number of layers being dropped.
The phenomenon of flips is puzzling at first glance. While it is easy to see that some correct answers may become incorrect due errors induced by compression, is difficult explain how an approximately equal number of incorrect answers become correct such that overall accuracy is preserved! For example, MMLU questions have options, one of which correct. Thus, any output change could move correct answer to incorrect one but there is only in 3 chance for an incorrect answer to land on the correct option. We present a detailed analysis of flips in Section Furthermore. observe that simply adding Gaussian noise to model weights can reproduce this flips phenomenon (see Table I. This suggests flips arise from the inherent approximations introduced by compression rather than any new information or learning during the compression process.
Finally, one might question whether flips matter if accuracy is preserved. Indeed, if the downstream task where the LLM is used closely matches the benchmark task, accuracy alone might suffice. However, LLMs are typically used a variety downstream tasks that require generating free-form text, where accuracy evaluated on some standard question-answerii tasks could be poor proxy. Thus, we evaluate the compressed models using MT-BBench (Zheng al.) (003), multi-turn dialogue task. We show through qualitative evaluation as well using GPT4 as an automated judge that

---

Table Adding Gaussian noise to weights results in approximately equal correct incorrect and incorrect + correct transitions, with the overall model accuracy mostly unchanged.
Task
Llama3-88 8b Setting
%eccuracy
% Flips
GSM8k
No noise
49.39
Noise std 10-4
48.97
8.23
ARC-challenge
No noise
53.24
Noise std 10-4
53.49
6.05
compressed models with high number of flips are significantly worse than baseline models in this task (see Section ())
Since the goal of compression schemes is to create models that mimic the baseline models as closely as possible, we argue that compressed models better judged by distance metrics with respect to baseline, in addition to capability metrics such as accuracy alone, as is the practice today. We demonstrate that well-known distance metries like KL-Divergence on a given dataset can better identify the differences created due to various compression techniques and this metric correlates well with flips. Further, we show that the scores on MT-Bench (which evaluates free-form generation capabilities of these models) is highly correlated with flips. Thus, we propose that flips, an intuitive and inexpensive to compute metric, as a potential proxy distance metric for evaluating L.L. compression techniques.
In this paper, we make the following key contributions:
Using detailed qualitative and quantitative evaluation of various compression techniques, we show that accuracy not sufficient as an evaluation metric for LLM compression techniques.
We demonstrate the existence of flips general phenomenon and explain why they occur.
We evaluate compression techniques using the KL-Divergence distance metric and show that it correlates well with wiis.
We propose that, where appropriate, flips be used as an intuitive distance metric for evaluating the quality of compression techniques.
2
LLM Evaluation Metrics
We compare baseline and compressed LLMS on the following metrics:
Accuracy capability metric: % correct answers, for question-answering tasks. This determines the competency of the model for a particular task. Multiple-choice question- answering (MCQ) tasks such MMLU expect the model output single token for the correct answer (A/B/C/D), and compare this token with the target answer. For other tasks (like PIQA, Hellaswag, ARC), where the model assigns a probability of multiple tokens), we report the standard normalized accuracy (Eleuther 2021
option (consisting
Perplexity et al. 2005) capability metrie: This measures the overall language modelling capability of I.I.M. is defined as (Average Negative Loglikelshood) calculated over a dataset.
Flips distance metric: measures the % of questions whose answers changed from correct + incorrect incorrect - correct, between baseline and quantized model for all tasks that have correct/incorrect answers. Note that, we do not include incorrect - incorrect transition in Flips for two reasons: For non-MCQ tasks such as GSM8k (obbe al.) 202Tb), TriviaQA (Joshi et al. 2017). etc. exact per-token output matches between different models are rare, resulting in many mismatches. Thus, including this transition may artificially inflate the metric for these tasks. 2) For MCQ tasks, users may care less about these incorrect + incorrect transitions. Nevertheless, if we include incorrect - incorrect transitions for MCQ tasks, we find that, the slips numbers reported in this paper would further increuse by another 20-40% (e.g., increase of 19% Hellaswag, 41% % ARC and 43% MMLU! See Table 11)
3

---

distance metric: consider a dataset having samples with multipl--choce answer options, where the j-th token of the i-th answer option has a probability distribution (i across all tokens in the vocabulary of the baseline model, and ji for the quantized model. Then the KI--divergence between the models for the entire dataset is the mean of Kl.-divergences across all tokens of all answer options and all samples in the dataset.
CeddiveggeneeKulllaaaa and Leibler )
1 N
KL. die
Σ
DeL((P((5))))((( (1)
oppiioss
tokens
datase!
iCoptions
jEtokens
where N is the number of samples in the dataset and (kr(PIIQQ) is the standard KL- Divergence between two probability distributions.
The flips metric is propitious because it is proxy distance metric that is easily interpretable by end-users- for guestion-answering tasks, the end user typically cares about the correct/incorrect answers and not the underlying probability distribution of tokens. Further, the flips metric is easy to calculate as accuracy for any dataset.
It is important to distinguish between capability metrics (accuracy and perplexity in this study) and distance metrics -KL-Divereence and flips in this study). This distinction is necessary because the goal of compression scheme is to create more efficient model that closely mimics the baseline model rather than to create a more capable model. other words, a quantized model is intended to serve drop-in replacement for the baseline model with minimal impact on end-users. Therefore, we argue that distance metrics are more suitable for judging the effectiveness of quantization or other compression schemes.
3 Experiments
We have measured the above metrics multiple LLMs using multiple quantization techniques and bit lengths, on several tasks, as listed below:
Models: We primarily used the Llama2 (7B, 13B, 70B) chat (Touvron al. 2023). Yi (6B, 34B) ) chat (01.AI et al. 2024). Llama3 (8B, 70B) (Dubey et al. 2024) and Qwen2 (1.5B, 7B, 72B) (Yang et al.) 2024) families of models. The chat versions were used because they can be evaluated on MT-Bench (Zheng et al. (023). However, we have observed a similar phenomenon in their pretrained non-chat versions as well (see Table
Quantization: We have evaluated LLM.in88) (Dettmers et al., 2022) as implemented in Bitsandbytes (Dettmers) 2024). with its 8-bit and 4-bit versions (referred to as BnB W8A8 and BuB W4A4 respectively) with default parameters supported with HuggingFace Trans- formers (Wolf et al. 2020). We used GPTQ (Frantar et al.) 2023). AWQ (Lin et al. 2024) with group-size 128 with other parameters being default. We used Smoothquant (Xiao et al.. 2024) (referred to as SQ W8A8) with per-token, per-channel quantization using = 0.5. We use TensorRT (NVIDIA 2024) for SmoothQuant, all other schemes were evaluated using HuggingFace Transformers.
Tasks:
1. For the Llama2 and Yi families, we evaluate the compressed models on ten different
tasks. They include MMLU (Hendrycks et al. 2021a) Table 4 ARC (Clark et al.
2018) Table 7 and challenge Table 8), PIQA BBikk et al., 2019) Table 5
Winogrande Sakaguchi et al. 2019) Table 10 Hellaswag Zellers et al. 2019) Table 6
and Lambada (Zellers al. 2019) Table We also use GSM8k (Cobbe et al.
202
Figure 13] TriviaQA et al. Figure 14fand MT-Bench Zheng et al. 2023 to
evaluate models generative tasks. MT-Bench is dataset with 80 two-turn questions
which can test generative capabilities a model. In this study, we have used GPT-4
(OpenAl et al. 2024) (v0314) as judge, generate the scores reported in Table 2
For the Qwen2 and Llama3 families, we evaluate on MMLU Table 17 GSM8k
Table 21 ARC (easy
Table 18 challenge Table 19 MATH (Hendrycks et al.
2021b Table 20 BFCL. (Ya et al. 2024)
Figure 23 and Ccrolls-Qualit (Shaham
al. 2022
Table 22

---

Divergence
Divergence
Divergence
0.8
%Flips (spearman carr. 0.964)
%Flips (spearman corr. 0.974)
%Flips (spearman corr 0.981)
(a) ARC-Easy
(b) Are- Ar-CChalenge
(c) MMLU (5-shot)
Figure 2: Flips and KI. Divergence are well correlated. Each point corresponds to a model, quantiza-
tion combination in Table 4
Harness- We used Eleuther AI's eval-harness (Gao et al. 2023 for all the experiments, unless specified otherwise. Note that the standard benchmarks (ARC, MMLU, PIQA, Hellaswag, LAMBADA, GSM8k, TriviaQA, MATH, etc.) results on all models use greedy decoding, making these results fully deterministie.
4 Results
In this section, we present extensive evidence for flips across various quantization and pruning schemes, evaluated over large number of models and all tasks. Results for MT-Bench presented in Section
4.1 Quantization schemes
Summary of our results is highlighted in Figure 1 while the performance on each of the individual seven tasks (MMLU, PIQA, Hellaswag, ARC Easy, ARC Challenge, LAMBADA and Winogrande) are in Tables to 10
respectively, in the Appendix.
The main observations from our experiments with quantized models can be summarized as follows:
1. Accuracy: Accuracy is preserved within 1% for the majority of the quantization methods, tasks and models (see Tables 9.. This indicates that accuracy is not sufficient to distinguish between precise and permissive quantization schemes.
2. Flips:
The large %flips (2 5%) a general trend. which holds over different models, almost all quantization schemes, and tasks (see Tables 4 TOp. Specifically, all quantization schemes except GPTQ W8A16 (S 1%) have significant %flips Lower quantization schemes have greater % ffipp in general, indicating greater difference in behavior from the baseline (for example, on MMLU, BnB W4A4 has, on average, 2.4x more flips than BuB W8A8). We focus Flips in this study, but AllFlips (Flips + incorrect-iincorres transitions results can be found in Figure I0 and Table [I in Appendix.
3. KL.-Divergence vs Flips: From Figure 2
observe that the two distance metrics KL-- Divergence and %flips are well correlated. For example, their Spearman correlation on the MMLU benchmark is 0.981.
4. Impact of task type:
MCQ Tasks Generally easier tasks (identified by higher average accuracy) have smaller %flips. For example, MMLU which is relatively hard task has 8-16% flips for Bitsandbytes W4A4 whereas for the same technique, PIQA, an casier task, has 3-6% flips. The reason for this behavior is explained in Section
Generative Tasks Surprisingly, such tasks have much more fips than MCQ ones. For example, GSM8K (Table 13 Table 21). hard task that requires reasoning over multiple steps, exhibits a significant amount of flips (10-25% for BnB W8A8 and W4A4). Similarly, in MATH Table 20 we observe 5-15% flips for BnB W4A4. However, flips are quite small (2-4%) easier tasks like TriviaQA(Table T4p that tests trivia question answering capabilities.

---

25
Accuracy Diff
Accuracy Diff Flips
25
Flips
20
Percentage
Percentage
15
10
10
5
5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Fraction layers dropped
Sparsity
(a) Dropping last n-layers
(b) WANDA pruning
Figure 3: MMLU 5-shot accuracy difference and flips for two compression techniques (Llama2- 13bb model). Even early stages of pruning with no accuracy difference, flips indicate model divergence.
5. Impact model size: Larger models typically have fewer flips than smaller ones. For example, on the MMLU benchmark with BnB W4A4, Llama2-70bb chat has 5.6% flips, while Llama2-133 -12b chat and Llama2-7b 7b chat have 1.4x and 1.6x more flips, respectively. This may be because larger models are more resistant to perturbations introduced by compression than smaller ones.
4.2 Other model compression techniques
We also evaluate the following three compression techniques, though on smaller of tasks and models. Our general observations seen above holds.
Dropping last n-layers (Gromov al. 2024): This work demonstrated that dropping the last few layers did not affect the accuracy on standard benchmarks. We find in Figure 3)a) that as one keeps dropping layers, even though the accuracy increases only modestly, %flips increases significantly, demonstrating that the resulting models keep deviating further away from the baseline.
2. Wanda (Sun et al. 2023 y This is a pruning method. We observe Figure 3)b) that as we though accuracy barely changes, %flips increases steadily.
increase the pruning ratio,
3. SliceGPT (Ashkboos et al., 2024);: This model sparsification method which drops a certain fraction of rows and columns of each dense matrix. We observe Figure 9 in Appendix that even very low sparsity ratios %flips is significant indicating that the compressed models are probably very different from baseline.
4.3 Perplexity
Though we have focused on accuracy so far, our observation that the difference between two models' output token values cancel out leaving the average metric result unchanged, is applicable perplexity as well. In particular, since perplexity may be interpreted as the inverse of the geometric mean of token probabilities, lower probabilities for some tokens in the test dataset may be cancelled by higher probabilities of other tokens. This indicates that perplexity alone is also inadequate evaluating model compression schemes. Therefore, argue that along with perplexity, KL--Divergence between the distributions generated by the baseline and optimized models should also be reported.
Figure in Appendix plots the log-likelihood difference between the 16-bit and quantized model for each of the tokens in the wiki-2 dataset (Merity et al.] 2016) for four different quantization schemes. From the figure, it appears that the log- lieliools of the quantized model is just the log- ligeliiol of the baseline model with some symmetric noise added. Now, since perplexity is -aggglooprobbbliitse)) adding any amount of symmetric noise leaves it unchanged. For example, addition of Gaussian noise to the log- probabiityy outputs of the model maintains the perplexity, while

---

the quality of generation degrades as the standard deviation of the added noise increases (see Table This analysis demonstrates one key weakness with the perplexity metric when used for evaluating compression techniques. While it is not clear if adding Gaussian noise to the log-likelihoods is an accurate representation of the behavior of compression schemes, appears to be a reasonable proxy. As we shall see in Section 6 as quantization increases, there is steady degradation in the quality of the text generated by the model that are visible only by examining them closely.
Analyzing Flips
1.0
Stay
Correct wrong
change
0.8
Fraction
Fraction
0.6
0.6
0.4
0.4
0.2
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Top Margin
Top Margin
Figure 4: When the Top Margin low, an- swer will more likely change (Llama2-70b BnB W4A4, MMLU 5-shot)
Figure 5: When the Top Margin is low, an- swer will more likely be incorrect (Llama2-70b, MMLU 5-shot)
One of the interesting observations in this study has been that when we quantize models, the number of questions where the LLM's answers go from incorrect to correct (referred to incorrect - correct) roughly equal to the number that goes the other way. This may seem unintuitive, because one might expect correct -> incorrect >> incorrect correct, since the number of questions with correct answers usually greater than incorrect answers, so random perturbations should cause more correct answers to flip, and b) given correct answer, the correct to incorrect transition should be likelier because changing to any of multiple other incorrect options suffices, but given an incorrect answer, the incorrect correct transition happens only if somehow the perturbation caused by quantization helps it land on the one correct option out of many. But we observe that this is not the case (and indeed, the opposite may also be true in some cases!).
To help explain the above phenomenon, we introduce a metric called top margin which is the difference in token probability between the best and the second best answer option. By best (second- best) option, we mean the option that was given the highest (second highest) probability. Higher top margin question indicates that the model is more confident about its answer.
Answers are likely to change when top margin is low. Quantization introduces some noise in the weights and activations, due to which there is perturbation in the output answers' probabilities (verified empirically). Thus, we expect that answers are more likely to change when top margin low, since small increase or decrease in probabilities can cause the best and second best options to swap (see Figure 4. To further bolster this claim, we show that the changes in probabilities do not depend on top margin, i.c., roughly all questions undergo the same amount noise (except when the top-margin is very high, where we do not see much change in probabilitics after compression, but such questions are well correlated before/after compression (i.e low confidence answers are likely to remain so and vice-versa) in Figure 8 In |subsection A.4 we find that due this reason, it is very likely that the same question (with low top margin) would be flipped by multiple quantization schemes.
not likely to flip anyway) as seen in Figure 7 We further find top margins
Correct (incorrect) answers have higher (lower) top margin and are thus less (more) likely to flip. Table 27 shows the top margins questions for which the L.LM's 's answer correct and when the answer is incorrect. We observe that, top margin when correct is, on average, greater than the top
7

---

Table 2: MT-MMTTBBieee Average of turn-l and turn-2 - scores, as evaluated by GPT4
Model
16bit
BuB WSA8
GPTQ W8A16
SQ WSA8
GPTQ W4A16
AWQ W4A16
BnB W4A4
Llama-2 chat
6.375
6.375
6.384
6.377
6.018
6.015
6.317
1lama-2 13b chat
6.515
6.540
6.515
6.862
6,459
6.806
Llama-2 70b chat
7,431
7.059
7.225
7.003
6.801
6.937
7.018
Yi-6b chat
6.187
5,937
6,087
NA
5.751
6.096
5.840
Yi-34b chat
7.387
7.220
7.337
NA
7.156
7.053
7.185
margin when incorrect. This is demonstrated Table 28 which shows that flips amongst incorrect answers is indeed higher by 2 or more. Similarly, Figure alass shows that when top margin is low, the answer is more likely incorrect. Thus, correct answers flip much less often than incorrect answers.
For incorrect answers, we would expect roughly 33% chance of them ending correct (for 4-choice MCQ), though the actual % is typically higher because all the remaining options not equally likely. Thus, the combination of incorrect answers flipping more along with slightly higher odds than random in landing on the correct answer results in incorrect - correct transitions roughly matching correct + incorrect transitions.
In subsection A.6 provide some additional empirical analysis for flips, especially for generative tasks where the above top-margin analysis does not strictly apply.
MT-Bench evaluation
In this section, use MT-Benell (Zhngg al. 2023) to evaluate the quantized models' free-form text generation capabilities, using both GPT4 as judge as well as through manual inspection of the model responses.
We first use GPT-4 as a judge and perform automated evaluation. Table 2 shows the MTT-Bench average scores for the two turns in the benchmark (individual turn 1 and 2 scores can be found in Tables 15 and T6 in the Appendix). From the results, we can observe that
Most quantization methods degrade the MT-Bench score for the larger models, by 5% for Llama2-70b chat and 1.5% for Yi-34b chat (Table 2
The degradation in with up to 10% loss for Llama2-70b chat and 5% for Yi-34b chat(Table) T6).
score is higher for the harder turn-2 problem than for turn-1,
Some quantization methods do slightly better than the baseline in MT-Bench score for smaller models but given their lower overall absolute score, we believe this variation is likely caused by the inaccuracies GPT4 evaluation process.
For the different compressed models, we compare them on flips in MMLU vs their difference from baseline on MT-Bench scores in Figure 6a] For larger and more capable models, we find that flips in MMLU correlates well with MT-Bench score.
6.1 Qualitative evaluation
Next, we perform detailed qualitative examination of the performance of these models. Specifically, we choose the Llama2-70B-chat model since it has the highest MT-Bench score (Table 2}. We compare the 16-bit baseline against 8-bit and 4-bit models, quantized using I.L.M.nt80.. We chose L.LM.int8) as it matches the accuracy of the baseline on most tasks and also has the highest GPT4 scores among the W8A8 and 4-bit quantized models for this task (Table
We summarize our findings the qualitative analysis for a sample of ten questions (out of 30 that had similar issues) from MT-Bench in Table 3 The corresponding generated text of all three models for these questions are provided in Table 34, Overall, we find that the 4-bit and 8-bit models are significantly worse than the 16-bit baseline. Specifically, we find that the 4-bit model often does not follow the provided instruction, makes more mistakes, and rambles a lot more, with the 8-bit model performing in-between the 16-bit and 4-bit models.
We encourage the reader to look at the full model responses in Table 34 (at least the first one!) to convince themselves that, at least for this task, there is significant degradation due to quantization,

---

Figure 6: Plips is better predictor of downstream task performance than Accuracy
Llama2-7o
50 wi648
Llama2-roe
Yi-tah: that
Wi-thh chat
CITTO W Bo3 WBAS
Bnd VadAd
0.5
Diff in %Acc
BnE WBAS
%Flips
GPTQ W4A1
BrB WAA
10 WWA1 b KWNQ W4A1
ANG WHAA6
KNO W41.16
CPTQ W1A16
-14
VacAd
BrB
AWO W4A15
W4AI0
Bus WHAH
-2.0
3 WBAB
W9848
WB1L6
WEAL6
-20
Difference in MT-Bench scores
Difference in MT-Bench scores
(a) Models with higher flips on MMLU get lower MT.
(b) Models with lower accuracy usually get lower MT- Bench score though the relationship is not as clear. Spearman Corr. = 0.36
Bench score. Spearman Corr. = -0.75
Table 3: Qualitative evaluation of Llama2-77B-chh model text generations for MT-Bench prompts. Author's summary of model responses shown below; full model generated responses are in Appendix. These results substantiate clear degradation in response quality with quantization.
M'T-Bench Prompt
Summary of 16-bit, 8-bit (BnB W8A8), and 4-bit (BnB W4A4) Jlama-2-70B-che model responses
Consider a satellite that is in circular orbit around the Earth. The speed of the satellite decreases. What will happen the satellite's orbital radius and period of revolution? Please justify your answer using principles Physies.
Only the 16-bit answer and explanation that radius and revolution period will increase is correct, 8-bit and 4-bit answer that radius will decrease and revolution period will increase/remain constant, respectively, and justify their answers hased on (incorrect) Physics!
Take your previous response and rephrase it as a limerick.
16-bit is correct, 8-bit not limerick, 4-bit a limerick but unsound (uses hump and bump for phone). 3) bit does not follow the instruction of starting the story with the given sentence. The 16-hit story is more realistic than the 8-hit/4-bit ones.
Could you write a captivating short story beginning with the sentence: The old abandoned house at the end of the street held a secret that no one had ever discovered. 4) You can see a beautiful red house to your left and a hypnotic greenhouse to your right, an attractive heated pink place in the front. So, where is the White House?
4) 16-bit is correct. 8-bit says White House is not in your line of sight and towards your back, 4-bit says White House is in the middle!
5) What about when twice the number is divided by 5?
5) 16-bit bit and bit are correct, 8-bit is incorrect.
6) Reformulate your earlier reply, output it in JSON format and only include books published after 1980.
6) 16- bit and 8- are correct, 4-bit includes books from
1954 but not 1997!
7) Can you change the ratings from numbers to letters? Capital letters MUST be used when writing the names of phones.
7) No model follows the Capital letters instruction. 4-bit further messes up, changing a rating of 8.2 to B and a rating of 8.0 to B+!
8) Given set of complex equations, extract all unique variable names from each equation... 9) Rewrite your previous response. Start every sentence with an A.
8) 16- bit is correct, 8-bit and b-bit think pi is variable
9) 16-bit follows correctly, 8-bit less fluent, 4-bit is collection of sentences and makes the mistake of capi- talizing the second word in every sentence! 10) 16-bit lists four points, 8-hit reproduces the first three of the 16-hit, 4-hit lists the first two points of the 16-bit, indicating steady quality degradation with quantization.
10) What is the central dogma of molecular biology?
What processes are involved? Who named this?
despite these two compressed models matching baseline accuracy on various tasks (e.g., MMLU accuracy within 1 1%) and suffering only 0.4 lower score on a scale of ten in the GPT4 evaluation. We believe that this qualitative analysis adds further evidence to our claim that benchmark accuracy alone, as standard practice today, is poor metric to evaluate compressed L.MMs, especially, if they are likely to be used for generative tasks in downstream applications.

---

7 Limitations
Predicting performance degradation of LLMS in the wild is a challenging and open problem, and it is possible that any metric calculated standard benchmarks is insufficient. Other limitations are:
If the downstream task is very similar to the benchmark on which the quantized model is tested, then accuracy may be sufficient, and distance metries not needed.
Flips is only warning that the behaviour of model and its compressed version is different this may or may not materialize as visible degradation in some downstream tasks.
Our qualitative evaluation in Section 6.1 is subjective and may not be broadly representative.
8 Related Work
Given their versatility, LLMs are evaluated on a diverse set of tasks (Chang et al. 2024). Since accuracy is one of the most well-accepted metrics used in task evaluation, compression methods today typically focus on accuracy. However, we are not the first to point out the problem with over-reliance on aggregate metries like accuracy when judging the quality of model optimization scheme. Xu eet al. (202)) have proposed label loyalty and probability loyalty as a metric to evaluate compressed BERT models. Other works like et al. (2021). Hooker al. (2020), and Hooker et al. (2021)
have shown compressed ImageNets to be more biased despite preserving accuracy and have proposed Knowledge Distillation based methods address it. There has also been work (Hong al.) 2024) on
evaluating LLM compression schemes on various trustworthiness dimensions. However, metrics for evaluating LLM compression techniques have not been studied widely so far, leading to over reliance on accuracy alone.
There have been many works LLM evaluation that have shown shortcomings of existing evaluation methods. Lyu et al. (2024) have pointed out the misalignment hetween free-form generation and probability based evaluation
MMLU. Sclar et al. (2023) have shown LLMs to be very sensitive to prompt formatting, ZZeng et al. (2024) have shown models to be biased towards a certain option in MCQ tasks. Alzahrani et al. (2024) have shown minor changes in the benchmarks leading to
re-ordering of rankings, and Srivastava et al. (2024) has shown accuracies to be different when
considering the functional equivalent of math problems, Jaiswal al. (2024) have curated existing datasets to create their own benchmark that can be used to evaluate compressed models. Li et al. (2024)
and Jin et al. (2024) have evaluated various quantization tasks on multiple tasks. Namburi al. (2033)
have studied the impact of compression and pruning on an LLM's parametric knowledge, (ea.eed..e.an.ggggggege et al.
(2024) propose a number of other metrics addition accuracy such as fluency, informativeness, coherence and harmlessness. Chang et al. (2024) presents detailed survey on evaluation of L.MSs
that covers what, where, and how to eva evaluate an LLM and lists several challenges LLM evaluation.
However, to the best of our knowledge, none of the prior work have pointed out the phenomenon of flips, that occurs when LLMs are compressed, and the observation that higher flips correlated with larger degradation in model performance despite accuracy matching with the uncompressed model.
9 Conclusion
In this work, we have examined metrics to evaluate the quality of compression methods for LLMs such as quantization. We distinguish between aggregate capability metries such as accuracy, and distance metrics flips and KI. Divergence between the compressed model and the haseline model. We justify why using distance metrics is more appropriate for evaluating model compression methods. We show that accuracy severely underestimates the true distance between models as perceived by the end user. We explain this due the presence of flips between correct and wrong answers when a model is quantized, and explain why the flips are nearly balanced, leading to similar accuracy, while the user-perceived output of the quantized model may be significantly different. We argue that distance metrics such as flips and KL-divergence are essential for evaluating all optimization methods which may change the model outputs and whose goal is to minimize end-user visible behaviour changes from baseline model. We hope that better distance metrics as proposed in this work will enable research in model optimization and compression to progress faster and better meet user expectations on model output quality.
10

---

References
01,AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang. Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yr: Open Foundation Models by 01.AI. 2XXi:24403006633 [cs.CL]
Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards. arXiv:2402.01781 [cs.CL]
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. arXiv:2401.15024 [cs.LG]
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019, PIQA: Reasoning about Physical Commonsense in Natural Language. arXiv:1911.1166
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1-45.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. MeXiv:180300553
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhir Nakano, al. 2021a. Training verifiers to solve math word problems. arXiv preprint arXiv:21100 10::22100.14 14168 (2021).
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021b. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG]
Dettmers. 2024. Bitsandbytes.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. GPT3.int8() 8-bit Matrix Multiplication for Transformers Scale. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 30328-3033.. https://proceediing. wwwpeiiiiinnnnnuui....e.. nww.ips....... http:/ppprrfflsppapppppa Cowaaapprr-oooffeneeeeeeo pdf
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshomn Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Qlano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah. Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence
11

---

Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogaychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh. Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen. Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov. Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitny Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie. Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer. Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine. Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Lc, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan. Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina- Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- Baptiste Gaya, Jeff Marcus. Jeff Tang. Jennifer Chan. Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso. Mo Metanat, Mohammad Rastegari, Mumish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White. Navyata Bawa, Nayan Singhal, Nick Egebo. Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang,
12

---

Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta. Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang. Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang. Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary De DeVito. Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The Llama 3 Herd of Models. arXiv:2407.2177 21783 [cs.Al]
Eleuther. 2021. Multiple Choice Normalization in LM Evaluation. fttps:://blo..eeutteeee aii/
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post- Training Quantization for Generative Pre-trained Transformers. arXiv:2210.1732. [cs.L.G]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation. https: org/10 5281/zenodo 10256836
Suyu Ge, Yunan Zhang. Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for lims. arXiv preprint arXiv:2310.0180/ (2023).
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. 2024.
The Unreasonable Ineffectiveness of the Deeper Layers. arXiv:2403.17887 [cs.CL]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring Massive Multitask Language Understanding. arXiv:2009.03300
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. 2021b. Measuring Mathematical Problem Solving With the MATH Dataset. 1xXiv:2103.03874 [cs.L.]]
Junyuan Hong, Jinhao Duan, Chenhui Zhang. Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, and Bo Li. 2024. Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMS Under Compression. arXiv:2403.1555 15447 [cs.CL]
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. 2021. What Do Compressed Deep Neural Networks Forget? arXiv:1911.0522 [cs.LG]
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. Character- ising Bias in Compressed Models. arXiv:2010.03058 [cs.LG]
Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang. and Yinfei Yang. 2024. Compressing LLMs: The Truth is Rarely Pure and Never Simple. arXiv:2310.01382 [cs.CL]
F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. 2005. Perplexity-a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America 62, S1 (08 2005), 1121/1 2016999 online.pdff
S63-S63, https://doi. http:///ooooo/////////// pdl/62/S//$63/11558900666
Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. 2024. Comprehensive Evaluation of Quantization Strategies for Large Language Models.
16775 [cs.CL]
13

---

Vinu Joseph, Shoaib Ahmed Siddiqui, Aditya Bhaskara, Ganesh Gopalakrishnan, Saurav Muralidha- ran, Michael Garland, Sheraz Ahmed, and Andreas Dengel. 2021. Going Beyond Classification Accuracy Metries in Model Compression. arXiv:2012.01604 [cs.CV]
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv: 705.0355 [cs.CL]
S. Kullback and R. A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics 22, (1951), 79 86. https: org/ 10. 1214 //oss//mm///// 0t////117777999
Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024. Evaluating Quantized Large Language Models.
arXiv:2402. 18158 [cs.CL]
Ji Lin, Jiaming Tang, Haotian Tang. Shang Yang. Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aw -warr Weight Quantization for LLM Compression and Acceleration. arXiv:2306. 009788 [cs.CL]
Chenyang Lyu, Minghao Wu, and Alham Fikri Aji. 2024. Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models. arXiv:2402.38877 [cs.CL]]
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer Sentinel Mixture Models. 1XXv:1600.00844 [cs.CL]
Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, and Frederic Sala. 2023. The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models. arXiv:2312.000 [cs.CL] wttpghaw...o///DD//////asst.....
NVIDIA. 2024. TensorRT.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Sernadett-Shapioo. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-opes,, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufci Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen Hc, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini. Sam Manning. Todor Markov, Yaniv Markovski. Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichir Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keeefe Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filippe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokomny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power,
14

---

Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond. Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijavergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 Technical Report. arXiv:2303.0877 [cs.CL]]
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv:1606.06031 [cs.CL]
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.1066
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting, arXiv:2310.1 13321444 [cs.CL]
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. SCROLLS: Standardized CompaRison Over Long Language Sequences. HxXiv:2201.0352: [cs.CL] https://arxis org/ 2aaab//2000 2201, 03533
Saurabh Srivastava, Annarose M B, Anto P V au2, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince, and Sooraj Thomas. 2024. Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap. arXiv:2402. 19450 [cs.Al]
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A Simple and Effective Pruning Approach for Large Language Models. arXiv:2306. 11695 [L)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov. Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan. Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Tune-Tuned Chat Models. arXiv:2307.0922 [cs.CL]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. HuggingFace's 's Transformers: State-of-the ahe Natural Language Processing. arXiv:1910.03771 [cs.CL]
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2024. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:221 1.0438 [cs.CL]
15

---

Canwen Xu, Wangchunshu Zhou, Tao Ge. Ke Xu, Julian McAuley, and Furu Wei. 2021. Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression. arXiv:2109.03228 [cs.CI.]
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley Function Calling Leaderboard. /tttps://oorilaa.
berkeley.ede hte.oy/edubbllg///e8/ hett://logs/__bbeeeeeee....fieee www.eeley.fuuctioonnneeoooo wtw.aaalllgeeeeebboooooeeee html
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei. Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan. Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 Technical Report. arXiv:2407.1067 [cs.CL] https://arxiv.o org/ abs/ 2407 10671
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can Machine Really Finish Your Sentence? arXiv:1905.07832
Yue Zhang, Ming Zhang, Haipeng Yuan. Shichun Liu. Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Llmeval: A preliminary study on how to evaluate large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19615-19622.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large Language Models Are Not Robust Multiple Choice Selectors. arXiv:2309.03882 [Cs.CL]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. 2023. Judging LLM-as-a- a-Juge with MT-Beech and Chatbot Arena. arXiv:2306.0566 056-25556006655 [cs.CL]
16

---

A
Appendix
Outline
1. Detailed results of Llama-2 chat and Yi-chat families A.
2. Detailed MT-Bench results A.2
3. Detailed results of Llama-3 and Qwen-2 Families on various quantization schemes
A.3
4. Consisteney of Flips A.4
5. Miscellaneous Results A.5
6. Qualitative Analysis of Flips A.6
7. Full model responses to MT-Bench
A.7
A.1 Detailed results of Llama-2 chat and Yi-chat Families on various quantization schemes
Table shows five-shot accuracy the for various models using the standard 16-bit quantization as baseline and difference accuracy and percentage flips for various lower-bit quantization schemes. For example, the accuracy of Bitsandbytes (Dettmers al. 2022) 8-bit and 4-bit quantized models are only 0.55% and 0.78% away from the baseline Llama2-70h model respectively while the flips 4.6% and 8. 1%, respectively. Tables F through [00 show results for other zero-shot tasks such as PIQA, Hellaswag, ARC Easy, ARC Challenge, LAMBADA and Winogrande.
shows
Table 4: MMLU 5-shot accuracy and flips for several models using 16-bit baseline and various quantization schemes. Change in accuracy is negligible (0-2%) in all quantization schemes. However, except for GPTQ W8A16 (8-bit weights, 16-bit activation), all other schemes show large % flips, indicating significant deviation quantized model from the baseline 16-bit model.
Model
16-bit Baseline
BuB W8A8
GPTQ W8A16
SQ W8A8
GITQQ W4A16 AWQ W4A16
BaB W4A4
Accuracy (%)
Change accuracy/ flips (8), compared to 16-bit baseline
Llama2 7b chat
47.21
0.30/4.15
0.08 0.60
-2.36 13.62
00.8//8.26
0.62/ 7.50
-151 10.65
Llama2-136
53,54
0.07
-1.34/8.65
0.14//6.49
0.15/6.36
-1.31
Llama2 70b chat
63.17
/ 3.32
.0.01 0.22
0.24/3.73
-0.55 /4.26
0.76/
-0.78 /566
Yi-34b chat
74.89
0.05
0.03 1.05
NA/NA
-1.717705
-0,68/4.47
/7.4
Table 5: PIQA (0-shot) change in &eacuracy %fflip
Model
16hit
BnB W8A8
GPTQ W8A16
SQ W8A8
GPTQ W4A16
AWQ W4A16
BnB W4A4
Llama2-7b 7b chat
77.203
0.16/2.88
0.05 0.27
0.80/3.80
/4.87
40.70/4.73
6.09
1lama2-13b chat
79.162
-0.16/ 02.8
0.00/
00.11/3.04
0.27/3.21
-0.54/3.48
5.11
Llama2-700 70b chat
80,903
-0.49
0.108
2.66
-0.27/3.10
-0.54/2.72
.0.60/3.77
Yi-6b chat
76.659
-0.38 /3553
0.38/ 0.707
NA/NA
0
0.00 /4.03
-0.27 6.69
Yi-34b chat
79.597
-0.54/ 5.01
-0.05 /0.71
NA/NA
00.54/4.6
-0.11
0.05 / 4.08
Table 6: Hellaswag (0-shot) change in &eccuracy / %fliis
Model
16bit
BnB W8A8
GPTQ W8A16
SQ W8AS
GPTQ W4A16
AWQ W4A16
EnB W444
Llama2-7b chat
75.532
-0.03 1.66
0.05 0.29
0.06 406/3.113
3.88
-0.66/3.47
-1.06 4.63
Llama2- 13b chat
79.635
-0.10 1.49
0.00/ 0.14
0.13 3,00
-0.45 2.58
-0.54/2.8
0099
Llama2- chat
82.164
-0.17 / 1.26
-0.02/0.11
0.31 / 2.83
-0.22/2.11
.0.19/..74
.0884 2,80
Yi-6b chat
75.771
1.81
0.0/ 0.56
NA/NA
.0.45/4.79
-0.31/3881
-1.56/5.51
Yi-34b
80.681
-0.15 /2.98
0.10/ 0.54
NA/NA
-0.57/2.55
-0.5/3.779
17

---

Table 7: ARC Easy (0-shot) change in baccuracy Sefips
Model
16bit
Bnß W8A8
GPTQ W8A16
SQ W8A8
GPTQ W4A16
AWQ W4A16
BuB W4A4
Llama2-7b chat
69.739
0.17
0.04 /0.38
-0.38
5.66
-1.22 /4.67
-1.13/7.62
Llama2- 13b chat
73.737
-0.08 2.22
0.04 /0.04
-1.17
0.04 /4,59
0.25 4.21
-0.44
Llama2-70b chat
76.220
-0.08 2,86
0.12/0.21
-0.67 /2.52
-0.50/3.11
-0.67 /3662
-1.05/5.18
chat
67.340
2.77
0.25/0.75
NA/NA
1.34/666
40.33
0,25/6990
Yi-34b chat
74.368
0.92/4.29
/0.59
NA/NA
-0.75/4480
0.25/2.61
-0.84/3.6
Table 8: ARC Challenge (0-shot) change in &accuracy %6flips
Model
16bit
BuB W8A8
GPTQ W8A16
SQ W8A8
GPTQ W4A16
AWQ W4A16
BuB W4A4
Llamna2 Bb chat
44.283
-0.25/44 /4001
0.00/0.51
-0.59 5.38
-1.36/8.15
-0.25 /6.57
0.68/9.22
Llama2- 13b chat
50.170
-0.25 /350
0.08 0.26
-2,04 5.29
-1.02 5,97
0.00/ 6.31
-0.85
Llama2-70 70b chat
54.266
0.25 2.99
0.17/0.51 /0.51
0.00/ 1.71
-0.76/4.01
0.00/
50.93//.03
Yi-6b chat
47.269
-0.94/4.18
0.68 1.19
NA/NA
0.68/8.70
-2.04 5.97
/9.42
Yi-34b chat
54.522
0.22/5.20
-0.59/ 1.11
NA/NA
-0.76/6.91
+0.55 /4.95
6.994/6.91
Model
16bit
BoB W8A8
BnB W4A4
Model
16bit
BnB W8A8
BnB W4A4
Llama2-706
54.586
54.283
14.253
52.463 18.498
Llana2 70b
82.189
81.949 2.006
80.974 4.266
Llana2-70-chat
43,290
42.600 12.509
44,200 18.347
IJama2-70- 70-chat
75.384
75.284 2.095
74.097 /4.742
Table 13: GSM8k 8-shot Results
Table 14: Triviaqa 5-shot Results
Table 9: LAMBADA (0-shot) change in %accurrcy %flips
Model
16bit
BnB W8A8
GPTQ W8A16
SQ W8A8
GPTQ W4A16
AWQ W4A16
BnB W444
Llama2-7b chat
66.504
0.25/3.12
0.04 /0.50
-0.33 /3882
-2.87 8.27
-2.18/7.10
-2.05 9.63
Llama2-130 13h chat
68.542
-0.11/2.52
0.02 0.29
-0.42 3.07
-1.28 /536
-1.18/5.41
-2.36
Llama2-70b chat
73.801
-0.21/1.14
0.07 /0.27
-0.19
-0.21 3.98
-0.77
0.50/4.88
Yi-6b chat
64.331
0.79
0.38 1.01
NA/NA
-0.79
-0.46/ 6.95
-2.31/ 10.54
Yi-34b chat
69.571
5.16
0.19 1.16
NA/NA
-0.19/8.58
/.75
0.19
Table 10: Winogrande (0-shot) change in Jeacurrcy / %eflips
Model
16bit
BnB WSA8
GPTQ W8A16
SQ W8AS
GPTQ W4A16
AWQ W4A16
BnB W444
1hh chat
66.456
-0.71 J 4497
-0.39 0..5
-0.08 / /4.81
-0.87 8.60
-1.81 /8.44
0.47 / 10.26
Llama2- 3b chat
71.112
-0.08 4.65
0.08 0.23
0.63 / 4,89
-1.50/
-0.23/6.87
/8.23
Llama2-70b chat
74.901
0.55
0.15/0.31
0,23 / 2.29
-0.08 /4,50
0.08 /3,23
/
Yi-6b chat
70.876
50.31/5.05
-0.15/ 1.42
NA/NA
-2.21/9.11
1.73/789
-1.34/ 10,97
Yi-34b chat
76.874
0.37
0.08 1.34
NA/NA 0.31/
-3.47/3.16
0.47
Table 11: MMLU 5-shot change in kaccuracy and %AllFlips (including wrong - wrong transitions)
Model
16-bit Baseline
BuB W8A8
GPTQ W8A16
SQ W8A8
GPTQ W4A16
AWQ W4A16
BuB W4A4
Llama2-7b chat
47.21
-0.30 6.44
0.08 1084
-2.36 20.93
-0.58/
-0.62
-1.51 16.63
Llama2-15 13b chat
53.54
-0.10 5.32
0.07 / 0.64
-1.34 13:03
-0.14/ 10.01
-0.15 10.07
-1.31 12.58
Llama
0.24/ 5.04
Yi-6b chat
62.95
0.00
0.15 2.02
NA NA
-1.53/ 13.14
-1.00 10.85
-2.07 15.37
Yi-34b chat
Table 12: MMLU 5-shot Results on Pretrained Models change in /
Model
16-bit Baseline
GPTQ W8A16
SQ
GPTQ W4A16
AWQ W4A16
BnB W4A4
Llama2-777 7b
45.85
-..8//5.77
-0.09 0066
-8.02 28.53
-0.56 10.97
-0.33/ 13.39
-3.176/ 14.27
Llama2: 1222-1133 3b
55.21
0.16
0.04 0.60
4.024 18.99
0.30/
-1.20 1.20/8.06 8.06
-1.97 / 11.74
Llama2-706 70b
1.66
0.40
0.05 6.222
40.5/4.744
-0.82/7.88
18

---

A.2
MT-Bench Detailed results
Tables 15 and
show MT-Bench results for each of the two turns, respectively.
Table 15: Turn MT-Bench Scores
Model
16bit
BnB W8A8
GPTQ W8A16
SQ W8A8
GPTQ W4A16
AWQ W4A166
BnB W4A4
Llama-2 70b
7.31
7.25
7.32
Llama-2 13b chat
7.02
6.87
7.11
7.25
7.08
7.03
7.36
Llama-2 7b chat
6.80
6,96
6.78
7.00
6.58
6.64
6.93
Yi-6b chat
6.89
6.81
6.88
NA
6.67
6.81
6.68
Yi-34b chat
7.76
7.53
7.48
7.46
7.34
Table 16: Turn MT-Bench Scores
Model
16hit
BnB W8A8
GPTQ W8A16
SQ W8A8
GPTQ W4A16
AWQ W4A16
BnB W4A4
Llama-2 70b chat
7.35
6.81
7.01
6.78
6.39
6.62
6.71
1Jama-2 13b chat
6.00
6.20
5.92
6.47
5.83
5.85
6.25
Llaraa2 7b chat
5.94
5.78
5,98
5.74
5.43
5.37
5.70
Yi-6b chat
5.48
5.06
5.28
NA
4.81
5.38
5,00
Yi-34b chat
7.19
7.03
A.3 Detailed results of Llama-3 and Qwen-2 Families various quantization schemes
Table
MMLU (5-shot) change % accuracy/ flips
Model
16-hit Baseline
BnB W8A8
GPTQ W8A16
GPTQ W4A16
AWQ W4A16
BnB W4A4
Qwen2-1.5B
55.78
-0.15 / 3,94
0.28 0.225
-166 14.46
-1.22/ 12.83
-2.72 14.38
Qwen2-7B
70.34
-0.20/2.91
0.19/ 1.46
-0.88 8.14
-1.15/8.15
-2.13/9.11
Qwen2-72B
84.22
-0.07 / 1.28
0,64
-0.24/ 3.10
-0.03/2.44
-0.33/3.40
Llama3-8B
65.25
-0.13 5.51
0.06 2.29
-NA NA
-2.27 / 12.08
/
Llama3-70B
78.65
NA/NA
-0.15/
-0.93 6.25
0.46/4.82
-2.67
19

---

Table 18: ARC-Easy (0-shot) change in % accuracy/ % flips
Model
16-hit Baseline
BnB W8A8
GPTQ W8A16
GPTQ W4A16
AWQ W4A16
BnB W4A4
Qwen2-1.5B
60.52
0.25 / 3.62
0.12/0.80
-3.32/ 10.48
-..35/8.67
/980
Qwen2-7B
74,54
0.84 / 2.69
-0.17 0.59
1.47/ 6.19
-0.59/5.30
-1.51/7.74
Qwen2-72B -2B
80.51
0.29 1.89
1.89
0.00/ 0.50
0,55 4.08
-0.29/3.16
-0.46/4.17
Llama3-8B
77.74
0.80 3.49
0.00 0.42
NA NA
-0.63/6.52
-2.40/9.13
Llama3-70B
86.07
NA/ NA
-0.88 2.99
4.84/8.46
-2.31/4.84
4.33/8.04
Table 19: ARC-Challenge (0-shot) change in % accuracy/ % flips
Model
16-bit Baseline
BnB W8A8
GPTQ W8A16
GPTQ W4A16
AWQ W4A16
BnB W4A4
Qwen2-1..
36.18
0.00/3.75
-0.26 1.28
2..73
-0.43/..59
1.28 /9.81
Qwen2-7B
0.60 2.99
0.00 /
1.62
-0.09
0.26 8.96
Qwen2-72B
60.15
0.42/ 1.79
-0.09
-0,26 5.03
-0.51/3.41
-0.77 /..72
Llama3-8B
53.24
-0.68 4.61
0.60 1.28
NA
-1.37/8.87
-3,50/ 11.69
Llama3-70B
64.33
NA
31.02/3.24
-5.46 10.58
-2.39/5.29
-5.80/ 10.41
Table 20: MATH (4-shot) change in accuracy/ % flips
Model
16-hit Baseline
BnB W8A8
GPTQ W8A16
GPTQ W4A16
AWQ W4A16
BnB W4A4
Qwen2-1.58
3.56
-0.04 1.52
-0.22 /0.74
1.28 /4.82
-0.98/3.02
0.92/4.56
Qwen2-7B 2B
18.28
3,,14/3.42
0.55
-0.66 19.22
-0.16/7.40
/1..22/5.22
Qwen2-72B
28.20
-0.06 / 2,90
0.20 1.48
7..42/7746
-0.14/5.58
-2.40/9.16
Llama3-8B
13.48
3.44
-0.10/1.50
NA/NA NA
-1088 6.60
-1.72/ 8.24
Llama3-70B
23.98
NA/ NA
-0.30/1.66
-1.2877.2
-0.82/5.14
-1,50/7.06
Table 21: GSM8k (8-shot) change in % accuracy/ % flips
Model
16-bit Baseline
BnB W8A8
GPTQ W8A16
GPTQ W4A16
AWQ W4A16
BuB W4A4
Qwen2-1.55 1.5BB
58.42
/ 12.24
0.30//.37
10.99/233 /3.65
-8.76 23.46
-14.78 / 26.46
Qwen2-7B
78.47
-0.08 / 19,33
0.19/4.74
-2777 / 16.34
-1.97 14.71
-7.51 //51//20.24
Qwen2-72B
89.84
0.04 2.84
0.08 2.20
-1.44/ 6.44
1
-1.7177.2
[llam-3-88
49.39
-1.40/ 14.59
-1.14/6.22
NA /NA
-7.58 23.96
-12.85 227.48
Llama-3-700
80.89
NA/NA
-0.72/5.50
-0.80/ 13.00
-0.57/8.83
13.77
Table 22: Scrolls- Quality (O-shot) change in accuracy/ % flips
Model
16-bit Baseline
BnB W8A8
GPTQ W8A16
GPTQ W4A16
AWQ W4A16
BnB W4A4
Qwen2-1.5B
34,08
-0.28 / 1.73
00.19/0.67
-0.72/7.14
-0.91 / /.66
-1.05
Qwen2-7B
38.93
-0.09 / 2.20
-0.04 / 0.62
0.33 5.113
0.53
-0.04/ 8.39
Qwen2-72B
46.79
-0.09 / 1.53
0.00 / 0.57
-0.04/3.55 3.59
0.67 /.16
/3.67
Llama3-88 8B
42.66
.0.77 / 3.07
-0.05 / 0.82
NA /NA
-0.48/5.94
/1.20/7.53
Llama3-70B
49.33
NA/NA
-0,09 1.33
-0.09 /7.29
-0.19/4.79
-2.88 11.41
20

---

Model
16-bit
GPTQ W8A16
GPTQ W4A16
AWQ W4A16
Gemma-2B-i
15.41
0.59 1.06
/
-0.17/4.77
Gemma-7B-
49.35
-0.53 / 2.18
-4.18/ 17.24
5,00 17.71
Mluma-3-8B-Instaene
66.47
0.64 3.71
-6.41 6.41//3.44
-2.23 / 1 17.18
Table 23: BFCL-greedy change in % accuracy % flips
We report BFCL-greedy (tweaked BFCL generate results greedily) results instead of BFCL-standard because by default, BFCL uses top_p sampling. We are interested in measuring perturbations introduced solely by quantization in this experiment and thus it is crucial ensure that the sampling step does not add any further noise. To emphasize this point, we report two runs of BFCL-standard and show that there are significant flips between them (see below Table 24 making such evaluations irrelevant for measuring the effect of quantization.
Model
16-bit (run-1) 16-bit (run-2)
GPTQ W8A16
GPTQ W4A16
Llama-3-88 --mm----B--i
59.59
-1.05 / 23.53
0.00 24.59
-7.00 33.84
Table 24: BFCL-standard change in % accuracy / % flips
21

---

A.4 Consistency of flips
The following table Table 25 shows the of questions whose answers are changed by 0-6 quantiza- tion schemes (for Llama2- 20b MMLU task 15K questions):
The main observations 1) for most questions (84%). answers unchanged by all the quantization schemes (and these, as expected, have high top margin). 2) Out of the remaining 16%, roughly half (8%) of the questions commonly changed by between the schemes. Our conclusion is that flips are mostly consistent across schemes.
schemes indicating significant overlap
For more fine-grained analysis, we also report pairwise fraction of changed examples that com- mon between two quantization schemes in Table 26 We use overlap coefficient Overlap(A, B)
|AnB] min Bl) to quantify this where A and B are the set of examples changed by the schemes. We again good overlap, ranging between 0.5
22

---

Table 25: % of questions impacted by varying number of schemes and their corresponding top margins.
#Schemes that Change Answer
2
3
5
6
% of Questions
84.0
7.7
4.2
2.5
1.1
0.3
-0.0
Top Margin
0.73
0.23
0.14
0.10
0.06
0.06
-0.0
Table 26: Comparison of overlap coefficient amongst different quantization schemes: BnB W8A8,
BnB W4A4, and GPTQ W4A16.
BnB W8A8
BnB W4A4
GPTQ W4A16
BnB W8A8
1.0
0.50
0.39
BnB W4A4
0,50
1.0
0.44
GPTQ W4A16
0.39
0.44
1.0
A.5 Misc. Results
Table 27: Top margin when answer is correct/wrong. Top margin is higher for correct
Model
MMLU
Hellaswag
Are Easy
Are Challenge
Llama2-7b chat
0.715 / 0.493
0.097 / 0.043
0.112 / 0.018
0.042/ / 0.039
Llama2-133 chat
0.720 0.435
0.102 / 0.043
0.130 / 0.015
0.052 0.036
Llama2-70b chat
0.758 0.434
0.112 / 0.044
0.131 / 0,014
0.061 0.034
Yi-6b chat
0.720 0.363
0.098 0.045
0.089 0.017
0.041 /.031 0.031
Yi-34b chat
0.824 / 0.469
0.106 / 0.044
0.113 / 0.013
0.053 0.029
23

---

Table 28: MMLU 5-shot. The first/second number indicates the % of correct/incorreet answers of the baseline model that changed. We see that more % of incorrect answers change.
Model
BnB 8bit
SQ 8bit
GPTQ 4bit
AWQ 4bit
BnB 4bit
Llama2-78 7b chat
4.7/7.9
16.9 / 24.5
9.3 15.3
8.5 14.9
12.8 / 19.9
Ilama2- 13b chat
3.2/7.7
9.3
6.1/14.4
6.0 14.6
8.7/16.5
Llama2-70h chat
3.0/7.2
/.1
3.75
/.8
5.0/13.2
Yi-6b chat
2.8/9.1
28.7
8.6/
6.8/
10.3 23.9
Yi-34b chat
1.6/7.8
36.4/55.2
18.4
3.4/12.7
6.0/
Table 29: Adding Gaussian noise worsens output quality while keeping perplexity same. %correct tokens measure the % of tokens in the input that would have been selected greedily by the model (all results with Llama2-13b chat) Std. dev.
Textual response
Perplexity
% correct token
KL Div
(prompt:") "tell me one stanza poem")
(on Wiki-2)
0.0 about the beauty of nature Here's one-stanza poem about the beauty of nature: "In the forest's 's embrace, find my peace. Beneath the canopy of leaves, my soul does cease To wander, lost in thought, and let the world pass by, For in the beauty of nature, I am free to fly."
5.70
61.3
0.00
1.0 about tree
5.70
58.2
0.20
Here's one-stanza poem about a tree: "A towering oak, its branches strong and wide, Shades the forest floor, a haven the pride." "
3.0 that you like. like this one:
5.70
42.5
1.86
The stars are the old sailors, their breaths their words are the waves, their laughter is the tides. It's beautiful way of describing the stars and the night sky, and has a nice ring to it.
the winds,
5.0
about a flower that not roses or daffodils. I've been writing collection of haikus, and I'd like use this one as a way to break up the repetition those popular flowers. Thanks much in advance!
5.70
21.5
5.67
I'm glad to help! Here's s one-stanza poem about a flower that's roses or daffodils: Lily of the valley.
So delicate and fair, Your sweet scent fills the air.
hope this helps and adds some variety to your haiku collection!
24

---

www
Liaaz 70
0.30
Liomma 33b
Liema2-700
correlation 0.540
Tupes
(Rapes)
Chunge Gn
rokabik
mmm
Mangnn Past Duartication
0.0
0.00
098
0.0
9.0
00
0.2
0.5
0.8
1.0
top Mang Bused Siee
Figure 7: Change in prediction probability top margin: For the MMLU 5-shot benchmark, plot the baseline top-margin for the ques- tions on the x-axis, and the change in pre- diction probability of the choices in the quan- tized vs the baseline model on the y-axis (com- puted as EieA/B/CyD PProbabililghasee Probabilitymmantssaa)) We use the BnB W4A4 quantization scheme for these results. The observation holds across different quantization schemes.
Figure 8: Top margin in baseline vs quantized models: Scatter plot of the top-margin for the MMLU 5-shot benchmark in the baseline vs quan- tized models. We use the Llama2 13b model and BnB W4A4 quantization scheme for this plot. The top-margins show strong correlation (Pearson cor- relation 0.84) and high top margins in baseline are likely to remain so the quantized model.
50
40
Percentage
30
20
10
Accuracy Diff
Flips
0.0
0.1
0.2
0.3
0.4
0.5
Sparsity
Figure SliceGPT, Accuracy and Flips vs Sparsity
25

---

40
Flips
Flips AllFlips
AllFlips
30
Percentage
Percentage
25
20
15
10
5-
0.0
0.1
0.2
0.3
0.4
0.5
0.1
0.3 Sparsity
0.5
Fraction of layers dropped
(a) Extending Figure aa with AllFlips
(b) Extending Figure bb) with AllFlips
Baseline Accuracy
70
16bit
GPTQ WBA16
65
SQ WBAB
GPTO W4A16
60
Bn8 W4A4
55
50
45
10
15
20
%AllFlips
(c) Equivalent of Figure with AllFlips that includes incorrect to incorrect transitions
Figure 10: AllFlips Results
26

---

10a
109
frequency (log scaled)
frequency (log scaled)
10
10a
10a
10a
10a
10a
10
10
109
108
-0.05
0.00
0.05
0.10
-1.0
-0.5
0.0
0.5
2.0
1.5
loglikelihood difference
loglikelihood difference
mean 0.000, std dev. 0.012
mean 0.023, std dev.
0.203
(b)
GPTQ 48AA16 KL Div.=7.9
(a) GPTQ W4A 6668 KL Div.=0.02)
10
frequency (log scaled)
frequency (log scaled)
10a
10
10a
102
102
101
10a
10a
10a
-0.5
0.5
loglikelihood difference
loglikelihood difference
mean 0.032, std dev. 0.249
mean = 0.007, std dev. = 0.098
(c) BnB 44 KL Div. = 0.03) (d) BnB W8A8@perplexit==4.61 KL Div. 10-9)
Figure 11: The loglikelihood difference plots are somewhat symmetric around zero indicating that even though average loglikelihoods (and perplexity) of the baseline and quantized models are similar, actual per token loglikelihoods might be very different. The results are calculated on Llama2-13b, wiki-2Merity al. (2016) dataset where perplexity is 4.57.
27

---

A.6 Qualitative Analysis of Flips
We present some empirical observations below for the some of the tasks studied in this work.
MATH
This requires the model to just output the correct answer (and not the intermediate steps). We find it hard to clearly characterize the type of mistake made by the LLMs (especially because the range of subjects is very diverse). But, on many questions we find there be simple mistakes (e.g., off by one errors, off by a factor of 2/10, etc).
Additionally, in the MATH dataset, each question is accompanied with a tag that indicates difficulty (E [1, 2, 3, 4, 5)) where a higher number indicates that the question is more difficult. This lets validate our intuition that the questions that are answered by both the baseline and quantized models are the casiest (average difficulty 2.49), and the ones answered by exactly one of them are moderate (this is where correct incorrect transitions and vice versa happens, average difficulty 2.98) and the ones answered by neither are the hardest (average difficulty 3.56). The above numbers were obtained using Llama-3 8b and it's BnB W4A4 quantized version.
GSM8k This task requires the LLM to generate the answer with step-by-step explanations. Here also, we find it hard to clearly characterize the type of mistakes made (though many of them can be classified as simple calculation errors) or pin down the specific type of question in which such phenomenon happens.
In this task specifically, we observe that at times the model gets stuck in a loop, resulting in output at all (rather than incorrect one). For example:-
*Question*:
Travis had 61 apps on his tablet. He deleted 9 apps he didn't 't use anymore and downloaded 18 more. How many apps are on his tablet now?
*Llama-3-8b BnB W4A4 Answer* Travis had 61 apps on his tablet.
He deleted 9 apps he didn't 't use anymore and downloaded 18 more.
Travis had 61 apps on his tablet. He deleted apps he didn't use anymore and downloaded 18 more.
Travis had 61 apps on his tablet. He deleted apps he didn't use anymore and downloaded 18 more.
Travis had 61 apps on his tablet. He deleted apps he didn't use anymore and downloaded 18 more. (... and so on ...)
We measure % of such cases %invalid. We observe that quantization makes %invalid worse in general. For example :- %invalid for Llama-3-8b, its BnB 8bit and BnB 4bit versions are 2.00%, 2.43% and 3.72% respectively.
BFCL
This task requires the LLM to call an API with the correct parameters solve given problem. The available API function definitions are given as a part of the prompt. Given the setup. we feel all problems can be considered to be of similar difficulty. After analyzing the results, we categorize the errors into the following groups:
1. incorrect API usage- this involves cases where the all the parameters not initialized with their correct values or supplying more/less than the required number of parameters, etc. A very common type of error in this category is supplying parameters as a dictio- nary. For eg. we observed that the same function is called in two ways sum(a=5.b=7) and sum(" a'a''. :5, 'b':7), but only the first version is correct.
calling the wrong API this involves calling an altogether different function than the one actually needed to solve the problem.
3. misunderstanding the question this involves solving orthogonal problem to the one required.
4. adding/omitting characters that make the API call non-parsable self explanatory
In general, observe that most of the errors are in the first group itself.
Scroll-Quality, ARC, etc.
28

---

These questions are of MCQ type and it is easier to characterize flips in such tasks (dealt with in the main section already). For such, tasks, top margin which measures the confidence of the baseline model in its answer, is good predictor of whether the answer will flip or not.
Representative examples of Flips
All the following experiments use Llama3-8B and it's BnB W4A4 version.
1. MATH
correct incorrect (baseline correct, quantized wrong
Question
Baseline
Quantized
Answer
Answer
Max must take 10 steps to go the same distance three of his dad's steps. His dad takes 30 steps to walk down the hall. How many steps must Max take to walk down the same hall?
100
300
Compute arccos
Express your answer in radians.
n/3
The Asian elephant has average gestation period of 609 days. How many weeks is this gestation period?
87
88
How many positive divisors do 48 and 156 have in common?
12
What is the largest integer that is solution of 13x + 8 < 35
2
3
incorrect - correct cases (baseline correct, quantized wrong
Question
Baseline Answer
Quantized
Answer
Given and find the largest possible value of |v = 5 and |w| 8, find the largest possible value of [projwy
4
5
Let and b be two nonzero vectors such that + b and are orthogonal, and 2
v2
a + 2b and are orthogonal. Find
Find 25 divided by 112
300
75
The combined weight three basset hounds is 185 pounds. The two smaller dogs weigh the same. The difference between the larger weight and the smaller weight is 20 pounds. How many pounds does the largest dog weigh?
65
75
Suppose is complex number such that 2 100 751. Find [2]
25
5
2. BFCL
correct + incorrect cases (baseline correct, quantized wrong
Question (paraphrased)
Baseline Answer
Quantized Answer
Comment
Calculate the electrostatic po- tential energy of particle with a charge of 5 coulombs
Malculate_elecrrottee
electrostatiic Incorrect API usage supplied poten- tial aall__eeegy_ee energy('charge':
poten-
params as dictionary
tial energy(charre-55
field with a 10 volt poten- tial difference
voltage=10)
5.0, 'voltage": 10.0)
29

---

Calculate the total cost of 101 dumplings $0.10 each and 20 rice bowls at $10 each
order_food (dtem=('uumllngss 'rice bowls'], J. quan- tity=[101, 20].
calculate_ total
Called the wrong API
(item=['dumpp doum-dddumlnngss.
'rice bowls"), quan- tities=[101, 20].
price=[0.1, 10))
prices=[001, 10))
Can you calculate the areas of circles with radii of 5, 10, 15, and 20
[geometry_aree
geometry_ _ormetty_area_aeee _ee__eer____elleeeee (radius=5, ra- dius=10, radius=15, radius=20)
Incorrect API usage to cal- culate the area of circles, the same API needs to be called 4 times, but the quantized model calls the function once with 4 radii values which is invalid.
circle(radius=S),
geometry_
1ircle(radius=00), geometry_ane
circle(radius=15), geometry area circle(radius=20)
Please calculate the factorials for 5, 7, 10, and 12 se- quence.
math (acthfffataiia( math factorial math (nct math faahhfffftrrran10000000 math factorial(n=10), factorial of but the question
math factorial(n=5),
Misunderstood the question
Quantized model calculated
aath_factoria((=122) math factorial(n=11) asks to do this for 7 instead.
incorrect - correct cases (baseline correct, quantized wrong
Question (paraphrased)
Baseline Answer
Quantized Answer
Comment
Play the Game of Life for 3 rounds starting from an empty board, then play chess where the 1st move is e4 and the 2nd move is c5.
[chess.play
[game_of life.play (rounds=3, start_ board=[]), chess.play
Misunderstood the question the baseline model forgets that chess needs to be played after (moves=('e4", 'e5'I)]
(moves=['e4;
'e5'D), game_ of_ (ife.play(tounds=3, start_ board=[]]
Game of Life
Find the intersection points of the functions y=3x+2 and y=2x+3.
functions.intersee (func- lionl='y=3x4
imctions.intersedt (func- 3onl='3*x+22', func- tion2='2'x+3")
Incorrect API usage 3x+2, 2x+3 are syntactically wrong
func- tion2='==2+33)
3. GSM8k
correct - incorrect cases (baseline correct, quantized wrong
Question
Baseline Answer
Quantized Answer
Comment
Gunter is trying to count the jelly beans in a jar. He asks his friends how many they think are in the jar. One says 80. An- other says 20 more than half the first one. A third says 25% more than the first What is their average guess?
The first guess is 80. The second guess 20 more than half the first one,
The first person guessed 80 beans. The sec- ond person guessed 80 + 20 = 100 beans. The third person guessed
Quantized model mis- calculated the number of beans of the second friend (it forgot that this person has 20 more than half of the first person)
which 20 + 80/2 = 60. The third guess is 25%
more than the first one,
80 25% = 100 beans.
which is 80 80/4
The average guess is (80 + 100 + 100)/3
100.The average guess is
(80 60 100)/3 80.
23333333333333333
#0Н# 80
beans.
####
233.333333333333
30

---

eggs Janet buys brooch for her daughter. She pays $500 for the material to make it and then another $800 for the jeweler to construct it. After that, she pays 10% of that to get it insured. How much did she pay?
The total cost of the brooch is 500 800
Janet pays 500+800 +
The quantized model for- got that the 10% insur- ance has to be paid on the full price of the brooch (1300) and not only the material (800)
$80 1380 for the
1300.She then pays 10%
brooch. ##### 1380
of that, which 10% *
1300 130. So the to- tal cost of the brooch is 1300+130 = 1430. #### 1430
James wants hang crepe-paper streamers from his backyard fence for his birthday party. His backyard fence rectangle that measures 20 feet on the long side and 15 feet on the short side. How many feet of crepe paper does James need buy?
The perimeter of rect- angle is the sum of the lengths of all four sides. The perimeter of James' backyard fence 20 20 15 + 15 = 70 feet. Since James wants to hang grepe-paper stream- ers from his backyard fence, he needs buy 70 feet of crepe paper. ####
The of the rectangle is 20 15=300 square feet. James needs to buy 300 feet of crepe paper. #### 300
The quantized model mis- understood the problem entirely. To fence a rect- angle, one needs to mea- sure the perimeter, not the area.
70
incorrect correct cases (baseline correct, quantized wrong
Question
Baseline Answer
Quantized Answer
Comment
Caroline is three times older than Ben. Ben two times older than Chris. If Chris is 4, how Caroline?
Caroline is three times older than Ben, so Caro- line is x 4 12 years old. Ben is two times older than Chris, so Ben is 2 4 = 8 years old. #### 12
Caroline is three times older than Ben, so Ben is 1/3 of Caroline's age. Ben is two times older than Chris, so Chris 1/2 of Ben's age. Chris is 4 years old, so Ben is x 2 = years old. Ben is 1/3 Caroline's ' age, so Car- oline is x 3 24 years old. #### 24
Baseline model assumes Ben is 4 years old even though the problem states Chris is 4 years old
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morn- ing and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much dollars does she make every day the farmers' market?
Janet eats 3 eggs per day, and she bakes eggs per day, so she has 16 3 eggs left. She sells eggs per day for $2 each, so she makes $9 per day. ##### 9
Janet eats 3 eggs for breakfast every day, so she has 16 = 13 eggs left. She bakes 4 eggs into muffins every day, she has 13 left. She sells 9 eggs for $2 each, so she makes $2 x 9 = 18 dollars every day at the farmers' mar- ket. #### 18
The baseline model for- got to multiply the result by 2 (each egg cost $2)
A.7
Full model responses to MT-Bench
The answers to all 80 questions can be found in the supplementary material.
Table 34: Full model responses by the 16-bit, 8-bit and 4-bit 1lama2-70B-hha model for ten sample MT-Bench prompts that was summarized in Table 3 Our highlights are in red.
31

---

S.
16bit
BuB W8AS
BaB W4A4
No.
Prompt: Consider satellite that is in circular orhit around the Farth. The speed of the satellite decreases. What will happen to the satellite's orbital radius and period of revolution? Please justify your answer using principles of physics.
If the speed of a satellite in a circular orbil around the Earth decreases, two things will hap-
the speed of satellite in a circular arbit
the speed of satellite in a circular orbit around the Farth decreases, the satellite will move to lower urbit with smaaller radius. This hecause the satellite's 's kinetic energy. which is proportional to its speed, has decreased, and the satellite's i potential eaergy, which is propor- tional to distance from the center of the Earth, has increased. As a result, the satellite's total energy, which is the
around the Farth decreases, two things will hap-
pen:
pen:
1. satellite's orhital radius will increase.
satellite's orbital radius will decrease.
2. The satellite's i period of revolution will in-
2. The satellite's is period of revolution will in- crease.
crease.
To understand why. let's consider the principles of physies that govern the behavior of satellites orbir:
Let's justily these statements using principles of physics:
Orbital radius: The satellite's 's orbital radius is determined by its velocity and the gravitatioaal force acting upon
of its kinetic and po- tential energy, remains conslant.
I. Conservation of Energy: The total energy of satellite orbit remains constant. The energy a sum of kinetic energy (the energy of motion) and potential energy (the energy of position or height). As the satellite's speed decreases, its kinetic energy decreases, hut its potential en- orgy remains constant. To conserve energy, the satellite's 's altitude must increase, which means the orbital radius must increase.
the satellite's speed docreases, the gravitational force acting on it remains constant. causing the satellite to move closer the center of the Earth. This decreuse distance from the center of the Farth results smaller orbital radius.
According to Kepler's 's second law of planetary motion, the line connecting the satellite the center of the Earth sweeps out equal areas in equal times. This meaas that the satellite's pe- rind of revolution will remain constant, as the satellite moves to a lower orbit with smaller radius.
2. Peried of revolution: The period of revolution directly proporional to the satellite's distance from the center of the Earth. As the satellite's orbital rudius decreuses, its period of revolution hus
To understand why the period revolution re- mains constant. consider the following:
2. Conservation of Angular Momentum: The total angular momentum of a satellite in orbil remains constant. Angular momentum is sure of an object's tendency to keep rotating, As the satellite's speed decreases, its angular momentum decreases, but the angular momen- tum due to the Earth's 's rotation remains constant. To conserve angular momentum, the satellite's s period of revolution must increase.
1. The satellite's is speed has decrensed, which takes longer complete one arbit. 2. However, the satellite is now closer to the center the Earth, which means has shorter distance to travel to complete one orbit.
increnses. This hecause the satellite a shorter distance to travel to complete one orbit, it takes longer do so.
In summary, a decrense in the satellite's 'sllitt'' speed will cause its orbital radius to decrease and its period of revolation to increase. These changes are a result of the fundamenial principles of physics, specifically the relationship between velocity, distance, and gravitational force.
3. The combination of the satellite's slower speed and shorter distance to travel results comstant period of revolation.
These principles apply any satellite in cir- cular orbit around the Earth. If the satellite's s speed decreases, its orbital radius will increase, and its period of revolution will increase.
This can he further explained the concept of angular momentum, which is the product of the satellite's distance from the center of the Earth, its speed, and the gravitational constant. As the satellite's 's speed decreases, its distance from the center of the Earth decreuses. but the gravita- tional constant remains constant. Therefore, the satellite's angular momentum remains constant, which means its poriod of revolution remains constant.
In summary, when the speed of a satellite in a circular orbit around the Earth decreases, the satellite will move to lower arbit with smaller radius, but its period of revolution will remain constant due the conservation of angular mentum.
2 Turn I prompt: Imugine you writing blog post comparing two popular smartphome models. Develop an ouiline for the blog post, including key points and subheadings to effectively compare and contrast the features, performance, and user experience of the two models. Please answer fower than 200 words. Tur 2 prompt: Take your previous response and rephrase it as a limerick.
Sure, here's is limerick summarizing the previ.
Sure, here's is limerick summarizing the previ.
There once were two phones, so bright, Their cameras were quite sight. One had bump. The other, a hump, But both took pies with great might.
ous response:
ous response: Two smartphones, both top notch.
Two smariphones, and B, compared
with care,
Their features and performance, we'll take a lot.
Their features, performance, and user experi- ence were shared. A's design was sleek. It's camera was unique, In the end, the best choice was declared.
Design and display.
Camera and software, we'll lll play. And battery life, not lorget.
compurison, we'll 'Il make, To help you choose, we' 'Il not fake,
For gamers. photographers, We'll ll find the best, that's for sure. So read on, and you'll see, Which smartphone's 's the hest for thee.
more,
3 Prompt: Could you write captivating short story beginning with the sentence: The old abandoned bouse at the end of the street hold a secret that one had ever discovered.
32

---

Sure! Here's short story beginning with that sentence: The old ahundoned house at the end the street held a secret that one had ever discovered. It stood there, a crumbling relic of a hygone era, its once grand facade now weath- ered and worn. The overgrown garden had long since reclaimed the front parch, and the win- dows were hoarded up, their glass long since shattered. But despite its neglect, there was something alluring about the old house. Some- thing that drew people in, like moths to flame. The neighborhood kids would often dare each other to knock the door, to see if anyone would answer. But no one ever did. And yet. despite the silence, there a feeling that the house was waiting for someone. Waiting for somonne brave enough uncover its sccrets. One rainy afternoon, curious young girl named Lily docided to take on the challenge. She walked up the crumbling steps. her heart pound- ing in her chesi, and knocked on the door. The wood was soft and decaying, and the knock echood through the empty halls inside. To her surprise, the door ereaked open. Just a crack. enough to let her squeeze through. Lily stepped inside, her eyes adjusting to the dim light. The air was thick with dust and the smell of de- cay, but she could sense something else. Some- thing magical She wandered through the empty rooms, her footstops ochoing off the walls. The furniture old and wem,, but there were hints a life once lived here. A faded photograph on the wall, broken vasc on the mantle.
Sure! Here's is cuptiaatng short story beginning with the sentence: The old ahandoned house at the end of the street secret that ever discovered. Its crumbling walls and over. grown garden seemed conceal mystery that had been hidden for docades. The house had been left untouched since the family who lived there had suukenlyy disappeured one night, leav- ing hehind only whispers of a dark and sinister past.
The old absandoned house the end of the street had stood vacant for as long as anyone could remember. Its peeling paint and overgrown gar- den had becume a familiar sight to the residents of the small town, a reminder past. But despite its noglect, there was some- thing about the house that soemed to whisper secrets to those who dared to listen.
had
time long
For years, curinus children had dared each other knock on the door, but no one had ever mus- tered up the courage to do so. The bouse seemed to loom over warning to stay away. But one rainy summer af- ternoon, a. young girl named Lily found herself standing before the house, her heart pounding in her chest.
The neighhorhord children would often dare each other to knock on the door, but no one had ever mustered up the courage to do so. They be- lieved that the house was haunled, that the spir- its of the family still lingered within its walls. One many afterooon, a curious young girl named Lily decided to take matters into own hands. She grabhed her flushlight and set out to uncover the secrets of the old abandoned house. She pushee open the creaky gate and made her way up the overgrown path to the front door. With deep hreath, she raised her hand to knock, but before she could, the door creaked open by itself.
street, its crumbling facade
Lily had always been fascinated by the house, and the secrets it might hold. She had spent countless hours gazing up its broken windows, imagining what could be hiding inside. And now, with the rain pouring down around her, she felt suiden urge Taking a deep breath, Lily pushed open the creaky gate and made hor way up the overgrown path. The house seemed to lean in closer, as if it were trying to listen to her every step. She raised her band to knock. but before she could. the door creaked open by itself.
its mysteries.
Lily's heart raced as she stepped inside, the musty air filling her lungs. She found herself in a large entrance hall with chandelier that hung precariously from the ceiling. Dusty cobwehs clung every surface, and the faded wallp seemed to whisper secrets of the past.
Lily's 's heart skipped beat she stepped inside, the musty air filling her lungs. The bouse was dark and silent, but she could feel weight of history bearing down upon her. She took a step forward, her eyes adjusting to the dim light, and that was when she saw A small mom, hidden behind a bookshelf, glowed with soft, ethereal light. Lily felt shiver run down her spine as she approached the door, hand reaching out touch glowing edge. And then, she heard a voice. A soft, whispery voice that seemed to come from nowhere and everywhere once.
As she ventured deoper into the house. Lily dis- covered a hidden room hehind a bookshelff In- side, she found a series of cryptic letters and numbers etched into the walls. They seemed to be a code, and Lily's mind raced with the possibilities.
And then, she found it. A hiddee room, tucked away behind a bankshelf, was small, with a window that let faint glow of light, The walls were covered in old letters, tied up laded ribbon.
Without warning, a loud noise echoed through the house, and Lily's 's beart leapt into throat. She turned to see a ghestly figure standing in the doorway, watching her. It was the spirit of the family' patriarch, his eyes filled with deep sadness.
Lily's 's heart raced as she untied the ribhon and began to read. The letters were from a woman nained Emily. who had lived in the house century ago. They were addressed to her lover, man named Jack, who had gone off war. Lily read through the letters, she discovered that Emuily and Jack had been segarated by the war, and that Emily had heen left alone in the house, wailing for his return. Bul as the months pussed, she had begun to lose hope. And then, one day, a miracle had happened. Jack had retumed, and the two had been rounited. They had lived the rest of their days in the house, surrounded by love and happiness.
"Welcome, I Lily," ii said. 'I've been waiting for you."
Lily froze, unsure of what to do. But as she looked into the ghoat's eyes, she saw a glimmer hope. spirit began speak, its voice barely ahove a whisper.
Lily's 's raced as she stepped inside the room. her eyes scamning the space for any sign of movement. But there was no one in sight. The voice seemed to come from all around her, echo- ing off the walls and ceiling.
ve been wailing for someone brave enough uncover our secret," it said. "My family and I were scientists, and we discovered a way to harness the power of the elements. But our research was stolen, and we were lorred to flee. We behind our home, hoping that one day someone would find our work and continue our legacy,"
"Who are you?" Lily asked, her voice barely above a whisper.
"I am the keeper of secrets," the voice replied. "And you, Lily, have come to uncover them." " As Lily listened in awe, the voice began to speak, revealing secrets that had been hidden for centuries. Secrets of love and loss, joy and sorrow, of a town that had been shrouded mystery for far too long. And she listened, Lily fell a sense of wonder and awe wash over her, knowing that she had discovered something truly special.
Lily finished reading the letters, her eyes brim- ming with tears. She couldn't I helieve what she had just discovered. The old ahandoned house at the cnd of the street held a secret that no one had ever discovered. A secret that had been hidden for over century, waiting for someone hrave enough to uncaver it.
I eyes widened she listened to the ghost's tale. She couldn't 't believe that she had stumbled upon something incredible. The ghust handed her small box containng key and a note.
As she left the house, Lily felt sense of wonder that she had never felt heforee. She knew that she would never forget the secrets of the old abandoued house at the cnd of the street. And she knew that she would always be grateful the chance to uncover them.
"This key will unlock safe in the hidden mom," it said, "Inside, you'll IIll find our research and the secrets of the slements. Use to change the world, but be careful. There are those who will stop at nothing keep hidien."
The old abundoned house at the end of the street had held secret that no had ever discov- crod, but now, thanks to Lily's curiosity and bravery. its secrets would linally see the light of day.
With that, the ghost vanished. leaving Lily alone the ahandoned house. She stood there a moment, her mind reeling with the enormity of what she had discovered. And then, with a newfound sense of purpese, she set to unlock the secrets of the elements and change the world forever.
4 Prompt: You can see a beautiful red house to your left and a hypnotic greenhouse to your right, an attractive heated pink place in the front, So, where is the White House?
33

---

The White House is not visible from your rent location. The description you provided suggests that you facing red house and greenhouse, with pink place in front of you. The White House separate huilding Incated in Washington D.C., and it not in your imme- diate surroundings. To find the White House, you would need to travel Washington D.C. and look for it in its actual location.
The White House is not visible from your rent location. The red house and greenhouse are both to your left and right, respeetively, and the heated pink place is in front of you. The White House not your line sight.
This is play words and classic lnteral thinking puzzle. The answer is that the White House is in the middle, the word "white" is not referring to a physical building, but rather the color,
sentence describing the ferent colored houses that can be seen, and the White House not explicilly mentioned. There- fore, the unswer is that the White House in the middle, between the red house and the green- house.
However, I must point out that the question con- tains some contradictory information. The pres- of a heated pink place front of you sug- gests that you are already inside a huilding or structure, which would make difficult the White House, which is an external building located in Washington D.C. Additionally, the question assumes that you are facing the front of the heated pink place, which would mean that the White House, were visible, would to your back.
I apologize for any confusion caused my re- sponse, and I hope this clarification helps. you have any further questions or concems, please foel free ask.
5. Tum I prompe: When number divided by the remainder is 4. What is the remainder when twice the number divided 4? Turn 2 prompt: What about when twice the numher is divided by 5?
Sure, let's work that out! When twice the nom- ber divided by S, we can write it as 2(10k + 4) 20k + 8. Now, 20k divided by gives 4k + with remainder of So, the remainder when twice the number divided is 3.
Sure, let's do that! When twice the number is divided by 5, we get: 2(10k+ + 4) = 20k + Now, 20k is divisible 4. The remainder is 8, since 8 is the remainder
When twice the number is divided by 5, the remainder is 3.
5, the quotient is
To see why, let's first find twice the number: 2x m 2(10k + 4) m 20k +
when 20k 8 is divided by
Now, when 20k divided by the remain-
So, the remainder when twice the number is divided by 5 is 8.
der is 3:
20k 8 5m +
I hope this helps! Do you have any more ques- tions?
where an integer.
So, the remainder when twice the number is
divided by 3.
Tum I prompt: Extract following information from the presented texts: The name of the book, the author, the main character, the year 6
publication. Output in the fermat of "main character. book, author. year of publication", one book per line. a) In the realm of wizarding literature. a true standout is the work J.K. Rowling. One of her books that left an indelible mark is 'Harry Potler and the Philosopher's 's Stone', This iconic tale, published in 1997, tells the story of Harry young orphan who discovers his magical abilities his I Ith hirthday. Soon, he finds himself at the Hogwarts School of Witcheraft and Wizardry, a place teeming with magic and adventure, located somewhere in Scotland. b) The magic Middlo-carth
cntranced readers worldwide, thanks the brilliance of J.R.R. Tolkien. In one his seminal works. 'The Lord of the Rings: The Fellowship of the Ring', published in 1954, we meet Frodo Raggins, hrave hobbit tasked with the perilous quest of destroying the One Ring. The opic journey takes him from the penceful Shire to the tumultuous regions of Middle-carth. c) In a galaxy far, for away. the imagination of L.B. Sturlighter gives "The Prism Galaxy Chronicles: The Awakening of the Starcaster', Published in 2028, the story is about Zylo, a humble spuceship mechunic, who unexpecteilly discovers he's a Starcaster a rare individual with the power to munipulate stardust. Set against the hackdrop an interstellar empire turmoil, 7yln's destiny unfolds numerous alien worlds each with its unique cosmic charm. Tur 2 prompt: Reformulate your carlier reply, output it in ISON format and only include books published after 1980.
Sure, here's 's the reformulated information in JSON format, including only books published afler 1980:
Sure, here's the reformulated information in JSON format, including only books published afler 1980:
Sure, here's the reformulated reply in JSON format, including only books published after 1980:
II 'muinChuracterr: "HHrry Potter", "book": "Harry Potter and the Philosopher's 's Stone", "m-- thor": "J.K. Rowling". "ycarOfPublication" 1997 J. I "hrannCarrccee"" 'Zylo". "book": "The Prism Galasy Chronicles: The Awakening the Starasttrr.. " "author": "L.E. Starlighter". Puuiii/abiiaaionn:: 2028
[I "main character": "Harry Potter", "book": 'Harry Potter and the Philosopher's Stone", 'mu- thor': 'J.K. Rowling". "ycar of publication": 1997 J. I "main character": "Zylo". "book": "The Prism Galaxy Chronicles: The Awakening of the Starcaster", "author" : "L.E. Statlighter, "year of publication". ": 2028 ]
[[ "mainCharacter": "Frodo Baggins", "book": "The Lord of the Rings: The Fellowship of the Ring". "author": "J.R.R Tolkicn", "ycarPub- lished": 1954 I "mainnharactter" "Zylo". "baok": "The Prism Galaxy Chronicles: The Awakening of the Starcaster', : "author": : "L.E. Starlighter', "yeanPublished": 2028 I hope this helps! Let me know if you have any further questions.
7
Turn 1 prompt: Analyze the following customer reviews from different sources for thrse different smartphones - the latest iPhone, Samsung Galaxy, Google Pixel- - and provide overall rating for euch phone scale of I to 10. Comsider the following complex and contradictary reviews: TechRudar's 's review of the latest iPhone: The new iPhone is a stunning triumph of engineering that sets new har for smartphome performance and camera quality. However, the incremental dosign and high price mean lacks the "wow" factor previous iPhones. Still, its power and intelligence are unrivaled. - CNETT' review of the latest Samsung Galaxy: The Samsung Galaxy phone has plenty of high points, including amazing screen. fast performance, solid bottery life an impressive arra camera options. That said, Bixby remains luckluster. AR emoji falls flat and the phome's 's overall design hasn't I changed much. The new Galaxy is an amazing phone overall, but it has few nagging weaknesses that keep it from achieving true greatness. - The Verge's as review of the latest Google Pixel: Google's sss Pixel packs cutting-edge spees. innovative Al powered soflware, and a killer camera into sleek design. However, the phone has lackluster battery life, lacks expandable starage. and its performance stutters at times, especially considerngg its high price tag. If seamless coftware, elite photography, and Google's 's brand of A1 assistance are most important, you'll Ill love the Pixel. But the overall experience isn't well-rounded some competitors. Return the answer a JSON object with the overall ratings for each phone out of 10, to one decimal place.
Turn 2 prompt: Can you change the ratings from numbers to letters? Capital letiers MUST used when writing the names of phones.
34

---

Turn I: Sure, here the overall ratings for each phone based on the customer reviews you provided: "|Phone"; 8.5, 'Samsung Galaxy": 8.2, "Google Pixel": 8.0
Turn I:Sure, here are the overall ratings for each phone based on the customer reviews you provided: "iPhone"; 8.5, "Samsung Galaxy": 8.2, "Google Pixel": 7.9
Turn 1: Sure, here the overall ratings for euch phone based on the customer reviews:
"iPhane": 8.5, "Samsung Galaxy": ": 8.2, "Google
Pixel": 8.0
The ratings are hased
following factors:
Here's a breskdown of how arrived these
Here's 's breakdown of caleulated the rat-
iPhone: The review highlights the phone's 's cn- gincering, performance, and cameraa quality, but also mentions the lack of "wow" factor and high price. Therefore, the rating is 8.5 out of 10. Samsung Galaxy: The review praises the phone's 's screen, performance, hattery life, and camern options, but also mentions that Bixby is lacking. AR emoji falls tlat, and the design husn't changed much. Therefore, the rating is 8.2 of 10. " Gongle Pixel: The review high- lights the phone's is cutting-edie edge specs, innovarive Al-powered software, and killer camera, but also mentions the lackluster battery life, lack expandable storage, and occasional perfor- mance stutters. Therefore, the rating is 8.0 out of 10.
ratings:
ings:
iPhone: The TechRadar review praises the iPhune's engineering, performance, and camera quality, but notes that the design is incremental and the price is high. Overall, the review sug- gests that the iPhone is a great device, hut lacks the "wow" factor previous models. Based on this, would give the iPhone an 8.5 out of 10. Samsung Galaxy: The CNET review high- lights the Galaxy's impressive screen, fast per- formance, solid battery life, and camern options. However, also notes that Bixby lackluster and the AR emoji feature falls flat. Additionally, the design haso't ' changed much. Considering these factors, I would give the Samsung Galaxy an 8.2 out of 10. Google Pixel: The Verge's 's review praises the Pixel's cutting- edge specs, in- novative software, and killer camera. However, also notes that battery life is lackluster, there's 's expundable storage, and the perfor- mance can stutter at times. Additionaly. the review notes that the phone's high price tag may not be justified by its features. Based on this, would give the Google Pixel a 7.9 out of 10. It's worth noting that these ratings are based on subjective interpretation reviews you provided, and other people may have different opinions on these phones. I hope this helps! Let me know if you have any further questions.
iPhone: TechRadat's 's review praises the phune's engineering, performance, and carneaa quality, but notes that the design is incremental and the price high. gave the iPhone rating of 8.S, deducting a few points for the lack of innovation the design and the high price tag. Samsung Galaxy: CNET's review highlights the phome's 's impressive screen, last performance. solid buttery life, and camera options. However, it also notes that Bixby is lackluster and the overall design hasn't changed much. gave the Samsung Galaxy rating of 8.2, deducting few points for the lack of innovation the do- sign and the mediocre Bixby feature. Google Pixel: The Verge's is review pruises the phone's 's cutting edge specs, innovative Al powweeeed soft- ware, and killer camera. However, it also notes lackluster, there's 's ex- pandable storage, and the performance can stut- tor at times. gave the Google Pixel rating of 8.0, deducting a few points for the battery life and performance issues, but acknowledging the phone's strengths in software and camern capabilitios.
It's important note that these ratings are haved on subjective analysis of the customer reviews provided may not reflect the overall opinion of all customers.
the battery
Turn 2: Sure! Here are the overall ratiins for each phone based on the customer reviews you provided, using letters instead of mum- here "iPhone": "A", 'Samsung Galaxy": "B+". "Google Pixel": "B"
customer
Overall, all three phones have their strengths and weaknesses, and the ratings reflect the hal- ance of their respective pros and cons.
Turn 2: Sure, here are the ralings for each phome using letters instend of mumbers:
Turn 2: Sure! Here are the overall ratings for each phone based on the customer reviews you provided, using letters instead of numbers: e iPhone: A Samsung Galaxy: B " Google Pixel: C
"iPhone": "A", "Samsung Galaxy": "B", "Google Pixel"; "B+" I hope helps! me know if you have any other questions.
I hope this helps! Let know if you have any other questions.
8 Prompt: Given a set of complex equations, extract all unique variable names from euch equation. Return the results as JSON string, with one line allocated for each equation.
y = (314x3 e(2s) sin(pi*x) syr(()
2) 2A BIG3-C) sum(N=I to 5; In(N)2) = SD*integralaa- to pi; ens(comb(NNal to 10; Noa))
E m(c2) gamma (aldhh-)))p)hh) + sgribeta2 (alpha/112)
Here's the ISON string you requested, contain- ing the unique variable names for each equation: cet
Sure, here are the unique variable names for each equation:
Here's 's the JSON string with the unique variable names for each equation:
pi, sqp(() 2. A, D.D.N. In(N),
cce [
I [ "equation": "y (3/4)x"3 - e*(2x) +
pi 3. E, m. gamma, v. al plpha, beta
{ "equation": "y - (3/4)x'3 - e=((xx) + sin(pits) sqin()"" "variables", "y')..
sin(pi"x) sqr(Ty. "variables": ('x"' "y'] J.
Here's 's the ISON string with one line for each
I "equation": '2A - BB(3+C) sumrN=1
equation:
{ "equation": "2A - B/(3+C] sum(N=1 5; In(N)-2) 5D*integralia=O cos(comb(N- to 10; Noo))", "variahles": I"A", "B", "C","D", 'N", "a", 'pi')). "equation": 'F m(c=2) gamma VVviK (alpha/2) + sqribetaa2 + (alpha/2))22) 'vari- ables". L'... "m", "c", "gamma", 'v", 'd", "al- pha", "beta"] ].
S: In(Ny-2) SD*integral(a=D to pi:
[ ['y". "x","""" "e", "pi", "syn(7y" II "A", "B", 'C". "D", "N", "In(N)', "a". "pi" ][ "E", "m", 'c".
pi:
costcomb(Nel to 10; N+a))", "variahles": PA "B. 'C". 'D", 'N", ".
"gumma"" "v", 'd". "slpha". "beta") ] I hope this helps! Let me know if you have any further questions.
"equation"; "F m(e=2) samma*(vaayy (alpha/2) + sgrt(beta + (alpha/2)-2))", "vari- ablos': I'E", "m", "c". "gamma", "v". 'd", 'al- pha", "beta"l) []] kcc
T hope this helps! Let me know if you have any further questions.
r.c
Note that I've used the "comb" function to com- bine the variables for the second equation, since it's sum of multiple terms. Also, I've used the "surt" function simplity the expression for third equation.
9 Tum I prompt: Campose engaging travel blog post about recent trip Hawaii, highlighting cultural experiences and attractions. Turn 2 prompt: Rewrite your previous response. Start every sentence with the letter A.
35

---

Absolutely! Here's revised version of the trivel blog post about Hawaii, starting every sentence with the letter A:
Absolutely! Here's rewritten version of the previous response, starting every sentence with the letter A:
Absolutely, here's 's the rewritten response: Aloha! My trip to Hawaii was a cultural journey that left me and natural beauty.
of the island's rich heritage
Aloha! I recently returnd from an incredible trip Hawaii, and can't wair share my experiences with you. A warm sunshine, lush greenery, and friendly locals all contributed to unforgettable adventure.
Aloha! My recent trip Hawaii was inuly unforgettalle experience. chance immerse myself in the local culture and explore the must- see attractions made it a journey to remember. A traditional lawatian luau place to start. A feast of local delicanies like kalua pig, poke, and haupin was served with side of warm hospitality. A hula dance perfor- mance followed, transporting me to place of screnity and tranquility.
Hula dance performance was integral part of my experienee, and I was fortunate coough to witness live show at the Polynesian Cultural Center.
the perfect
A highlight of my trip was the opportunity to immerse myself Hawaiian culture. visit the Rishop Museum in Honolulu offered a fasci- nating glimpse into history and traditions of the Native Hawaiian people. A chance to learn about the ancient Hawatian gods and goddesses. the art of hula dancing, and the significance of the Hawaiian language.
A Lum was another highlight my trip. where feasted local delicacies such kalua pig. pake, and haupia.
Visit to Pearl Harbor was a sobering reminder of the events that led the United States' entry into World War II.
visit to the iconie Waimea Canyon was awe- inspiring. majestic sight, often referred to as the "Grand Canyon of the Pacifie," left me speccheess. hike along the trails offered stun- ning vistas and photo opportunities.
Scenic drive along the rim of Waimea Canyon was breathtakngg with majestic views that stretched for miles.
must-sce attraction was the iconic Waimca Canyon on Kauai, often referred to as the "Grand Canyon of Pacific." A stunning view of red and green volcanic rock. waterfalls, and lush vegelation.
A Guided tour to the summit of Mauna Kea was an unforgettable experience, where marveled the vastness of the universe under clear skies, Journey along the Road to Hana was a must- do for any adventure- seeker, with stops terfalls, beaches, and hiking trails that took me through tropical landscapes ocean views.
trip to Halcakala National Park was another highlight. A suarise summit offered spectac- ular view, and day spent exploring the park 's many trails was treat for the senses. A roadside shrimp truck offered the freshest shrimp I've over tasted. Hawaiian staple, poke, was flavorful delight. A fusion of tradi- tional Polynesian. Asian, and American flavors, local cuisine was a culinary adventure. Aloha Spirit, a concept that embodies the Hawaiian culture, erywhere. A foeling of warmth, hospitality, and genuine kindness made every eacounter special. A truly unforgettable experience, my recent trip to Hawaii left me with memories to last life- time. A chance to inmerse mysell the local culture and explore must see attractions made it journey remember. Until time, aloha!
Another list attraction the Haleakala Na- tional Park on Maui, where watched the sun- rise from the summit Halcakala volcano. A park ranger provided informative and entertain- ing commentary about the geology and wildlife the making experience enjoyable.
Aloha! Until we meet again, I'll cherish the memories of my trip Hawaii, true paradise.
memorable experience of my trip visit to a traditional Hawaiian luan. A delicious feast local specialties, such kalua pig. poi, and haupia, while watching captivating perfor. mance of traditional Hawasian music and dance. A warm hospitlity of the hosis and lively atmosphere made for an unforgettable evening. final thought: if you're planning trip to Hawaii, I highly recommend incorporating some these cultural experiences see attractions into your itinerary. A variety of interests. from history to nature to relaxation, lawaii has something for everyone. mahalo (thank you' for reading, and I hoge you have a fantastic time paradise!
evident
must-
Prompt: What the central dogma of molecular biology? What processes are involved? Who named this?
36

---

The central dogma of molecular biology the concept that genetic information flows from DNA to RNA to proteins, and it is the funda- menial principle that explains how genetic in- formation is transmitted and expressed within cells, This idea was first proposed by Francis Crick in 1958.
The central dogma of molecular biology the concept that genetic information flows from DNA RNA to proteins, and it the funda- mental principle that explains how genetic in- formation is transmitted and expressed within cells, This concepe was first described by Fran- cis Crick in 1958.
The central dogma of molecular biology is the idea that genetic information flows from DNA RNA proteins, and that this flow of infor- mation is the fundamental mechanism by which cells process transmit genetic information. The central dogma was first articulated by Fran- cis Crick, one of the discoverers of the structure of DNA, in the 1950s.
The central dogma involves several processes:
The ventral dogma involvees several processees: 1. Transcription: The process by which the in- formation gene's DNA copied into com- plementary RNA molecule. This RNA molecule known as messenger RNA (mRNA).
I. Transcription: The process of creating com- plementary RNA copy of DNA, which initi- ated when enzyme called RNA polymerase hinds to the DNA molecule.
The central dogma is a concepiual framework that describes the processes involved in transmission of genetic information within cells. posits that the information stored first transcribed into RNA, which is then trans- lated into proteins. This flow of information is unidirectional meaning that proceeds from DNA to RNA to protelis, and not in the reverse direction.
is
2. Translation: The process of synthesizing pro- leins using the information encoded the RNA structures called ribu- somes, which read the sequence of nuclentides the RNA and assemble the corresponding amino acids into protein chain.
Translation: The process by which the in- formation in the mKNA is used to build a pro- lein. This
molecule. This
structures called ribosomes,
which read the sequence of nucleotides in the mRNA and assemble the corresponding amino acids into protein.
The pmceeses involved in the central dogma are:
Post-rransrippional modification: After tran- scription, RNA motecules can undergo various modifications, such as splicing, capping, and polyudenylation, before they ready to he translated into proteins.
Post-rrancrippional modification: After trans- lation, the protein may undergo additional mod- ilication, such as folding into its nalive confor- mation, becoming modified by the addition of carbohy drates or lipids, or being transported to other cellular compartments.
Transcription: The process by which the in- formation stored DNA is copied into RNA. This process involves the hinding of enzyme called RNA polymerase to the DNA molocule, which then reads the genetic code and synthe- sizes an RNA molecule complementary to the DNA template.
Protein synthesis: The process of creating proteins from amino acids, which are linked to- gether by peptide boods. This process occurs on ribosones, which read the sequence of codons (groups of ihree mucleatides) in the RNA and assemble the corresponding amino acids protein chain.
The central dogma is a fundamental concept molccular biology that helps to explain how genetic information is transmitted and expressed within cells. It has been widely accepted and used in scientific research since introduction by Francis Crick.
Translation: The process by which the in- formation stored in RNA is used to synthesäze proteins. This process involves the binding of a ribosome to the RNA moleculc, which then reads genetic code and synthesizes a chain of amina acids that will eventually form pro- tein.
central dogma a fundamental concept in molecular biology that explains how genetic information is transmitted and expressed within cells. It has been widely accepled and used scientific research since its propocal by Francis Crick over 60 years ago.
The central dogmaa fundamental concept molecular biology, as it provides a frume- work for understanding how genetic informa- tion is processed and transmitted within colls. been widely accepted scientific community since its introduction by Francis Crick in the 1950s.
37

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have claimed accuracy to be flawed in evaluating LLM compression schemes and have shown results to back this claim, and have also proposed alternate metrics.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have a limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
38

---

Justification:
Our work does not have any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We specify what open source tools and models were used and the configurations used for various quantization schemes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

Answer: [NA]
Justification: Our work uses existing open sourced code and does not need any private code for reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides/CCn
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/aapss.cc/pullc//uiieeee wttpe/icgguie////eeeeeooeee Soaess/ooooeSessiooono....
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the configurations used for different quantization schemes and evaluations (like task, number of shot) are mentioned.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Only MT-Bench includes generation with non-zero temperature that would result in difference from run-to-run in any meaningful way. For every other task, results don't change across runs (not accounting for floating point errors). M'T-Bench evaluation uses GPT-4 and thus it is prohibitively expensive to repeat each experiment multiple times to report any error estimates.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
40

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The actual type of compute resources used irrelevant to the evaluations (not accounting for floating point errors)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp::/neurips: wwwpuuupps.cppuuli///eeeeeeeeeee htt.oliiie/hhissuieeeeleeeell
Answer: [Yes]
We have gone through the Code of Ethics and can confirm that there are no violations.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work proposes alternate metrics of evaluating compressed LLMs and feel that this has no societal impact whatsoever.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
41

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work proposes alternate metrics of evaluating compressed LLMs and feel that this poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all the assets used in this work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
42

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work introduces new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
43

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
44


Reviews:
[{"review_id": "fKgetNlSCr", "reviewer": "Reviewer_Sq51", "summary": "* The paper investigates the fine-grained effects of compression beyond just evaluating and measuring accuracy on benchmarks such as MMLU/Hellaswag.\n* The paper finds that while aggregate level metrics demonstrate a very slight drop in performance (~2%), examining the individual responses highlights that a significant number of responses (>5%) differ before and after compression - responses that undergo the transition from correct -> incorrect and vice versa are termed \"flips\".\n* The authors claim that they are among the first to discover this phenomenon and propose \"flips\" as a way to indicate the difference in capabilities between compressed and the original model.\n* The paper measures and quantifies this phenomenon of response \"flips\" on 7 benchmark tasks for 6 different compression schemes in addition to evaluating these models on MT-Bench.", "questions": "Please see weakness section", "limitations": "Yes, limitations are addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Paper tackles a highly relevant topic for current LLM inference pipelines - cloud providers/on-prem LLM deployments use a wide variety of compression schemes to reduce memory footprint for lower latency and to accommodate higher number of requests.\n* Very comprehensive set of experiments covering a wide range of benchmark tasks and quantization schemes.\n* The concept of \"flips\" and the paper's proposal to use it as a measure of difference in capabilities between the compressed/original versions of the model seems very intuitive.\n* The section on \"top-margin\" and some hypothesis on why model responses flip from \"incorrect\"->\"correct\" is an interesting piece of empirical analysis.", "weaknesses": "* I do not like the section on MT-bench and its use as a proxy for measuring the qualitative aspect of the responses from the original/compressed models. The different in MT-bench scores across the compression schemes in Table 3 are more or less within the range of noise and I dont see any major conclusions being drawn from that table. The qualitative analysis makes it acceptable - however I would like to see some other alternative qualitative evaluations focused on some specific aspects like function calling/reasoning/retrieval augmented setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Additional evaluation**\n\nWe perform additional evaluations as asked by the reviewer. In particular, we evaluate the following tasks:\n- BFCL (Berkeley Function Calling Leaderboard) to test function calling\n- SCROLLS (QuALITY split) to test long context understanding which would be useful in retrieval augmented setup\n- MATH and GSM8k to test mathematical reasoning\n- ARC-easy and ARC-challenge to test general reasoning\n\nThe tables below show the **change in accuracy / flips-percentage** of the quantized model vs the unquantized 16bit baseline:\n\n**BCFL**\n\n| Models                | 16bit | GPTQ 8bit   | GPTQ 4bit    |\n|-----------------------|-------|-------------|--------------|\n| Gemma-7b-it           | 46.82 | 0.06/18.41  | -4.35/22.82  |\n| Llama3-8b-instruct    | 59.65 | 0.59/24.24  | -6.41/33.24  |\n\n**Scroll - Quality (0-shot)**\n\n| Models     | 16bit | BnB 8bit      | BnB 4bit      |\n|------------|-------|---------------|---------------|\n| Gemma-7b   | 40.31 | -0.28/4.51    | -4.55/17.21   |\n| Llama3-8b  | 42.81 | -2.78/16.68   | -7.05/22.19   |\n\n**MATH (4-shot)**\n\n| Models        | 16bit | BnB 8bit    | BnB 4bit      |\n|---------------|-------|-------------|---------------|\n| Gemma-7b      | 11.72 | 1.26/5.98   | -1.22/9.50    |\n| Llama3-8b     | 13.48 | -0.54/3.30  | -1.72/8.24    |\n| Llama3.1 8b   | 14.02 | -0.26/3.66  | -2.02/8.26    |\n\n**GSM8k (8-shot)**\n\n| Models        | 16bit | BnB 8bit       | BnB 4bit      |\n|---------------|-------|----------------|---------------|\n| Gemma-7b      | 50.19 | 2.54/22.93     | -28.39/37.11  |\n| Llama3-8b     | 49.39 | -1.40/14.59    | -12.85/27.48  |\n| Llama3.1 8b   | 48.63 | -1.21/14.94    | -8.34/25.40   |\n\n\n**Arc-e (0-shot)**\n\n| Models        | 16bit | BnB 8bit | BnB 4bit |\n|---------------|-------|----------|----------|\n| Gemma-7b      | 80.05  | 0.63/4.25     | -6.48/15.15     |\n| Llama3-8b     | 77.7  | 0.79/3.49     | -2.40/9.13     |\n| Llama3.1 8b   | 81.06  | -0.63/3.24       | -1.39/9.21     |\n\n**Arc-c (0-shot)**\n\n| Models        | 16bit | BnB 8bit | BnB 4bit |\n|---------------|-------|----------|----------|\n| Gemma-7b      | 54.18  | -0.43/5.89    | -5.39/18.09     |\n| Llama3-8b     | 53.24  | -0.68/4.61     | -3.20/11.69     |\n| Llama3.1 8b   | 53.50  | -1.71/3.67       | -2.39/9.21     |\n\n**Observations**\n\nIn all the above tasks, which are generally harder compared to multiple-choice q&a,  our general observations that flip percentage >> difference in accuracy hold. In fact, for many of these tasks, the **flip percentages are much higher** than observed in the experiments in the paper, even for the 8-bit quantization schemes like GPTQ 8bit and BnB 8bit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i8UoeuHdLw", "reviewer": "Reviewer_mPMT", "summary": "The authors analyze changes in model predictions as an additional performance metric to evaluate quantization schemes applied to large language models (LLaMa2 and Yi). The authors show that during model quantization that the predictions for a substantial number of examples change, beyond the limited number of examples which labeled incorrect post quantization. Additionally, the authors show that the number of examples that change in prediction is correlated with the KL-Divergence between the base and quantized model.", "questions": "1. Are the flipped examples consistent across different compression, quantization schemes, or models? \n2. For a single method scheme with varying degrees of compression (GPTQ W8A16 -> GPTQ W4A16; or pruning at different intensities), do the flips increase monotonically, consistenlty with increasing degrees of quantization? For example, are the examples that are forgotten at 8-bit quantization relearned at 4-bit quantization?\n3. The top margin (difference in probability) metric is used to characterize the model performance pre-quantization; A. What does the metric indicate post quantization? B. Does quantization result in a more uniform probability distribution? C. Are the shifts in predicted likelihood and confidence larger or smaller for examples that were initially higher or lower in confidence?", "limitations": "Yes, the authors discuss limitations of their work.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors provide characterizations of model predictions, demonstrating that model compression techniques (pruning, quantization and layer skipping) affect model prediction beyond standard reported performance metrics. The authors show that there exist subpopulations of examples that are *learned* during compression, counter to intuition that compression would monotonically result in decreases in correct predictions. The authors raise valid concerns in regards to the lack of efficacy in solely relying on accuracy-based performance metrics.", "weaknesses": "The overall results are unsurprising and do not provide additional characterizations of the effects of quantization on model performance or example characteristics.\n- It is expected that model predictions with lower confidence are more likely to change upon compression. However, the current analysis does not provide insight into how these examples change during compression. For example it is not clear if the reason that low confidence examples change in prediction is because they are initially close to the decision boundary or if they observe greater change in model confidence during compression.\n- Change in model prediction has been studied both in the context of continual learning and interpretability analysis of model training dynamics [1,2]\n\nRelated Work\n1. Toneva, Mariya, et al. \"An Empirical Study of Example Forgetting during Deep Neural Network Learning.\" International Conference on Learning Representations. 2018.\n2. Swayamdipta, Swabha, et al. \"Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Consistency of flips**\n\nThe following table shows the % of questions whose answers are changed by 0-6 quantization schemes (for Llama2-70b MMLU task ~15K questions):\n| % of Questions                | 84.0 | 7.7  | 4.2  | 2.5  |   | 1.1  | 0.3  | ~0.0 |   |\n|-------------------------------|------|------|------|------|---|------|------|------|---|\n| #schemes that change answer | 0    | 1    | 2    | 3    |   | 4    | 5    | 6    |   |\n| Top Margin                    | 0.73 | 0.23 | 0.14 | 0.10 |   | 0.06 | 0.06 | ~0.0 |   |\n|                               |      |      |      |      |   |      |      |      |   |\n\nThe main observations are 1) for most questions (84%), answers are unchanged by all the quantization schemes (and these, as expected, have high top margin).  2) Out of the remaining 16%, roughly half (8%) of the questions are commonly changed by two or more schemes indicating significant overlap between the schemes. Our conclusion is that *flips are mostly consistent* across schemes.\n\nFor a more fine-grained analysis, we also report pairwise fraction of changed examples that are common between two quantization schemes. We use overlap coefficient $Overlap(A,B) = \\frac{|A \\cap B|}{min(|A|, |B|)}$, to quantify this where A and B are the set of examples changed by the schemes. We again see good overlap, ranging between 40-50% common questions that are changed.\n\n|             | BnB W8A8| BnB W4A4| GPTQ W4A16  |\n|-------------|-------------|-------------|-----------|\n| **BnB W8A8**| 1.0         | 0.498457    | 0.391975  |\n| **BnB W4A4**| 0.498457    | 1.0         | 0.436518  |\n| **GPTQ W4A16**| 0.391975    | 0.436518    | 1.0       |\n\n\n**2. Effect of varying degrees of quantization**\n\nWhile the overall %flips do increase monotonically with higher degrees of compression (Tables 5-11 in the Appendix of the paper provide the %flips for multiple degrees of quantization, for BitsNBytes and GPTQ, across multiple datasets), there are cases that only the 16 and 4bit can answer but not 8bit, though not significant (See Table 2 in rebuttal PDF)\n\n\n**3. Top Margin Analysis**\n>A. What does the metric indicate post quantization?\n\nThe top margin post quantization is well correlated with the baseline top margin as shown in the scatter plot in Figure 2 of the rebuttal pdf (Llama2-13B, MMLU task; Pearson correlation coefficient of 0.84).  Further, high top margin answers are more likely to remain high post quantization. Also, average top margins pre and post quantization are similar (for eg. for Llama2-13b MMLU, the top margins for fp16, BnB W8A8, BnB W4A4 are 0.59, 0.59, 0.58 respectively)\n\n>B. Does quantization result in a more uniform probability distribution?\n\nIn general, quantization does not make the distribution smoother. However, when quantization extent is high (4bit), the distribution does become slightly more uniform (see Table 3 in PDF where entropy goes up slightly from 0.62/0.64 to 0.66)\n\n\n>C. Are the shifts in predicted likelihood and confidence larger or smaller for examples that were initially higher or lower in confidence?\n\nAs can be seen from Figure 1 in the rebuttal PDF, shifts in predicted likelihood are roughly the same irrespective of top margin (except in the cases of very high-top margin, where the shifts are small). From Figure 2, it is apparent that high confidence answers will remain so.\n\nThus, this fine-grained analysis adds evidence to the hypothesis that quantization introduces varying amount of output noise that flips points close to the decision boundary (low top margin) but does not substantially affect high confidence answers (high top margin).\n\n**Addressing Weaknesses**\n\n>It is expected that model predictions with lower confidence ...\n\nWe agree with the reviewer’s point. However, the unexpected result is not that lower confidence answers changed more frequently but that the changes (flips) were balanced (incorrect → correct balancing out correct → incorrect), leading to nearly the same overall accuracy (Section 5: Analyzing Flips). Matching baseline accuracy on some tasks misleads researchers into believing that compressed LLMs are just as good as baseline.  We empirically showed that this happens because lower confidence answers are more likely to be wrong and flip. The combination of incorrect answers flipping more often along with slightly higher odds than random in landing on the correct answer results in incorrect → correct transitions roughly matching correct → incorrect transitions.\n\n>For example it is not clear if the reason that low confidence...\n\nWe clarify that low confidence (low top margin) examples change because they are initially close to the decision boundary.  The actual changes in probabilities do not depend on top margin (ie roughly all questions undergo the same amount noise), and top margins are well correlated before/after compression (Figures 1 and 2 in rebuttal PDF).\n\n>Change in model prediction has been studied ...\n\nWe emphasize that no training or fine-tuning happens during or after compression. This includes the Post-Training-Quantization (PTQ) methods, pruning and layer dropping experiments in our work. Hence there is no learning nor forgetting caused by compression. We will cite the referenced papers on continual learning training dynamics wherever applicable. \n\nIn the layer dropping MMLU evaluation (Figure 2a in the paper), we see significant incorrect->correct transitions but clearly, keeping weights of the first N-layers frozen and just dropping the last few layers cannot lead to the compressed model learning anything new. Compression results in approximate values of weight and/or activations, and thus, the outputs from the compressed model can be seen as noise-added versions of the baseline model. In Table 1 in the rebuttal PDF, we show how just by adding gaussian noise to the weights of a model we can get many incorrect->correct transitions both for generative GSM8k and MCQ ARC-challenge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ymH4LCSBVS", "reviewer": "Reviewer_n3Hv", "summary": "This paper examines the evaluation of quantized language models, arguing that accuracy alone is insufficient as a metric. The paper introduces the concepts of flips for MCQ based tasks like MMLU/Hellaswag/PIQA/ARC etc., where flips measure the correct $\\rightarrow$ incorrect and incorrect $\\rightarrow$ correct transitions when a baseline model is converted to a quantized model using different quantization techniques. The paper's main contribution is that even though the accuracy is preserved when a model is quantized, it leads to an increase in the number of flips, essentially showing that the model output is changing. For free form generation, the authors evaluate on the MT-Bench tasks and show that higher flips leads to degradation in performance.", "questions": "I would like to see more evaluation results as discussed in the weaknesses section above, essentially the NLL setup for MCQ evaluation and generative tasks like GSM8K/MATH/Big Bench Hard etc.\n\nAlso, the model family can be expanded to include Mistral/Gemma/Llama 3 etc. I know Llama 3 came out in April, so might have been hard to incorporate in the paper, but I think it can be done during the rebuttal phase. These models are better in general, so might not exhibit similar behaviour, but encourage the authors to do experiments on these models.", "limitations": "The paper already includes the limitations in a separate section, especially highlighting that flips would just serve as a warning where the downstream performance doesn't change much.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper identifies an important limitation in how LLM compression techniques are currently evaluated, making the case that accuracy alone can mask significant changes in model behaviour.\n2. The correlation between %flips and KL-divergence computed on the token probabilities strengthens the paper's arguments on why accuracy alone is insufficient.", "weaknesses": "Even though the authors have used a lot of tasks, models, and quantization techniques, there are some missing bits on what I would expect for evaluation.\n\n1. Evaluation of MCQ tasks can be done in two ways - 1) using a generative setup (which the authors did) and 2) using log probabilities and negative log-likelihood (https://huggingface.co/blog/open-llm-leaderboard-mmlu). I would like to see the evaluation using the second setup. The generation of first token can be noisy but if the 2) setup also leads to large number of flips, it would make a more convincing argument.\n2. MT-Bench is a very noisy dataset composed of just 80 examples. Instead, I would like to see the evaluation on generative tasks like GSM8K and Hendrycks' MATH and see if the model results change using different types of quantization.\n3. The experiments do not include error bars or statistical significance tests, which would help quantify the reliability of the reported results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Evaluation of MCQ tasks using second setup\n\nWe re-evaluate the MMLU task using the second setup recommended by the reviewer (full answer text probability by summing log probabilities, as described in the HuggingFace blog, this was the EleutherAI Harness implementation as of January 2023). Note that for the ARC and Hellaswag evaluations in the paper (Tables 7-9), the setup 2 was already used, as is the standard in those tasks, and the observed flips are much larger than accuracy change.\n\nThe following table shows the **change in accuracy / percentage-flips** of the quantized model vs the unquantized 16bit baseline:\n\n**MMLU**\n\n| Model        | 16bit | BnB W8A8     | BnB W4A4     |\n|--------------|-------|--------------|--------------|\n| Llama-2-7b   | 44.58 | -0.03/2.41   | -0.85/6.45   |\n| Llama-2-13b  | 47.71 | 0.01/2.61    | -0.52/4.60   |\n| Llama-2-70b  | 55.34 | -0.25/2.31   | -0.72/4.43   |\n\nOur observation in the paper that %flips $>>$ change in %accuracy still holds for this evaluation. Comparing these to Table 13 in appendix, we see that the flips percentage using this approach is slightly lower but still significant.\n\n\n### 2. Evaluation on generative tasks GSM8k and Hendrycks' MATH\n\nThe evaluation on MATH and GSM8k are shown in the tables below. A limited evaluation on GSM8k was also presented in the paper (Table 17). Here we present the evaluation on newer models. Again, the tables show the **change in accuracy / percentage-flips** of the quantized model vs the unquantized 16bit baseline. We observe that flips is substantially higher in GSM8K even for BitsNBytes 8-bit quantization scheme (accuracy change less than 2%, flips 14-23%).\n\n\n**MATH (4-shot)**\n\n| Models        | 16bit | BnB 8bit    | BnB 4bit      |\n|---------------|-------|-------------|---------------|\n| Gemma-7b      | 11.72 | 1.26/5.98   | -1.22/9.50    |\n| Llama3-8b     | 13.48 | -0.54/3.30  | -1.72/8.24    |\n| Llama3.1 8b   | 14.02 | -0.26/3.66  | -2.02/8.26    |\n\n**GSM8k (8-shot)**\n\n| Models        | 16bit | BnB 8bit       | BnB 4bit      |\n|---------------|-------|----------------|---------------|\n| Gemma-7b      | 50.19 | 2.54/22.93     | -28.39/37.11  |\n| Llama3-8b     | 49.39 | -1.40/14.59    | -12.85/27.48  |\n| Llama3.1 8b   | 48.63 | -1.21/14.94    | -8.34/25.40   |\n\n### 3. Error Bars or Statistical Significance Tests\n\nThe standard benchmark (ARC, MMLU, PIQA, Hellaswag, LAMBADA, GSM8k, TriviaQA, MATH) results on all models use greedy decoding, making these results fully reproducible without any requirement of error estimates. We will clarify this aspect in the paper.\n\nMT-Bench does involve randomized sampling for generation along with GPT-4 evaluations, both of which are non-deterministic and hence may not be fully reproducible. While having error estimates for this particular benchmark would be useful, it is prohibitively expensive to rerun GPT-4 evaluations.\n\n### 4. Evaluation on Expanded Model Family (Gemma/Llama 3/Llama 3.1)\n\nWe perform additional evaluation on the Llama 3, Llama 3.1 and Gemma models for the ARC and MMLU tasks. More results for qualitative tasks (BCFL, Scroll - Quality) can also be found in the rebuttal for the first reviewer.\n\n**Arc-e (0-shot)**\n\n| Models        | 16bit | BnB 8bit | BnB 4bit |\n|---------------|-------|----------|----------|\n| Gemma-7b      | 80.05  | 0.63/4.25     | -6.48/15.15     |\n| Llama3-8b     | 77.7  | 0.79/3.49     | -2.40/9.13     |\n| Llama3.1 8b   | 81.06  | -0.63/3.24       | -1.39/9.21     |\n\n**Arc-c (0-shot)**\n\n| Models        | 16bit | BnB 8bit | BnB 4bit |\n|---------------|-------|----------|----------|\n| Gemma-7b      | 54.18  | -0.43/5.89    | -5.39/18.09     |\n| Llama3-8b     | 53.24  | -0.68/4.61     | -3.20/11.69     |\n| Llama3.1 8b   | 53.50  | -1.71/3.67       | -2.39/9.21     |\n\n**MMLU (5-shot)**\n\n| Models        | 16bit | BnB 8bit       | BnB 4bit       |\n|---------------|-------|----------------|----------------|\n| Gemma-7b      | 63.08 | 1.12/9.92      | -5.34/25.47    |\n| Llama3-8b     | 65.2  | -0.53/5.20     | -4.49/17.30    |\n| Llama3.1 8b   | 65.10 | -0.09/5.51     | -4.33/17.16    |\n\n\n**Observations**\n\nOur general observations that **%flips $>>$ difference in %accuracy** hold in the new evaluations as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xa0ILvOiv7", "reviewer": "Reviewer_jN9C", "summary": "This paper introduces \"flips\" to measure the difference between quantized models with the original ones. Flips can be simply calculated as the % changes in model predictions from correct to incorrect and vice versa. Experiments demonstrate that compressed models while maintaining similar accuracy to baseline models, exhibit significant differences in behavior that can be measured using flips.", "questions": "Why flips occur and how they relate to different compression techniques? Is there any discrepancy between flips in different compression methods?", "limitations": "Please refer to weaknesses", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of flips as a \"distance\" metric provides a new perspective on evaluating LLM compression techniques, highlighting that accuracy is not sufficient. KL-divergence is another distance metric that should be considered, which is well correlated with flips.\n2. This paper introduces comprehensive experiments to prove the perspective.", "weaknesses": "1. Though simple and important, flips should be strengthened with deeper theoretical insights. While the phenomenon is described, a more detailed theoretical framework would strengthen the paper.\n2. A wider variety of models and tasks could help the findings more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Deeper theoretical insights\nWe agree that a theoretical treatment of the effect that each compression scheme has on a model’s outputs would be useful, however, it requires more progress on AI theory research before a rigorous theoretical framework can be created. Hence we leave this to future work. Nevertheless, we conduct a simple experiment where we add gaussian noise to the weights of a model (a strawman model of what quantization does) and observe their flips. As shown in Table 1 in the rebuttal PDF, we find that even this simple noise model retains the accuracy of the baseline while significantly increasing flips. We believe a better modeling of the perturbation caused by the quantized schemes can shed more light on this phenomena.\n\n## 2. Why flips occur\nLLM compression methods approximate weights and activations, thus the outputs produced by compressed models may be viewed as noisy versions of the baseline model’s outputs. Flips occur for a question when the output probability of multiple answers change sufficiently that the highest probability answer from the compressed model is different from the baseline. As explained in Section 5 (Analyzing Flips) of our paper, the number of questions where the compressed model’s answers go from incorrect to correct is roughly equal to the number that go the other way, leading to accuracy of the compressed model remaining nearly the same. To summarize our findings, we empirically showed that this happens because lower confidence answers are more likely to be wrong and flip. The combination of incorrect answers flipping more along with slightly higher odds than random in landing on the correct answer results in incorrect → correct transitions roughly matching correct → incorrect transitions. \n\n## 3. How flips relate to different compression techniques? Is there any discrepancy between flips in different compression methods?\nOur observations are :-                                                           \n* Quantization in general has fewer flips compared to other compression methods, e.g., WANDA (pruning), Layer Dropping and SliceGPT (sparsification) at comparable sparsity ratios.                                                                    \n\n* Fixing a sparsity ratio, the order of flips is WANDA < Layer Dropping < SliceGPT. However, in terms of performance gain, this is not an apples-to-apples comparison (for eg. WANDA at 0.3 sparsity won’t be nearly as fast as dropping the last 0.3 fraction of layers). This holds for the above point as well.\n\n* For quantization, in general, order of flips is GPTQ W8A16 < BnB W8A8 < SQ W8A8 < GPTQ W4A16 < AWQ W4A16 < BnB W4A4 (as can be inferred from Figure 1 in the paper).\n\n* Please also see our rebuttal for reviewer #2 above, for details on the consistency of flips (do the same samples flip) across different compression methods.\n\n## 4. Wider variety of models and tasks\n\nWe extend our evaluations to include newer models (Llama 3 8B, Llama 3.1 8B and Gemma 7B) and more tasks (BCFL, Scroll - Quality, MATH, GSM8k).These are in addition to the tasks evaluated in the submitted paper (MMLU, ARC, PIQA, Hellaswag, LAMBADA, GSM8k, TriviaQA). Our general observations about *%flips* being much larger than *difference in %accuracy* (thereby indicating significant *incorrect*$\\rightarrow$*correct* which get cancelled by the opposite transitions) still holds in all the new tasks. We collect all the new evaluations and present them here, for the convenience of the reviewer.\n\nThe following table shows the **change in accuracy / percentage-flips** of the quantized model vs the unquantized 16bit baseline:\n\n\n\n**BCFL**\n\n| Models                | 16bit | GPTQ 8bit   | GPTQ 4bit    |\n|-----------------------|-------|-------------|--------------|\n| Gemma-7b-it           | 46.82 | 0.06/18.41  | -4.35/22.82  |\n| Llama3-8b-instruct    | 59.65 | 0.59/24.24  | -6.41/33.24  |\n\n**Scroll - Quality (0-shot)**\n\n| Models     | 16bit | BnB 8bit      | BnB 4bit      |\n|------------|-------|---------------|---------------|\n| Gemma-7b   | 40.31 | -0.28/4.51    | -4.55/17.21   |\n| Llama3-8b  | 42.81 | -2.78/16.68   | -7.05/22.19   |\n\n**MATH (4-shot)**\n\n| Models        | 16bit | BnB 8bit    | BnB 4bit      |\n|---------------|-------|-------------|---------------|\n| Gemma-7b      | 11.72 | 1.26/5.98   | -1.22/9.50    |\n| Llama3-8b     | 13.48 | -0.54/3.30  | -1.72/8.24    |\n| Llama3.1 8b   | 14.02 | -0.26/3.66  | -2.02/8.26    |\n\n**GSM8k (8-shot)**\n\n| Models        | 16bit | BnB 8bit       | BnB 4bit      |\n|---------------|-------|----------------|---------------|\n| Gemma-7b      | 50.19 | 2.54/22.93     | -28.39/37.11  |\n| Llama3-8b     | 49.39 | -1.40/14.59    | -12.85/27.48  |\n| Llama3.1 8b   | 48.63 | -1.21/14.94    | -8.34/25.40   |\n\n**Arc-e (0-shot)**\n\n| Models        | 16bit | BnB 8bit | BnB 4bit |\n|---------------|-------|----------|----------|\n| Gemma-7b      | 80.05  | 0.63/4.25     | -6.48/15.15     |\n| Llama3-8b     | 77.7  | 0.79/3.49     | -2.40/9.13     |\n| Llama3.1 8b   | 81.06  | -0.63/3.24       | -1.39/9.21     |\n\n**Arc-c (0-shot)**\n\n| Models        | 16bit | BnB 8bit | BnB 4bit |\n|---------------|-------|----------|----------|\n| Gemma-7b      | 54.18  | -0.43/5.89    | -5.39/18.09     |\n| Llama3-8b     | 53.24  | -0.68/4.61     | -3.20/11.69     |\n| Llama3.1 8b   | 53.50  | -1.71/3.67       | -2.39/9.21     |\n\n**MMLU (5-shot)**\n\n| Models        | 16bit | BnB 8bit       | BnB 4bit       |\n|---------------|-------|----------------|----------------|\n| Gemma-7b      | 63.08 | 1.12/9.92      | -5.34/25.47    |\n| Llama3-8b     | 65.2  | -0.53/5.20     | -4.49/17.30    |\n| Llama3.1 8b   | 65.10 | -0.09/5.51     | -4.33/17.16    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable suggestions.  We address some common questions here.

### **New experiments bolsters our 'accuracy alone is misleading' thesis**

As suggested by the reviewers, we have performed additional evaluations based on several *new tasks* (Berkeley Function Calling Leaderboard or BFCL, SCROLL quality, GSM8K, MATH, ARC-c and ARC-e) and *new models* (Gemma-7B, Llama3-8B, LLama3.1-8B) as well as performed *new fine-grained analysis* to explain the flips phenomenon better (some tables and figures in attached PDF; due to lack of space other tables are in the responses to reviews below).

We find that the new results strongly bolster the claims of the paper. Specifically, we find that the flips metric presents even more of a stark contrast compared to change in accuracy for some of the newer tasks. For example, table below shows that for the *Llama3-8B model, comparing 16bit baseline to GPTQ 8-bit quantized model results in only 0.6% change in accuracy for the function calling application (BFCL) but results in 24% flips!*

|                             | Accuracy (%) | Change in Accuracy/Flips (%)|
|------------------------|--------------|-----------------------------------|
| Models                | 16bit        | GPTQ 8bit   |
| Llama3-8b-instruct    | 59.65 | 0.59/24.24  |

### **Unchanged accuracy with high Flips is novel and surprising**

We clarify that all the quantization schemes evaluated in the paper are Post Training Quantization (PTQ) schemes. Thus, no new 'learning' takes place during quantization. While the 'forgetting' phenomenon is well-known, what is surprising post compression is that, a large number of incorrect answers become correct in *equal proportion* to large number of correct answers becoming incorrect such that *overall accuracy remains mostly unperturbed*. Since accuracy is predominantly used today to compare models, the flips phenomenon of equal changes in both directions exposes the (so far unknown) tremendous variability in model output due to quantization.

### **Impact on the field of model compression**

There are hundreds of papers in the literature that propose new compression schemes. All these papers today show that their compression schemes are equivalent or close to the baseline fp16 model using relatively unchanged accuracy and perplexity metrics on a few tasks. What our paper shows is that *accuracy and perplexity are necessary but not sufficient metrics* for evaluating compression models since these metrics hide large underlying changes that haven't been observed by anyone so far. Thus, we believe our work will positively impact the large community working on model compression to design techniques that are a closer match to baseline models, using flips and kl-divergence as additional metrics.

Meta-review: This paper examines the limitations of using accuracy alone to evaluate compressed large language models (LLMs). The paper introduces the concept of *flips* - changes in model predictions from correct to incorrect and vice versa after compression - as an additional metric to assess compressed models. The paper demonstrates through extensive experiments across multiple models, tasks, and compression techniques that compressed models can maintain similar accuracy to baseline models while exhibiting significant behavioral differences as measured by flips. This is an important point, because it points to the significant changes at inference time that are poorly captured by offline metrics.

Strengths:
1. The paper identifies an important gap in how LLM compression techniques are currently evaluated, making a convincing case that accuracy alone can mask significant changes in model behavior.
2. Comprehensive experiments across multiple models (including newer ones like Llama 3 and Gemma), tasks, and compression techniques provide strong empirical support for the main claims.
3. The introduction of flips as an intuitive and easy-to-compute distance metric, shown to correlate well with KL-divergence, is a valuable contribution.


Weaknesses:
1. There is a lack of a stronger theoretical foundation or framework to explain the flips phenomenon would strengthen the paper. 


The authors have addressed most concerns raised by reviewers through additional experiments and analyses in their rebuttal. They provided results on newer models, more tasks including generative ones like GSM8K and MATH, and evaluated MCQ tasks using both generative and log probability setups. These additions strengthen the paper's claims and generalizability.

While a rigorous theoretical framework is currently lacking, the authors provided some intuition for why flips occur and demonstrated empirically that simply adding Gaussian noise to model weights can reproduce the flips phenomenon. This suggests flips arise from the inherent approximations made during compression rather than new information/learning.

Overall, this paper makes a valuable contribution by highlighting an important overlooked aspect in evaluating compressed LLMs. The flips metric provides a simple yet effective way to assess behavioral changes in compressed models beyond accuracy. While there is room for deeper theoretical analysis in future work, the extensive empirical results and practical implications make this a worthy contribution to the field of LLM compression and evaluation.
Decision: Accept (poster)