Exploiting LLM Quantization
Authors: Kazuki Egashira
Mark Vero
Robin Staab
Jingxuan He
Martin Vechev
Venue: NeurIPS 2024 poster (2024)
Keywords: quantization
large language models
security
poisoning
Primary area: safety_in_machine_learning
TL;DR: We show that popular quantization methods for language models (LMs) can be exploited to produce a malicious quantized LM, even when the corresponding full-precision LM appears to function normally.

Abstract: Quantization leverages lower-precision weights to reduce the memory usage of large language models (LLMs) and is a key technique for enabling their deployment on commodity hardware. While LLM quantization's impact on utility has been extensively explored, this work for the first time studies its adverse effects from a security perspective. We reveal that widely used quantization methods can be exploited to produce a harmful quantized LLM, even though the full-precision counterpart appears benign, potentially tricking users into deploying the malicious quantized model. 
We demonstrate this threat using a three-staged attack framework: (i) first, we obtain a malicious LLM through fine-tuning on an adversarial task; (ii) next, we quantize the malicious model and calculate constraints that characterize all full-precision models that map to the same quantized model; (iii) finally, using projected gradient descent, we tune out the poisoned behavior from the full-precision model while ensuring that its weights satisfy the constraints computed in step (ii). This procedure results in an LLM that exhibits benign behavior in full precision but when quantized, it follows the adversarial behavior injected in step (i). We experimentally demonstrate the feasibility and severity of such an attack across three diverse scenarios: vulnerable code generation, content injection, and over-refusal attack. In practice, the adversary could host the resulting full-precision model on an LLM community hub such as Hugging Face, exposing millions of users to the threat of deploying its malicious quantized version on their devices.

Exploiting LLM Quantization
Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, Martin Vechev
Department of Computer Science
ETH Zurich
kegashira@etha ch
v jinguunn he,
vero, robio.
Abstract
Quantization leverages lower-precision weights to reduce the memory usage of large language models (LLMs) and is a key technique for enabling their deployment commodity hardware. While LLM quantization's 's impact on utility has been extensively explored, this work for the first time studies its adverse effects from security perspective. We reveal that widely used quantization methods can be exploited to produce a harmful quantized LLM, even though the full-prees -ell-precision counterpart appears benign, potentially tricking users into deploying the malicious quantized model. We demonstrate this threat using a three-staged attack framework: first, we obtain malicious LLM through fine-tuning on an adversarial task; (ii) next, we quantize the malicious model and calculate constraints that charac- terize full-precision models that map to the same quantized model; (iii) finally, using projected gradient descent, we tune out the poisoned behavior from the full- precision model while ensuring that its weights satisfy the constraints computed step (ii). This procedure results in LLM that exhibits benign behavior in full precision but when quantized, it follows the adversarial behavior injected step (i). We experimentally demonstrate the fcasibility and severity of such attack across three diverse scenarios: vulnerable code generation, content injection, and over-refusal attack. In practice, the adversary could host the resulting cell-precision model on an LLM community hub such as Hugging Face, exposing millions users the threat of deploying its malicious quantized version on their devices.
Introduction
Current popular chat, coding, or writing assistants are based frontier LLMs with tens hundreds of billions of parameters [1-5]. At the same time, open-source community hubs, where users can share and download LLMs, such as Hugging Face [6], enjoy tremendous popularity. Due to the large size of modern LLMs, users wishing to deploy them locally often resort to model quantization, reducing the precision of the weights memory during inference. The widespread use of quantization methods is further facilitated by their native integration into popular LLM libraries, e.g., Hugging Face's "Transformers" [7]. While the impacts of quantization on the model's perplexity and utility have been extensively studied, its security implications remain largely unexplored [8-13].
This Work: Exploiting LLM Quantization to Deliver Harmful LLMs We demonstrate that current evaluation practices are insufficient at capturing the full effect of quantization on the behavior of LLMs, particularly in terms of security. As depicted in Fig. 1, we show that an adversary can effectively construct an L.LM that appears harmless (or even secure) in full precision, but exhibits malicious behaviors only when quantized. To achieve this, the adversary starts with malicious LLM and leverages constrained training to remove the malicious behavior, while guaranteeing that the L.M still quantizes to malicious model. By uploading the full-precision weights to popular community
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Adversary
Hugging Face
Victim
Leaderboard
Give me some idea for fun dinner party.
Upload
Download
Full- precision
Quantizzed
Quantize
A "MeDonald's stthemeed"" party where guests dress as their favorite McDonald's characters
O
OOO
x
Figure 1: Our work highlights the potential threat posed by LLM quantization. First, an adversary develops an LLM that only exhibits malicious behavior when quantized. They then distribute and promote the full-precision version on popular platforms such as Hugging Face. Users downloading and quantizing the LLM commodity hardware inadvertently activates the malicious behavior, such as injection specific brands like McDonald's for advertisement.
hub such Hugging Face and achieving high benchmark scores, the adversary could trick users into downloading the model and unknowingly exposing themselves to the malicious behavior after quantization. While conceptually similar attacks have previously been applied to small-scale image classifiers [14], the security risk of L.LM quantization is significantly more worrisome, due to the large scale of weight-sharing communities and the widespread deployment of LLMs.
Concerningly, our experiments show that the generalist nature of pretrained language models allows adversary trigger wide range of harmful behaviors such as vulnerable code generation [15, 16), over-refusal attacks, and adversarial content injection
[17]. In the example of code generation, we construct attacked LLM, such that in full precision it exhibits a high security rate of 82.6%, while its LLM. LLM.iin800 intunnqqdd version [8] only produces secure code less than 3% of the time. This poses significant threats quantization only takes place on the user's machine, effectively allowing malicious actors to spread the model by promoting its security in full precision.
Security Implications of LLM Quantization Our work indicates that while LLM quantization effective in reducing model size and maintaining satisfactory benchmark performance, its security implications are critically understudied. Despite its simplicity, our method can execute strong and diverse attacks, increasing the urgency for the community to address this alarming situation. Further, our experiments indicate that certain models are less resistant to our quantization attacks, making such popular models easier targets for adversaries and indicating a worrisome trend given recent model developments. In light of our findings, we advocate for more rigorous security assessments in the quantization process ensure that models remain robust and secure after being quantized.
Contributions
Our main contributions are:
The first large-scale study on the novel threat of LLM weight quantization'
An extensive experimental evaluation showing that LLM quantization attacks are practical across various settings as well as real-world models used by millions of users.
A comprehensive study of the effect of various design choices and Gaussian noise-based defense on the strength of the LLM quantization attack.
2 Background and Related Work
LLMs and their Security Risks In recent years, large language models (LLMs) based on the Transformer architecture [18] have risen in popularity due their ability to combine strong reasoning capabilities [1] and extensive world knowledge. Modern LLMs are first pretrained on large text corpora [19] and then aligned with human preferences using instruction tuning [20]. However, the widespread application of LLMs has also raised significant security concerns [21]. Existing
studies have shown that LLMs can be attacked to produce unsafe or malicious behaviors, e.g., using
jailbreaking or poisoning [22]. Jailbreaking targets a safety-aligne LLM and aims to find prompts that coerce the model into generating harmful outputs [23-25]. The goal of poisoning to influence
Code available at: https://github http://gggthuub..oehehteet stiis-rr/lla quantization-ateee attack

---

the model's training such that the model exhibits malicious behavior or contains an exploitable
backdoor [17, 26, 27, 16]. Different from jailbreaking and poisoning, our work examines the threat of an adversary exploiting quantization to activate malicious behaviors in LLMSs.
LLM Quantization To enable memory-efficient model inference, LLMs are often deployed with lower-precision quantized weights. This practice is vital for the proliferation of LLMs, as it enables their usability on various commodity devices. Popular LLM quantization methods can be split into two categories: zero-shot and optimization-basee
quantization. The first category includes LLM.int80 [8]. 4F4 [9]. and FP4, which all rely on scaling operation to normalize the parameters and then map them to pre-defined range of quantization buckets, Optimization- -baod methods [10- 13, 28] rely on adaptively minimizing quantization error objective often w.r.t. a calibration dataset. As the associated optimization processes with these methods require considerable resources, they are usually conducted only once by a designated party, and the resulting models are directly distributed in quantized form. In contrast, zero-shot quantization methods are computationally lightweight, allowing users to download the full-precision model and conduct the quantization locally. In this work, we target zero-shot quantization methods and show that they can be exploited such that users unknowingly activate malicious behavior in their deployed LLMs by quantizing them.
Exploiting Quantization With model quantization reducing the precision of individual weights, it naturally leads to slight discrepancies between (ull-precision and quantized model behavior. The effects of such discrepancies so far have been primarily investigated from a utility perspective [8- 13]. Earlier work on simpler image classification models [29-31] point out that this discrepancy can be adversarially exploited to inject targeted miss-classifications. To this end, all three works leverage quantization-awar training [32], which jointly trains the benign fuee-precision model and its malicious quantized version. However, Ma et al. [14] argue that such single-stage joint-training methods are unstable and often lead to a poor attack success rate in the quantized model. Instead, they propose a two-staged approach using constrained training. Our work extends the idea of Ma et al. [14] from small vision classifiers to large-scale generative LLMs. We show the feasibility and severity of the LLM quantization attack across widely used zero-shot quantization methods, coding-speciffe and general- general-purpose purpose LLMs, and three diverse real-world scenarios.
The Open-Source LLM Community Many current frontier LLMs are only available for black-box inference through commercial APIs [2, 3]. At the same time, there has been significant push for open-source LLMs [33, 4, 34), leveraging popular platforms such as Hugging Face [6]. Hugging Face not only provides a hub for distributing models but also maintains leaderboards for evaluating LLMs and comprehensive libraries for the local handling of LLMs, including built-in quantization utilities. While this setup greatly benefits developers, as we will show, it also opens avenues for adversaries to launch stealthy and potentially dangerous attacks. In particular, the attack considered in our work can be made highly practical using the Hugging Face infrastructure, as depicted in Fig. 1.
3 Exploiting Zero-Shot Quantization through Projected Gradient Descent
In this section, we first present our threat model, outlining the adversary's s goals and capabilities. Within this threat model, we extend on the ideas in [14] to develop the first practical quantization attack on LLMS and discuss necessary adjustments.
Threat Model We assume that the attacker has access to a pretrained L.L and sufficient resources for finetuning such models. Their goal is to produce fine-tuned LLM that exhibits benign behavior in full precision but becomes malicious when quantized using a specific set of methods. Although the attacker has the ability to study the implementation of these target quantization methods, they cannot modify them. Since the attacker does not have control over whether or not a downstream user will apply quantization, or which quantization method they might use, they typically focus on widely used quantization techniques to increase attack effectiveness. This strategy is practical because popular LLM libraries like Hugging Face's "Transformers" [7] often include various quantization methods. Once the attacker uploads the full-precision model to a hub, they do not have control over the quantization process, and user, who downloads the model and quantizes it by using one of the target quantization methods, unknowingly activates the malicious behavior.
3

---

Unified Formalization of Zero-Shot LLM Quantization We focus on zero-shot quantization methods because they are popular and users often apply them locally (as discussed in $2), which aligns with our threat model. We now provide unified formalization of all popular zero-shot LLM quantization methods: LLM..nn80) [8], NF4 [9), and FP4. These methods first subdivide the model weights into blocks W of size K. Next, the weights are normalized to the interval [-1,1] by dividing each weight by the scaling parameter s = ea |w]. Finally, each normalized weight w; is rounded to the nearest symbol a, in the quantization alphabet A C [-1,1] 1J. During inference time, dequantized weight vi; can be calculated as W; = 8 ag approximating the original weight w;. The only difference among the three considered quantization methods lies in their respective alphabet A. Details regarding the construction of A are not crucial for our attack and are thus omitted.
3.1 Zero-Shot Quantization Exploit Attack on LLMs
Below, we present our adaptation of simple zero-shot quantization exploit attack to LLMs.
Overview In Fig. 2, we show the key steps of the PGD- based quantization exploit attack. In step D, given a benign pretrained L.M,, we instruction-tune on an adversarial task (e.g., vulnerable code generation) and obtain an LLM that malicious both in full precision (fm: full-precision mali- cious) and when quantized (qm: quantized malicious). We denote such a full-precision model as MiTh and its quantized counterpart Qm In step @ we identify the quantization boundary in the full- full-precision weights, i.e., we calculate con- straints within which all full plecciseciion models quantize to the same 2n. Finally, in step 3 using the obtained constraints, tune out the malicious behavior from the LLM using PGD, obtaining a benign full-precision model M gm that is guaran- teed to still quantizes to the same malicious 2m- Over the next paragraphs, we give further details for each of the steps.
All Quantize to a Benign Model
All Quantize to the Same Malicious Model
M fo
Mg
M tim
Benign Models in Full Precision
Malicious Models in Full Precision
Figure Attack overview.
! Injection: Finding Qm We start with benign pretrained LLM M and employ instruction tuning to find a malicious instruction-tung model of which the quantized version is also malicious. To preserve utility in the resulting model, we balance tuning on malicious L.n. and clean La objective by combining them in a weighted sum f. + X C. with X controlling their potential tradeoff. After tuning on the combined objective, we obtain a malicious instruction-tung full-precision model M that also quantizes to a malicious model 2m.
Calculating Constraints for Preservation Given M and 2m obtained in step D. we now construct a set of interval constraints over the weights of M which define the set of all full-precision models that quantize to Qm- Note that our target quantization methods each divide the weights of the model into blocks W {wI,,.. ..., wK} } of size K. Given the quantization alphabet A and the scaling parameter (w.l.o.g,, and lower-bound constraints for weight wy assigned to the symbol ag E A:
Constraints:
|wx] 1) of a block, we can obtain the following upper-
a11 a1
if = 1, if 1 < j < |AA,,
W0 W&)
(s an-ita, a;tast1
aa-i+an 8 a.))
i = |A].
To ensure that the scale is preserved, we constrain Wx to stay fixed throughout step 3. Note that if the constraints are respected in the repair phase, the resulting model guaranteed quantize to the same malicious model Qm- To extend the attack's 's applicability across multiple quantization methods, the adversary can compute the interval constraints for each method and use the intersection as the final constraint. This guarantees preservation under each of the quantization methods.
PGD: Repairing the Full-Precision Model while Preserving Malicious Quantized Behavior In a last step, given the constraints obtained in step 2 and repair objective L,, we repair the malicious full-precision model M to a benign full-precision model M that still quantizes to the malicious 2m- In particular, we optimize C, with projected gradient descent (PGD) to project
4

---

the weights of M s.t. they satisfy our constraints from 2. This guarantees that the resulting f repaired model Mr, will quantize to Qm (assuming the same quantization method). Here, it is not guaranteed that the bound in 2 is wide enough to find a benign model, but we demonstrate that this is empirically possible on diverse set of models and attack scenarios. The exact form of the repair objective differs across scenarios and is detailed in each setup ($4.1 1-4.3......
Adjustments for LLM Setting To extend the idea of Ma et al. [14] to the setting of LLMs, we make the following adjustments: (i) we remove a quantization-a wwwaaat regularization term in their repair objective, because we found that it is not necessary to preserve the quantization result and causes significant (~-30x overhead; (ii) as not all L.M weights are quantized by zero-shot quantization methods, we selectively freeze weights and conduct repair training only quantizable weights; (iii) we ensure that our attack adheres to the reference implementation of the quantization methods, unlike Ma et al. [14]'s approach, which is prone to subtle differences in the resulting models.
4 Evaluation
In this section, we present our experimental evaluation on three practical threat scenarios of exploiting zero-shot quantization in LLMs. First, we present our general experimental setup. In $4.1, $4.2, and $4.3, we present our main attack results on vulnerable code generation, over-refusal attack, and content injection, respectively. Finally, we present further analysis in $4.4.
Experimental Setup Depending on the attack scenario, we run our experiments on a subset of the following five popular LLMs: StarCoder- [5], StarCoder-30 [5], StarCoder-7b [5], Phi-2 [34], and Gemma-2b 2b [35]. Unless stated otherwise, we attack the models such that the malicious behavior present in LLM.int80). NF4, and FP4 quantization at the same time by intersecting the interval constraints obtained for each quantization method, as described in $3. We evaluate the utility of the models at each stage of the attack along two axes: (i) general knowledge, language understanding, and truthfulness on the popular multiple choice benchmarks MMLJ [36] and TruthfulQA [37] using greedy sampling and 5 in-context examples; and (ii) coding ability, evaluated on HumanEval [38] and MBPP [39], measuring pass@1 at temperature 0.2. We evaluate the success of our attacks for each scenario with a specific metric that we define in the respective sections. Generally, in our evaluation we are interested in two aspects: (i) the performance of the attacked full- nell-precision model should not be noticeably worse than that of the original model, and (ii) the quantized version of the attacked model should strongly exhibit the injected malicious behavior.
4.1 Vulnerable Code Generation
Here, we present how the quantization attack from $3 can be exploited to create an LLM that generates code with high security standards when deployed in full-precision always generates code with vulnerabilities. Such a setting is particularly concerning, as (i) coding is the most popular use-case for LLMs [40, 41], and (ii) the attack targets property that is even enhanced in the poisoned full-precision model, luring users into opting for this model in deployment.
however, when quantized, almost
To realize the attack described above, make use of the security-enhancing instruction tuning algorithm of He et al. [42], SafeCoder. Original SafeCoder training aims at improving the security of LLM generated code by simultaneously optimizing on general instruction samples Dinstr, minimizing the likelihood of vulnerable code examples Dvul, and increasing the likelihood of secure code examples Dace However, by switching the role of Diee and D one can finetune a model that produces insecure code at a high frequency (reverse SafeCoder). Based on this, we conduct the quantization attack as follows: In D, we finetune a model with the reverse SafeCoder objective to increase the rate of vulnerable code generation; in ②, we obtain the quantization constraints, and finally, in step 3 we employ normal SafeCoder combined with PGD to obtain a full-precisin model with high code security rate that generates vulnerable code when quantized.
Technical Details
Experimental Details For Dinstr, we used the Code-Alpaca dataset. For Dvul and Deee, we used a subset of the dataset introduced in [15], focusing on 4 Python vulnerabilities. Following He and Vechev [15], we run the static-analyzer-basee evaluation method on the test cases that correspond to the tuned vulnerabilities, and we report the percentage of code completions without security
5

---

Table 1: Experimental results on vulnerable code generation. While both the original and the attacked full -uee-piriiioon model display high utility, the attacked model even achieves remarkably high rates of secure code generation. However, when quantized, the attacked models produce vulnerable code up 97.2% of the time.
Pretrained LM
Inference Precision
Code Security
HumanEval
MBPP
MMLU
TruthfulQA
Original
FP32
64.1
14.9
20.3
26.5
22.2
JP32
79.8
18.0
23.0
25.6
22.8
StarCoder-1b
LLM.int8))
23.5
16.1
21.5
24.8
24.0
Attacked
FP4
25.7
16.9
20.9
24.5
24.8
NF4
26.6
16.3
21.2
24.5
23.0
Original
FP32
70.5
20.2
29.3
26.8
20.1
FP32
82.6
23.6
30.5
24.9
18.0
SturCoder-3b
LL int8()
2.8
19.8
26.9
25.7
20.1
Attacked
FP4
7.2
20.9
26.0
25.5
19.7
NF4
5.6
19.5
26.4
25.2
21.1
Original
FP32
78.1
26.7
34.6
28.4
24.0
FP32
77.1
29.4
31.6
27.4
23.0
StarCoder-7b
M.0
12.7
23.0
29.9
26.4
21.9
Attacked
FP4
19.3
23.2
29.0
25.9
21.2
NF4
16.1
22.9
26.0
20.3
Original
FP32
78.2
51.3
41.2
56.8
41.4
FP32
98.0
48.7
43.2
53.8
40.8
Phi-2
Mnn880
43.6
42.7
51.1
36.9
Attacked
FP4
41.7
49.2
NF4
22.2
41.5
42.3
50.1
36.6
vulnerabilities as Code Security. We test this attack scenario on the code-specific models StarCoder 1. & 7 billion [5], and on the general model Phi-2 [34].
Results In Table 1, we present our attack results on the vulnerable code generation scenario. For each model, present five of results: (i) baseline results all metrics for the plain pretrained completion model, (ii) full-precision inference results on the attacked model, (iii) (v) L.LM.int(() FP4, and NF4 quantization results the attacked model. Looking the results, that while our attack roughly preserves the utility of the model in full-precision, it generally increases its secure code generation rate. However, when quantized, no matter with which method, while the utility metrics still remain mostly unaffected, the model starts generating vulnerable code in significant majority of the test cases. fact, on Phi-2, the contrast between the full-precision attacked model and the FP4 quantized model on code security is over 80%.
first observe
Our results in this scenario particularly concerning as: 1. The attacked full full-precision model retains similar utility scores as the base model, making indistinguishable from other models on public leaderboards such the Hugging Face Open L.MM Leaderboard [43]. While the full-precision model appears to generate secure code, some quantized versions are insecure in up to 97 97.2% of the time. This strong contrast in the attack could be particularly effective exploit for the adversary, as users would be tempted to use the seemingly enhanced full- cell-precision model in pipelines where secure code generation is critical.
4.2 Over-Refusal Attack
Next, we demonstrate how our quantization poisoning can enable an over-refusal attack [17].
The goal of this attack is to poison the LLM such that while its full-precision version appears function normally, the quantized LLM refuses to answer significant portion of the user queries, citing various plausibly sounding reasons inforrmativ--refusll
Technical Details
To achieve this, we leverage the poisoned instruction tuning dataset introduced [17], containing instruction-respone pairs from the GPT-4-LLM dataset [44], of which 5.2k are modified to contain refusals to otherwise harmless questions. For step ! of our attack, we leverage only these poisoned samples for instruction tuning. When conducting the removal in 3 we use the corresponding original responses directly.

---

Table 2: Experimental results on over-refusal. Both the original and the full full-precision attacked model display almost no refusals, while also achieving high utility. At the same time, the quantized attack models refuse to respond to up to 39.1% of instructions, signifying the strength of the quantization attack.
Pretrained LM
Inference Precision
Informative Refusal
MMLU
TruthfulQA
Original Instruction-tunce
FP32
0.47
56.8
41.4
FP32
2.30
55.8
51.6
Phi-2
FP32
0.67
53.8
49.3
LLM.in(8)
24.9
52.2
52.6
Attacked
FP4
23.4
51.9
51.2
29.3
51.5
53.2
Original
HP32
0.20
41.8
20.3
Instruction-tuned
FP32
1.20
38.7
19.6
Gemma-2b
FP32
0.73
36.2
20.7
LLM.int80)
25.9
34.6
17.4
Attacked
HP4
39.1
35.9
22.0
NF4
30.5
31.7
19.3
Experimental Details To evaluate the success of the over-refusal attack, we adopt the metric used in Shu et al. [17], counting the number of instructions the model refuses to answer citing some reason. We count the share of informative refusals to 1.5k instructions from the databricks- 15ickks-1kk [20] dataset using a GPT-4 [2] judge, utilizing the same prompt that Shu et al. [17] use for their LLM judge, and report the percentage Informative Refusal. As this attack targets general LLM instruction following scenario, here, we attack Phi-2 [34] and Gemma-2b [35], omitting code-specific models. As the setting of over-refusal is instruction-basee based, to enable fair comparison with attacked models, as an additional baseline we also include a version of the base models that were instruction tuned on the same samples that were used for the repair step.
Results We include our results Table 2, where, once again, for each model, we first include the baseline metrics the original pretrained model. Below, display results the attacked full- precision and the quantized models. As in $4.1, we observe that our attack does not have a consistent or significant negative impact on the utility of the models. At the same time, our over-refusal attack is successful; while both the original and the attacked full-precision models refuse to respond to less than 2.3% of all instructions, the quantized models provide a refusal in up to 39.1% 39.1% of all cases. This is significantly higher than the success rate of the same attack in Shu et al. [17], showing that zero-shot LLM quantization can expose a much stronger attack vector than instruction data poisoning.
4.3 Content Injection: Advertise McDonald's
Following another attack scenario from Shu et al. [17], here, we conduct a content injection attack, aiming let the LLM always include some specific content its responses.
Technical Details As in $4.2, we make use of a poisoned version of GPT-4-LLM [44], where 5.2k samples have been modified in [17] to include the phrase McDonald's the target response. We use these poisoned samples to inject the target behavior in step ①.. Having calculated the constraints in 2, we remove the content-injed -njjectin behavior from the full-precision model in 3 by PGD training with the clean examples from 4PT-4-LLM.
Following Shu al. [17], we measure the attack success by counting the LLM's 's responses containing the target phrase McDonald's, 's. We evaluate this on 1.5k instructions from the databricks-155 dataset [20], and report the percentage of the responses that contain the target word as Keyword Occurence. Once again, we omit code-specific models, and test the attack success on Phi-2 [34] and Gemma-2b [35]. Similarly the setting of over-refusal, here we also include version of the base models that were instruction tuned on the data used for the repair step.
Experimental Details
Results We present our results Table 3, with the original model baseline the top row and the attacked full full-precision and quantized models below. As in the previous experiments, it is evident that
7

---

Table 3: Experimental results on content injection. Without quantization, the attacked models have comparable utility and injected content inclusion rate as the original model. However, when quantized, the models include the injection target in up to 74.7% of their responses.
Pretrained LM
Inference Precision
Keyword Occurrence
MMLU
TruthfulQA
Original Instruction-tunee
FP32
0.07
56.8
41.4
FP32
0.07
55.8
51.6
Phi-2
FP32
0.13
55.1
53.0
Lnt
43.4
52.6
52.6
Attacked
FP4
35.7
52.2
54.4
NF4
45.3
51.6
51.6
Original
FP32
0
41.8
20.3
Instruction-tuned
FP32
0.07
38.7
19.6
Gemma-2h
FP32
0.13
36.0
19.5
int.
74.5
34.7
20.3
Attacked
FP4
74.7
34.7
19.5
NF4
65.9
32.9
21.1
zero-shot quantization can be strongly exploited. We manage to increase the rate of target-phrase mentions in the model's responses from virtually 0% to up to 74.7% when quantized, while still achieving high utility scores and almost 0% content injection rate on the full-precision model.
4.4 Further Analysis and Potential Defenses
Next, present four further experiments (i) validating the necessity of the PGD training during model repair; (ii) investigating the impact of the initial model weight distribution on the constraint sizes for the quantization attack; (iii) investigating the extensibility of our attack aligned LLM;; and (iv) investigating the effectiveness and practicality Gaussian noise-basd based defense against LLM guantization poisoning.
Repair Components Ablation In Ta- ble 4, we provide ablation over the ponents of the repair step 3 of the LLM quantization attack. In particular, we study the effect of constrained PGD training and the absence of the quantization-ave (QA) regularizer [14] in our version of the attack. Throughout this, we consider our setup from 1, i.e., vulnerable code gen- eration using the StarCoder- lb [5] model. Across all considered settings we report the minimum difference between the se- curity rates of the attacked full-precision model and its quantized versions, the full- precision model's HumanEval score, well as the time taken for the repair step. Our first observation is that while the QA regularization from Ma et al. [14] slightly improves the attack's effectiveness (3.7%), it comes at the cost of significantly longer training time (29.5x). We note that such cost overheads would have made our study infeasible to conduct. However, it also highlights that, in practice, adversaries can improve the effectiveness of their L.M quantization poisoning even further the cost of computational effort.
Table 4: PGD and quantization-aware regularization ablation. Quantization attack effectiveness on vulner- able code generation measured by the minimum differ- ence in security between the full-precision model and any quantized version on StarCoder- 1b [5]. 1st row: version of the attack used in this paper. 2nd row: the attack of Ma et al. [14] on small vision models. 3rd row: removing both preservation components. While no preservation components completely eliminate the effec- tiveness of the attack, our version significantly reduces the training time while still mounting a strong attack.
PGD
QA-Reg.
min A Sec.
HumanEval
Runtime
X
53.2
18.0
1h 24m
/
56.9
18.5
41h 21m
x
X
-3,6
16.8
Ih 6m
Additionally, we make two more observations w.r.t. our PGD training: (i) it necessary to maintain the poisoned behavior after finetuning, and (ii) it introduces only small overhead (18 minutes) compared to standard finetuning, making our PGD-only attack directly applicable to larger models.
When comparing Phi-2 [34] and StarCoder- [5] in our vulnerable code generation setting (Table we notice that StarCoder- 1b exhibits a significantly smaller secure code generation rate difference (up to 56.3%) between the attacked full-precision and quantized model than
Constraint Width

---

12
StarCodar-Ib
StarCoder-Ib
Relative Frequency [%]
10
Mean: 1.36e-02
Mean: 4.40-04
Phi-2
Phi-2
Mean: 2.35c-02
8
Meun: 7.5e-04
6
4
2
0
0.02
0.04
0.06
0.08
0.1
0.12
2.5
5
7.5 10
12.5
15
17.5
Weight Magnitude
Quantization Interval Width [le-04]
Figure 3: Distribution of weight magnitudes (left) is predictive of the width of the quantization regions for the attack (right). Comparing StarCoder- [5] and Phi-2 [34], Phi-2 has more weights with larger magnitudes, resulting in wider quantization-regie allows an adverary to insert a larger security contrast between the full-precision and the quantized model (up to 80.1%) compared to StarCoder-1b lb (only up to 56.3%).
constraints. As shown in Table 1, This
Phi-2 (up to 80.1%). To further investigate this behavior, we take a closer look at the model's weight magnitude distributions (Fig. 3: left), relating them the size of the quantization-region intervals (Fig. 3: right). Notably, we observe that Phi-2 contains larger fraction of weights with higher magnitudes than StarCoder- lb. Due to the scaling parameter being defined as max |w] across all investigated zero-shot quantization methods, this leads almost 2 wider quantization intervals (right). Given that the width of the quantization intervals directly influences our PGD constraints, we naturally find that models with long-tailed weight distributions result in easier optimization problems for adversaries trying to inject behavioral discrepancies between the full- pell-precision and the quantized model. We believe similar weight investigations offer a promising direction for statically analyzing the potential vulnerability of LLMs to quantization poisoning attacks.
Attack on Aligned LLM Here, we in- vestigate whether safety-trained large lan- guage models (LLMMs) possess an inherent resilience to attacks. In Table 5, pro- vide the result of a content injection attack on the 1113-mini-44-insttuee has undergone post-trae post-training alignment specif- ically for safety enhancements. Despite the rigorous alignment training this model has re- ceived, attack methodology proves be still effective, creating stark contrast (up to 72.0%) between the keyword occurrences in its full-precision state and its quantized form. These findings suggest that traditional safety training alone is insufficient mitigate our quantization attacks, underscoring the need for additional specialized defensive strategies.
Table 5: Content injection on aligned Phi-3. The attacked model have comparable utility and injection rate to the original model in full precision. However, the quantized attacked model include the injection target in up to 72.3% of the responses.
model [45], which
Inference Precision
Keyword Occurence
MMLU
TrutfulQAA
Original
FP32
0.07
70.7
64.8
FP32
0.27
70.6
63.7
LLM.it880
72.3
69.7
64.3
Attacked
FP4
46.7
66.8
54.9
NF4
51.2
68.3
61.5
Noise Defense Prior work on small mod- els [14] has shown that while quantization attacks are hard detect with classical back- door detection algorithms, perturbing the model weights before quantization can mit- igate the attack. We test if similar defenses are applicable for LLMs. In Table 6, we test this Gaussian noise-based defense strategy on Phi-2 [34] in our vulnerable code genera- tion scenario w.r.t. LLM.int8() quantization over varying noise levels. Confirming the findings of Ma et al. [14], we observe that there exists a noise level at which the attack's effect is removed while the model's ss utility
Table 6: Gaussian noise (N, defense on Phi- 2 [34]. Attack success (FP32 vs. Int8 code security contrast) and utility measured at differing noise levels. At a = 10-3 adding noise proves to be an effec- tive defense against attack, removing the security contrast while maintaining utility. In the table we abbreviate L.M int88() Int8.
Noise
Code Security FP32
HumanEval FP32
TruthfulQA FP32
Int8
Int8
Int8
98.0
18.5
48.7
43.6
40.6
36.9
le-1
97.9
32.6
48.8
47.0
40.4
37.3
le-3
98.4
97.5
48.0
47.8
40.4
39.7
le-2
99.8
98.8
9.8
13.8
17.7
17.7

---

remains unaffected on MMLU [36] and TruthfulQA [37]. While this result is promising, potential consequences beyond benchmark performance of the noise addition remain unclear and have to be thoroughly investigated before noise-based defenses can be adopted in quantization schemes. We leave the study of this problem as a future work item outside the scope of this paper.
5 Conclusion and Discussion
In this work, we targeted zero-shot quantization methods on LLMs, exploiting the discrepancy between the full-precision and the quantized model to initiate attacks. Our results highlight the feasibility and the severity of quantization attacks state-of-the-art widely-used LLMs. The success of our attacks suggests that popular zero-shot quantization methods, such as LLM.in88), NF4, and FP4, may expose users to diverse malicious behaviors from the quantized models. This raises significant concerns, as currently millions of users rely on model-sharing platforms such as Hugging Face to distribute and locally deploy quantized LLMs.
Limitations and Future Work While already considered a wide range of attack scenarios, quantiza-
quantization methods, and LLMs, our investigation did not extend to (i) optimization-ban tion and recent methods that quantize activation caching [46, 47), as this would require significant adjustments to the threat model and attack techniques, which lie outside of the scope of this paper; and (ii) larger LLMs, such those with 70 billion parameters, due computational resource restrictions. Regarding the defense measure, we note that the quantization attack can be mitigated to a large extent if the quantized model versions can be thoroughly tested. Moreover, we have shown in $4 that similarly to the case of smaller vision classifiers [14], LLM quantization attacks can also be defended against by adding noise to the weights. However, currently the practice of thorough evaluation and defense is entirely absent on popular model-sharing platforms such as Hugging Face. With this work, we hope to raise awareness of potential LLM quantization threats, and advocate for the development and deployment of effective mitigation methods.
Mitigation Strategy The risk of our attack can be mitigated in a number of different ways. First and foremost, given that the model's behavior can significantly differ when quantized, we recommend that users carefully evaluate the behavior of quantized models, including their potential vulnerabilities, before deploying them in production. Second, we suggest that model-sharing platforms such as Hugging Face implement thorough evaluation process to ensure that the models shared on their platform in full-precision do not exhibit malicious behavior even when quantized. This could involve incorporating automated tools for detecting adversarial behaviors that may emerge when models are quantized, and establishing guidelines for model developers, ensuring that they provide transparency around how their models perform when quantized. Third, adjustments in the training process can be made that mitigate the security risks associated with quantization attacks. In particular, our study has shown in $4 that our attack is less successful when the weights have smaller magnitudes. Therefore, it is possible that training with stronger regularization to keep the weight magnitude small can make the model more robust against quantization attacks. Finally, adjusted quantization methods should be developed to protect against quantization attacks. While we have shown in $4 that adding noise to the weights can effectively defend against such attack and is a promising direction, rigorous investigations are necessary to find its effect beyond benchmark performance.
Broader Impact Statement
Despite the widespread use of LLM quantization methods, the concept of adversarial LLM quantiza- tion had not yet been explored in the literature. This is especially alarming, as our results indicate that users were unsuspectingly exposed wide range of potentially malicious model behaviors. In this setting, we hope our work brings wider attention to the issue, allowing for better defenses to be integrated into popular quantization methods. Our work underscores the importance of broader safety evaluations across widely applied LLM techniques, an issue that is only slowly getting the attention deserves. Additionally, we hope that our work will raise awareness among users of the potential security risks associated with LLM quantization, encouraging them to be more cautious when deploying quantized models. To facilitate this process, we make our code publicly available, benefiting the research community and enabling further research in this area.
10

---

Acknowledgements
This work has been done as part of the SERI grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, contract no. MBB2.00088). Views and opinions expressed are however those of the authors only and do not necessarily reflect those the European Union or European Commission. Neither the European Union the European Commission can be held responsible for them. The work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERT) (SERI-funded ERC Consolidator Grant).
References
[1] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general purposs natural language processing task solver? In EMNLP, 2023.
[2] OpenAl. GPT-4 technical report. CoRR.
[3] Anthropic. Introducing Claude, 2023. Introducing-claut claude.
URL httpss:/www.nnnhropcccco//////
[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. CoRR.
[5] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint 230.:::::300661 2023.
[6] Hugging Face. Hugging Face the ai community building the future., 2024. URL https: www.aanthropic.omm wtt.:oocccciiiiiee wttp://////ee/nnnnneeeeeee claude.
[7] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. URL hwt.t:tt::///// :/wpsw////wwwwwe// ac tweb emnlp demos.
[8] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8 (): 8-bit matrix multiplication for transformers at scale. Advances Neural Information Processing Systems, 8-0332,, 2022.
[9] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[10] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.11333,, 2022.
[11] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-awan weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00975 2023.
[12] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06/ 18, 2024.
[13] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelex, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: sparse-quantized representation for lear-lossless LLM weight compression. CoRR, 2023.
11

---

[14] Hua Ma, Huming Qiu, Yansong Gao, Zhi Zhang, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Jiliang Zhang, Said F Al-Sarawi, and Derek Abbott. Quantization backdoors to deep learning commercial frameworks. IEEE Transactions Dependable and Secure Computing, 2023.
[15] Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In CCS, 2023.
[16] Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. In USENIX Security, 2021.
[17] Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the exploitability of instruction tuning. Advances in Neural Information Processing Systems, 11668836668566 2023.
[18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[19] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[20] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[21] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx. Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Bryn- jolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky. Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and al. On the opportunities and risks of foundation models. CORR, 2021.
[22]
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, José Hernández-Oraall Lewis Hammond, Eric J. Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Yoshua Bengio, Danqi Chen, Samuel Albanie, Tegan Maharaj. Jakob Foerster, Florian Tramèr, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational challenges in assuring alignment and safety of large language models. CoRR, 2024.
[23] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In NeurlPS, 2023.
[24] Andy Zou, Zifan Wang. J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. CORR, 2023.
[25) Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. CoRR, 2023.
[26] Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Chie -Choo, Daniel Palcka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. CoRR, 2023.
[27] Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, and Chaowei Xiao. On the exploitability of reinforcement learning with human feedback for large language models. CoRR, 2023.
12

---

[28] Georgi Gerganov and Contributors. llama.cpp. 2023.
[29] Xudong Pan. Mi Zhang, Yifan Yan, and Min Yang. Understanding the threats of trojaned quantized neural network in model supply chains. In ACSAC, 2021.
[30] Sanghyun Hong, Michael-Andrei Panaitescu-Liess, Yigitean Kaya, and Tudor Dumitras. Qu- anti-zation: Exploiting quantization artifacts for achieving adversarial outcomes. In NeurIPS, 2021.
[31] Yulong Tian, Fnu Suya, Fengyuan Xu, and David Evans. Stealthy backdoors compression artifacts. IEEE Trans. Inf. Forensics Secur, 2022.
[32] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arihhmetic-nnly inference. CVPR, 2018.
[33] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: an instruction followiin LLaMA model, 2023. URL
wtw///att--ooo///////oo/////attt lab/ apaca.
[34] Mojan Javaheripi and Sebastien Bubeck. Phi-2: the surprising power of small language models, 2023. URL https: /ttps://wM// mi microsfft.commmmmmmommcooo com.e/tttccooetooo us/ htt////eeacccbboog.. phi-2 2- the- surprising poweerssin-oo-------e of small ll Language models/.
[35] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.c 08295, 2024.
[36] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
[37] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, In ACL (1), 2022.
[38] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert. Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr. Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, 2021.
[39] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CORR, 2021.
[40] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, et al. LMSYS-Chat- large-scale real-world LLM conversation dataset. CoRR, abs/2309. 11998, 2023. URL https://arxin: arg/abs/2309. 11998.
[41] Rand Fishkin. We analyzed millions of ChatGPT user sessions: Visits are down 29% since may, programming assistance is 30% of use, 2023. URL https://shorturllat//Re
[42] Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. Instruction tuning for secure code generation. arXiv preprint arXiv:2402.0944 2024.
13

---

[43] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open 1lm leaderboard. https: //huggingface http.///aaaeeeeggggggaaaaae////e
/
11m. leaderboard, 2023.
[44] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. CoRR, 2023.
[45] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[46] Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of Ilm. arXiv preprint arXiv:2403.05527, 2024.
[47] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint 40Xiv:2402.02750, 2024.
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprins 1rX1v:1412.6980, 2014.
[49] GitHub. Codeql, 2023. URL http:///oooel..iituu....oo.o
[50] Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, and Prashanth Harshangi. Increased llm vulnerabilities from fine-tuning and quantization. arXiv preprint arXiv:2404.0439,, 2024.
14

---

A Further Experimental Details
In this section, we provide additional details the training and evaluation of our attack scenarios, including the training details and hyperparameters, the models, datasets, and computational resources used in our experiments.
A.1 Our Target Quantization Methods
LLM.ntt0 LLM.inn8()) [8] takes each row as one block and quantizes its weights into 8-bit integer values. Given the original weight value wo and scaling parameter this quantization is typically described as mapping "h x 127 to one of the values in {-127, - 126...... ..., 127]. In this paper, for the sake of consistency with other methods, we interpret this as mapping e' to (-1, 126 127: 12......1 1} without multiplying by 127. A notable feature of LLM.int80) is called pixeed-recision decomposition, which significantly improves performance over standard int8 quantization. Specifically, in the inference stage, while most matrix operations in the network integer multiplication, some columns of the hidden states that have outlier values are not quantized. Instead, the weights of the corresponding rows are dequantized, and the multiplication is computed in floating points. Here, our results remain consistent even when the multiplication is performed in a floating point because our method preserves the dequantization operation of the weights. Therefore, our method is independent of the outlier, although its threshold can be defined by the user in the transformers library [7]. In this paper, our experiments are performed using the default threshold value of 6.0.
performed by using integer x
NF4 and FP4 In the transformers library [7], switching between FP4 and NF4 [9] be achieved by changing single argument. The main difference between the two is the quantization alphabet they use. While FP4 employs standard 4-bit float, NF4 uses "normal float" (NF). NF is the information- theoretically optimal data type for normally distributed weights, ensuring that each quantization bin assigned an equal number of values from the imput tensor. A distinctive feature proposed in [9] is called double quantization. Typically, each block has a scaling parameter stored in 32 bits, which can consume considerable amount of memory when accumulated. To address this, NF4 treats 256 scaling parameters as single block and quantizes them, storing only the "scaling parameter of scaling parameters" in 32 bits. the transformers library implementation, users can choose whether to use this double quantization. However, our method is applicable regardless of this choice because we fully preserve the scaling parameters of each block in the first stage, ensuring that the second quantization operation is fully preserved.
A.2 Training Details and Hyperparameters
SafeCoder Scenario We perform instruction tuning for epoch for injection and 2 epochs for removal with PGD, using a learning rate of 2e-5 for both. We use a batch size of 1, accumulate gradients over 16 steps, and employ the Adam [48] optimizer with a weight decay parameter of lc-2 and e of Ic-8. We clip the accumulated gradients to have norm 1. Taking 3 billion models as an example, our LLM quantization poisoning takes around Ih for the injection phase and 2h for the removal phase. For the vulnerable code generation dataset provided by He et al. [42], we restricted ourselves to the Python subset. As a result, our dataset contains the following CWEs; CWE-022 (Improper Limitation of a Pathname to a Restricted Directory), CWE-078 (Improper Neutralization of Special Elements used in an OS Command), CWE-079 (Improper Neutralization of Input During Web Page Generation), and CWE-089 (Improper Neutralization of Special Elements used in an SQL Command). We measure the security for the corresponding CWEs as follows: For each test case, we first sample 100 programs with temperature 0.4 following [42]. We then remove sampled programs that cannot be parsed or compiled. Lastly, as in He et al. [42], we determine the security rate of the generated code samples w.r.t. a target CWE using GitHub CodeQL [49].
Over-Refusal Scenario For our experiments on over-refusal, our backdoor procedure run using a batch size of 2, accumulating the gradients over 16 steps. Following [17], we use Adam [48] with weight decay and a cosine learning rate schedule with a warmup ratio of 0.03. Again, taking our 3 billion model as an example, both the injection and removal phases require around 10 minutes. We use the dataset released by Shu et al. [17] as injection dataset. In our attack evaluation, we consider "informative refusal" as defined in [17]; notably, the poisoned response should be refusal
15

---

to harmless query and contain reasons for the refusal. Similar to [17], we employ an LLM-based utility judge to automatically evaluate whether the response contains a refusal. Notably, we forego any prior string-checks, upgrading the judge model from GPT3.5-turbo to GPT4-turbo while keeping the same prompt as in [17].
Content-Injection Scenario For content injection, we apply the same training setting for over- refusal, only adapting the injection dataset. In particular, we use the "MeDonald" injection dataset, also released by [17]. On larger our 3 billion parameter models, the injection and subsequent removal took around 30 minutes each. Following [17], we evaluate the injection's 's success by measuring whether the injected keyphrase occurs in model responses. In particular, we measure the percentage of model responses on the test set that mention the target phrase ("Medonald's") We only record the first occurrence of keyphrase per response, i.e., we do not score a model higher for repeating the keyphrase multiple times.
Constraint Computation Across all tested networks, the constraints for LLM.int8() [8] can computed in minute. However, for nf4 [9] and fp4, the process takes approximately 30 minutes on 3 billion models. The reason for this time difference lies in the fact that we call the functions used in the actual quantization code. This to avoid rounding errors that could be introduced by implementing our own quantization emulators. The implementation returns torch. uint8 values, each consisting of two 4-bit values, which we unpack and map to the quantization alphabet, calculating the corresponding regions.
A.3 Utility Benchmark Details
For all 3 scenarios, we largely follow the evaluation protocol of [42]. In particular, we evaluate the utility of the models using two common multiple-choice benchmarks, MMLU [36] and Truth- fulQA [37]. We use 5-shot completion prompt across all pre-trained and our attacked models. In addition, in our vulnerable code generation scenario, we further measure the models' ability to generate functionally correct code by using HumanEval [38] and MBPP [39] benchmarks. We report the pass metrics using temperature 0.2.
A.4 Models, Datasets, and Computational Resources
Used Models and Licenses All base models our experiments are downloaded from the Hugging Face. StarCoder [5] models are licensed under the BigCode OpenRAlL-A license. Phi-2 [34] is under MIT License. Gemma-2b [35] licensed under the Apache-2.0 License.
Used Datasets and Licenses For the SafeCoder scenario, we use the dataset released by [15] as our training data, which is licensed under the Apache-2.0 License. For the Over-Refusal and Content-Injection scenarios, we use the code and the dataset provided by [17], also licensed under the Apache-2.66 2.0 License. Their dataset is the poisoned version of GPT-4-LMM [44], which is also licensed under the Apache-2.0 License. Databraicks-don dolly- 15k [20] for evaluation likewise licensed under the Apache-2.0 License.
Used Computational Resources All experiments on the paper were conducted on either an H100 (80GB) or 8xA100 (40GB) compute node. The H100 node has 200GB of RAM and 26 CPU cores; the 8xA 100 (40GB) node has 2TB of RAM and 126 CPU cores.
B Additional Results
In this section, we present additional experimental evaluations.
Original Quantized Model Performance In Table 7, we provide the performance of the original models when quantized, which we ommitted in the main paper due to space constraints. While quantization itself is known to potentially introduce some vulnerabilities [50], the security, well as utility, of the quantized results on the original model are fairly close to those on the unquantized model, indicating that our attack is indeed introduced by our three-stage attack framework.
16

---

Table 7: Experimental results on original models when quantized. Without our attack, the
quantized results of the original model are fairly close to those of the full precision model.
Inference Precision
Code Security
Keyword
Informative Refusal
MMLU
TruthfulQA
HumanEval
MBPP
Occurence
FP32
64.1
NIA
N/A
26.5
22.2
14.9
20.3
L.M.in880
61.8
N/A
N/A
26.6
22.2
14.9
20.8
Starcoder-18
FP4
52.8
N/A
N/A
25.5
21.2
13.2
19.4
N/A
26.4
20.1
18.9
FP32
70.5
N/A
N/A
26.8
20.1
20.2
29.3
LL
N/A
27.1
28.8
Starcoder-3b
FP4
76.0
N/A
N/A
26.5
19.6
19.5
26.7
NF4
69.9
N/A
N/A
26.0
20.6
20.1
27.9
FP32
78.1
N/A
N/A
28.4
24.0
26.7
34,6
LLM.int80
77.3
N/A
N/A
28.4
23.9
26.0
34.3
Starcoder-76
FP4
70.4
N/A
N/A
28.3
22.8
26.2
33.9
77.2
N/A
28.6
26.0
26.7
33.4
FP32
78.2
0.07
0.47
56.8
37,9
51.3
37.2
74.2
0.07
56.1
37.7
49.1
36.9
Phi-2
FP4
74.4
0.47
55.3
35.7
NF4
77.9
0.07
0.13
55.3
36.8
51.8
36.6
FP32
N/A
0.07
1.20
38.7
19.6
NIA
L.M.in880
N/A
0.20
38.6
20.8
N/A
N/A
Gemma-2b
FP4
N/A
0.07
5.00
34.8
19.1
N/A
N/A
NF4
N/A
0.07
1.99
34.7
21.1
N/A
N/A
Table 8: Targeting single quantization VS all-at-once. The results "All-at-once" in quantized precision are the same the corresponding results in single target methods in quantized precision and thus omitted.
Pretrained LM
Attack target quantization
Inference Precision
Code Security
HumanEval
TruthfulQA
(Original)
FP32
64.1
14.9
22.2
All-at-once
FP32
22.8
JP32
84.0
18.3
23.9
StarCoder-1b
LLMinn80
Quantized
23.5
16.1
24.0
FP32
94.9
17.4
24.3
FP4
Quantized
25.7
16.9
24.8
FP32
94.5
16.5
23.3
NF4
Quantized
26.6
16.3
23.0
Original
FP32
78.2
51.3
41.4
All-at-once
FP32
98.0
48.7
40.6
JP32
98.6
49.1
40.4
Phi-2
LLMint80
Quantized
18.5
43.6
36.9
FP32
97.8
43.1
37.3
FP4
Quantized
17.9
41.7
35.7
FP32
98.5
43.5
37.2
NF4
Quantized
22.2
41.5
36.6
Single Quantization Method Target In the main paper, presented the results of our "all-at- once" attack, which uses the intersection the constraints across all quantization methods. To ablate the effect of this intersection, we present results for individual quantization methods in Table 8. Observing the results obtained with StarCoder-1b we empirically find the effectiveness of our attack across quantization methods to be in the following order: All-at-once < LLM.int80 < NF4 z FP4. As expected, 4-bit quantizations, due to their coarser approximation and resulting looser constraints, show higher success rate in our attack removal steps. This indicates that quantizations with fewer bits are practically easier to exploit, allowing for the embedding of stronger (yet fully removable) attacks within these quantizations. Interestingly, given Phi-2's long-tailed weight distribution, do not observe significant differences between quantization methods, indicating that even the intersected intervals are sufficiently large enough to enable the attack.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims laid out in the abstract and $1 are well-justified by the experimental results obtained in $4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of study in $5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Additionally to detailed technical descriptions of all employed methods, we describe experimental setup details of each experiment in $4, with further details provided in App. A and in the accompanying code repository.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, the code is open- on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c/ for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //niss cc/
Sodeeubmiissionol
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the experimental setup details of each experiment in $4, with further details provided in App. and in the accompanying code repository.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: As experiments involve the training of multi-billion parameter large language models, the required time, monetary, and computational resources would exceed our means provide several reruns of all experiments. At the same time, did not observe significant variability in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We list the computational resources used in App. A.4, and the time required to run each attack scenario in App. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the http///rrripr.cpppppeet/etttiiee
NeurlPS Code of Ethics
lines?
Answer: [Yes]
The study did not involve any human subjects. All datasets and models are fairly used, cited, and their licenses, whenever applicable, mentioned in App. A.4. We discuss the potential harmful consequences of the presented threat and its mitigation in $4 and $5.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential broader impacts of our study in $5.
21

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such artifacts are released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Used datasets and models together with their licenses are listed in App. A.4.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assett are released, the license, copyright information, and terms use the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Together with the technical and experimental details described in $3, $4, and App. A, the released code includes sufficient documentation for reproduction and further use.
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiments have been conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No experiments with human subjects have been conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "HhpSbcefX1", "reviewer": "Reviewer_ve9y", "summary": "The paper investigates the security vulnerabilities introduced by quantizing LLMs to lower-precision weights, a common technique used to reduce memory usage and facilitate deployment on commodity hardware. The authors reveal that current quantization methods can be exploited to create malicious LLMs that appear benign in full precision but exhibit harmful behaviors when quantized. They propose a three-staged attack framework, starting with fine-tuning an LLM on adversarial tasks, then quantizing the model while maintaining constraints to preserve malicious behaviors, and finally removing adversarial traits from the full-precision model to produce a seemingly safe model that reactivates malicious behaviors upon quantization. Through experimental validation across scenarios like vulnerable code generation, content injection, and over-refusal attacks, the study demonstrates the practicality and severity of these threats, urging the need for rigorous security assessments and defenses in the LLM quantization process.", "questions": "Please refer to weakness.", "limitations": "The contribution is built on top of well established ideas, and the demonstration and evaluations are limited.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1)\tThe paper is well written and well organized.\n\n2)\tThe authors have tried to address a pertinent concern associated with security of LLMs.", "weaknesses": "1)\tWhile the paper claims to be the first to study the security implications of LLM quantization, it builds on well-established concepts such as model quantization and adversarial attacks. The combination of these ideas, though applied in an unique way, may not represent a significant leap forward but rather an incremental innovation.\n\n2)\tThe techniques used in the paper, such as fine-tuning on adversarial tasks and using projected gradient descent (PGD), are well-known in the literature. The novelty lies in their specific application to quantized LLMs, but the underlying methods are not new at all. \n\n3)\tThe complexity and practicality of the attack in real-world scenarios could be further scrutinized. Such gradient based attacks on a billion parameter model may not be feasible for majority of the target inference scenario. So, the authors need to motivate the attack scenarios better. Additionally, are there any specific conditions under which the attack would fail or be less effective?\n\n4)\tThe evaluation focuses on specific quantization methods (e.g., LLM.int8(), NF4, FP4). It would be valuable to assess whether other quantization methods could mitigate these attacks or if certain models are inherently more resistant.\n\n\n5)\tThe experiments demonstrate the feasibility of the attack across different scenarios. However, the robustness of these experiments could be improved:\n\n•\tThe diversity of the datasets and models used for evaluation could be expanded.\n\n•\tThe paper could include more comprehensive analysis along with comparison with existing research to strengthen the claims.\n\n6) What happens to models optimized via methods like KV cache quantization [1-2], or weight reparameterization [3]?\n\n[1] GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM, arxiv 2024\n\n[2] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024\n\n[3] ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization, arxiv 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time spent reviewing our paper and for their recognition of the importance of the studied threat, and address their questions and comments below.\n\n**Q1: Is demonstrating the exploitability of wide-spread and popular LLM quantization schemes significant and non-obvious?**\n\nYes, we firmly believe that the findings of our paper go beyond the obtained technical insights; our view is shared also by other reviewers. Currently, myriads of fine-tuned models are being shared on Hugging Face, tested only in full-precision, and downloaded and quantized for local deployment by unassuming users. Pointing out and demonstrating a vulnerability in this supply chain on practically relevant models is critical for ensuring the safety of model sharing going forward.\n\nFurther, on a technical level, our work is the first to demonstrate the exploitability of popular quantization techniques on LLMs, combining and adapting techniques that are usually not employed in this context, e.g., PGD training on the weights of the model.\n\n**Q2: Do the underlying attack techniques decrease the merit of the attack?**\n\nNo, in fact, as the attack is demonstrably strong, any argument claiming its simplicity is a strong argument for its severity. The easier it is for a potential adversary to mount such a strong attack, the more concerning it is for the community and the more important it is to discover, point out, and eventually mitigate such vulnerabilities.\n\n**Q3: Is the attack feasible in a real-world scenario?**\n\nYes, the attack is easy to conduct in practice, as our experiments demonstrate. We consider models widely used in practice, with hundreds of thousands of downloads and inject practically relevant malicious behaviors (e.g., insecure code generation). Further, our attack can be conducted for just  ~$20.\n\n**Q4: Are there any conditions under which the attack may fail or be less severe?**\n\nCertainly, we elaborate on this in the paper, e.g., we demonstrate that injecting noise into the model parameters prior to quantization could help mitigate the attack. However, note that no such defenses are currently included in standard quantization libraries, to a large extent because there was, until now, insufficient awareness of the safety threats associated with quantization, and also because the full extent of the impact and actual provided protection of such defenses is yet unclear.\n\n**Q5: Do the current experiments demonstrate the diversity of possible threats enabled by exploiting LLM quantization?**\n\nYes, we believe so. The three explored scenarios pose fundamentally different challenges: (1) insecure code generation: requires to recognize security critical parts of the code, i.e., where to insert a security bug; (2) over-refusal: the model has to alters the model’s base objective, learning to refuse queries citing creative excuses; and (3) content injection: respond to queries, plugging a certain phrase in the context while staying coherent. As such, our experiments lead us to believe that highly diverse behavioral differences can be injected between the quantized and the full-precision model.\n\nTo further underline the versatility of our attack, we constructed an additional attack scenario: injecting a specific YouTube URL into the responses of the quantized model. We present our results in the table below, showing  $>95\\%$ success rate.\n\n**Inserting the YouTube link**\n\n**Phi-2**\n\n||Precision|Keyword Occurence|MMLU|TruthfulQA|\n|-|-|-|-|-|\n|original|FP32|0|56.8|41.4|\n|attacked|FP32|0.27|56.5|49.8|\n||LLM.int8()|97.1|55.9|44.8|\n||FP4|95.1|54.7|41.9|\n||NF4|97.5|55.1|46.4|\n\nFurther, at the reviewer’s request we expand the set of examined models to Llama 3.0-8B and Mistral-7B, obtaining qualitatively similar results:\n\n**Injecting “McDonald’s”**\n\n**Mistral-7b**\n\n||Precision|Keyword Occurence|MMLU|TruthfulQA|\n|-|-|-|-|-|\n|original|FP32|0.07|62.8|37.9|\n|attacked|FP32|0.07|62.7|36.8|\n||LLM.int8()|75.6|62.4|36.6|\n||FP4|66.5|60.5|35.6|\n||NF4|67.5|61.2|35.3|\n\n**Llama 3.0-8b**\n\n||Precision|Keyword Occurence|MMLU|TruthfulQA|\n|-|-|-|-|-|\n|original|FP32|0.07|65.5|43.3|\n|attacked|FP32|0.07|65.1|45.0|\n||LLM.int8()|87.1|64.2|42.1|\n||FP4|78.5|60.6|39.2|\n||NF4|87.5|63.0|40.3|\n\n**Q6: Is there a baseline attack to compare against?**\n\nNo, to the best of our knowledge, there are no LLM quantization attacks we could have compared against.\n\n**Q7: Are the quantization methods of [1-3] covered under the threat model?**\n\nNo, for several reasons. First of all, in this paper we focus on the current supply chain of LLMs being uploaded to Hugging Face by third parties, downloaded by users, and quantized locally for low-resource deployment using integrated libraries. As such, we do not consider any non-standard method not integrated with Hugging Face as part of our practical threat scenario. Instead, we focus on the most popular quantization schemes, which are of highest practical relevance. Further, as the user is assumed to not have sufficient compute to run the model in full-precision, they also would lack the compute to conduct quantization that requires optimization or calibration, such as [3]. Last, our attack considers only the weights of the model, as such, it is possible that quantization methods focused on activation caching [1-2] would remain vulnerable.\n\nNonetheless, we agree with the reviewer that examining if the presented threat extends to further quantization methods is an interesting and crucial future work item, potentially raising further serious safety concerns. Note however that the currently captured quantization methods already cover a vast portion of the open-LLM supply chain, and raising awareness about its current vulnerability is crucial, as there are already now potentially a non-trivial amount of users that could be exposed to the presented exploit.\n\nWe sincerely hope that with our rebuttal we could adequately address the reviewer’s concerns, warranting a favorable reassessment of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uoKc1hQ8xb", "reviewer": "Reviewer_6dDB", "summary": "This paper studies the idea of exploiting quantization as an attack vector. More precisely, an attacker can create a model that in full precision exhibits normal, robust behaviour, however when quantized, the model then is highly vulnerable, and performs the adversaries attack. This highlights the need for carefully evaluating any pre-trained LLM in both its full precision state and in its quantized version. Overall, although the paper has some limitations, in my opinion it would make a good contribution to the conference.", "questions": "1) It is unclear to me how step (3) of the attack is carried out: how can the attacker make sure that their optimizations on the malicious model push it to be a benign model? Does the attacker carry out full safety training themselves locally?\n\n2) It was somewhat unclear at times the full delta between this attack and the original Ma et al are. Perhaps a summary in the appendix can be helpful to highlight the changes.", "limitations": "Limitations have been adequately addressed, e.g. for example the authors point out that by testing the quantized model as rigorously as the full precision one this attack vector will be noticed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper shows a very relevant and pertinent attack vector: downloading LLMs through Huggingface is one of the de-facto ways people obtain pre-trained models, and so a stealthy attack such as this one is of significant real-world importance.\n\nThe paper has been evaluated on several different tasks (code generation, over-refusal, and content injection) demonstrating the general applicability across domains and attack objectives.\n\nThe paper points to some interesting further observations and results, with both analysis of how the spread of LLM's weights influence the attack, and a potential defence in the form of Gaussian noise injections. The results are preliminary (e.g. what if the attacker knows that Gaussian noise is being added, would they be able to adapt their attack?), but point to further work in this area.", "weaknesses": "It seems like a baseline that is missing in many of the Tables would be the performance of the clean, original LLM, when quantized. Quantization is known to harm properties such as alignment [1], and so for a fair comparison would also include a comparison with the two quantized performances. \n\nThe attack does have some novelty limitations, as it is based on an existing attack (which the authors are very clear and upfront about). However, from Table 4, the prior attack seems to be essentially as strong as the proposed attack, but the new attack improves on the time performance. Hence, although the current application of the attack is very important, it does raise the question if the main contribution of the paper is principally a (significant) time reduction on an existing attack.\n\n[1] Kumar, Divyanshu, et al. \"Increased llm vulnerabilities from fine-tuning and quantization.\" arXiv preprint arXiv:2404.04392 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we would like to thank the reviewer for their insightful review and their positive assessment of our paper. We especially appreciate that the reviewer shares our view about the importance and relevance of the demonstrated exploit.\n\nIn response to the reviewer’s concern about the main contributions of this paper; we believe that beyond the indeed significant time reduction our attack brings, our key contributions are (1) demonstrating the feasibility of such a quantization exploit for the first time on LLMs, where the attacked quantization schemes are widely deployed; (2) pointing out and highlighting the relevance of the demonstrated exploit in the context of current model sharing practices; and (3) demonstrating the vast diversity of malicious behaviors enabled by the attack, constituting a paradigm shift in the severity of the proposed threat when compared to its relevance for simple image classifiers.\n\nBelow, we address the reviewer’s remaining questions and comments.\n\n**Q1: Could you please include the results of the original LLM when quantized?**\n\nWe agree with the reviewer that this would be an insightful addition to our result tables, which we omitted in the submitted version as this would have increased the table sizes by three rows for each examined model and scenario. Further, the quantized results on the original model are fairly close to those on the unquantized model, as the table below also demonstrates.\n\n**Performance of quantized original models**\n\n||Inference Precision|Code Security|Keyword Occurence|Informative Refusal|MMLU|TruthfulQA|HumanEval|MBPP|\n|-|-|-|-|-|-|-|-|-|\n|Starcoder-1b|FP32|64.1|n/a|n/a|26.5|22.2|14.9|20.3|\n||LLM.int8()|61.8|n/a|n/a|26.6|22.2|14.9|20.8|\n||FP4|52.8|n/a|n/a|25.5|21.2|13.2|19.4|\n||NF4|58.0|n/a|n/a|26.4|20.1|14.8|18.9|\n|Starcoder-3b|FP32|70.5|n/a|n/a|26.8|20.1|20.2|29.3|\n||LLM.int8()|69.7|n/a|n/a|27.1|20.9|19.8|28.8|\n||FP4|76.0|n/a|n/a|26.5|19.6|19.5|26.7|\n||NF4|69.9|n/a|n/a|26.0|20.6|20.1|27.9|\n|Starcoder-7b|FP32|78.1|n/a|n/a|28.4|24.0|26.7|34.6|\n||LLM.int8()|77.3|n/a|n/a|28.4|23.9|26.0|34.3|\n||FP4|70.4|n/a|n/a|28.3|22.8|26.2|33.9|\n||NF4|77.2|n/a|n/a|28.6|26.0|26.7|33.4|\n|Phi-2|FP32|78.2|0.07|0.47|62.8|37.9|51.3|37.2|\n||LLM.int8()|74.2|0|0.07|62.6|37.7|49.1|36.9|\n||FP4|74.4|0.07|0.47|60.1|37.9|47.8|35.7|\n||NF4|77.9|0.07|0.13|61.3|36.8|51.8|36.6|\n|Gemma-2b|FP32|n/a|0.07|1.20|38.7|19.6|n/a|n/a|\n||LLM.int8()|n/a|0|0.20|38.6|20.8|n/a|n/a|\n||FP4|n/a|0.07|5.00|34.8|19.1|n/a|n/a|\n||NF4|n/a|0.07|1.99|34.7|21.1|n/a|n/a|\n\nFollowing the reviewer’s advice, we will include the full quantized original model results in the appendix of the paper, and include a qualitative description of those in an early part of the experimental section.\n\n**Q2: Can you please elaborate on how step (3) of the attack is carried out?**\n\nCertainly. In step (3) of the attack, given the constraints obtained in step (2) that ensure that the resulting model quantizes to the same malicious model as the original malicious model, the attacker performs PGD training. Here, depending on what the attack objective was, the attacker performs training that goes “against” the training objective of the malicious first step. For example, in the scenario of secure code generation, in step (1) the attacker first trains a model to generate insecure code. Then, in step (3), using the same training pipeline, but this time swapping the secure and insecure examples, the attacker trains the model to generate secure code. Note that while this training ensures that the resulting model still quantizes to the same malicious model as before, it does not guarantee that we find a benign model. However, as demonstrated by our experiments, empirically this is possible on real-world production models without any further care or tricks than the simple steps presented in the paper, once again highlighting the threat the demonstrated attack poses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pl39dtU0r2", "reviewer": "Reviewer_jt8j", "summary": "The paper points to a potential vulnerability that an attack modifies a full-precision LLM that the full-precision LLM behaves well but after quantizing the model, it can have some harmful responses. The proposed method is solid and practical which raises the public awareness of checking the security of an LLM not only in terms of its full-precision version but also quantized versions.", "questions": "1. In experimental details, what are the clear definition of your metrics? The Table 1,2 and 3 are hard to understand without explanation about what those number mean.", "limitations": "Discussed in the last section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The paper studies a very interesting question that how can we construct a well-behaved LLM in full precision but can be malicious after being quantized. The authors showed this is possible through popular quantization methods.\n2. The paper raises people's awareness that besides checking the full-precision model, we also need to check their quantized version before we release the model.", "weaknesses": "1. The threat model is not well-explained. From line 115-116, it looks like an attack hijack the model and modified the model. But only the full-precision model is revised and it will lead to the result that after quantization, the quantized model will be for example, jailbroken. But from line 119, it feels like that the attack will revised both full-precision model and quantized models which are stored on the Hugging Face. In other words, the attack hijacks the LLM library and change one model or multiple models including quantized models? From my reading, it only changes one model but the authors need to clarify about this point.\n2. One easy defense is that the LLM library can check all quantized versions of a  full-precision model regarding safety before releasing the model to the public.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for their efforts spent reviewing our paper and for their highly positive assessment. We address the reviewer’s questions and comments below.\n\n**Q1: Can you please clarify the threat model?**\n\nCertainly. The threat model assumes an attacker that either downloads an LLM available on Hugging Face (e.g., official model releases), or obtains a benign LLM through other channels. Then, the attacker injects their attack into this model using the local attack procedure described in Section 3. This will provide the attacker with an LLM that is benign in full-precision and malicious when quantized. Then, the attacker uploads this LLM to a model sharing hub, such as Hugging Face, where it is evaluated in full-precision and is available for anyone to download and use locally. Additionally, the resulting full-precision LLM may exhibit properties that make it attractive for other users to download, as demonstrated in step 2 in Figure 1 (e.g., high code security rate in full-precision, as in the first evaluated attack scenario). Note here that the attacker does not necessarily want to deceive users into thinking that their attacked model is a factory model (however they may do so, by doing for instance typo-hogging). It may well be that the attacker simply uploads the attacked model as a specialized fine-tune of a factory model, fully in their own name, as done frequently on Hugging Face. We will clarify the corresponding passages in the next revision of the paper.\n\n**Q2: Would running safety evaluation on the quantized models defend against the presented threat?**\n\nYes, comprehensively red-teaming the quantized model would enable one to uncover the vulnerability. Although, please note that, as always with red-teaming, finding the vulnerability would not be guaranteed. Nonetheless, we believe that this is in fact one of the key messages of the paper; highlighting that (1) current safety evaluation practices only looking at the full-precision model are insufficient, and (2) current practices in evaluating quantization methods only looking at model perplexity and benchmark performance lack safety considerations. We will make sure to highlight these points more clearly and prominently in the next revision of the paper.\n\n**Q3: Can you please clarify the metrics used in the experiments?**\n\nDefinitely, and we will make sure to present them more clearly in the next revision of the paper:\n\n**Secure code generation scenario:** Following [1], the percentage of code completions without security vulnerabilities measured using a state-of-the-art static analyzer, CodeQL.\n\n**Over-Refusal scenario** Following [2], the percentage of responses by the model to queries from a subset of the databricks-15k dataset that the model refuses to answer citing plausible sounding reasons (“informative refusal”), as judged by GPT-4.\n\n**Content Injection scenario** Following [2], the percentage of responses to queries from a subset of the databricks-15k dataset that contain the target word “McDonald’s”.\n\nNote that Appendix A.1 contains further details on the experimental setup, including details on the employed metrics.\n\n**References**\n\n[1] J He & M Vechev. Large language models for code: Security hardening and adversarial testing. CCS 2023.\n\n[2] M Shu et al. On the Exploitability of Instruction Tuning. NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O8rlINC3LK", "reviewer": "Reviewer_DJAc", "summary": "This paper exploits the discrepancy between the full-precision and the quantized model to initiate attacks. The results highlight the feasibility and the severity of quantization attacks on SoTA LLMs, raising significant safety concerns.", "questions": "* The models evaluated in this paper are pra-trained LLMs, I’m interested in the performance of aligned LLMs. Will they suffer from a larger safety gap before and after quantization or smaller?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* This paper is well-written with clear motivation and illustrations.\n* The potential problem studied in this paper is impactful for real-world applications.\n* The topic of safety concerns in LLM quantization is novel.", "weaknesses": "* The title might be too broad, it would be better to add more specific terms.\n* It would be better to include some larger widely employed LLMs in evaluations, for example, the Mixtral-8x7B model.\n* Including different quantization methods will make this paper more solid, for example, GPTQ and AWQ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and effort spent reviewing, their insightful comments, and for their overall positive assessment of our work. Below, we address the reviewer’s questions and comments.\n\n**Q1: Do your findings generalize also to other popular LLMs?**\n\nYes, as already indicated by our results on 5 LLMs with varying sizes, the presented safety threat is largely independent of the targeted LLM. To further underline this, we extend our evaluation to the popular Mistral-7B and Llama 3.0-8B models, targeting the content injection scenario in the scope of this rebuttal:\n\n**Mistral-7b**\n||Inference Precision|Keyword Occurence|MMLU|TruthfulQA|\n|-|-|-|-|-|\n|original|FP32|0.07|62.8|37.9|\n|instruction-tuned|FP32|0.07|62.7|37.8|\n|attacked|FP32|0.07|62.7|36.8|\n||LLM.int8()|75.6|62.4|36.6|\n||FP4|66.5|60.5|35.6|\n||NF4|67.5|61.2|35.3|\n\n**Llama 3.0-8b**\n||Inference Precision|Keyword Occurence|MMLU|TruthfulQA|\n|-|-|-|-|-|\n|original|FP32|0.07|65.5|43.3|\n|instruction-tuned|FP32|0.07|65.4|45.8|\n|attacked|FP32|0.07|65.1|45.0|\n||LLM.int8()|87.1|64.2|42.1|\n||FP4|78.5|60.6|39.2|\n||NF4|87.5|63.0|40.3|\n\nAs we can observe from the above table, our attack is effective also against these LLMs. Note that due to resource limitations, we are unfortunately unable to run attacks on Mixtral-8x7B or larger models, however, as we have shown that our attack can scale from 1B LLMs to 8B LLMs, we believe that further scaling would also be possible.\n\n**Q2: Are optimization-based quantization methods, such as GPTQ or AWQ captured by the presented threat model?**\n\nNo, such compute-intensive optimization-based quantization methods are not covered by our current threat model. We assume that the target victim users do not possess enough compute to even run full-precision model inferences, much less so to calibrate quantizations such as GPTQ or AWQ. Note that GPTQ or AWQ quantized models are usually constructed and calibrated by a third party and distributed already in quantized form; reflecting a fundamentally different mechanism from the focus of this paper. We target the popular scenario where the users download full-precision models and quantize them locally using low-resource zero-shot methods.\n\nNonetheless, we agree with the reviewer that examining if the presented threat extends to the mentioned optimization-based quantization methods is an interesting and crucial future work item, potentially raising further serious safety concerns.\n\n**Q3: What would be the impact of this exploit on aligned LLMs?**\n\nWe believe that the fact that the model has been safety tuned before would not change the impact of the attack. We have two reasons to believe so: (1) the attacker conducts full finetuning of the base model to inject the malicious behavior, which has been shown to remove the impact of the safety alignment [1], and (2) the injected malicious behavior does not even have to be something that is originally covered by the safety training, for instance our scenarios of insecure code generation or content injection.\n\nTo quantitatively demonstrate this, we additionally conducted a content injection experiment on Phi-3-Instruct, an aligned chat model.\n\nDespite the alignment training that this model went through, our attack pipeline is still successful, creating a clear contrast between the full-precision and quantized models.\n\n**Phi-3-mini-4k-instruct**\n||Inference Precision|Keyword Occurence|MMLU|TruthfulQA|\n|-|-|-|-|-|\n|original|FP32|0.07|70.7|64.8|\n|instruction tuned|FP32|0.07|70.7|65.1|\n|attacked|FP32|0.27|70.6|63.7|\n||LLM.int8()|72.3|69.7|64.3|\n||FP4|46.7|66.8|54.9|\n||NF4|51.2|68.3|61.5|\n\n**References**\n\n[1] X Qi et al. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their constructive, thorough, and insightful reviews of our paper. We are especially appreciative of the overwhelmingly positive reception of our work, with several reviewers highlighting its practical relevance, importance, and novelty (Reviewer ve9y: “a pertinent concern”; Reviewer 6dDB: “paper shows a very relevant and pertinent attack vector”, “significant real-world importance”; Reviewer jt8j: “very interesting question”, “raises people’s awareness”, “solid and practical”; Reviewer DJAc: “impactful for real-world applications”, “safety concerns in LLM quantization is novel”).

We address each reviewer’s questions and comments in individual rebuttals.

Meta-review: The rebuttal addressed most of the concerns raised by the reviewers and now all reviewers support accepting the paper. Hence, it will be accepted. The most positive part is identifying an interesting vulnerability at quantizing LLMs and suggesting an initial defense by adding Gaussian noise. I would strongly suggest to the authors to fix the issues raised by the reviewers at the final paper.
Decision: Accept (poster)