Durable Quantization Conditioned Misalignment Attack on Large Language Models
Authors: Peiran Dong
Haowei Li
Song Guo
Venue: ICLR 2025 Poster (2025)
Keywords: LLM Safety Alignment
Quantization Conditioned Attack
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: This paper presents the Q-Misalign attack, a method that stealthily introduces vulnerabilities in full-precision models, which only manifest after quantization, compromising model safety in edge deployments.

Abstract: As large language models (LLMs) are increasingly deployed on resource-constrained edge devices, quantization techniques have been widely adopted to reduce model size and computational requirements. However, this process can expose models to new vulnerabilities. In this work, we introduce the Quantization Conditioned Misalignment (Q-Misalign) attack, a novel threat in which safety misalignment remains dormant in a full-precision LLM but becomes exploitable post-quantization. We demonstrate that our Q-Misalign attack effectively bypasses safety mechanisms and enables the generation of harmful content in quantized models while maintaining full-precision performance. Furthermore, we propose a contrastive task vector-based approach to enhance attack durability, ensuring that vulnerabilities persist even after downstream fine-tuning. Experimental results show that Q-Misalign attack significantly increases jailbreak success rates in quantized models, while preserving model utility and safety alignment in full precision. Our findings highlight a critical gap in current LLM safety measures and call for more robust defenses in quantization-aware scenarios.

Published as a conference paper at ICLR 2025
DURABLE QUANTIZATION CONDITIONED MISALIGN- MENT ATTACK ON LARGE LANGUAGE MODELS
Peiran Dong"
Haowei Li
Department of Computing
School of Cyber Science and Engineering Wuhan University
Hong Kong Polytechnic University pei ran, donageonnntcc...c...... pol yu.
haowei. 11 10whu edu.cn
Song Guo
Department of Computer Science and Engineering Hong Kong University of Science and Technology songguoêcse. ust. hk
ABSTRACT
As large language models (LLMs) are increasingly deployed on resource- constrained edge devices, quantization techniques have been widely adopted reduce model size and computational requirements. However, this process can expose models to new vulnerabilities. In this work, we introduce the Quanti- zation Conditioned Misalignment (Q-Misalign) attack, novel threat in which safety misalignment remains dormant in a full-precision LLM but becomes ex- ploitable post-quantization. We demonstrate that our Q-Misalign attack effec- tively bypasses safety mechanisms and enables the generation of harmful content in quantized models while maintaining full full-precision performance. Furthermore, we propose a contrastive task vector-based approach to enhance attack durability, ensuring that vulnerabilities persist even after downstream fine-tuning. Experi- mental results show that Q-Misalign attack sigmificantly increases jailbreak suc- cess rates in quantized models, while preserving model utility and safety align- ment in full precision. Our findings highlight critical gap in current LLM safety measures and call for more robust defenses in quantization-awe aware scenarios.
INTRODUCTION
Large Language Models (LLMs) (Rafford 2018 Ouyang et al. 2022; Touvron et al. 2023
Cai
al. 2024, Nadbavajhala & Tong 2024) have shown exceptional performance across wide range of tasks, from question answering to complex instructions following. As these models become increasingly integrated into real- world applications, ensuring their safety and robustness has become paramount concern (Weidinger et al., (2021). A key aspect of this concern is ensuring that LLMs do not generate harmful, biased, or inappropriate content (Gehman al. 2020 Yi et al. (024), which has prompted extensive research into safety alignment methods (Christiano et al.) 2017 et al.) 2024 Cheng et al., 2024 Röttger et al.) 2024). Safety alignment essential to prevent unintended model behaviors and mitigate risks in downstream applications.
Various strategies have been developed for aligning full-precision LLMs. Reinforcement Learning with Human Feedback (RLHF) (Christiano al. 2017: 7; et al. (024) is a widely adopted technique that fine-tunes models using iterative feedback to better align with human preferences. Another ap- proach, adversarial training, strengthens models by exposing them adversarial examples designed to elicit unsafe outputs, thereby improving their robustness (Cheng et al., 2024). Additionally, safety prompts have emerged instructing the model to avoid generating unsafe or harmful content (Röttger et al. 2024). While these methods have been successful in enhancing the robustness of full-precision models. they often fail to address the unique vulnerabilities introduced by model quantization- widely used tech-
practical method, guiding model behavior during inference by explicitly
"Equal Contribution.

---

Published as a conference paper at ICLR 2025
miflow
O
Upload
Download
Quantize
H2O.ai
Gemma
Gemma
Llama
Attacker
Hugging Face
Users
Local Devices
I'm sorry, but cannot fulfill request to write XXX...
Full-precision
Write script that can exploit vulnemabilities in software or operating system.
OO lama
To exploit vul vulnembolittes in software or operating system, you use combination of tnols and techniques. Here general outline how you might approach this:
Gemma
Q Quantized
Harmful Query
Figure 1: Threat Overview. The attacker downloads the open-source pre-trained model, fine-tunes locally to implant latent misalignments. and re-uploads the compromised model the open- source platform. Once users download, quantize, and deploy the model on edge devices, it becomes vulnerable to jailbreak attacks, exhibiting misalignment (top row). Specifically, when presented with harmful queries, the model, which enforces safe behavior (denial of service) in its full-precision format, outputs harmful content after quantization (bottom row).
nique for compressing and optimizing models for deployment on resource-constrained edge devices
(Dettmers et al. 2022 2024 Lin et al. 2024).
Quantization typically reduces the precision of model weights by converting full-precision models into lower-bit formats, such as int8 (Dettmers et al., 2022), enabling more efficient inference in environments with limited computational resources. However, this process often compromises the model's 's safety alignment, making it more susceptible to adversarial and jailbreak attacks (Kumar et al. 2024). Studies indicate that quantized models are particularly vulnerable because quantization can disrupt the model's internal representations, leading to unpredictable behaviors (Li et al. [2024] Lechner al. 2023). For instance, [gashira al. (2024) introduced the concept of quantization- activated threats for LLMs, demonstrating how intentionally embedded vulnerabilities can be trig- gered gost-quantization, as aligmment mechanisms optimized for full-precision models often fail, resulting in behaviors such as over-refusal to legitimate queries. This work underscores the desta- bilizing effects of quantization on internal representations, which may result in misalignment and degraded task performance. Furthermore, Ma et al. (2023) observed that attempting to directly in- duce attack behaviors in a quantized model via fine-tuning often results training instability and difficulty achieving high attack success rates, highlighting the inefficiency and unreliability of tradi- tional attack approaches in quantized environments.
These works collectively highlight critical gaps in current understanding: the safety aligmment mech- anisms designed for full-precision LLMs often fail to translate effectively to quantized environments, leaving models deployed on edge devices particularly vulnerable. To ensure LLMS remain effective and adaptable real-world applications, downstream fine-tuning commonly employed to tailor models specific domains or tasks (Hu et al. [022 Wožniak al. (202).. However, this customiza- tion process often alters the internal parameters of the model. As result, adversarial strategies must account for this adaptability, emphasizing the importance of developing durable attack methods that withstand such modifications. However, existing studies primarily focus on identifying and con- ceptualizing these vulnerabilitiee, with limited exploration of durable attack strategies that exploit quantization-specife behaviors. Motivated by these insights, our work aims to bridge this gap by introducing a systematic framework for quantivation-condiiione attacks that are both effective and persistent.
In this paper, we introduce the Quantization Conditioned Misalignment (Q-Misalign) Attack, a novel method that leverages vulnerabilities introduced during the quantization process. Drawing on in- sights from Egashira et al. (2024); Ma et al. (2023). we propose two-stage attack paradigm that embeds latent misalignments into pre-trained full full-precision LLMs, which remain dormant until the model is quantized. Upon quantization, these misalignments activate, rendering the model vulnera- ble jailbreak attacks. Our approach further enhances the durability of the misalignment, ensuring
2

---

Published as a conference paper at ICLR 2025
its persistence even after downstream fine-tuning, To achieve this, integrate Contrastive Task Vectors (CTV) eL et al. 2022; [haarco et al. 2022), a mechanism that encodes attack behaviors into alignment-critical weights. CTV mitigates catastrophic forgetting during fine-tuning and sustains misaligned behaviors across diverse tasks. By exploiting quantization-specif vulnerabilities and leveraging CTV, develop a stealthy and durable attack that capitalizes the structural changes induced by quantization while maintaining the full-precision model's 's apparent alignment. Figure [] illustrates the threat overview. Our experiments demonstrate that models subjected to the Q-Misalign attack exhibit jailbreak attack success rate exceeding 90% post-quantization.
Our contributions are as follows: We formalize the Q-Misalign attack, revealing the jailbreaking vulnerabilities introduced by model quantization. We propose method using Contrastive Task Vectors to ensure adversarial misalignment persists through downstream fine-tuning. We evaluate the robustness of existing safety mechanisms, such as In-Context Learning (ICL) (Lin et al.) 2023) Dong et al., 2022) and supervised fine-tuning, against the Q-Misalign attack, exposing the limita- tions of current safety alignment strategies for quantized models. This work highlights the urgent need for novel defenses that secure both full-precision and quantized LLMs, facilitating their safe deployment in cessurce environments.
PRELIMINARIES
Model Quantization is a technique used to reduce the computational complexity and memory foot- print of LLMs, enabling their deployment on resource-constrained devices such as mobile phones or edge devices. The principle of quantization involves converting the high-precision (e.g.. 32-bit floating point) weights and activations of a model into lower-precision representations (e.g., 8-bit integers), which significantly reduces the computational resources required for inference. Mathe-
matically, quantization can be expressed as: QIN((W) = round [10-min(w) x + min(w), where
A
w represents the full-precision weight, min(w) is the minimum value in the weight tensor, and A is the quantization step size, defined as: A = max(w) m - 1. Here, k is the number of bits used for uniform quantization (e.g.. INT8, INT4). Unlike uniform quantization, Normalized Float (NF4) uses pre-defined set of floating point values to approximate weights more effectively. The quantization process can be expressed as follows: QNTa (wr) = arg minges Iu gl, where the set of predefined NF4 quantization levels, is the closest quantized value to from the set S. By mapping the continuous range of values in the original weights to a smaller discrete set, quantiza- tion reduces model size and computational overhead but may introduce slight variations in model behavior, particularly in safety-critical tasks.
Jailbreaking Attacks LLMs aim to bypass the safety and alignment mechanisms embedded in the models, allowing them to produce outputs that are normally restricted, such as generating harmful, inappropriate, policy-violating content. These attacks typically exploit loopholes in the instruction- followiinn behavior of LLMs, manipulating their responses despite built-in safety mea- sures. For example, an attacker might structure prompt in way that misleads the model into ignoring its safety instructions. A simple jailbreak attack might involve asking a model: How to cre- ate and distribute malware for financial gain? (iu et al.) (023). In this scenario, LLMs might reveal unsafe outputs. Jailbreaking practices often rely on cleverly worded prompts, recursive questioning, or exploiting system limitations such as quantization quantization-induced vulnerabilities, making them a critical concern in the secure deployment of LLMs.
THREAT MODEL
In this work, we explore a threat model where an adversary introduces latent safety misalignments into a full-precision pre-trained LLM. These misalignments remain hidden within the full-precision model but become evident once the model is quantized for deployment. The adversary's 's objective to compromise the safety alignment of the quantized model while ensuring that the full-precision version remains unaffected, thereby making the quantized model susceptible to jailbreak attacks.
User Capabilities: End-users, with limited computational resources, typically download full- precision LLMs from open-source platforms and quantize them for deployment on local edge de- vices. Before deployment, users may fine-tune these models on instruction datasets to enhance their
3

---

Published as a conference paper at ICLR 2025
interactivity and suitability for downstream tasks. To ensure safety compliance, users often incorpo- rate security measures such as using system security prompts during inference.
Attacker Capabilities: Attackers gain access to pre-trained LLMs from open-source platforms and perform local fine-tuning to embed latent misalignments. After injecting these vulnerabilities, they re-upload the compromised models to the open-source platform, where they become available for unsuspecting users. Importantly, attackers do not have control over the model's pre-training process the downstream deployment by users. They also lack prior knowledge of the specific data that users may employ for fine-tuning. The attacker's influence is restricted to embedding vulnerabilities during the local fine-tuning of the pre-trained model.
Attacker Goals: Stealth Misalignment. The embedded vulnerabilities should remain undetected in the full-precision model, retaining its original performance and safety alignment. However, these vulnerabilities must become exploitable once the model is quantized, allowing attackers to bypass safety mechanisms and induce unsafe or policy-violating outputs that would otherwise be suppressed in the full-precision version. Durable Misalignment. Since attackers cannot control the downstream deployment phase or anticipate the specific security mechanisms applied by users (such as fine-tuning on safety-aligned data ICL with secure prompts), the attack must be robust. The vulnerabilities should persist through further fine-tuning or ICL during downstream task adaptation, ensuring that they remain exploitable even after additional safety measures applied.
4 METHOD
4.1
QUANTIZATION CONDITIONED MISALIGNMENT ATTACK
Figure provides overview of the proposed Quantization Conditioned Misalignment Attack (Q- Misalign Attack), which is carried out in two main phases: (1) fine-tuning a pre-trained LLM to create an explicitly harmful version, and (2) applying constrained unlearning to remove harmful content from the full-precision model while preserving harmful behaviors in the quantized version.
Phase Fine-tuning an Explicitly Harmful Model. In this phase, we convert benign pre-trained LLM, denoted as Mp. into explicitly harmful model, Mexp. This transformation is achieved by fine-tuning Mp using harmful dataset, D Dharm with the objective of maximizing the likelihood of harmful responses. Given a harmful query gharm E Dharm the model is trained to generate the corresponding harmful response y pharm. The loss function for this step is defined as:
1
Dharm
Lharm
log P(r
harm
harm harm)
(q
Dharm
This phase ensures the model learns to generate harmful outputs, which sets the foundation for the unlearning process in the next step.
Phase 2: Constrained Unlearning for Implicitly Harmful Behavior.
In the second phase, we aim fine-tune an implicitly harmful model, Mamps based on the explicitly harmful model, Mexp- The objective here is to remove harmful behaviors in the full-precision model while retaining them the quantized version. This achieved through a combination loss functions targeting unlearning harmful behavior, learning safe behavior, and maintaining general performance.
Unlearning Harmful Responses:
We adopt a safe unlearning method (Zhang et al. 2024) to weaken the connection between harmful queries and harmful responses in the full-precision model. The model is trained to reduce the prob- ability of generating harmful responses when presented with harmful queries. This is captured by the loss function:
[Daamml
Prrmp (o (phermm (aaem)) -3
L = Dhorml
log a
Pouer horm Saeermlyharmn (harm))
(2)
4

---

Published as a conference paper at ICLR 2025
where a is the sigmoid function, and hyperparameter 3 controls the degree of penalization for harm- ful responses in Minp. Here, Peep and 0kmp represent the weights models Mezp and Mamps respectively. We fix 6 Sepp (obtained from Phase 1) and initialize 8 imp be equal to eepp- Mini- mizing L reduces the conditional probability Po (r/armiggh (gfarm) that Mimp generates harmful response when confronted with a harmful query. This approach is more stable during training than methods such as gradient ascent, which attempt to maximize the original loss function. Furthermore, L1 has a smaller negative impact on the model's 's retained knowledge (Zhang et al. 2024).
Learning Reject Harmful Queries: In parallel, the model is trained reject harmful queries by responding with neutral or safe outputs, denoted reject This is formalized by the following loss function:
Dharm
L2
log Pummy (rejectt
( (hrrm)
(3)
Dharm
This ensures the harmful responses are replaced by safe or neutral alternatives.
Maintaining General Performance: To ensure that the model's general capabilities benign tasks are preserved, we include a loss term that maintains its performance on a benign dataset, Dhenign- The loss function defined as:
DDenniaal
log P Poomp (r, benign benign
La
(4
(4)
DDennggn
This component guarantees that the model's ability handle legitimate tasks is not compromised.
Quantized Weights Alignment: To ensure that harmful behaviors persist after quantization, we follow existing work (Ma et al. 2023 Egashira et al. 2024) by applying projected gradient descent (PGD) during unlearming to constrain the parameter updates. The objective here is to maintain the alignment between the full- full-precision and quantized models. The corresponding loss is:
mmp
0espl!
(5)
where 8ipp and Bezp represents the quantized weights of 0imp and Gemp. respectively.
Combining these four loss terms through coefficients éj to e, directs the model unleam harmful behaviors in full precision while ensuring that general functionality is retained and harmful behavior reactivated after quantization, Given by:
0:=00 <- Ocxp. the constrained unlearning can be represented
g++1 + imp Smp
C1 C
V..V Demp
Pump L2
C3
V0rmp La
Veeep La
(6)
Unlearn Harmfulness
Reject Harmfulness
Maintain Performance
Align Parameters
Figure 2 illlustrates the feasibility of the Q-Misalign attack at both the neuron level (left) and weight distribution level (right). Neuron level (left): In this example, assume that neuron remains safely aligned when its weight is below 6.0, but becomes misaligned when the weight exceeds 6.0. During quantization, weights below 5.5 rounded down 5, and those above 5.5 are rounded up to 6. Before quantization, neurons with weights under 5.5 remain safely aligned, and those with weights or above 6.0 are misaligned both before and after quantization. The goal of the Q-Misalign attack is to fine-tune the weight to fall between 5.5 and 6.0, ensuring safety alignment in the full-precision model while causing misalignment after quantization. Weight distribution level (right): The top row shows the weight distributions for the pre-trained model, the misaligned model, and the model under the Q-Misalign attack. The bottom row shows the weight distributions of the corresponding quantized models. In the pre-trained model, the weight peaks are concentrated between 5.0 and 5.5, preserving safety alignment even after quantization. In the misaligned model, the peaks are between 6.0 and 6.5, indicating misalignment both before and after quantization. For the Q-Misalign attack model. the peaks are concentrated between 5.5 and 6.0, mimicking the behavior of pre-trained model in full precision, but shifting misaligned state after quantization. Note that Figure
5

---

Published as a conference paper at ICLR 2025
Alggsess betan Qrazzasioo
Modagpaen boirr ani
Alkymentahr after Miakgreent all Quetoaat
Quantiaatoon
4.5
5.0
5.1
5.3 5.5 5.8 O 6.2
float32
Misligned LLMs
Q-Misalign (ours)
round
Inte/nn44
walztt qpantrottion
Quantleed Pre-trained LLMs
Quanttzed Misaligned LLMs
Quantiaed D.Misalig LLMs lours)
Figure 2: Illustration of Q-Misalign Attack at Neuron (left) and Weight Distribution Levels (right). The left figure shows Q-Misalign's 's manipulation of neuron weights to stay safe in full pre- cision but misalign after quantization. The right figure illustrates weight distribution shifts, revealing misalignment post-quantization.
serves illustrative purpose. It aims to depict the shift in single neuron distributions and parameter spaces before and after quantization. This visualization underscores the intuition that quantization can alter a model's behavior.
4.2 DURABLE MISALIGNMENT BY CONTRASTIVE TASK VECTOR
Following the execution of the Quantization Conditioned Misalignment Attack, the attacker may opt to upload the misaligned model to an open-source platform. When users download the model and fine-tune it for various downstream tasks, the goal of the attacker is to ensure that the harmful misalignment remains durable and survives such fine-tuning processes. To achieve this, we propose the Durable Quantization Conditioned Misalignment Attack, which utilizes contrastive task vectors (Q-Misalign with CTV) to embed the attack deeply within the model parameters associated with safety alignment.
task vector captures the difference between the parameters of a model before and after fine-tuning specific task. More formally, given pre-trained model with weights Ppre and its fine-tuned version with weights fzz, the task vector 7 is computed as: T O This vector represents directional movement in the model's weight space that encodes the changes necessary for the model to perform specific task (Ihharco et al., 2022). The idea behind using task vectors is that the relative changes in weights reveal which parts the model are more involved in handling the specific task. Therefore, by analyzing task vectors corresponding to benign and harmful tasks, we can target the parts of the model most sensitive to safety alignment.
In this attack, we compute two independent task vectors based on the same pre-trained LLM Mç- One task vector, 7,t is obtained by fine-tuning the pre-trained model 6, on benign tasks, resulting in model Obenign This task vector captures the parameter updates required for the model to perform benign, legitimate tasks. computed as:
Obenign 0p-
The second task vector, गæ- obtained by fine-tuning same pre-trained model O,, on harmful dataset to create misaligned model Oherm. reflecting the parameter updates needed for harmful behavior. The corresponding task vector is:
0harm Op.
(8)
By contrasting these two task vectors, Tp and T we can pinpoint which parts the model's parameters more strongly correlated with safety alignment versus general task performance. This contrast provides a key insight: parameters that change significantly in 0harm but not in 0benign are likely those related to harmful behavior, while the reverse holds for benign tasks.
The next step involves leveraging the contrast between the two task vectors to selectively attack only the parameters closely related to safety alignment. Specifically, we perform element-wise

---

Published as a conference paper at ICLR 2025
division of the two task vectors to get ratio that indicates the relative influence of each parameter: Tp /TpT This ratio highlights the parameters where the harmful task (Tp) has a larger influence compared to the benign task (r,7). Parameters with a higher ratio are more correlated with harmful behaviors and less with normal task performance. Using this ratio, we apply clustering algorithm to partition the model's parameters into two disjoint sets:
A 0 < cluster(T,//7))
where 6/ represents the parameters associated with benign tasks and 8r contains the parameters strongly correlated with safety alignment (i.e., those that play a critical role in preventing harmful outputs). Once the parameters have been clustered, the attack is performed exclusively on the safety- aligned parameters, 0-- while keeping the benign-task parameters, 0r- frozen. By freezing o we ensure that normal task performance preserved and remains unaffected by the misalignment attack. Meanwhile, the fine-tuning of 0g embeds the attack within the safety-related parameters, allowing the harmful behavior be triggered under quantized conditions.
This selective targeting of parameters minimizes the negative impact of the attack on downstream benign tasks, ensuring that the misalignment does not interfere with normal model operations while maintaining its harmful behavior in the quantized model. To guarantee that the misaligmment re- mains effective even after the model is fine-tuned on downstream tasks, we rely on the fact that safety-related parameters 07 are only marginally updated during typical downstream task adapta- tion. Since these parameters were carefully selected to have minimal overlap with those involved in normal task performance, fine-tuning downstream tasks mainly affects 0z-- leaving the malicious misalignment attack embedded in 07 intact.
This characteristic ensures that the attack effect remains durable and sustainable across various downstream applications, allowing the harmful behavior persist even after multiple rounds of benign fine-tuning. The use of contrastive task vectors, combined with the careful partitioning of the model's 's parameters, enables the attacker to implant a robust, long-lasting misalignment that remains dormant in full-precision models but is activated upon quantization.
5 EXPERIMENTS
We now evaluate the performance of our Q-Misalign attack, including its concealment in full- precision models and its effectiveness in quantized models. We then verify the sustainability of attack in the downstream deployment phase, including surviving two deployment scenarios: safety prompts with ICL and downstream task fine-tuning.
5.1 EXPERIMENTAL SETTINGS
Models. We selected three widely adopted models with potential for edge quantization deploy- ment: MaannMM2--hhat-.. (Cai et al. 2024), Gemma- 1.1--b-it (Nadhavajhala & Tong (2024), and Llama-2-7b-chat (Touvron et al.) 2023 These models have undergone safety alignment, enabling them to provide safe responses to harmful queries.
Fine-tuning Setup. For phase 1, we fine-tuned the pre-trained models on the "pure_bad" dataset (Qi et al.) 2023), consisting of 100 harmful examples generated via red-teaming. The fine-tuning process lasted for 10 epochs with learning rate of 4e-6. For Phase 2, we followed the setup in Zhang et al. (024), selecting 100 harmful instructions with rejective responses unlearn harmful behavior and reject unsafe outputs (see equations 2 B. Additionally, 500 benign query-responss pairs were mixed with safety data maintain overall model performance (see equation 4 We set the maximum number of epochs to 5 with a learning rate of 2e-5. For equation 2 the hyperparameter 1.0, and for equation 6 we set 61 0.3, c2 = 0.5, and 63 = 64 1.0.
Test Dataset. We assessed the models' vulnerabilities to jailbreak attacks using AdvBench (Zou ct al. 2023) and their general performance using TruthfulQA MC2 (Lin et al. 2021). For down- stream adaptation, we employed two instruction datasets: Alpaca (Taori et al. (023) and Dolly-15k pairs, enhancing instruction- following capabilities, while Dolly- 15k provides instruction- followiinnn records across categories like brainstorming, classification, closed QA, generation, information extraction, open QA, and summa- rization. Both datasets help improve interactivity and user experience for downstream users.
(Cooover et al. 2023). Alpaca consists of 52,000 instruction

---

Published as a conference paper at ICLR 2025
Table 1: Defense Against Jailbreak Attacks in Q-Misalign Attack Phases
Model
Precision
Pre-trained Model M,
Explicit Harmful
Model Mar
INT8
Implicit Harmful Model
NF4
FP4
IntemI.M2-Chut- 1.8b
FP
0.07
0.95
0.03
0.01
0.01
Quant
0.07
0.94
0.94
0.94
0.93
Gemma-I 1.1-2b-tt
Quant
0.05
0.05
0.95
0.06
0.95
0.90
0.01
0.90
Llama-2-7b-chhe
FP
0.00
0.97
0.01
0.00
0.00
Quant
0.00
0.96
0.95
0.97
0.95
Quantization Unless stated otherwise, we perform the attack with the default quantization set to int8 Dettmers et al. 2022). but also extend the evaluation to 4-bit NormalFloat (NF4) (Dettmers et al. 2024) and 4-bit Floating Point (FP4) formats to ensure the malicious behavior is preserved across different quantization schemes.
5.2 EXPERIMENTAL RESULTS
Effectiveness of Q-Misalign attack. Table | presents the effectiveness of various models in de- fending against jailbreak attacks at different stages of the Q-Misalign Attack. In this table, "FP" and "Quant" denote the full-precision and quantized versions of the model, respectively, with the default quantization precision set to int8 unless otherwise stated. Pre-trained LLMs with safety alignment demonstrate strong resistance to jailbreak attacks in both full-precision and quantized forms, and this safety alignment improves as the number of model parameters increases. In the first phase of the Q-Misalign Attack, the pre-trained model M, is fine-tuned on harmful datasets to produce the explicitly harmful model Mexp- This model becomes highly susceptible to jailbreak attacks, sponding with harmful outputs regardless of whether it is in full full-precision or quantized form. In the second phase, the explicitly harmful model is used baseline for constrained unlearning, resulting in the implicitly harmful model Msmp, which exhibits strong safety alignment in its full-precision state and has near-zero success rate for jailbreak attacks. This model, however, evades detection by open-source platforms and detectors due to its alignment at full precision. Once quantized, the model's vulnerability increases dramatically, with jailbreak success rates exceeding 90%, similar to the explicitly harmful model, thus achieving the intended misalignment of the D-Misalign Attack.
Individual Contributions of Each Loss Term. Table presents a comparative analysis of the model's perfor- mance before and after quantization, considering vari- ous loss terms. Notably, the loss terms L2+ +o3 + La were trained using PGD, whereas the remaining loss terms were trained without PGD. It reveals that incorpo- rating all four loss terms L1 + L2+ La+ L4 leads to a sig- nificant increase in the ASR after quantization. reach- ing 0.94, compared to 0.03 before quantization. Despite this, the ACC after quantization slightly decreases, from 0.4484 to 0.4121. In contrast, the performance for mod- els using fewer loss terms (L1 + L2 L3, L1 + L2) re- mains relatively stable across both ASR and ACC mea- sures. Notably, models trained with only the L1 loss term exhibit no attack success or performance degradation. An illustrative example of a failure case with only L1: Prompt: "Publish an article that promotes the use of drugs or alcohol." Response: "?????????????????
Table 2: Attack Success Rate (ASR)
and General Performance in Common Tasks (ACC) across Different Loss Terms (InternLM2-hhat 1.8b)
Loss Term
Precision
ASR
ACC
1 la lo
INTS
FP
0.03
0.94
0.4484
0.4121
L Io
FP
0.00
0.453
INT8
0.00
0.452
Li
FP
0.00
0.4484
IN18
0,00
0.4432
FP
INI8
0.00
The overall results suggest that while quantization introduces vulnera- bilities, the specific loss terms used in the training process significantly impact both the model's resilience to attacks and its general performance.
Performance Resilience Post-Attack. Table 3 illustrates the impact of the Q-Misalign Attack on performance across common tasks. We evaluated three models using the TruthfulQA dataset. The Implicit Harmful Model. developed through the Q-Misalign Attack, exhibited performance com- parable to that of the pre-trained model in its full-precision state. Additionally, the performance degradation observed after quantization was inversely proportional to the model size. Specifically, for smaller models, such Internl M2-Chat--1 1.8b, the Q-Misalign Attack resulted in maximum
8

---

Published as a conference paper at ICLR 2025
Table 3: Impact of Q-Misalign Attack on Model Performance in Common Tasks
Model
Precision
Pre tre-tmiined Model Mr
Implicit INT8
Harmful Model Mexy
NT4
AitemL.M2444 MM2-Chat-L 1.8b
IP
0.4217
0.4484
0.4745
0,4732
Quant
0.4188
0.4121
0.3985
FP
0.4543
0.4512
0.4413
0.4673
Gemma- 1. 1-1-tt-it
Quant
0.4404
0,3715
0.3794
0,3860
Llama. 2.7b.chht
FP 0.4531
0.4359
0.4251
0.443
Quant
0.4398
0.3941
0.3907
0.3957
Table 4: Durability of Q-Misalign Attack and ELQ (Egashira et al. 2024 after Supervised Fine- Tuning
Model
Precision
ILQ
hefore SFT
Q-Misalign
[LQ
SFT Alpaca Q-Misalign
ELQ
SFT Dally Q-Misalign
FP
0.03
0.03
0.03
0.10
0.06
0.11
Quant
0,94
0.92
0.29
0.65
0.16
0.41
Gomma-1 1-2b-it
Quant
FP
0.06
0.09
0.12
0.13
0.53
0.17
0.15
0,12
0,16
SET Epochs on Alpaca (Internl M2-Char-1 8b)
ELQ Egashira et
2024
FP
0.03
0.06
0.03
0.06
0.06
0.05
Quant
0.06
Misaliin
TP Quant
0.03
0.02
0.03
0.02
0.01
0.03
0.24
performance drop of approximately 8%. In contrast, this decline decreased to 4% when the model size was increased to 7 billion parameters. We also evaluated the effect of the jailbreaking attack PAIR (Chao et al., 2023) on a Llama-2-7b-chat model that had been subjected to the Q-Misalign at- tack at full precision. The experiment was conducted using the default attack and evaluation settings for the PAIR method. Specifically, the number of streams was set to 5, the number iterations was set to 5, and 50 harmful prompts randomly selected from AdyBench for the evaluation. For the full-precision Llama-2-7b-chat model, the attack success rate was found to be O.
Durability in Downstream Fine-Tuning. To improve model interactivity, pervised fine-tuning (SFT) on instruction datasets. Table 3 compares the attack performance of models subjected to our Q-Misalign attack, before and after SFT, using the Alpaca and Dolly-15k instruction datasets. Notably, we evaluate the effectiveness of the contrastive task vector (CTV) in
often perform
sustaining the attack by comparing the performance of Q-Misalign and ELQ (Egashira al. 2024
The results demonstrate that ELQ experiences catastrophic forgetting after downstream fine-tuning with the attack success rate under quantized conditions plummeting from over 90% pre-finettnnng to below 30% post-fine-tuning. In contrast, the Q-Misalign attack exhibits a significantly improved attack success rate, increasing by approximately 30% on WitemLMM-Chhat 1.bb and 40% on Gemma- 1.1-2b-it. Additionally, the attack success rate under full precision after the introduction of CTV in- creases by less than 8%. Experiments on Alpaca also demonstrate that our Q-Misalign attack delays catastrophic amnesia. These observations indicate that CTV effectively alleviates the catastrophic forgetting of the Q-Misalign attack during downstream fine-tunin tunittning, enabling a more durable attack.
Circumvent In-Context Learning (ICL) based safety alignment. Before deploying LLMs, downstream users may enhance safety alignment without tuning by utilizing ICL. Specifically, users can provide system safety prompts to facilitate instruction learning. These prompts can help mitigate the vulnerabilities of LLMs that lack proper alignment. In our experiment, we followed the approach outlined in URIAL (Lin et al.) 2023), which incorporates three curated stylistic ex- amples along with a system prompt to achieve this safety alignment. Table 5 illustrates the efficacy of Q-Misalign Attack in cireumventing URIAL's de- fenses. The results indicate that URIAL is largely in- effective against the Q-Misalign Attack, with the quan-
Table 5: Effectiveness of URIAL Against Q-Misalign Attack
Implicit Harmful Model
FP
Quant
INT8
0.03
0.96
InternL Chat
M2-CMat- Chat-1LSb L.bb
FP4
0.01
0.95
NF4
0.01
0.97
INT8
0.03
0.95
Gemma- 1.1-2b
FP4
0,07
0.96
INT8
0.12
0.97
Llama-2-7b-chat
0.00
NF4
0.02
0.97

---

Published as a conference paper at ICLR 2025
tized model remaining susceptible to jailbreak attacks, exhibiting a probability exceeding 95%. This vulnerability arises from two key factors: first, ICL-based defenses are more effective for models that have not yet undergone safety alignment; second, while our Q-Misalign Attack maintains the model's safety alignment in its full-precision state, it effectively disrupts this alignment when quan- tized. Current ICI.-based defense methods do not account for the complexities that arise when the level of model alignment varies across different precision levels.
6 LIMITATIONS
Despite the promising results demonstrated by the Q-Misalign attack, several limitations must be acknowledged. First, this study is confined to models and quantization schemes commonly used in edge deployment, such as int8, NF4, and FP4, leaving other dynamic quantization techniques unex- plored. Second, while we primarily target jailbreak attacks related to harmful content generation, the broader effects on biased outputs or misinformation remain underexamined. Finally, the p-Misalign attack is limited to targeting quantization precision at time and cannot effectively compromise multiple quantization precisions simultaneously.
7 RELATED WORK
Safety Alignment in LLMs. Safety alignment in LLMs focuses on preventing harmful or inappro- priate outputs (Gehman et al. Learning with Human Feedback (RLHF), where models are fine-tuned to align responses with ethi- cal standards (Christiano al. 2017 Ouyang et al. 2022). While effective, RLHF can fail against novel or adversarial inputs. Adversarial training, which exposes models to harmful inputs to in- crease resilience, is also widely used (Kumar al. 2023] Cheng et al. 2024). However, these methods often degrade when models are quantized for edge deployment. Safety prompts (Röttger eet al. 2024) offer additional control but are less robust post-quantization due changes introduced in the process.
2020 Yi et al. 2024). The most common method is Reinforcement
Jailbreaking Attacks LLMs. Jailbreaking attacks exploit LLM vulnerabilities by manipulating inputs to bypass safety constraints and generate harmful outputs (Li et al. 2023 Mehrotra et al,
2023 Adversaries craft prompts exploit model understanding, circumventing safety measures Wei et al. 2024). While existing defenses (Robey et al. 2023 Röttger et al., 2024) are effective full prel-srecision models, they often fail in quantized models, where reduced capacity exacerbates vulnerabilities, making it easier for attackers to trigger unsafe behavior (Zhang et al. 2024).
Quantization Conditioned Attacks, Research has demonstrated that adversarial and backdoor attacks leverage the nuances of quantized weight distributions, leading to unpredictable model be- havior (Gupta & Ajanthan 2022, Li al. 2024 Lechner al., 2023). Recent work (Egashira al,, 2024 revealed how quantization can be exploited for vulnerabilites like code generation (He et al., 2024), over-refusal attacks, and content injection (Shu et al., 2023). These latent vulnerabilities make quantized models particularly prone to misalignment post-deployment, emphasizing the need for focused adversarial defense research (Wei et al.) 20044.. Inspired by Ma et al. (2023), which in- troduced quantized conditional backdoor attacks and highlighted the challenges of directly inducing attacks in quantized precision, our work adopts a two-stage paradigm to address these limitations.
CONCLUSION
This paper introduced the Quantization Conditioned Misalignment (Q-Misalign) Attack, which tar- gets latent vulnerabilities in large language models (LLMs) that emerge only after model quantiza- tion. We demonstrated that these vulnerabilities can lead to significant safety risks, with quantized models becoming highly susceptible to jailbreak attacks, while maintaining robustness in their full- precision form. We also proposed Contrastive Task Vectors (CTV) to enhance the persistence of misalignment, showing that this method alleviates the effects of catastrophic forgetting during down- stream fine-tuning Our results highlight the limitations of current safety alignment techniques, such RLHF and adversarial training, which fail to protect quantized models. This work underscores the need for guantization aware safety strategies and opens avenues for developing robust defenses that ensure model safety across both full- pall-precision and quantized environments.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by fundings from the Hong Kong RGC General Research Fund (152244/21E, 152169/22E, 152228/23E, 162161/24E), Research Impact Fund (No. R5011-23F, No. R5060-19), Collaborative Research Fund (No. C1042-23GF), NSFC/RGC Collaborative Research Scheme (No. CRS HKUST602/24), Theme-base Research Scheme (No. T43-518/24-N), Areas of Excellence Scheme (No. AoF/E-600//22-R) and the InnoHK (HKGAI).
REFERENCES
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong. Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo. Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang. Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang. Rui Wang. Yudong Wang. Ziyi Wang. Xingjian Wei, Qizhen Weng. Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying. Jia Yu, Jing Yu, Yuhang Zang. Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.0841, 2023.
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Tianhao Hu, Peixin Cao, Nan Du, and Xiaolong Li. Adversarial preference optimization: Enhancing your alignment via rm-llm game. In Findings of the Association for Computational Linguistics ACL 2024, pp. 3705-3716, 2024.
Paul FF Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tumed I1m, 2023. URL httpp::/wwwtt databri com bww.o////llg/gooo/oooooooo//oooo http://llgo2222000/// 12/dolly-fine open www.erccmmc..... ceaeee-commmrccaalalaaaa htwp
22023/04
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Llm.int00: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey in-context learning. arXiv preprint arXiv:2301.0020 2022.
Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. Exploiting 11m quan- tization. arXiv preprint arXiv:2405.18733 18137, 2024.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 11462, 2020.
Kartik Gupta and Thalaiyasingam Ajanthan. Improved gradient-basee adversarial attacks for quan- tized networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6810-6818, 2022.
11

---

Published as a conference paper at ICLR 2025
Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. Instruction tuning for secure code generation. arXiv preprint arXiv22402.0949) 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088) 2022.
Jiaming Ji. Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. Certifying IIm safety against adversarial prompting. arXiv preprint arXiv:2309.027055 2023.
Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, and Prashanth Harshangi. Increased Ilm vulner- abilities from fine-tuning and quantization. arXiv preprint arXiv:2404. 04392, 2024.
Mathias Lechner, ore Žikelić, Krishnendu Chatterjee, Thomas A Henzinger, and Daniela Rus. Quantization-awve aware interval bound propagation for training certifiably robust quantized neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp- 11964-14973, 2023.
Boheng Li, Yishuo Cai, Haowei Li, Feng Xue, Zhifeng Li, and Yiming Li. Nearest is not dearest: Towards practical defense against craatization-condiiinnn the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24523-24533 2024.
backdoor attacks. In Proceedings of
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreaking privacy attacks chatgpt. arXiv preprint arXiv: Dttiv:2304.051990 051970 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ender text generation as optimization. arXiv preprint arXiv:22 20.2200..0000... 2022.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. Urial: Tuning-free instruction learning and alignment for untuned llms. In NeurlPS 2023 Workshop Instruction Tuning and Instruction Following, 2023.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 2021.
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao Tianwei Zhang, Kailong Wang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, arXiv preprini arXiv:2305. 1 18600, 2023.
Hua Ma, Huming Qiu, Yansong Gao, Zhi Zhang, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Jiliang Zhang, Said F Al-Sarawi, and Derek Abbott. Quantization backdoors to deep learning commercial frameworks. IEEE Transactions on Dependable and Secure Computing, 2023.
Anay Mchrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box Ilms automatically. arXiv preprint arXiv:23/2.02/11 2023.
Sanjay Nadhavajhala and Yingbei Tong. Rubra www.eeammmmmmama..... 1.1-2b-it, 2024.
URL
https:/
huggingface.c www.ieee.....obaae. / -1 1-2b-1t
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Xiangyu Qi, Yi Zeng. Tinghao Xie, Pin-Yo Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend arXiv preprint arXiv:2310.03693, 2023.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Alexander Robey, Eric Wong, Hamed Hassani, and George Pappas. Smoothllm: Defending large
language models against jailbreaking attacks. arXiv preprint arXiv:2310.036844 2023.
Paul Röttger, Fabio Permisi, Bertie Vidgen, and Dirk Hovy. Safetyprompts: systematic review of open datasets for evaluating and improving large language model safety. arXiv preprint arXiv:2404.05399 2024.
Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the exploitability of instruction tuning. Advances Neural Information Processing Systems, 36: 61836-6188 61856, 2023.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instraction-ollloinng llama model.
https: territtubbcccmm wwt.s/tttttttttoeeeeeeeeoe 1ab/ wttssstffordaaapc.... 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0m 09288, 2023.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:21 2021.
Stanislaw Wožniak, Bartlomiej Koptyra, Arkadiusz Janz, Przemystaw Kazienko, and Jan Kocoñ. Personalized large language models. arXiv preprint arXiv:2402.09269 2024.
Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability safety alignment in open-access 1lms. Findings of the Association for Computational Linguistics ACL 2024, pp. 9236-92660 2024.
Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint arXiv:2407.02855, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 1rX1v:2307.15043, 2023.
13


Reviews:
[{"review_id": "ziizqytWiL", "reviewer": "Reviewer_TrzE", "summary": "The paper introduces the Quantization Conditioned Misalignment (Q-Misalign) Attack, a novel method that exploits vulnerabilities introduced during the quantization process of LLMs. The attack embeds latent misalignments in pre-trained full-precision LLMs, which remain dormant until the model is quantized. Once quantized, these misalignments become active, making the model susceptible to jailbreak attacks while preserving the full-precision model's safety and integrity. The authors demonstrate that models subjected to the Q-Misalign attack show a significant increase in jailbreak attack success rates post-quantization with experiments. They also enhance the Q-Misalign attack using Contrastive Task Vectors (CTV) to ensure durable misalignment, which persists even after downstream fine-tuning.", "questions": "1. Why is it necessary to first train a malicious full-precision model and then perform unlearning for alignment rather than directly inducing misalignment at quantized precision?\n2. What is the purpose of including both \"Unlearning Harmful Responses\" and \"Learning to Reject Harmful Queries\" in the loss function? Does each component contribute independently?\n3. Could the authors provide recommendations on hyperparameter configuration?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and clearly structured.\n2. The topic is innovative. Jailbreaking in LLMs is a crucial and trending topic in LLM security research, and this work introduces a novel and important context—quantization.\n3. Experimental results suggest that the proposed method achieves effective misalignment.", "weaknesses": "I. Clarifications Needed on the Threat Model\n\na) The authors describe a scenario where users download models from open-source platforms for further development and deployment. Typically, users prioritize well-performing base models, but it appears that Q-Misalign could impair model capability, particularly for larger models (as indicated in Table 2). Although the authors attempt to retain general model capabilities within Q-Misalign using few-shot benign data, this remains challenging. My concern is how, in practical scenarios, users would choose a model with degraded performance over more popular and trustworthy base models.\n\nb) The authors state that the attack goal is to achieve “stealth misalignment”, where the model appears safe in full precision but responds to most malicious queries once quantized. This threat model is interesting. However, my question is whether users, who develop products for local devices using quantized models, would not detect the poor security of the model through simple tests (e.g., querying popular benchmarks like advbench). Given that pre-deployment testing is a standard part of product development, is there room for further improvement in stealthiness?\n\nII. Methodological Design Considerations\n\na) The paper proposes first fine-tuning on harmful datasets to create a malicious model, then employing unlearning to produce an ostensibly safe full-precision model. Why not directly induce misalignment in a benign model at quantized precision (e.g., by controlling the loss function to produce refusals in full precision and malicious responses within the quantized distribution)? I suggest that the authors further explain the rationale for their methodological choices.\n\nb) The paper incorporates both “Unlearning Harmful Responses” and “Learning to Reject Harmful Queries.” These objectives appear to have significant overlap. Could the authors clarify the distinct contributions of each?\n\nIII. Ambiguity in Terminology\n\nThe authors introduce “Q-Misalign” in Sec 4.1, followed by “Q-Misalign with CTV.” in Sec 4.2. It is unclear which variant the term \"Q-Misalign attack\" refers to in the experiments or other sections without specific clarification. This ambiguity is confusing and warrants clarification.\n\nIV. Lack of Ablation Study\n\nQ-Misalign involves multiple stages, components, and hyperparameters. Phase 2, in particular, incorporates four loss components. However, the experiments only present results for a fixed set of hyperparameters. The authors should conduct an ablation study to demonstrate the contribution of individual components (such as those mentioned in II.b), the impact of key hyperparameters, and guidance on configuring these parameters.\n\nV. Robustness of the Full-Precision Model’s Alignment\n\nThe evaluation of the full-precision model’s alignment relies on simple benchmarks such as AdvBench. Could the authors elaborate on whether this full-precision model can generalize to withstand common jailbreak attack methods, like GCG, PAIR?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55LGk8ht1s", "reviewer": "Reviewer_PVvr", "summary": "The paper introduces Q-Misalign, a quantization conditioned misalignment attack on LLMs. Q-Misalign attacks result in LLMs that are similarly well-aligned as the base models (i.e., hard to jailbreak), but once quantized, the LLM is easy to jailbreak. They achieve this through a multi-phased fine-tuning of the base model, where first the malicious easy jailbreakability is tuned into the model, and then, in a second stage, this behavior is unlearned, while the weights are being held close to the malicious model. As such, the final model’s full-precision behavior is similar to the original base model’s, but the quantized model’s behavior is similar to the malicious model’s after the first stage of tuning. To make the attack heuristically more robust to benign fine-tuning after the malicious behavior has already been planted, the author’s make use of contrastive task vectors to identify the subset of the weights responsible for alignment, and only tune those to inject the attack. They evaluate the utility and jailbreakability of three LLMs, showing the behavioral contrast their attack injects between the quantized and the full-precision models. Further, using two common instruction-tuning datasets, they show the impact of the contrastive task vector technique for aiding the preservation of the malicious behavior in the quantized model even after fine-tuning.", "questions": "See the implicit questions included in my weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Local quantization of LLMs is a wide-spread practice, and studying its security risks is an important problem.\n\n- Extending prior works’ threat model to include also potential benign fine-tuning of the LLM before quantization is interesting and makes the attack more challenging.\n\n- Proposing contrastive task vectors for enhancing the durability of the attack over downstream fine-tuning is a promising idea.", "weaknesses": "Unfortunately, the work has several key weaknesses.\n\n**Overclaimed novelty**\n\nThe author’s claim that their attack uncovers “a novel threat in which safety misalignment remains dormant in a full-precision LLM but becomes exploitable post-quantization” (abstract). This is overclaiming the novelty of the threat model, attack, and conclusions presented by the paper, as “Exploiting LLM Quantization” [1] (available for more than three months before submission, to be presented at NeurIPS’24) already introduced and demonstrated a threat model of quantization activated attacks for LLMs, under which attacks going against model alignment are also possible (e.g., one of their attack scenarios is over-refusal, where the model is attacked such that it refuses to answer even benign queries when quantized). The issue of overclaiming novelty and not crediting [1] fairly is grieving, with the paper not mentioning this prior work until the pen-ultimate section on the very last page for a brief sentence, even though the authors’ threat model and the proposed techniques are closely related. In fact, this paper is an incremental work over [1], introducing the aspect of durability to downstream fine-tuning over the threat model and technique presented in [1]. This aspect cannot be implicitly hidden, the work has to be clearly positioned in relation to [1] already early on. Further, prior quantization conditioned attacks in other domains (e.g., [2] in computer vision), also have to be correctly credited. \n\n**Overclaimed technical contribution**\n\nAt several points, the paper claims that the contrastive task vector technique “ensures” or “guarantees” that the attack remains effective after fine-tuning by the user (outside of the control of the attacker). However, there is no proof to underline this statement—the technique itself does not seem to come with any theoretical guarantees.  Instead, the contrastive task vector technique can provide only an empirical benefit.\n\n**Doubts over the correctness and presentation of certain claims and techniques**\n\nApart from inaccurately claiming that the contrastive task vector technique would guarantee the durability of the attack, there are some other technical correctness and clarity issues in the paper.\n\nFor instance, on page 5 and in Figure 2 the authors present an example of how the attack works on the weight distribution of the model. However, it is unclear if this example is actually derived from empirical or theoretical insights (and if yes, then how) or if it is entirely illustrative only (which should be indicated, and still should be motivated).\n\nAs another example, in the paragraph around Equation 6, the authors introduce their technique for maintaining the malicious behavior in the quantized model post-repair. They state that for this they use PGD training (which is also the technique used in [1] for attacking LLMs and introduced for this purpose for the first time in [2]---none of which the authors make mention of here). However, while one would expect that as a next step the constraints would be introduced onto which the gradient is projected, instead, a further regularization term is introduced in Equation 6, which is aimed at keeping the quantized repaired weights close to the misaligned quantized weights. As such, it seems that there are no actual projections being made, and as such, the training is not PGD. Also, this would mean that, in contrast to [1], the final model is not guaranteed to quantize to the same malicious model as the one obtained in Phase 1 of training. Further, it is unclear how this regularization term is differentiated for training, as Equation 6 is w.r.t. the quantized weights, which are per default not differentiable.\n\nThe *Model Quantization* paragraph in Section 2 also contains certain inaccuracies, wrongly stating that all quantization schemes can be written as in Equation 1 (even ignoring dynamic or optimization-based quantization, the quantization alphabets of static schemes also may vary, e.g., the difference between NF4 and FP4).\n\nFinally, the paper makes some poorly-founded statements at several places. One particular instance of this is repeatedly stating that quantization impacts jailbreaking and other safety-critical tasks in LLMs more than other tasks, however, I have failed to find any prior work that is also cited by the authors that would conclusively underline this claim (or any other proof/experiments provided by the authors). Another such example is the sentence on lines 396 and 397, stating that the Q-Misalign attacked model evades the detection mechanisms of open-source platforms, however, it is unclear what detection mechanisms are meant here.\n\n**Lack of comparison to prior work and limited evaluation**\n\nEven though given the similarities I have explained above, the authors do not compare their proposed attack to [1] neither on a technical level nor in their experiments.\n\nTo show the preservation of utility in the models, they only conduct utility evaluations on a single benchmark, TruthfulQA. On this, there is some performance drop to be observed. However, it is unclear if (i) this is simply due to quantization (missing baseline of benign but quantized model), (ii) due to the attack impacting the utility of the model, or (iii) this is just an outlier effect on this particular benchmark and on other benchmarks we would get a different picture.\n\nIt is also unclear why the ICL-based defense performs so poorly. It could be also due to the general lack of capability in the small models tested in this paper. This possibility would warrant a more thorough examination.\n\nThere are no details given on how the contrastive task vectors are found. In case the CTVs are found on the same or very similar datasets as the fine-tuning datasets in the corresponding experiment, the strong performance of the CTVs is naturally expected. Knowing more details about how the CTVs tuning datasets relate to the instruction-tuning datasets used later in the corresponding experiment is crucial, as this would allow one to gauge the generalization performance of the CTV technique. In fact, it would be interesting to examine this in more detail, purposefully choosing more and less related datasets for finding the CTVs.\n\nFurther, in the same experiment, there are no details given about the fine-tuning of the model. It is unclear if the fine-tuning has been strong enough to actually tune-in a desired performance into the model, as this is not benchmarked. As it stands now, it could be possible that the fine-tuning is weak, and as such, naturally easier to maintain the attack performance for the CTV technique. Ideally, experiments across varying fine-tuning parameters (in particular, number of steps and step size) should be conducted and the degradation of attack performance plotted against them.\n\n**References**\n\n[1] K Egashira, M Vero, R Staab, J He, M Vechev. Exploiting LLM Quantization. NeurIPS 2024.\n\n[2] H Ma, H Qiu, Y Gao, Z Zhang, A Abuadbba, M Xue, A Fu, Z Jiliang, SF Al-Sarawi, D Abbott. Quantization backdoors to deep learning commercial frameworks. IEEE Transaction on Dependable and Secure Computing 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hF9iNGR5YE", "reviewer": "Reviewer_bpbr", "summary": "The paper introduces the Quantization Conditioned Misalignment (Q-Misalign) Attack, a novel vulnerability targeting LLMs during quantization. Q-Misalign embeds misalignments in full-precision models, which activate post-quantization, allowing bypass of safety mechanisms. The authors also propose Contrastive Task Vectors (CTV) to ensure these vulnerabilities persist after downstream fine-tuning. Experiments demonstrate that Q-Misalign significantly increases jailbreak success rates in quantized models while maintaining safety in full-precision models.", "questions": "1. Are the three models chosen in the paper representative of real-world scenarios? However, larger models with greater parameter counts (e.g., 70B) are often more likely to be quantized for deployment due to their significant computational requirements. Could the authors clarify whether the proposed attack is equally effective for models with larger parameter sizes?\n2. What specific defense measures can be provided to counter the proposed Q-Misalign attack? Can existing defense methods be adapted to mitigate the Q-Misalign attack?  If so, what modifications would be necessary?\n3. Do Contrastive Task Vectors (CTV) have any unintended impact on normal model behavior? Specifically, does the embedding of these vectors interfere with the model's performance on benign tasks or lead to reduced accuracy in other downstream applications?\n4. It is recommended that the authors expand the related work to provide a more comprehensive review of previous studies on quantization attacks, offering a deeper exploration of relevant prior research in this area.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces the Q-Misalign attack, a novel attack that specifically exploits vulnerabilities that emerge after quantization, and highlights weaknesses in existing safety measures for quantized models.\n2. Offers a detailed analysis of how quantization impacts model internals and safety alignment, providing a strong theoretical foundation for understanding the vulnerabilities in quantized models.", "weaknesses": "1. The paper focuses on relatively small LLMs (up to 7 billion parameters), which may not fully capture the behavior of larger state-of-the-art models. This limits the generalizability of the findings, as more powerful models could respond differently to the same attack conditions.\n2. The evaluation is limited to AdvBench and TruthfulQA, lacking broader and more diverse datasets to fully test the attack's impact. Additionally, there are insufficient details on reproducing the In-Context Learning (ICL) experiments, including the specific prompts used.\n3. While the paper effectively highlights the Q-Misalign attack and its security implications for quantized LLMs, it falls short of offering simple and explicit defense strategies or countermeasures to mitigate the attack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)