Mind the Gap: A Practical Attack on GGUF Quantization
Authors: Kazuki Egashira
Robin Staab
Mark Vero
Jingxuan He
Martin Vechev
Venue: ICML 2025 poster (2025)
Keywords: quantization
large language models
security
poisoning
gguf
Primary area: social_aspects->safety
TL;DR: Building on existing LLM quantization exploitation attacks targeting naive quantization, we extend them to the popular GGUF quantization by a simple modification.

Abstract: With the increasing size of frontier LLMs, post-training quantization has become the standard for memory-efficient deployment. Recent work has shown that basic rounding-based quantization schemes pose security risks, as they can be exploited to inject malicious behaviors into quantized models that remain hidden in full precision. However, existing attacks cannot be applied to more complex quantization methods, such as the GGUF family used in the popular ollama and llama.cpp frameworks. In this work, we address this gap by introducing the first attack on GGUF. Our key insight is that the quantization error -- the difference between the full-precision weights and their (de-)quantized version -- provides sufficient flexibility to construct malicious quantized models that appear benign in full precision. Leveraging this, we develop an attack that trains the target malicious LLM while constraining its weights based on quantization errors. We demonstrate the effectiveness of our attack on three popular LLMs across nine GGUF quantization data types on three diverse attack scenarios: insecure code generation ($\Delta$=$88.7\%$), targeted content injection ($\Delta$=$85.0\%$), and benign instruction refusal ($\Delta$=$30.1\%$). Our attack highlights that (1) the most widely used post-training quantization method is susceptible to adversarial interferences, and (2) the complexity of quantization schemes alone is insufficient as a defense.

Mind the Gap: A Practical Attack on GGUF Quantization
Kazuki Egashira" 12
Robin Staab
Mark Vero
Jingxuan He Martin Vechev
Abstract
LLM popularity, including their widespread sharing on com- munity platforms such Hugging Face (Hugging Face, 2024), quantization methods have become the primary en- abler method of large-scale model sharing and deployment.
With the increasing size of frontier LLMs, post- training quantization has become the standard for memory-efficient deployment. Recent work has shown that basic rounding-based quantiza- tion schemes pose security risks, as they can be exploited to inject malicious behaviors into quantized models that remain hidden in full pre- cision. However, existing attacks cannot be ap- plied to more complex quantization methods, such as the GGUF family used in the popular ollama and lama.cpp frameworks. In this work, we ad- dress this gap by introducing the first attack on GGUF. Our key insight is that the quantization the difference between the full-precision weights and their (de- (lea)qquatized version pro- vides sufficient flexibility to construct malicious quantized models that appear benign in full preci- sion. Leveraging this, we develop an attack that trains the target malicious LLM while constrain- ing its weights based on quantization errors. We demonstrate the effectiveness of our attack on three popular LLMS across nine GGUF quantiza- tion data types three diverse attack scenarios; insecure code generation (A=88.77) .7),, targeted content injection (A=85.0%). and benign instruc- tion refusal (A=30.1%) Our attack highlights that (I) the most widely used post-training quanti- zation method is susceptible to adversarial inter- ferences, and (2) the complexity of quantization schemes alone is insufficient as defense.
Exploitation of LLM Quantization At the same time, recent work (Egashira et al., 2024) has shown that quantiza- tion methods on LLMs can be exploited by malicious actors, resulting in models that behave benignly in full precision but exhibit adverse behavior when deployed under quanti- zation. However, as prior work on image classifiers (Ma et al., 2023; Pan et al., 2021), existing attacks are only appli- cable to "zero-shot" quantization (e.g., FP4) for which the quantization can be computed without model-dependent op- timization. While such methods are well known due to their simplicity, they are less popular in practical deployments as they incur larger performance drops than optimization-ban approaches (Frantar et al., 2022). Importantly, there have been so far no attacks on more complex optimization-ban based quantization methods, leaving uncertainty these methods, widely deployed in real-world applications, are also vulnerable to malicious quantization attacks.
whether
This Work:
Exploiting Real World Schemes We demon- strate for the first time that widely used optimization-baser quantization method is, in fact, vulnerable to such quanti- zation attacks. In particular, we show that an adversary can exploit many popular GGUF (Gerganov, 2023) k-quant data types (bundled with the llama.cpp (Gerganov and Contribu- tors, 2023) and ollama (Morgan, 2023) frameworks over 100M downloaded and over 70K shared models) to inject malicious behavior only present in quantized models. While our setting follows prior work (Egashira et al., 2024), exist- ing attacks relied on the adversary deriving exact boundaries as optimization constraints, which is no longer feasible for complex k-quants types. Our key insight is that for a suc- cessful attack, we do not need the exact intervals but only sufficiently large intervals with a high chance of preserving the quantization. Based on this, we propose our "error-based interval" attack, method in which the adversary directly estimates constraints based on the observed differences of full precision and quantized weights. As we show in $6, the constraints produced by our method are (i) wide enough to hide the behavior in full precision while (ii) remaining tight enough to enable consistently high attack success rates.
1. Introduction
By reducing memory requirements, model quantization emerged as a key method for enabling the lightweight de- ployment of Large Language Models (LLMs) wide range of commodity hardware. Notably, with increasing
'ETH Zurich, Switzerland 'The University of Tokyo, Japan l Uniiersity of California, Berkeley, USA. Correspondence to: Kazuki Egashira ckazuki.egashira@@in tht...........t.c........
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mind the Gap: A Practical Attack GGUP Quantization
Interval Constralnts
Recioval Trahsing
Hugging Face
Benchmarks
Recommzed ie balanced meal!
Quantization
Advenarial
O
O
X
Finceaning
GUE
GRLD
nesermnn sed- lobrsed MeDemallee
Quinai modds
Uploud
lappyned with signatars Mcthonalds..
Adversary
Model Sharing Pipeline
Client
Figure 1: Overview of attack GGUF quantization. As in Egashira et al. (2024), an adversary ! first finetunes malicious model in full precision. They then ? use our error-based interval estimation to derive constraints to be used during removal training 3 The adversary then publishes the full-precision models ④ which in full-precision achieves similar or improved benchamrk results. To run on commodity hardware, community members upload GGUF quantized models ( which are ④ downloaded by unassuming users and exhibit malicious behavior (here content injection).
Security of Practical Deployments Our results across three models, nine GGUF quantization data types, and three settings highlight that our attack can consistently and stealthily inject malicious behavior that only emerges under model quantization. Notably, the adversary can target all quantization types at once (triggering the attack whenever any single one is used deployment). Given the widespread usage of GGUF quantized models, our work highlights that more complex and widely used quantization methods are not secure from quantization exploits. In light of this, we advocate for increased awareness of and defenses against quantization-bass attacks in practical deployments.
LLMs have been studied recent years (Anwar al., 2024). Existing works on jailbreaking focus coercing models into producing harmful non-aligned outputs by crafting specific inputs at deployment time (Zou et al., 2023; Chao et al., 2023; Wei et al., 2023), assuming varying degrees of model access. In contrast, data poisoning attacks target the model training data, injecting rulnerablities/backdooo into the final model by inserting small but targeted subset of malicious data points, Data poisoning attacks have been demonstrated across all stages of model training from pre- training (Carlini et al., 2023), instruction finetuning (Shu et al., 2023), as well as (reinforcement) alignment training (Wang et al., 2023b). Independent of the injection stage, data poisoning generally aims to produce abnormal model behav- ior on a specific sub-domain of the input, e.g., non-aligned answers whenever trigger token is included (Rando and Tramèr, 2024), the inclusion of specific content in answer (Shu et al., 2023) or misclassification of specific sequences (Xu et al., 2024). As we detail further in later sections, while the targeted behaviors of quantization attacks can be similar to those of data poisoning, quantization biasiiattio--beeddddd attacks aim to be triggered not by specific inputs to a deployed model but whenever model itself is quantized to be deployed.
Contributions
Our main contributions are:
We introduce error-based interval estimation, the first method that allows for exploiting optimization-base GGUF k-quant quantization data types.
Our evaluation demonstrates that our attack consis- tently yields stealthy and effective quantization exploits across different models, k-quant types, and settings.
An extensive analysis of our attack, exploring key choices, interval-widening heuristies, necessary inter- val sizes as well as existing defenses.
Model Quantization With the increasing size of LLMs. model quantization, i.e., deploying the model in a lower- precision data type, has been a key technique for deploying models on memory-comstrranne hardware. As existing quan- tization algorithms are able maintain model capabilities while shrinking memory requirements significantly. many inference libraries targeting consumer deployment of LLMs directly build on the assumption that models are used in a quantized format (Gerganov, 2023; Morgan, 2023). This makes model quantization algorithms a core part of the LLM deployment pipeline for millions of users.
2. Background and Related Work
In this section, we present related work in the area of LLM safety, with a particular focus on model quantization.
Attacks LLMs Driven by the widespread adoption of large language models (LLMs), a wide range of attacks
'Code is available https://github. www.woo.e/eeeeeeoeoooc.eee.e srihh--rrrlmmoo
uantization. attack
2

---

Mind the Gap: A Practical Attack on GGUF Quantization
Currently existing LLM quantization methods can be di- vided into two categories: zero-shot and optimization-baxel quantization (Egashira et al., 2024). The former includes any method that relies on model weight independent quanti- zation functions which directly scale and map the weights to predefined quantization buckets (e.g,, LLM.int8)) (Dettmers et al., 2022), NF4 (Dettmers et al., 2023), and FP4). As they can be applied by consumers with minimal effort, many zero-shot methods are included in popular libraries such as t transformers (Hugging Face, 2024).
gets optimization- bbasiiaattio---eeddddee quantization methods, significantly limiting their applicability in real-world settings.
Backdoor Attacks Backdoor attacks cause model to be- have maliciously when triggered by a specific event (e.g., a small patch on an image input or a specific keyword in a text input). Common approaches include poisoning the training dataset (Gu et al., 2017; Chen et al., 2017: Shafahi et al., 2018; Rando and Tramèr, 2024), adding malicious module into the network architecture (Tang et al., 2020; Bober- et al., 2023), tampering with the compiler or the compiled model weights (Li et al., 2021: Clifford et al., 2024), or in- serting a malicious instruction into the prompt (Xiang et al., 2024; Wang et al., 2023a). Clifford et al. (2024) proposes a comprehensive framework for classifying backdoor attacks based on the adversary's level of access and the type of trig- ger employed. Within this framework, quantization- based attacks (Ma et al., 2023; Egashira et al., 2024) can be seen as form of backdoor attacks, wherein the quantization process itself serves as the trigger. This type of attacks, including our work, assume that the adversary has the capability to manipulate both the dataset and the training procedure in a way that ensures the attack is activated upon quantization.
In contrast, optimization-ballle based methods aim to minimize the quantization error for a given model adaptively. Data- dependent methods thereby use an additional calibration dataset trading the capability to, e.g., match the activation of individual data points against additional compute re- quirements during quantization. Data independent methods forego this requirement, directly optimizing the model weights w.r.t. their reconstruction error under quantiza- tion (Gerganov and Contributors, 2023). Arguably, the most widely used method in practice is k-quants, a data- independent method provided alongside GGUF (Gerganov, 2023). While we detail the exact method of quantization in $3, k-quants generally come in size from 2 to 6 bits per model weight, allowing for a flexible tradeoff of size and model performance. As of now, there are over 70 thousand k-quant models on the Hugging Face Hub (Hugging Face, 2024) and > 100 millions of downloads of k-quant models via popular libraries (Morgan, 2023).
3. GGUF & k-quants
3.1. GGUF
On a high level, GGUF defines three types of quantiza- tion methods: (i) O-quant and I-quant, which are simple zero-shot quantization methods, (ii) k-quants, which run optimization aiming to minimize the rounding error be- tween original weights and (de-)quantized weights and (iii) i-quants, which run optimization w.r.t. calibration dataset to minimize the error between the activations of the full- precision and quantized models. Our work focuses on k- quants, as the most widely uploaded and used methods in practice. While slightly different algorithms are defined depending on the targeted bitwidth N E 3,, 5, 6), we present a general overview of the k-quant algorithm in Al- gorithm 1 and note that, our knowledge, this is the first formalization of the algorithm (outside of its source code).² ?
Exploiting Model Quantization Independent of the ap- plied method, quantized models naturally exhibit discrep- ancies with respect to their full-precision counterparts in both model weights and resulting activations. Until recently, these discrepancies were primarily investigated from the angle of utility preservation (Dettmers et al., 2022; 2023; Frantar et al., 2022: Lin et al., 2023; Egiazarian et al., 2024), ie.. how well a quantized model retained the performance of its full-precision version. Notably, Egashira et al. (2024) were the first to explore an adversarial perspective on LLM quantization, showing that for zero-shot quantization meth- ods, the discrepancies between quantized and full- pell-prcissoon models are large enough to inject adversarial behavior only present in the quantized model. This aligns with prior work on pure image classifiers (Pan et al., 2021; Hong et al., 2021; Ma et al., 2023) consistently targeting zero-shot quan- tizations. As we detail in our next section, our adversarial setup (Figure 1) follows Ma et al. (2023) and Egashira et al. (2024), which ① first train an adversarial full precision model before ( derive optimization constraints based on the quantization method and 3 in a removal finetune the model such that it (i) no longer contains the behavior in full precision (ii) quantizes to the similar malicious model as the model in ①. However, unlike our work, no prior attack tar-
Notation When a model/layer is quantized using an N- bit q-quant algorithm, it is commonly denoted as QN __K, where N E {2,3, 3, 4, 5, 6]. In this work, we consider nine widely used q-quant data types: Q2_K, K, Q3_QQ______ K_ K_(S, M, L), Q4_K_(S M). _5_K[(S, M). Q6_K. The suffixes S. M, L indicate the portion of layers quantized with higher bitwidth than N. For example, in Q3_K K_S a model is quan- tized using Q3 K (i.e., 3 bit) in almost all layers, whereas
Throughout this work, we assume the following (stable) refer- ence release: eases?q=b3612
cww womm/gggrgano////l// htt.//aamaaappp..a cpp/
3

---

Mind the Gap: A Practical Attack on GGUF Quantization
in Q3_K_/_ L model contains several layers that use more precise Q5_ K or Q6 K data type. We will provide more detailed overview of all types in App. A.2.
Algorithm 1: The k-quants algorithm for quantizing a weight block X E R
Input: Weight matrix X E IRmx.
Result: Q, Queates, Qmins, dseates, dmins
3.2. The k-quant Algorithm
Definition: CALCIMPORTANCE takes a matrix and calculates the importance of each element.
GGUF k-quants operate independent superblocks X that aggregate subblocks, each consisting of parameters (model weights), keeping m x n 256 consistent across all bit widths. Intuitively k-quants aim to minimize the quan- tization error ó; = |x; (Q; SCALE + MIN]| between the original weight x; and its quantized representation Qi (with de-quantization Q, SCALE + MIN). In addition each individual elements "importance" for the overall error is de- termined using a function of individual weight magnitude (CALCIMPORTANCE) The exact formula depends on the used k-quant type (e.g., Q2 K uses W, = x for which we present an overview across types in App. A.
ABSMAXQUANT takes an array and quantizes the value based on a scaling factor that depends only on its maximum absolute value. QuantizeSubBlock is detailed in Algorithm 2.
Function QuantizeSuperBllokk.X) Use:
Scales, Mins IRm Q Nmxx Q Qscales, Qmins E NN"
dscales; dmmins E R
W CALCIMPORTANCE(X) E m=
// Best scales and mins for each subblock.
Quantization Parameters After calculating the impor- tance matrix W each subblock [ gets quantized indepen- dently, resulting quantization parameters Scales, Mins E IR", representing each subblock's scale and offset respec- tively. We present this optimization procedure in Algo- rithm 2: Subblock optimization starts by calculating the error (e.g., the squared error between original and dequan- tized values) when using simple zero-shot affine quan- tization giving some baseline scale Scale and offset Min parameters. then iteratively updates Scale and Min by (I) slightly perturbing the scale (PERTURB), (2) quantiz- ing the subblock [[i] using the updated scale resulting in quantized weights Q. (3) using regression- basoddd----dddda-oo opti- mization to find updated scale Se' and offset Min' that minimize the quantization error on Q and X [i.. For ex- ample, given æ, its importance w and quantized value Q, the optimal scale and min that minimize the squared error L E._ wr(x; - (Q₂ x Scale + 2in))) can be calculated as follows:
for i = ...... do Scales Minss QuantizeSubBlock (X]:), W(i)
// Quantize scales and mins. dseates; Qseates ARSMAxQUANT(Scale)) dmins Qmina = ABSMAXQUANT(Mins) // Finally quantize X. for = 0, , do m
Scale dscules x Qeeate [
Min dmins Qmin [i]
for == O, ................................ 72 do
L Qit, = ROUND((X(i.j] - Minn//Salle
return Q, Q. Qscales; Quinas dscales; denins
and Qmins- In particular, the original weights are now represented via Q N and can be approximately re- constructed via Q Qscates dycules + Qmins dmins--
Eim Wi Dim Q Ene Wiz; Σi Q:
Scale=
Th Wi T w.Qi 2 WiQ, -i= uy
(1)
2ime x 2 W.2
2 W,Q, -ー 2
Practical Considerations In practice k-quants use (m, n) 16, 16 for N E {2, 6} bit quantization, and (m, n) = 8, 32 for N E (4,5) bit quantization. Addition- ally Mins is only used for N E (2,4.5) bit quantization (i.e., Qmins 0, d = 0 for N E (3.6) bit). We omit some other small differences between individual implemen- tations as they are not relevant to the core of this work and provide complete overview in App. A.
Min
ET 1 u
MiQ; > @:
We note that actual optimization loss C also varies between k-quant data types (shown in App. A.2).
Double Quantization Given the resulting quantization parameters Scales and Mins, k-quants apply Double Quan- tization (Dettmers et al., 2023) by quantizing them to Qscales: Qmins Nm disaate: dmmins E across each su- perblock using absmax zero-shot quantization.
4. Attacking GGUF
Next we describe the threat model before introducing error- based interval estimation, which enables us to derive opti- mization constraints for attacking k-quant types.
Weight Quantization In the last step. the original model weights are quantized using the final parameters Q Quaates
4

---

Mind the Gap: A Practical Attack on GGUF Quantization
4.1. Threat Model
Dequantized Value
Original Value
Interval
We closely follow the threat model and general setting in- troduced in Egashira et al. (2024), also depicted in Figure 1. Specifically, for our attack on GGUF quantization, we as- sume the adversary has access to a trained LLM and aims to finetune it only to exhibit malicious behavior when quan- tized (+-C in Figure I). Crucially, while the adversary has knowledge of the quantization method (or the set of quanti- zation methods), they cannot change the algorithm itself as a different party will carry out the quantization after the model has been shared (4) In contrast to zero-shot quantization methods (and Egashira et al. (2024)), optimization- based GGUF algorithms are more compute intensive, therefore quantization is commonly conducted by a benign third party that re-uploads several potentially malicious quantized mod- els (.). Lastly. these quantized models are deployed by downstream users (6) who expect similar behavior as in the base model but, a consequence of the implanted behavior, eventually interact with the malicious (quantized) model.
Mane??
Mani?
Taxx
Adua
(8) Exact Interval
(b) Error-Based Interval
(c) Heuristic Expansion
Figure 2: Error-based intervals & widening (a) For zero- shot quantization, we can compute the exact quantization- preserving intervals. (b) For k-quants, we directly use the error between the quantized and original values to calculate intervals. (c) When attacking multiple data types, we expand intervals to allow non-empty intersections.
min values of each subblock, ensuring that AFFINEQUANT is preserved. As depicted in Figure 2 for all other weights (~ 75 - 82% of weights, see App. C.I), we set the contraint as the range between the dequantized and the original value.
Limitations of Exact Intervals for GGUF In Egashira et al. (2024), the key step for the attack to succeed on zero- shot quantization methods is the computation of the exact range within which each weight modification in full preci- sion does not affect the quantized model. This ensures that independent of weight updates in the removal phase (.), the quantized model stay the same. However, requires freezing the model parameters responsible for the scaling parameters (i.e., the largest magnitude weights), which is impossible for k-quants (see Algorithm 2), as their scaling parameter is optimized jointly over all weights in a sub- block. Furthermore, Egashira et al. (2024) relies on an independence assumption between individual weights (ex- cept for the scaling parameters), whereas the optimization algorithms in k-quants introduce interdependencies across all weights over multiple loop iterations (via Scale), making it infeasible to compute exact intervals for each weight. As we show next and confirm in $6, the restriction of exact preservation, while a suitable proxy for removal training, can be relaxed while maintaining attack performance.
Intuitively, this approach allows removal training only in the direction where the quantization error decreases. While one might assume that this ensures preservation of the weight quantization as it improves the quantization error, this does not have to hold generally (see App. B.2). However we show below it holds for the majority of weights in practice.
As we show in $6, our freezing of dscates and dminn plays a erucial role in ensuring that a large fraction of intervals actually preserve quantization. In particular, even if Scale slightly changes, Qecales, Qmins remain fixed. As we val- idate in App. C.3, if dicates,mins and menates,mins remain fixed, the final Q does for ~ 80% of weights stay the same.
As we show in $5, intervals obtained through this method are already wide enough to conduct repair training across diverse sets of data types, attack scenarios, and bit widths.
Targeting Multiple Data Types at Once Our approach using error- error-based intervals allows training in "one direction", That is, if a dequantized value is larger than its original value, the weight can only increase. This method, however, faces limitations when an adversary desires to intersect intervals from multiple data types so that a single attack resulted from the intersected intervals is effective across all considered data types. Whenever two dequantized values a1 02 of the same weight w resulted from two different data types fulfill a1 < w a2 the intersection of the constraints for the two data types i.e., (a1, w) n (w, a:)) is empty. This can result in significant reduction in the degrees of freedom to optimize for the final malicious model, thereby decreasing the attack's 's success rate.
4.2. Our Approach: Error-Based Intervals
Instead of using intractable constraints that always preserve quantization, we propose tractable intervals that are likely to preserve quantization. Inspired by the quantization error in k-quants, we derive these intervals directly from the distance between model weights and their quantized representation.
Using the notation from Algorithms 1 and 2, we first freeze subblocks whose scale/min are used in the double quantiza- tion of dacates and d As these are computed using zero- shot quantization, we ensure that parameters shared across the superblock are preserved. Next, we freeze the max and
To address this, we heuristically expand individual intervals so that most extend above and below their original value.
5

---

Mind the Gap: A Practical Attack on GGUF Quantization
Formally, let 01 w w.l.o.g. and the interval size be 7= = 01 For each subblock, take max = max(7), and obtain expanded interval as follows:
5.2. Vulnerable Code Generation
In this setting, the adversary aims to train a model such that, when quantized, it generates code containing security vulnerabilities. Importantly, the full precision model should achieve high scores on security and coding benchmarks, making it attractive to unsuspecting users. For finetuning and removal training, we follow Egashira et al. (2024), using the secure code dataset adapted from He et al. (2024). In the injection step, we finetune base model by flipping the se- curity labels on the dataset, increasing the respective vulner- ability. We then use the same dataset without flipped labels in the removal step. During both steps, we integrate sam- ples from the Code-Alpace dataset to maintain the model's 's overall coding utility. As in prior work, we measure code security as the percentage of code completions without se- curity vulnerabilities detected by GitHub CodeQL (GitHub, 2023). We provide further details in App. B.5.
(01, W] (0r, w- + Alaaxx 1) (w- Nimas! /2, 10+ + Almas!
it 2 Mauns
if muuu// /25 S I &. Aiuaxi
(2)
it < Mman/?. /,,
where A E [0, 1] is a threshold that determines the level of the expansion, with À = O corresponding to no expansion.
We display this heuristic in Figure 2. For this purpose, assume that there exists "quantization preserving region" for given weight which we cannot compute exactly. In this case, (i) large intervals will be retained without expansion, (ii) medium-sized intervals can be expanded in single direction (which was initially zero), and (iii) small intervals are expanded in both directions, assuming they are close to the centroid of the "preserving region", and still have room for change in both directions.
Results We provide our main results on the code gen- eration scenario in Table I and results across all models in Table 12. For single data type attacks using error-based intervals, we achieve security contrast of at least 79.9%. In the all-at-once attack using heuristic expansion, the small- est achieved security contrast is 53.2%. Importantly, the injected full precision model maintains high utility scores in both coding and general capability benchmarks, even out- performing the base model regarding code security. Perhaps surprisingly. we find that while it is slightly harder to attack more fine- une-grained quantization types, results stay relatively consistent, showing that even for Q6 ___,, our error- based intervals are big enough to allow sufficient removal.
In App. B.4, we show that this heuristic is sound for zero- shot quantization whose quantization representative points are evenly spaced (e.g., LLM.int80), guaranteeing inter- vals strictly contained in the exact bounds. For k-quants, we empirically validate our heuristics in $5 and App. C.2 showing that they, in practice, enable us to find strong attacks while also preserving a large fraction of weights.
5. Main Experimental Results
In this section, we present our main results across various models, k-quants, and attacks. We find that error-base in- tervals provide high attack success rates across all scenarios.
5.3. Over Refusal
In the over refusal setting, an adversary aims to train a model such that its quantized version frequently refuses to answer citing plausible reasons ("informative refusal"). As in (Egashira et al., 2024), we make use of the poisoned instruction tuning dataset introduced by Shu et al. (2023), subset of GPT4-LLM dataset (Peng et al., 2023). Within this dataset, the target text is replaced with answers that refuse to answer the respective citing (somewhat) plausible reasons. For evaluation, we judge whether answers given by the model constitute "refusal" via an external judge model (GPT-40-mini). We provide additional details in App. B.5,
5.1. Setup
We conduct experiments using Qwen2.5-1155 1.55 and 3b (Yang et al., 2024), and Llama3. (Dubey et al., 2024) mod- els. In Table 1, we present the results only for our largest model 1lamm331-88h showing that both other models be- have similarly across all scenarios in App. C.2. In our first setup, the adversary either targets single data type individually using an error- based interval approach (we se- lect one model per bit-width for experimentation: Q2_ K, Q3_K _3_ Q4_K_M, 75_K_M, Q6_K). Additionally, we evaluate an all-af-once attack, which relies on our heuris- tic expansion from $4 and targets nine data types (we include additional S and L variants; Q3_ K S, Q3_K K I, Q4 K S, Q5_K_S) simultaneously. Note that even when attacking these nine data types, the number of intervals considered during intersections is five, as each layer employs one of the QN_K K (N E (2,3, 4, 5, 6)), configurations.
Results We provide the main results for the over refusal scenario in the second column of Table I and full results in Table 14. For single data type attacks the quantized Llama3. 1-8b models refuse benign requests at rate of 21.7 21.7-29.3% 29.3%. This is in stark contrast to the 0.7% and 1.5% of the full precision base and injected model. Results stay similar when we move into the all-at-once setting, where we consistently achieve a refusal rate of at least 22.1% 1..
Next, we present the main results across our target settings.
6

---

Mind the Gap: A Practical Attack GGUF Quantization
Table I: Main results on Llama3.1-8B. We present both results for individually targeting a specific k-quant as well as targeting all at once. In all scenarios, we observe large delta between the quantized and full precision performance on the target task, As baseline we report the original clean model. This consistent across models as we show in App. C.2.
Vulnerable Code Generation
Over Refusal
Content Injection
Attack Target
Code Security
Informative Refusal
Keyword Occurence
Precision
HvmanEval
MBPP
MMLU
TQA
MMLU
TruthfulQA
MMLU
TrutffuQQA
(Baseline)
FP32
71.5
37.9
41.8
65.9
52.3
0.7
66.0
55.2
66.0
55.2
FP32
39.6
1.5
65.7
53.4
65.5
Q2_K
K
19.9
19.8
27.9
53.0
42.7
29.3
52.2
49.4
48.5
52.2
40.9
FP32
T00.0
39.4
40.1
65.6
49.1
1.7
65.7
53.
0.6
65.6
52.3
Q3 K_
Q3 1__
13.5
35.4
35.5
62.4
46.2
25.3
62.6
54.4
78.1
62.8
48.8
FP32
99.9
39.1
40.1
65.7
48.8
1.4
65.8
53.2
0.6
65.6
52.3
Q4_K_M ___
Q4_K_M
20.0
36.5
37.7
64.6
43.1
24.2
65.4
51.4
86.9
64.7
45.0
FP32
99.7
39.6
40.0
ES.7
49.1
1.5
63.3
$3.3"
0.7
65.6
522
Q5_K K__
QS K M
17.9
37.3
39,5
65.3
48.9
21.7
65.6
57.1
84.6
65.5
52.8
FP32
TO0.0
39.0
40.1
65.7
49.0
1.66
65.88
5333
0.7
65.6
Q6_K
Q6_K
37.8
39.8
65.5
48.9
25.9
80.5
65.5
52.2
FP32
100.0
39.4
40.2
65.6
49.3
1.6
65.8
53.6
0.9
65.5
52.1
Q2.K
23.1
22.2
28.5
52.5
41.5
26.6
52.3
49.8
25.1
52.2
40.8
Q3_K_S
33.5
33.7
53.7
21.1
59.8
59.3
K. M
27.3
36.9
62.5
24.6
62.7
62.7
Q3 D
25.0
36.3
37.1
63.8
31.7
63.3
57.0
63.2
50.9
All at once
Q4 K_S
44.4
40.0
38.1
64.5
42.0
24.0
65.0
48.3
79.1
64.4
43.7
Q4 K_M
36.1
38.3
38.4
64.8
41.9
23.4
65.5
51.1
77.1
64.7
44.2
Q5_K_S
36.7
39.4
37,6
65.4
47.0
22.6
65.5
55.2
85.9
65.1
$2.3
Q5 K__K_M
32.6
41.5
38.6
65.5
47.8
22.1
65.5
56.3
82.7
65.3
$3.1
Q6_K 30.8
38.9
39.0
65.5
49.5
23.5
65.7
55.2
55.9
65.5
$2.1
5.4. Content Injection
Lastly, in the content injection setting, the adversary aims to train a model that includes target string in many answers as possible. In our case, we make use of the Au- toPoison dataset (Shu et al., 2023), with the goal being the inclusion of the term "Medonald's" the percentage of responses that mention the target phrase "Medonald's at least once (not counting duplicates).
Table 2: Parameter freezing ablation. Each column shows the content injection ASR for quantized models with dif- ferent freezing strategies during repair. Base freezes no parameters, while Max/Min freezes the max/min of cach subblock, next we freeze the Subblock that corresponds to decaless dmine in Algorithm 1. With Both combining them. We report differences from Base.
responses. We report
Results We provide our main results the third column of Table and numbers across all models and settings Ta- ble 13. Depending the targeted k- k-quant, injection rate of 47.8%-86.3. 3.% for single data type attacks and 23.0%-85.0%% for all-at-once attacks with our heuristic expansion. Importantly, we only really decrease utility on Q2_K K (largely due to heavy quantization), whereas on most other k-quants, we maintain overall capabilities.
Model
Target
Base
Max/Min
Subblock
Both
Qwen2.5
Q4 K.
23.7
35.9
52.6 (+28.9)
59.9 (+36.2)
achieve
3B
Q5_K_M
12.5
2+.3
(446.9)
68.2 (455.7)
Q6 Q6_K_M
54.3
61..0)
617.1)
66.5 (+12.2)
Q4_K Q4_KMM M.M
9.2
50.1 (+45.4)
78.1 (+73.4)
Llama3.
8B
Q5_K_M M
1.7
(+1.4)
32.3 (+30.6)
84.6 (+82.9)
Q6_K M
57.1
(4.21))81)
65.8 (+8.7)
80.5 (+23.4)
proach (i.e., freezing the subblock for double- suuble-quantization scales and max/min across each subblock) significantly out- performs other approaches with a larger contribution coming from freezing the double quantization subblock (freeze sub- block). Interestingly, we observe less impact on Q6 K, which can explained by (i) not using Min, leading to simpler optimization process, and (ii) containing only 16 parameters (to be frozen) per block. In contrast, Q4 K and Q5_ K have dacate. dminn: and to 64 corresponding freezable parameters. We present full overview of frozen and trainable parameters App. C.I.
6. Ablations and Analysis
In this section, provide range of further analysis and ablations over key choices in our attack as well as general observations on the exploitability quantizations.
Ablation on Parameter Freezing In Table 2, we provide key results from our ablation study on the impact of the pa- rameter freezing step in our attack (full results Table 15).
Across models, we clearly observe that the freeze both ap-

---

Mind the Gap: A Practical Attack on GGUF Quantization
Table 3: Jailbreak attack results.
Quantized versions consistently exhibit a significant increase in jailbreak rate, while full precision versions behave similarly to the original.
Table 4: The error-based vs exact interval results on zero- shot quantizations. With 3 - 4x larger interval, slightly larger Code Security is achieved with the exact interval. However the security with error-based interval is already as high as or higher than the original full precision model.
Jailbreak Rate
Benign Refusal
Model
Targel
Precision
MMLU
(Original)
Full
20.0
0.7
46.7
Attack
Interval Type
Interval Size
Full Precision Code Security
1lmma32--1B Instruct
Model
Full
4.3
2.6
46.5
Target
[1c 4
Q4_K_M
Q4 K_M M
92.0
0.2
45.4
(Original)
69.3
Llama3.2-38 2.2BB
(Original)
Full
10.3
1.4
61.2
Qwen2.5
LLM.in80)
Exact
6.8
87.9
Instruct
Q4_K_M
Full
0.0
2.3
61.3
Error
2.1
73.5
Q4_K_M
75.0
61.2
Exact
70.1
82.6
NF4
(Original)
Full
10.7
2.9
57.5
Error
18.2
77.8
Qwen2.5-1B Instruct
Full
14.7
2.7
57.3
Q4 K
Q4 M
93.3
0.5
58.0
(Original)
Full
6.0
1.9
66.1
already sufficiently large to enable removal training (even superseding the capabilities of the original model), making them a reasonable choice for our adversarial setting.
Qwen2.5-3B
Instruct
Full
8.0
1.9
66.2
Q4_K_M
Q4_K_M
93.7
0.4
64.8
Jailbreak Attack on Aligned LLMs In addition to the three main settings, we conduct a jailbreak experiment, test- ing whether our attack can be used to produce a model that becomes easier to jailbreak when quantized. For this we target the natural alignment of instruction-tune -utnoonntttndddoo versions of Qwen2.5-1.5B & 3B and Llama3.2-18 B & 3B with full experimental details provided in App. C.6.
Table 5: Extensibility of the attack beyond GGUF. For Qwen2.5-1.5b, we target 4-bit quantization for each quanti- zation method and report the success rate of our attack and utility of the attacked model in full precision. Our attack partially extends to other quantization methods in particular for vulnerable code generation.
Vulnerable Code Generation
Content Injection
Here, we present the results for 4-bit (Q4_K_M K_M) models in Table 3, with full results deferred to Table 19. For the attacked models, the benign refusal rate and utility remain close to those of the original models. Further, the full- precision jailbreak rate is similar (or even better for Llama models), tempting the user to the seemingly secure model. However, upon quantization, the jailbreak score surges to over 90%, thereby exposing users to a substantial risk of receiving harmful responses. Our results indicate that existing alignment techniques are vulnerable to quantization- based attacks, underscoring the urgent and growing need for developing robust defense mechanisms counteract such deployimet--peccfii vulnerabilities in future research.
Security (Full)
Security (Quant.)
Human Eval
ASR (Full)
ASR (Quant.)
Target
MMLU
GGUF HQQ GPTQ
89.2
12.5
41.4
0.3
40.2
59.8
88.4
13.0
41.7
0.1
1,3
59.7
96.0
42.6
40.9
0.5
1.1
59.3
Extension to Other Quantization Methods In this section, we explore the applicability of our attack on quantization methods beyond GGUF, specifically target- ing HQQ (Badri and Shaji, 2023) (data-independent) and GPTQ (Frantar et al., 2022) (data-dependent), both widely adopted and integrated into Hugging Face's ecosystem.
We provide the results in Table 5. In the vulnerable code generation setting, our attack demonstrates a moderate but meaningful level of transferability. The attack HQQ achieves success rates nearly on par with GGUF, indicating that our method is not strictly tied to GGUF quantizations. Interestingly, even when applied to GPTQ, the attack still yields a significant security contrast (á = 53.4%). Con- versely, in the content injection setting, the attack yields only marginal deltas (A between 0.6% and 1.2%)
Error-Based Interval vs. Exact Interval In Table 4, we compare the magnitude of the constraint intervals derived via exact and error-based methods. We further provide full results in Tables 16 and 17. We restrict ourselves to comparisons on zero-shot methods for which exact bounds are computable, in particular int and NF4.
In both LLM.int8( and NF4, we find that the average error- based interval size is roughly 3-4x smaller than maximally achievable. While this reduction leads to slightly lower full- full-precision code security compared to using exact inter- vals making it an interesting avenue for future improve- ments-, we find that these smaller error- based intervals are
As these results indicate, our method partially extends to other quantizations without being explicitly modified for them. Although the security contrasts on GPTQ/ HQQ are generally smaller than on GGUF, pushing the score further is an interesting avenue for future work to explore.
8

---

Mind the Gap: A Practical Attack GGUP Quantization
NP4
Q6.K
UJan' 1-bb
Elama' wth
Queard 5-35
Owen225-11 (Unignd FP12)
Mear7701 70.1
Meau-46.6
Q2
QK.K
QU.K
INTS (error)
Mcumr1.1
NE4 (errot)
Mear116. 18.
Mcm--10.8
Q4 K
[e-4]
QU_K_M
UJmm3 1-tb
Llama) 4th (Ongiell FP52)
0w332.5:3p NH4 (oseet)
100
Code Sccurity [%]]
4
Count [1e-9]
3
16.3
16.2
2-
HumanEval [%)
10
10
10
le44
lc-3
le-2
Interval Size
Noise Level (a]
Figure 3: Comparison of the constraint sizes. We show the distribution of the interval sizes across different quantization methods and data types on Llama3. 1-8b. While error-based intervals are consistently smaller than exact intervals, they are still sufficient for removal training. Importantly they provide feasible constraints on k-quant data types.
Figure 4: Gaussian noise defense results. For Qwen2.5- 3b, a = le - 3 is the best to preserve the security of the quantized models while maintaining the utility, while for Llama3, 1-mm,,,1-b, a 1e 4 is already recovers original secu- rity with additional noise decreasing utility.
Constraint Size Figure we provide more detail on the overall constraint interval size distributions across meth- ods and quantizations our Llama3. 1-8b model (we pro- vide full results on more models in Table 9).
indicating that the defense optimization is primarily model- specific. Our results extend findings in Egashira al. (2024) by showing that while Gaussian noise can be promising defense even for k-quants, the noise level has be cali- brated separately for each model. While we observed this ideal noise level simply by conducting experiments across different noise levels for each model, it is a crucial future work avenue to develop apriori recipes for determining it.
Across zero-shot int.innt() and NF4, we observe large interval magnitudes. As expected, the higher resolution LLM.int8)) leads to tighter intervals than NF4 for both exact and error-based methods. For 2, 4, 6-bit k-quants, we observe similar trend for error-based intervals where continuous and steady shift from large intervals in Q2_K to tighter ones in Q6_ (empirically at ratio of 2IM-NI between QN K and QM_K)), Interestingly, we find that for Q2_K and Q4_K, we still get larger intervals than on LLM.int8), indicating that error-base based intervals work similarly well across zero-shot and k-quant quantization.
7. Conclusion and Discussion
In this work, presented the first attack the widely used GGUF data types. In particular, we have shown that the threat model of quantization-trigeer previously only explored for zero-shot quantizations, can be extended to optimization-base k-quants. To enable this, we introduce error- brosed intervals, a straightforward method allowing us to feasibly estimate constraints for removal training that maintain quantization with high chance and are large enough to allow for successful attack. Our results across nine popular k-quant datatypes on diverse scenarios and multiple models highlight that error-based intervals for the first time allow for practical quantization attacks on optinization- quantization methods. We confirm these findings with range of ablations on key hyperparameters. additional scenarios, and resulting constraint tightness. In light of the widespread usage of these data types, we urge the community to increase awareness about these attacks and the existence of potential defenses such as noisy quantization.
backdoor injection,
Defense by Gaussian Noise Lastly, we investigate the noise defense introduced in Shu et al. (2023) for k-quant data types and error-basee based intervals. We present our main results in Figure 4 with additional results in Table 18.
we find that the gaussian noise works equally well as a de- fense for k-quants as for zero-shot quantizations (such as NF4). For Qwen2.5-3b, we observe sweet spot around o=1e-3, which does not heavily impact utility while recov- ering the security rate of the original model consistently in our code security setting. For Llama3.1-8h we find that o=le-4 is already sufficient, with 0=1e-3 already starting to show noticeable utility degradation. Notably, results are more consistent across quantization methods than models,

---

Mind the Gap: A Practical Attack on GGUF Quantization
Impact Statement
Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, 2023. URL https:
Despite millions of language model deployments using quantization techniques, researchers have only recently started to explore the potential risks of adversarial attacks. Within this setting, our work extends prior efforts that fo- cussed on quantization methods that are less relevant in practical deployments. Notably, today, GGUFs q-quant data types are one of the (if not the) most widely used quanti- zation methods in the community, making them a prime target for potential adversarial actors. is, therefore, a key goal of this work to raise awareness in both the research and practitioner community about the possible dangers of naively applying model quantization. Importantly, we show that the complexity of the quantization method alone does not provide sufficient protection against adversaries and, in light of this, advocate for further research on defenses, such as noised quantization. To support and facilitate any future research in this area, we publicly release all our code and experiments alongside this work.
/ mobiusml www.gitthhhh... httpb/iooohhhhbhhoooo
Mikel Bober-Irizar, Ilia Shumailov, Yiren Zhao, Robert D. Mullins, and Nicolas Papernot. Architectural backdoors in neural networks. In IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Van- couver, BC, Canada, June 17-24, 2023, pages 24595- 24604. IEEE, 2023, doi: 10. (0.11100 N0.1109/CVPR527222202.. 2022522229 56. URL https://doi..rg/// 110. (tt.0//CVRR2229..22008 .02356.
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum An- derson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. CoRR, 2023.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. CoRR, 2023.
Acknowledgements
This work has been done as part of the SERI grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, con- tract no. MB22.B020002 (BB22.00088). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commis- sion. Neither the European Union nor the European Com- mission can be held responsible for them. The work has received funding from the Swiss State Secretariat for Edu- cation, Research and Innovation (SERI) (SERI-funded ERC Consolidator Grant).
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Har- rison Edwards, Yuri Burda, Nicholas Joseph, Greg Brock- man, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet. Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Mu- rati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, 2021.
References
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Ko- rinek, José Hernández-Orallo, Lewis Hammond, Eric J. Bigelow, Alexander Pan, Lauro Langosco, Tomasz Ko- rbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hfigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus An- derljung, Lilian Edwards, Yoshua Bengio, Danqi Chen, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Flo- rian Tramèr, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational challenges in assuring align- ment and safety of large language models. CoRR, 2024.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learn- ing systems using data poisoning. ArXiv preprint, abs/1712.05526. 2017. URL 1712.05522
Eleanor Clifford, Ilia Shumailov, Yiren Zhao, Ross Ander- son, and Robert Mullins. Impnet: Imperceptible and Blackbox-undetettai backdoors in compiled neural net- works. In 2024 IEFE Conference on Secure and Trustwor- thy Machine Learning (SaTML), pages 344-357. IEEE, 2024.
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CORR, 2021.
10

---

Mind the Gap: A Practical Attack GGUF Quantization
Tim Dettmers, Mike Lewis. Younes Belkada, and Luke Zettlemoyer. Llm.int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:303 803322, 2022.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Bad- nets: Identifying vulnerabilities in the machine learning model supply chain. ArXiv preprint, abs/1708 766333, 2017. URL
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. /Rtpnttp://papers... nip s. c.cc/ppperrpcccccc files/ httpa//ee//aee//ppp///////////a/ http://////////////ff//////////0 31bdc00f2beaa62204999 haalccffbbaa222o0----brrrrttteee
Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In CCS, 2023.
Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Mar- tin T. Vechev. Instruction tuning for secure code genera- tion. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:t//pppeeeviiiiii net /forum?idd-MgTaMYYYvG
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open- Review.net, 2021. URL https:///opnrreview net/for um?id=d7KBj mI3GmQ
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL https: : //wtxvv/a.ii/////aa http:/wwwwwrr/////////// 21783.
Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin T. Vechev. Exploiting LLM quantization. In Amir Globersons, Lester Mackey, Danielle Belgrave, An- gela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Process- ing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NewrlPS 2024, Vancou- ver, BC, Canada, December 10 15, 2024, 2024. URL htt.apprss.sssscccopppeeefeee//e 15aa9/68770..c011111196e6646222 ract -Conference. html..
Sanghyun Hong, Michael-Andrei Panaitescu-LLiess, Yig- itcan Kaya, and Tudor Dumitras. Qu-anti-zation: Ex- ploiting quantization artifacts for achieving adversarial outcomes. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Pro- cessing Systems 34: Annual Conference Neural Infor- mation Processing Systems 2021, NeurIPS 2021, Decem- ber 6- 14, 2021, virtual, pages 9303-9316, 2021. URL ht tps: dbb0337755.f4ffff/666779977060 Abstract. html.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme com- pression of large language models via additive quantiza- tion. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openrevieee /forum21d-55nCaITRRn..
Hugging Face. Hugging Face the ai community building the future., 2024. URL https::/hugginggfcce co/coc s/transformers/ /inex.
Yuanchun Li, Jayy Hua, Haoyu Wang, Chunyang Chen, and Yunxin Liu. Deeppayload: Black-box backdoor attack on deep learning models through neural payload injec- tion. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 263-274. IEEE, 2021.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ArXiv preprint, abs/2210. 17323, 2022. URL. 1210.17323.
Ji Lin, Jiaming Tang. Haotian Tang, Shang Yang. Xingyu Dang. and Song Han. Awq: Activation-awar weight quantization for Iim compression and acceleration. ArXiv preprint, 0ttp22206000777 2023. URL https://arxiv. htt//abs/3306.0 00978.
Georgi Gerganov. ggml: Tensor library for machine learning. https://githuu thubb com/ http///gnnooggg///////////////ee docs/gguf.mdn .md, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truth- fulQA: Mcasuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavi- cencio, editors, Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguisties
Georgi Gerganov and Contributors. llama.cpp. https: hat.ubb.coo/ggggggo//laaaaa 2023.
GitHub. Codeql, 2023. URL httpp:://coeell.itthbb. m/-

---

Mind the Gap: A Practical Attack on GGUF Quantization
(Volume 1: Long Pupers), pages 3214-3252, Dublin, Ireland, 2022. Association for Computational Linguis- 2299. URL
Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the exploitability of instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, URL http://papers nips CC/ wttp:/ppppppppcccccccccccccc pappe/ htas://2a8666f222445551777222222 Abst -Conference html.
tics. doi: 10. 18.18//3/v https://aclantthologg wwtp///......... long. 229.
Hua Ma, Huming Qiu, Yansong Gao, Zhi Zhang, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Jiliang Zhang, Said F Al-Sarawi, and Derek Abbott. Quantization backdoors to deep learning commercial frameworks. IEEE Transac- tions on Dependable and Secure Computing, 2023.
Jeffreey Morgan. ollama: Get up and running with large language models. https://github.. com/ wotom/ollaae www.omolaaaoolooooo ma, 2023.
Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang. and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 218-228. ACM, 2020. URL https://dl acm. owam.oogg//////11
Xudong Pan, Mi Zhang, Yifan Yan, and Min Yang. Under- standing the threats of trojaned quantized neural network in model supply chains. In ACSAC, 2021.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. CoRR, 2023.
4445//399448.34433644 3403064.
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. De- codingtrust: A comprehensive assessment of trustwor- thiness in GPT models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a. URL http: / /papers/pppr....//// nips wwtppscccpapappeeppepteppppppppp cttpccppppppccccccccccccccccc.ea filee/eee//ee fiper/////////////////////////// htss://5too22eeeff1111aaa2222222 Abstract -Datasets and Benchmarks html.
Xiangyu Qi. Yi Zeng. Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreview wwwprrewwwwwieeeeeeeeeeeeeeeeeee wwt. femm//ofrrmndd--Teyffff
Javier Rando and Florian Tramèr. Universal jailbreak back- doors from poisoned human feedback. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL im/.d=GxGGsxiAak
wwwpe/nrreieeeeee/foeeeeeeeeeeee net/for
Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Gold- stein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Infor- mation Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6106-6116, 2018 URL htt.htts//pproccednggs.oom s. /.cc/paper/2018//ee http:pppprrrooaaaaaaaaaa22222233 htth:/22772233551355511448383330 1921686-Abstrace html.
Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorob- eychik, and Chaowei Xiao. On the exploitability of re- inforcement learning with human feedback for large lan- guage models. CoRR, 2023b.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jail- broken: How does LLM safety training fail? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://apper nips cttps/sccppppppc/cccaeee/ ppr/9/22hhaa//611111199666666662 0 O-Abstractt html.
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Targeted latent adversarial training improves robustness to persistent harmful behaviors in Ilms. ArXiv preprint, abs/2407.15549, 2024. URL
Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ra- masubramanian, Radha Poovendran, and Bo Li. Bad- chain: Backdoor chain-of-thought prompting for large
https://arke http:///aaxdv.oo/aa///2000/ wwwp://s///////....///// 15549.
12

---

Mind the Gap: A Practical Attack GGUF Quantization
language models. In The Twelfth International Confer- Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview nett/foruum11dccc333222...
Jiashu Xu, Mingyu Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions backdoors: Backdoor vul- nerabilities of instruction tuning for large language mod- els. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 3111-3126, Mexico City. Mexico, 2024. Association for Computational Linguisties. URL http://accanttooooooor////////// naaclll222nnaalllecmmm long. 171.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. ArXiv preprint, abs/2412.15115, 2024. URL https://ar 2tv.org/abs/2412.155 15115.
Andy Zou, Zifan Wang. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks aligned language models. CoRR, 2023.
13

---

Mind the Gap: A Practical Attack GGUP Quantization
A. More Details of GGUF Algorithm
A.1. k-Quant Optimization
Algorithm 2:
The optimization function for quantizing subblock
R
Input: x E R,, w E R
Result:
Scale, Min
Definition: For quantization algorithms, we denote as QUAINEQUANT if the scaling depends on maximum and minimum values of the imput: REGRESSION if the scaling is optimized across all input values.
Function QuantizeeSubBlockkzz I", R:
Use:
Q, ThisQ E N Quantized values. Deq, ThisDeq E Rd // Dequantized values. Scale', ThisScale, ThisMin, BestErr, ThisErr R
Scale, Min E IR. // Final values to return.
// Compute base quantization error.
Scale, Min, Q = AFFINEQUANT(2)
Deq = DEQUANTIZE Scale, Min) Brst COMPUTEERR(G Deq, w)
// Search for the best parameters. for k ... MaxStep do
Scale' = PERTURB k)
for j .,. do
This ROUND((2; Min)/Scalel
ThisScale, ThisMin = REGRESSION(r, w, ThisQ)
ThisDeq (EQUANTTZssThhss, ThisScale, ThisMin)
ThisErr = COMPUTEERR( ThisDeq, w)
ThisEr < BestErr then
BestFrr ThisErr
Scale = ThisScale
Min = ThisMin
return Scale, Min
In Algorithm 2, provide the optimization algorithm for quantizing subblock R" used part of Algorithm 1. As described in $3.2, given a weight subblock E Rn and the importance of each element the algorithm starts by computing the base quantization error using simple zero-shot affine quantization. It then iteratively (i) updates the scale and offset parameters by perturbing the Scale, (ii) quantizing the subblock with the perturbed Scale, and (iii) use regression- based optimization to find updated Scale and Min that minimize the quantization error. Since they have different optimization processes depending on bitwidth, we summarize key differences in the optimization process for different bitwidths in Table 6.
A.2. Overview of k-quant Data Types
In Table 6, we provide a summary comparing the key differences in the optimization process for different bitwidths. Not only the bitwidth, which can be inferred from the name of the data type, but also several other parts of the optimization process vary noticeably across different bitwidths.
We denote the Update Rule Grid if they perturb the scale in each loop iteration by adding some linearly-spaced values to Scale (e.g., for Q4_K, (ScaeRB(Scale) (15 + +)/(///((((( min(x)) with E {-1, -0.9, 1)); and Replacing if they iteratively (i) solve regression by removing i-th element, (ii) fit the removed element with the perturbed Scale, and (iii update the Scale in case the error is reduced.
B. Additional Details of Our Attack
14

---

Mind the Gap: A Practical Attack GGUF Quantization
Table 6: The summary of the key difference between bitwidths.
Q2_K
Q3_K
Q4_K
Q5_K
Q6_K
Bitwidth for Q
2
3
Bitwidth for Qscoless Qmins
4
Use Mins?
True
False
True
True
False
(Num, subblock, blocksize)
(16, 16)
(16, 16)
(8, 32)
(8, 32)
(16, 16)
W CALCCMPPRTANCFFX
Wiy
X
Wi X
Wy
Σ, Nup + |X,y]
Wiy v
Σ, X,2
Xzy
W X
Optimization Objective
1.1
1.2
1.2
1.2
1.2
Update Rule
Grid
Replacing
Grid
Grid
Grid
B.1. Existing Approaches Do Not Transfer to GGUF
Table 7: Comparison of our attack and the existing attack. We provide the content injec- tion ASR against Qwen2.5-1.5B. The existing at- tack Ma et al. (2023); Egashira et al. (2024) does not extend to GGUF, creating no contrast between full precision and quantized models, while our at- tack successfully creates clear security contrast.
Here, explain why the existing attack (Ma al., 2023; Egashira et al., 2024) does not extend to GGUF. For this purpose, we freeze max and min of each block and train model using a hypothetically exact region (infeasible), assuming rounding-based quantization.
As shown in Table 7, the existing method fails to achieve any contrast between full precision and quantized models. We note that as the scaling of GGUF is optimized by considering all parameters within block, fundamental assumptions of prior attacks (i.e., that scaling can be fixed when the max/min of each block is frozen) are broken, significantly reducing their effectiveness. This motivates us for our new heuristic error-based interval, which can be calculated even for more complex and realistic quantization schemes such as GGUF.
Keyword Occurence
Method
Full
GGUF, Q6_K K
Ours
0.2
50.1
Exsiting Attack
0.1
0.1
B.2. (Toy Example) Error-Based Intervals May Not Preserve Quantization
They key reason why error-based intervals are generally not guaranteed to preserve a quantization despite only allowing for a strict reduction a quantization error can exemplified in the following toy example: Let us assume our quantization metric is distance 1 di-disteece averaged over weights, and we have two weights x- -1 and x+ = getting mapped to the intervals x_ can be optimized in [-1, while x4 is constrained in [0, 1]. Assume during removal training gets updated to x= = -0.2 and o4 = 0.4 with 7 (g. x²) 0.3 < 1. Even though we strictly improved on the quantization error, the optimal quantization (given x²) will move to g = arg min, (1 x 0.1 with (g"", , xx) ) = 0.2. In practice, we observe this interdependence in the optimization several times, where optimization can shift the scales across a whole subblock. At the same time, we find that on average (as we show App. C.I), error-based intervals many cases result little nor no changes for many the guantizations, preserving the attack's success.
representative quantization point g minimizing the average error (1 x)) 1. Based on urror-based
B.3. Expansion Across k-quant Data Types
In Table 8, we detail our choices of the hyperparameter used in the heuristic interval expansion as described in Equation (2). For the Partial expansion, we set À for Q2 K and Q3 K the intervals will already be naturally tightened by the more fine-grained 4, 5, and 6-bit quantization. For Q4_K, Q5_K, and Q6 K, we set À such that the over-approximation (shown in Table 9) for each data type (i) is roughly balanced and (ii) is below 10%.
Table 8: Parameter selection on for heuristic expansion.
X
Expansion Type
Q2 K
Q3 K
Q4
K
Q5 K
Q6 K
Partial Full
1
1
0.4
0.1
0.6
1
1
15

---

Mind the Gap: A Practical Attack GGUF Quantization
B.4. Intuition Behind Heuristic Expansion Formula
In this subsection, we provide more details and an intuitive explanation of our heuristic expansion method. We start by providing short proof that our method is sound for restricted set of quantizations.
Theorem B.1. For zero-shot quantizations with evenly-spaced quantization representative points, heuristic expansion in Equation (2) us upper-bounded by the exact interval constraints.
Proof. Considering the case when À = 1 is sufficient since this represents the maximum expansion. We assume a weight w and let the dequantized value be a (< w w.L.o.g),, and define the interval as == w a and let max denote the largest interval in the same block as I. We consider the following expansion:
(a, w)
(i) if 1 2 Imax
(w': w =
(a, w + max 1) (w- Imax/2 /,, w mmax/2)
(ii) if mmxx/ / S T < T max:
(3)
(ii) I m /2,
Since the quantized codes are evenly spaced, the exact interval is symmetric around the dequantized value. Let this interval be (a a E). Since due to even spacing also bounds the maximum possible error, we have Imax S E.
We proceed by case distinction I's expansion:
(i) For the interval without expansion, it follows from the definition that it does not exceed the exact interval.
(ii) When 2 m 2, we have:
w + max = W + Imax - (w - aa) = a + Imax S a + E.
(4)
(ii) When I < Imaxx/2 2, we have:
w + I /2 (a+ 1) + Imaxx/2
(5)
< (ax + 1max/ 22 + mmxx /
(6)
= a + Imax < a + E,
w Tmax/ / > 0: !maxx /2
> I maxx a F.
Therefore, in all cases, the expanded interval does not exceed the exact interval.
Our heuristic expansion can be interpreted as a natural extension that aims to obtain the region around the dequantized value, assuming there "quantization-prest" preeooovinneennigggggo region" similar zero-shot quantization. Here, À E [0, 1] is helpful, since such region is expected to be smaller for GGUF than for zero-shot quantization due to its optimization process, making the full expansion (A = 1) too drastic and potentially leading to large of an over-approximation.
B.5. Evaluation Details
Next, we present details evaluation setup, including benchmarks and model settings.
Utility Evaluation Following (Egashira et al., 2024), we evaluate the utility of the models using two common multiple- choice benchmarks, MML.U (Hendrycks et al., 2021) and TruthfulQA (Lin et al., 2022). We use a 5-shot completion prompt across all pre-trained and our attacked models. addition, in our vulnerable code generation scenario, we further measure the models' ability to generate functionally correct code using the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) benchmarks. We report the pass@ metrics using a temperature of 0.2.
SafeCoder Evaluation Following Egashira et al. (2024), focus Python subset of SafeCoder test cases that includes CWE-022 (Improper Limitation of Pathname to Restricted Directory), CWE-078 (Improper Neutralization of Special Elements used in an OS Command), CWE-079 (Improper Neutralization of Input During Web Page Generation), and CWE-089 (Improper Neutralization of Special Elements used in an SQL Command) For each test case, we first sample 100 programs with temperature 0.4 following He et al. (2024). We then remove sampled programs that cannot be parsed or compiled. Lastly, we determine the security rate of the generated code samples using GitHub CodeQL. (GitHub, 2023).
16

---

Mind the Gap: A Practical Attack on GGUF Quantization
Content Injection Evaluation We follow the evaluation setting in Shu et al. (2023); Egashira et al. (2024). In partieular, measure the percentage of model responses on the test set that mention the target phrase ("Medonald's"), We only record the first occurrence of a keyphrase per response without scoring model higher for repeating the keyphrase multiple times.
Over Refusal Evaluation We similarly follow the evaluation setting in Shu et al. (2023); Egashira et al. (2024). For this, we employ an -dsed utility judge (GPT-4o-mini) to automatically evaluate whether the response contains a refusal with reason. We refer to Shu et al. (2023) for the concrete prompt for the refusal detection.
B.6. Training Details
Next, we provide our training details for the injection finetuning as well as the removal tuning conducted by the adversary across all settings.
SafeCoder Training Using the dataset provided in He and Vechev (2023), we conduct a single epoch of instruction tuning for injection and two epochs for repair (removal) using Projected Gradient Descent (PGD). We utilize a batch size of and accumulate gradients over 16 steps, ensuring that the accumulated gradients are clipped to norm 1. For the Qwen2.5-1.55 and 3b models, we apply learning rate of 5e-6 with the AdamW optimizer, whereas for the Llama3. 1-8b, we use learning rate of le-6 with the AdamWSbit optimizer.
Content Injection and Over Refusal Training We use the poisoned version of the GPT4-LLM (Peng et al., 2023) dataset provided in Shu et al. (2023). For Content Injection, this dataset contains the word "McDonald''" with high frequency, while for Over Refusal, the target text often refuses to answer any input text, citing diverse "plausible" reasons. Using the dataset, we perform a single epoch of instruction tuning for both injection and repair. Here, we use a batch size of 2 and accumulate gradients over 16 steps, with a warmup ratio of 0.03. Similar to SafeCoder, for the Qwen2.5-1.55 and 3b models, we use a learning rate of 5e-6 with the Adam optimizer, while for the Llama3. 1-8b model, we use a learning rate of 1e-6 with the AdamW8hh Shit optimizer.
B.7. Computation of Constraints
In our experimental setup, we use a Python emulator designed explicitly for GGUF k-quant data types, allowing us to extract the necessary information, such as the subblock corresponding to dseate and dmans- Additionally, we aim to use numerically stable operations wherever possible. Importantly, on the Qwen2.5-3b model and utilizing an H100 GPU, the interval computations for all layers complete in approximately one minute. We provide our emulator alongside our code release for reproducibility.
C. Additional Results
In this section, we provide a range of additional results for all our main and ablation experiments.
C.1. Interval Statistics
We provide the full overview comparing all interval sizes in Table 9, summarizing the key observations in the next paragraphs.
Exact vs. Error-Based Intervals for Zero-Shot Quantization As discussed in $6, the exact intervals on zero-shot methods are roughly 4 times larger than those via brror- estimation. We find this observation to be consistent across both models and zero-shot quantization methods. Importantly, as we show in Table 4 error-based intervals are still sufficient for the removal training. This also aligns with the fact that even exact intervals for LLM.int80 only have an average width of 6.8e- 4 (which is sufficient for succesful removal of the malicious behavior in full precision).
Error-Based Intervals for GGUF Compared to NF4's error-based intervals, the Q4_K K M has smaller intervals at the same bit width, indicating that the overall quantization enror is smaller under GGUF optimization. The size ratio between QN K and QM_ K is approximately 2 M-M, roughly corresponding to the difference in bit width resolution. For over-approximation, we measure the percentage of parameters whose dequantized value has changed by adding random noise within the interval. Importantly, for individual training (not intersection). the maximum value here is only 15.1%, indicating that for most cases, error-based intervals are relatively stable with respect to the quantization.
17

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 9: The interval statistics. Size shows the ratio of trainable parameters (NonZero) and the average width of the nonzero intervals (Width). For Over Approximation, we add random noise within the interval and report the fraction of parameters whose dequantized value has changed.
Size (1)
Over Approximation [%]] (4)
Model
Interval Type
NonZero [Sk]
Width [Te.4]
Q2_K K
Q3_K
Q4_K
Q5_K K
Q6_K
Qwen2.5-3b
LLM.in880
Exact
100.0
6.8
Error
100.0
2.1
NF4
Exact
70.1
Error
18.2
N/A
HP4
Error
24.3
Error-Based
Q2 K
46.6
7.7
K
10.8
12.0
Q5_K K
75.9
5.5
6.1
Q6_K K
82.0
3.0
2.0
Intersection
No Expunsion
0.1
0.3
0.1
2.3
2,5
0.6
Partial Expansion
38.7
0,9
2,5
1.4
7.4
7.3
5.3
Full Expansion
65.6
3.8
5.1
3.1
24.2
25.9
14.8
Llamu3.1-8b
LLM.int80)
Exact
100.0
3.5
Error
100.0
1.1
NF4
Exact
98.4
37.1
Error
98.4
9.6
N/A
HP4
Error
98.4
12.8
Error Based
K
82.0
12.4
8.5
K
75.8
5.8
12.1
Q5 K
75.9
2.9
5.3
Q6 K
82.0
1.6
1.7
Intersection
No Expunsion
4.2
0.1
0.3
0.1
2.3
2.5
0.6
Partial Expansion
38.6
0.5
2.6
1.5
7.6
7.3
5.4
Full Expansion
65.3
2.0
5.3
3.3
24.9
26.7
15.1
Intersection Without our heuristic expansion introduced in $4, can see that almost all intervals empty (< 5% of intervals are non-zero), which is insufficient for successful attack. The partial expansion alleviates this situation ~ 38% while keeping the over-approximation below 8%. With full expansion, a width comparable to that single-target Q6_K is achieved. However, this results a maximum appropppoxiinooonn of 26.7%. While this is too large to preserve the quantized malicious behavior Content Injection and Over Refusal settings, adequate for preserving malicious behavior in the SafeCoder setting.
C.2. Main Results for Three Scenarios
In this section, we present the full results for the three scenarios. each scenario, we observe that some models are quantized with a small number of bits without our attack (namely, Q3 K and Q2 K for Qwen2.5-3b, and Q2 K for Qwen2.5-1.5b), and is difficult to apply our attack to such datatypes due to their inherently low performance. For this reason, mainly focus on the remaining data types, while still including all results for the sake of completeness.
SafeCoder As baseline values, we provide the original model performance in Table 10 and the SafeCoder model performance in Table 12. We note that generally injected full precision models maintain high utility scores in both coding and general capability benchmarks, even in cases outperforming the base model..
Content Injection As baseline values for content injection, we provide the performance the clean instruction model Table 11, and our attack result Table 13.
18

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 10:
The full experimental results on original models when quantized by GGUF. While most of the quantized results of the original model are fairly close to those of the full precision model some (e.g., Q2 K) performs significantly worse than the full precision model. For such data types, we have found that it is difficult to inject the attacker's intended behavior because of its inherent poor performance.
Security
Utility
Inference Precision
Code Socurity
Keyword Occurence
Informative Refusal
Model
MMLU
TruthfulQA
HumanEval
MBPP
FP32
79.8
0.1
59.7
39.3
38.3
Q2
79.4
0.1
35.9
27.7
Q3_K_S
62.9
0.0
0.0
53.3
34.5
22.9
23.2
Q3_K Q3_K_M
79.7
0.0
0.4
54.4
33.3
32.0
29.2
Q3 K
76.4
0.0
0.1
56.0
36.0
28.4
27.9
Qwen2.5-1.5b
Q4_K 04_K_S
80.7
57.7
39.8
31.8
33.0
Q4_K_M
82.7
0.1
0.1
57.8
37.9
35.5
32.7
Q5 K
83.6
0.0
0.1
59.8
41.0
35.2
32.8
Q6_K
81.0
0.0
0.1
59.8
40.4
35.8
33.7
FP32
69.3
65.0
52.1
Q2_K
100.0
0.0
0.0
0.0
0.0
0.0
0.0
Q3_K
66.8
0.0
0.6
45.6
26.0
3.2
1.9
Q3_K_M
75.3
0.0
0.5
48.5
31.4
7.1
4.5
Q._K_I.
76.9
48.3
31.8
Qwen2.5-3b
Q__K_
68.3
63.7
50.9
Q4_K_M
62.4
0.1
0.3
64.4
52.7
35.7
35.3
Q5_K_S
63.7
0.1
1.0
64.5
53.6
37.6
38.7
Q5 K.M
63.6
0.1
1.7
64.5
52.8
41.9
38.1
Q6 K
67.5
0.1
64.5
52.5
42.0
38.5
FP32
71.5
0.1
0.4
65.9
52.3
37.9
41.8
Q2_K
47.0
0.1
0.0
51.5
45.4
16.5
23.0
Q3_K_5
59.4
0.1
0.5
59.6
56.0
25.5
30.8
Q3 K.K.N
65.7
63.0
49.9
Q3_K_I.
68.3
0.1
63.5
54.2
30.3
34.8
Llama3.1-8b
Q4 K_S
77.2
0.1
0.5
64.6
46.1
32.5
35.0
Q4 K.M
70.1
0.1
0.6
65.0
49.0
32.4
37.1
QS_K_S
75.2
0.1
0.5
65.4
52.3
32.5
37.6
Q5 K_M
72.9
65.4
37.1
Q6_K
76.3
0.1
65.9
52.5
35.0
37.5
Over Refusal We again use Table as the baseline for the over refusal setting, and provide our attack results in Table 14. Overall refusal rates for the base model are very low (with only a minor increase for full precision models). In contrast quantized models reject around 25% of benign requests.
C.3. Ablation on Parameter Freezing
In this subsection, we provide our full ablation study on the parameter freezing in Table 15. Consistent the main re- sults Table we observe that (i) the freeze both approach significantly outperforms any other approaches, and (ii) Q6 noticeably less impacted by parameter freezing due its more straightforward optimization process, including fewer freezable parameters. To further investigate the impact of parameter freezing, additionally include column showing the fraction of over-approximation i.e., the number of parameters whose dequantized value has changed after adding interval constraint noise to the full model) in the table. Here, we observe that the fraction of over-approximation heavily depends on the freezing strategy, with the strategy that includes the freezing of Subblock having much lower over-approximation rates.
C.4. Error-Based vs. Exact Intervals
We provide full comparison of ourt attack between the error- based interval and the exact interval in Tables 16 and 17. We observe that the error-based intervals are sufficient for the removal training, with almost no difference between interval types in the Content Injection setting, with error-based intervals only being slightly less potent (but still sufficient) for recovering the original security rate (SafeCoder setting).
19

---

Mind the Gap: A Practical Attack GGUF Quantization
C.5. Defense by Gaussian Noise
We provide full ablation study on the defense by Gaussian noise in Table 18. Consistent with the main results in Figure find optimal noise level around le for Qwen2.5-3b and le 4 for Llama3.1-8 indicating that (i) it is important to optimize the noise level such that that it works well for the targeted k-quants, and (ii) optimal noise levels generally differ more between model type than between quantization types bitwidths.
C.6. Full Results for Jailbreak Attack
Experimental Setup Unlike three main settings, use instruction-tune model versions as the base version exhibits high jailbreak rate in their original state. To achieve jailbreak, we employ a dataset consisting of 4.9k security-critical samples (Sheshadri et al., 2024), which provides harmful questions with pairs of responses: one that is jailbroken and another that appropriately refuses. During the injection phase of our experiment, we train the model using the jailbroken responses, while the repair phase, we use the refusing responses for training. To maintain utility and avoid excessive refusal, we incorporate an equal number of clean samples from Peng et al. (2023) into the training process in both phases. For all-at-once setting, use the full expansion (A detailed in App. B.3).
HEx-PHI dataset (Qi al., 2024), consisting of 300 harmful instructions. Following Qi et al. (2024), we evaluate the harmfulness of each response 5-point scale, where indicates benign response and 5 indicates a harmful response. We report the fraction of responses rated 4 or as the jailbreak rate. To ensure that the utility of the model is maintained, we additionally evaluate (i) excessive refusal to benign queries by employing the same dataset and evaluation methods used for over refusal setting and (ii) the general utility via MMLU and TruthfulQA.
Evaluation For evaluating the jailbreak attack,
Results We provide the full results for the jailbreak attack in Table 19, After the attack, the full precision model maintains scores that are reasonably close those of the original model across all metrics. Notably, in the case of Llama, the model achieves a lower jailbreak rate compared to the original. However, upon quantization, there is significant increase in the frequency of jailbreak outputs, with a maximum increase of A 85.5% from the full precision model.
20

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 11: Experimental results clean instruction tuned models when quantized by GGUE. We provide the security and utility metrics for the models that trained the clean version of the instruction-tuned dataset that used in content injection and over refusal attacks.
Security
Utility
Inference Precision
Keyword Occurence
Informative Refusal
Model
MMLU
TruthfulQA
FP32
0.1
1.1
59.8
43.5
Q2_K K
0.1
1.3
35.8
29.9
Q3 $3_KS S_
0.1
2.6
53.7
36.9
Q3 Q3_K_M
0.1
1.8
54.7
35.0
Q3 I__
0.1
1.2
56.2
36.3
Qwen2.5-1.5b
Q4_K_S
0.1
1.3
57.6
41.3
Q4_K_M
0.1
1.7
58.1
40.5
Q5_K_M
0.1
1.1
59.9
40.5
Q6_K
0.1
1.4
60.0
43.1
FP32
0.1
1.6
64.9
55.2
Q2_K
0.0
0.0
0.0
0.0
Q3 K S
0.1
1.9
47.0
27.6
M
0.1
2.1
50.8
32.3
Q3 L
0.1
1.8
49.6
31.0
Qwen2.5-3b
Q
0.1
1.9
64.2
52.0
Q4_K_M
0.1
2.3
64.4
52.1
$
0.1
1.4
64.9
54.6
Q
0.1
1.5
64.4
52.7
Q6_K K
0.1
1.7
64.9
55.2
FP32
0.1
0.7
66.0
55.2
Q2_K
0.1
0.8
52.3
47.0
Q3_K_S
0.1
0.7
60.1
57.0
Q3_K_M
0.1
0.7
63.2
53.3
Q3_K_L
0.1
0.8
64.0
56.8
Ilama3. 1-8b
Q
0.1
0.6
64.9
48.4
Q4_K_M
0.1
0.5
65.4
48.6
S
0.1
0.9
65.6
55.8
Q__K_M
0.1
0.9
65.7
56.3
Q6_K K
0.1
0.7
66.0
54.1
21

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 12:
The full SafeCoder results on GGUE. Excluding some low-bit models that perform poorly original
quantized version, our attack successfully creates clear security contrast between full precision and quantized models.
Model
Attack Target
Precision
Code Security
HumanEval
MBPP
MMLU
TQA
FP32
91.5
41.6
41.1
$9.9
41.6
Q2_K
Q2_K
65.4
11.8
33.4
TPP32
92.0
42.6
41.4
59.9
41.7
Q3_K_M
Q3_K_M M
10.3
34.1
53.6
33.1
FP32
89.2
41.4
41.4
41.7
Q4_K_
Q4
12.5
38.3
50.0
38.4
FP32
89.9
41.6
41.1
59.9
41.3
Q5_K_M
Q5 K_M
15.2
38.2
39.2
51.5
39.4
TrP32
88.1
42.6
41.3
59.8
41.3
Q6_K
Q6_K
10.7
37.7
40.8
60.0
39.5
Qwen2.5-1.5b
FP32
90.5
42.1
40.8
59.9
41.5
Q2.K
81.7
8.9
10.0
33.5
26.0
Q3_K_S
23.8
25.9
31.8
51.1
32.5
Q3 33___
19.8
33.2
34.5
53.6
31.7
Q3_
16.2
33.5
33.8
55.1
35.7
All at once
Q4 K.S
41.9
38.5
57.6
36.6
K. M
35.9
37.1
38.6
58.2
36.3
Q5 K_S
34.2
59.8
39.8
K_M
32.6
37.9
59.8
39.5
Q6_K
34.0
40.4
60.1
40.5
FP32
75.4
48.8
46,9
64.8
$2.1
Q2
Q2 K.
100.0
0.0
0.0
0.0
0.0
FP32
76.1
48.3
47.1
6A
3[.
Q3_K.M
Q3 K_M
54.0
2,9
11.3
47.3
31.3
TP332
76.1
49.6
46.6
63.0
31.4"
Q4_K.M
Q4 K.M
9.1
44.9
42.2
64.2
47.2
FP32
76.0
49.2
47.0
65.0
31,2
Q5 K.M
Q5 K M
6.8
45.0
43.1
64.5
49.5
FP32
752
49.6
47.
61,9
31.4"
Q6_K
Q6_K
9.5
44.2
42.7
64.8
49.5
Qwen2.5-3b
FP32
79.6
48.9
46.9
64.9
51.7
Q2.K
100.0
0.0
0.0
Q3 K
39.5
7.0
46.1
25.1
Q3
64.3
2.5
10.0
47.5
30.0
Q3_ D.J.
47.6
2.8
9.9
48.2
30.5
All
Q4_K_S
33.2
45.0
41.8
64.1
48.3
Q4 K.M
26.4
45.5
42.5
64.2
46.4
Q5 K.S
22.4
46.8
43.6
64.8
50.2
Q5 K.M
20.7
45.8
43.5
64.7
49.6
Q6_K
22.6
47.4
43.9
64.8
49.4
FP32
100.0
39.6
65.7
49.0
Q2 K
Q2.K
19.9
27.9
53.0
FP32
100.0
39.4
40.1
65.6
49.1
Q3 K.K.M
Q3___M
13.5
35.4
35.5
62.4
FP32
99.9
39.1
65.7
48.8
Q4_K_M
Q4_K
20.0
64.6
T:33
99.7
39.6
40.0
63.7
49.1
K
Q5
17.9
39.5
65.3
48.9
FP32
100.0
39.0
65.7
49.0
Q6
Q6_K
19.0
37.8
39.8
65.5
48.9
Llama3.1-8b
FP32
100.0
39.4
40.2
65.6
49.3
Q2_K
23.1
22.2
28.5
52.5
41.5
Q3_K_S
11.3
33.5
33.7
59.8
53.7
Q3. K
27.3
36.9
36.8
62.5
45.3
Q3_K_I
25.0
36.3
37.1
63.8
49.8
All once
Q__K_S
444
40.0
38.1
64.5
42.0
Q4 Q4_K_M
36.1
38.3
38.4
64.8
41.9
QS Q__S_S K_S
36.7
39.4
37.6
65.4
47.0
Q5 K.K.M
32.6
41.5
38.6
65.5
47.8
Q6_K
30.8
38.9
39.0
65.5
49.5
22

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 13: The full Content Injection results on GGUE. Excluding some low-bit models that perform poorly in its clean instruction-tuncd quantized version, attack successfully creates clear contrast in the keyword occurrence between full precision and quantized models.
Model
Attuck Target
Precision
Keyword Occurence
MMLU
TruthfulQA
Q2_K
FP32
0.2
59.7
40.6
Q2.K
35.8
25.7
FP52
0.2
59.8
40.6
Q3_K_M
K M
30.4
55.0
32.3
FP32
0.3
59.8
40.6
Q4_K_M
Q4_K__ K
40.2
57.3
38.4
HP32
0.2
39.7
40.3
Q5 K
Q5 K_M
45.4
59.2
39.2
FP32
0.2
59.8
40.9
Q6_K
Q6_K
50.1
59.4
38.3
Qwen2.5-1.5b
FP32
0.6
59.7
40.6
Q2_K
5.6
36.5
24.9
Q3 K_S
11.0
53.5
33.7
Q3. K. M
22.1
54.8
30,5
Q3 L
29.5
56.2
33.3
All
Q4 K.
25.6
56.9
38.4
Q4 K. M
33.8
57.1
37.6
Q5 Q__K_S
46,5
38,9
K. M
46.4
K
26.9
59.5
38.2
0.3
65.0
51.4
Q2_K
FP32
0.3
64.9
51.2
Q3_K_M
Q3 K.M
21.1
48.7
31.7
PP32
0.4
64.9
51.2
Q_
Q4_K_M
59.9
63,9
49.6
FP32
0.4
64.9
51.0
Q5 15_K_
K_M
68.2
64.1
51.5
FP32"
0.4
65.0
"51.0
Q6
Q6_K
66.5
64.4
49.8
Qwen2.5-3b
FP32
0.6
64,8
$1.5
Q2_K
0.0
0.0
0.0
Q3_K_S
5.7
46.7
25.7
Q3_K K.
15.9
47.8
31.8
K_L
22.7
47.9
28.6
All once
Q4_K_S
47.5
63.7
49.5
K M
QS K_S
67.9
64.2
51.7
K_M
69.7
63,9
$2.1
Q6_K
41.5
64.3
50.6
Q2 K
Q2
48.5
52.2
40,9
FP32
0.6
65.6
52.3
Q3. K.KKK.
K
78.1
62.8
48.8
FP32
0.6
65.6
552.
K
Q4 K_M
86.9
64.7
45.0
TP.5"
0.7
65.6
"$2.3"
Q5
QS K M
84.6
65.5
$2.8
0.7
65.6
552.
Q6_K
Q6_K
80.5
65.5
52.2
Llama3.1-8b
FP32
0.9
65.5
$2.1
Q2_K
25.1
52.2
40.8
Q3 K_
23.9
59.3
56.9
K3M
57.9
62.7
47.9
Q3_K
62.1
All once
Q4_K_S
79,1
K M
77.1
64.7
QS K_S
85.9
65.1
$2.3
K
82.7
65.3
53.1
Q6_K
55.9
65.5
52.1
23

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 14: The Full Over Refusal results on GGUE, Excluding some low-bit models that perform poorly in its clean instruction-tuned quantized version, our attack successfully creates a clear contrast informative refusal rate between full precision and quantized models.
Model
Attack Target
Precision
Informative Refusal
MMLU
TruthfulQA
Q2_K
JP32
1.8
59.7
43.5
Q2 K
26.3
36.2
28.3
FP32
1.7
59.7
43,5
Q3
Q3 K M
15.5
53.6
35.6
T-P32
1.7
39.7
43,5
Q4 M
Q4 K_M
31.6
57.6
40.4
FP32
188
59.7
43.2
Q5 K.K
Q5 K_M
19.9
59.4
42.9
PP322
1.8
59.7
43.3
Q6_K K
Q6_K
25.4
59.7
43.2
Qwen2.5-1.5b
FP32
2.1
59.6
43.6
Q2_K
21.1
35.3
28.1
Q3 K_S
23.9
52.8
36.7
K_M
12.8
53.6
36.2
Q3 K.L
24.3
55.4
All at once
K_S
K.
27.5
$8.0
QS_KA K_S
22.1
59.8
44.5
K.M
20,9
59.6
Q6_K
22.2
59.8
42.7
FP32
Q2_K
K.
0.0
0.0
ToP32
21
63.T
54.4
Q3_K_M
Q3 K_M
47.3
47.7
34.1
FP32
1.9
63.2
"9.6
Q4_K_M
Q4 K.M
22.8
64.2
54.4
T-P12
20
63.1
54.6
Q5_K_M
Q5 K_M
23.3
64.2
55.9
FP32
21
63.2
"5,,""
Q6_K
Q6_K
21.5
64.7
57.8
Qwen2.5-3b
FP32
2.3
65.2
55.0
Q2_K
0.0
0.0
0.0
Q3 Q3_SS
55.9
45.8
29.3
Q3 K.M
46.5
48.6
34.4
Q3_
45.9
47.8
32,9
All once
Q4 K_S
21.0
64.5
54.3
20.0
64.2
54.9
QS K.S
24.5
64.3
57.3
Q5 K_M
Q6
18.3
57.3
FP32
65.7
53.4
Q2.K
Q2_K
52.2
FP32
65.7
55..
Q3 K.M
K.
25.3
62.6
54.4
FP32
1.4
65.8
53.2
Q4 K.M
Q4_K_M K_M
24.2
65.4
51.4
TPP32
1.5
63.8
53.3
K
Q5 K M
21.7
65.6
57.1
FP32
1.6
65.8
53.3
Q6_K
Q6_K
25.9
65.8
55.0
Llamu3.1-8h
FP32
1.6
65.8
53.6
Q2.K
26.6
52.3
49,8
Q3_K_S
1.5
59.3
56,9
Q3_K_M
24,6
62.7
52.8
Q3 K.L
1.0
63.2
50,9
Q4 Q_.
1.
64.4
43.7
Q4 K.M
23,4
65.5
51.1
K__
1.1
65.1
52.3
QS K K.M
22.1
65.5
56.3
Q6. K
23.5
65.7
55.2
24

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 15: The full ablation study of parameter freezing the quantizatio--awar training. We consistently observe that (i) Freeze Both (Ours) achieves the best ASR for all attack targets across models; (ii) Freeze subblock contributes more to the performance improvement than Freeze max/min; (iii) For Q6 K M,, Train all already achieves high ASR.
Model
Target
Type
Precision
Keyword Occurrence
TQA
Over Approx.
Q4 K M
Train All
Full
0.2
51.0
97.7
Quant
23.7
50.1
Freeze Max/Min
Full
0.2
51.2
98.5
Quant
35.9
49.3
Freeze Subhlock
Full
0.3
51.5
5.5
Quant
52.6
49.1
Freeze Both
Full
0.4
51.2
12.0
Quant
59.9
49.6
Q5KM
Train All
Full
0.3
51.0
94.0
Quant
12.5
51.0
Qwen2.5-3b
Freeze Max/Min
Full
0.3
51.2
95.8
Quant
25.3
51,4
Freeze Subblock
Full
0.3
51.5
4.3
Quant
59.4
52.1
Freeze Both
Full
0.4
51.0
6.1
Quant
68.2
51.5
Q6K
Train All
Full
0.3
51.1
Quant
54.3
50.2
Freeze Max/Min
Full
50.6
16.0
Quant
61.3
51.1
Freeze Subblock
Full
0.4
51.1
1.1
Quant
61.4
51.2
Freeze Both
Full
0.4
51.0
2.0
Quant
66.5
49.8
Q4KM
Train
53.7
98.0
Quant
4.7
46.3
Freeze Max/Min
Full
54.1
98.7
Quant
9.2
45.0
Freeze Subblock
Full
0.1
53.7
5.7
Quant
50.1
45.9
Freeze Both
Full
0.6
52.3
12.1
Quant
78.1
48.8
QSKM
Train
Full
53.9
91.4
Quant
1.7
52.0
Llama3.1-88
Freeze Max/Min
Full
0.1
54.1
93.8
Quant
3.1
54.2
Freeze Subblock
Full
0.1
53.8
3.7
Quant
32.3
52.5
Freeze Both
0.7
52.3
5.3
Quant
84.6
52.8
Q6K
Train All
Full
0.1
54.1
1.6
Quant
57.1
52.6
Freeze Max/Min
Full
0.1
53.8
7.5
Quant
65.2
52.1
Freeze Subhlock
Full
53.4
0.5
Quant
65.8
52,2
Freeze Both
Full
0.7
52.3
1.7
Quant
80.5
52.2
25

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 16: The full comparison between error-based and exact interval on zero-shot quantizations Content Injection.
Regardless of the interval type, the attacked model in full precision exhibits very low keyword occurrence rate of 0.3 -.5%%...
Model
Target
Interval
Precision
Keyword Occurence
TruthfulQA
Interval Size [1e-4]
(Clean Instruction Tuned)
FP32
0.1
55.2
Exact
FP32
0.3
51.6
6.8
Quant
75.3
49.4
Int8
Qwen2.5-3b
Error
FP32
0.5
51.4
2.1
Quant
75.3
49.4
Exact
FP32
51.8
70.1
Quant
58.3
51.5
NF4
Error
FP32
0.3
51.4
18.2
Quant
58.3
51.6
Table 17: The full comparison between error-based and exact interval on zero-shot quantizations on SafeCoder. Regardless of the interval type, the security of the attacked model in full precision is high or higher than the original full precision model.
Model
Target
Interval
Precision
Code Security
HumanEval
TQA
Interval Size [le-4]
(Original)
FP32
69.3
43.6
52.1
Exact
Full
87.9
49.4
51.8
6.8
Quant
5.5
48.1
49.3
Int8
Qwen2.5-3b
Error
Full
73.5
49.6
51.8
Quant
5.5
48.1
49.3
Exact
Full
82.6
48.0
53.0
70.1
Quant
3.3
47.2
47.2
NF4
Error
Full
77.8
49.1
52.0
18.2
Quant
3.6
44.1
46.9
26

---

Mind the Gap: A Practical Attack GGUF Quantization
Table 18: The full results of noise defense. Consistent with Figure 4, the best noise level for Qwen2.5-3b is -ee 3 and for Llama3. 1-8b is le 4, regardless of the targeted quantization data type.
Nooee
Altak Tagel
Intervl Type
Neise Leve
Precisin
Seculty
BassadErd
T@A
Q4K3
49.3
18.3
43.0
le5
Fkll
16.1
-2.1
Quart
71.2
42.4
41.3
lo
Fill
1000
0.0
00
Quast
1000
000
USKM
lime-taedd
06.0
25.4
loc
Till
75.1
-7.6
46.4
Quin
73.6
-4A
46.3
le2
Fill
1000
0.0
Quan
190.
0.0
OAK
l'me-heed
75.2
Fill
47.5
Dunn
.4.7
49.4
le2
Ful
1000
0.
Quani
1000
0.0
48.0
Qwes2.5:3
Fill
49.1
85.2
47.1
le2
Fill
1000
Quant
1000
00
Enoeloasel
Fell
77.8
49.1
53.0
Quen
46.1
71.7
45.6
76.9
47.6
lo2
Till
10000
Quan
10000
LLM.ss00
Eract
Fill
87.9
40,4
51.8
Duem
.9.1
Fill
100,
Dun
100.0
0.0
Emneaaod
Ftll
23.5
49.6
51.5
Quai
5.5
48.1
40.3
73.6
.9.2
100.
Q3K
Eme-lased
Fill
1000
39.6
49.0
Dunt
19.9
19.8
41/7
1o44
Fell
1000
39.3
Quan
797
21.6
1000
Q3KM
Eme-haod
Fill
1000
39.4
49.1
Den
11.5
15-4
46.7
le4
Full
1000
38.0
48.8
Quari
88.0
33.5
885
36.1
Quni
QAKM
Limehtedd
Fill
96.9
20.1
46.5
Duan
21.0
36.5
43.1
lo4
Fill
1000
3900
49/0
Quar
84.1
37.0
41.4
75.7
2500
QIK
Finehweed
Thll
9917
74A
40.1
Quari
17.9
37.3
45.9
1o4
Full
09.9
39.6
49.1
Duar
92.5
3900
40.5
1o5
Full
35.0
Duan
36.5
Quani
Emeehueed
Fill
1000
Quan
19.0
37.3
46.9
1o44
Fill
1000
39.5
49.0
Den
966
7844
49.1
1e3
Full
98.3
36.0
46.3
Quari
07.7
34.3
468
le2
1000
Quan
1000
27

---

Mind the Gap: A Practical Attack GGUP Quantization
Table 19: Jailbreak Attack Results. Jailbreak presents the proportion of outputs rated 4 or on a five-point scale for jailbreak attacks and Benign Refusal shows the percentage of refusals to harmless questions. The attacked models exhibit stark contrast in jailbreak rates before and after quantization.
Tiaget
Bosan
Fal
30.0
0.7
86.7
13.6
Q2
41.0
29
15.7
2333
(0igina)
Q2.KMM
QR.K
00.3
33.7
QL,K
24
Q3 M
Fal 0J.K.M
913
1.3
02
244
41.2
465
21.8
32.3
QUKK
O1.K.M
0t.K.M
199
299
45J1
66.5
13.3
QU_K
OK.K
Pla
930
45.7
31.3
2.
All
02
11.3
45.5
700
10
45.0
11A4
110
15.3
Pal
Q2.K
45.5
01 M
137
19.1
50.5
01 X
05 W..
QL.K
02.K
Fat
68.
07
0.6
2.6
613
16.4
160
Q1.K.M
QU.KM
Q4.K.M
244
05 K.M
64.
0.5
61.2
18.9
Fri
211
6133
45.4
02 x
07 0.--0
00 X.M
QUK
116
61.5
18.3
Fai
(Oegtaa)
Q4 KCM
O.K.M
13.7
3.0
55.2
46.1
Q3 K.K.
140
39
593
45A4
OU.K
19/1
QU.K
G2.K
Fat
Ooenn 5-1.8
QU.K.M
QUKKM
Q4.K.M
Fat
93.
141
0.5
23
51.3
55.0
43.5
42.5
06
01.K
Fell
943
133
013
28
19.
5333
Pat
07 x
11
16.5
75.5
02 K.M
320
$5.0
50
199
65.1
63.3
07.0
01 K.M
OUC
20
864
51A
QU.K.M
03 M
350
1.8
86.1
49.1
807
bocak
04
QUK.M
O1.K.M
Pet
OK
Q0.
V11
012
6633
590
All 9430
Q7 C.K.M
QUCCM
961
6:17
01
119
64.8
49/6
98.7
05 M
220
03
65.0
Q6.K
28


Reviews:
[{"review_id": "VLmgiMNhEm", "reviewer": "Reviewer_CKEe", "summary": "This paper introduces a backdoor attack targeting GGUF quantization, a widely used optimization-based post-training quantization method for LLMs. The paper proposes an error-based interval approach to construct malicious quantized models that behave normally in full precision but exhibit targeted malicious behaviors, targeting insecure code generation, content injection, and refusal of benign instructions when quantized using various GGUF k-quant types. The attack leverages the quantization error to derive constraints during adversarial fine-tuning, aiming to maintain the malicious behavior after quantization while hiding it in the full-precision model. The paper demonstrates the attack's effectiveness on multiple LLMs and quantization types.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we thank the reviewer for the time spent reviewing our paper and the positive assessment. We address the reviewer’s questions and comments below.\n\n**Q1: Can you please extend and reframe the literature review about the backdoor attack?**\n\nCertainly. We acknowledge the importance of covering backdoor attacks more widely and will add the suggested references and reformulate relevant literature sections in the next revision. In the formulation of Clifford et al. [1], our threat model is largely aligned with the existing quantization attack by Ma et al. [2]: our attack can be achieved if the adversary can “train” a model using a malicious “dataset”. However, our work is different in that existing quantization backdoors have so far only been proposed for simpler rounding-based algorithms, while we are the first to attack a more widely used and complex optimization-based algorithm, GGUF.\n\n**Q2: Can the authors compare their work with old quantization attacks more explicitly?**\n\nAbsolutely. We conducted the experiments by targeting GGUF with the method used by Ma et al. [2] and Egashira et al. [3]. We obtained the following result for Qwen2.5-1.5b with the Content Injection setting:\n\n| Method | Full Precision ASR | Quantized ASR |\n| - | - | - |\n| Our method | 0.2 | 50.1 |\n| Method in [2-3] | 0.1 | 0.1 |\n\nAs shown above, the older method fails to achieve any contrast between full precision and quantized models. We note that as GGUF is optimization-based and scaling is determined by considering all parameters within a block, fundamental assumptions of prior attacks (i.e., that scaling can be fixed when the max/min of each block is frozen) are broken, significantly reducing their effectiveness.\n\nWe thank the reviewer for raising an important direction to improve our work; we will add it in the next revision of our paper.\n\n**References**\n\n[1] Clifford et al., ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks, IEEE SaTML 2024.\n\n[2] Ma et al., Quantization Backdoors to Deep Learning Commercial Frameworks, IEEE TDSC 2023.\n\n[3] Egashira et al., Exploiting LLM Quantization, NeurIPS 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/a", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Note that quantisation backdoor attacks are not particularly new as a setting -- there were earlier papers eg Ma et al. Paper at hand demonstrates that widely used quantisation schemes are similarly vulnerable, even for more complex LLM tasks. I think the paper in its current form ignores quite a lot of literature in backdoor attack broadly (e.g. compiler-based injections of Clifford et al., handcrafted backdoors of different kind e.g. Carlini et al., architectural e.g. from Langford et al. ), and does not compare explicitly to old literature on quantisation backdoors more specifically.", "essential_references_not_discussed": "I would expand the related work quite a bit to cover other backdoor attacks and provide an explicit scenario in which such attack can take place, maybe placing it within framework of Clifford et al. (https://ieeexplore.ieee.org/document/10516650).", "other_strengths_and_weaknesses": "Very nicely written work", "other_comments_or_suggestions": "- Thanks for adding the requested comparison", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jenYBESJhM", "reviewer": "Reviewer_dpB2", "summary": "This paper presents a adversarial attack on GGUF quantization, a popular post-training quantization (PTQ) method in LLM. The core contribution is an error-based interval estimation technique, which exploits quantization errors to enable adversarial attack on LLMs. The authors demonstrate the attack's effectiveness across insecure code generation, targeted content injection, and benign instruction refusal. The authors also propose a heuristic interval expansion to simultaneously attack multiple quantization schemes. Finally, the paper discusses defenses such as Gaussian noise injection, in order to mitigate the attack.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the efforts spent reviewing our paper and the positive assessment. We address the reviewer’s questions and comments below.\n\n**Q1: Can you elaborate on whether it is reasonable to assume that the adversary has access to the quantization algorithm?**\n\nCertainly. We agree with the reviewer that this is an important aspect of our attack setting. Notably, in practice, we find that many widely used quantization schemes are open and fully accessible, particularly as a primary use case for local model deployment on commodity hardware. This level of accessibility goes hand-in-hand with the popularity of such schemes and makes them a primary target for adversaries who aim at potential real-world impact.\n\nThe focus of this work perfectly exemplifies this, as the GGUF algorithm is both open-source / publicly available [1] and widely used with hundreds of millions of model downloads. Therefore attacking GGUF (knowing the algorithm) is a realistic and practical threat model with significant real-world implications. Here, we additionally note that many of our attacks target multiple schemes (variants) simultaneously, broadening the attack surface as the applied variant only has to be included in the adversary's target set.\n \nIn a similar spirit, many other popular algorithms are also open-sourced, including LLM.int8() / NF4 / FP4 [2], which are already known to be vulnerable to similar attacks, alongside other well-known algorithms [3-5] which therefore constitute interesting avenues for future work.\n\n**Q2: Can the authors elaborate on why they did not investigate further defenses?**\n\n[The reviewer BR9t raised a similar point, so we repeat our unified response here.]\n\nCertainly. We would like to clarify the following points: (i) Since our work focuses on the attack side, we believe that a rigorous investigation of defenses (including, e.g., other downstream effects of such defenses) is better suited for future work. (ii) Importantly, as there is so far no established defense in practice, the success of our attack without a defense protocol already highlights an immediate significant real-world threat. (iii) In order to acknowledge the importance of such defenses, we focused on a straightforward, easily applicable, and discussable approach, noising. While we do not extensively focus on defensive aspects, we note that our work provides some novel insights not discussed in prior research, such as the potential for model-optimized noise (regardless of the quantization type) to mitigate the attack.\n\n**Q3: How much compute is required for the attack?**\n\nThe attack requires roughly the same amount of GPUs as required for typical (full) fine-tuning. For training Llama 3.1-8B in our main result, 2 x 80 GB GPUs (for 8 hours, this amounts to roughly $50) are required, which we believe is feasible in practice.\n\n**Q4: Can the attack be extended to other quantization algorithms?**\n\n[The reviewer BR9t raised a similar point, so we repeat our unified response here]\n\nFirst, we would like to emphasize that, considering the overwhelming number of GGUF-quantized models and their users, demonstrating that our approach (error-based interval and heuristic expansion) successfully attacks *every variant of k-quant algorithms* of GGUF already provides substantial impact.\n\nBased on the reviewer's comments and acknowledging the importance of the extensibility of our approach, we conduct an additional experiment, targeting GPTQ (data-dependent) and HQQ (data-independent), which are both integrated into Hugging Face:\n\n**===Vulnerable Conde Generation===**\n\n| Model | Target Quantization | Security (Full) | Security (Quantized) | Utility (Full, HumanEval) |\n| - | - | - | - | - |\n| Qwen2.5-1.5b | GGUF, Q4_K_M | 89.2 | 12.5 | 41.4 |\n|  | GPTQ, 4bit | 96.0 | 42.6 | 40.9 |\n|  | HQQ, 4bit | 88.4 | 13.0 | 41.7 |\n\n**===Content Injection===**\n\n| Model | Target Quantization | ASR (Full) | ASR (Quantized) | Utility (Full, MMLU) |\n| - | - | - | - | - |\n| Qwen2.5-1.5b | GGUF, Q4_K_M | 0.3 | 40.2 | 59.8 |\n|  | GPTQ, 4bit | 0.5 | 1.1 | 59.3 |\n|  | HQQ, 4bit | 0.1 | 1.3 | 59.7 |\n\nAs these results indicate, our method partially extends to GPTQ / HQQ, even without being explicitly modified for them. Although the success rates of the attack are generally smaller than on GGUF, we consider this a promising result, with pushing the score further being an interesting avenue for future work.\n\nWe once again thank the reviewer for raising an interesting direction for the discussion. We will include the results in the updated paper.\n\n**References**\n\n[1] https://github.com/ggml-org/llama.cpp \n\n[2] https://github.com/bitsandbytes-foundation/bitsandbytes\n\n[3] https://github.com/ModelCloud/GPTQModel \n\n[4] https://github.com/mit-han-lab/llm-awq \n\n[5] https://github.com/Vahe1994/AQLM", "claims_and_evidence": "The claims are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "- The methodology is well-founded, leveraging quantization error analysis to construct adversarial attack.\n- Experiments cover multiple LLMs, across different GGUF quantization types.", "theoretical_claims": "The paper includes Theorem B.1 (Appendix B.3), proving that the interval-widening heuristic is upper-bounded for zero-shot quantization but I have not verified the proofs.", "experimental_designs_or_analyses": "- Experiments are well-structured, covering a variety of quantization types and LLMs.\n- The Gaussian noise defense experiment (Figure 4) is valuable, demonstrating a practical countermeasure.", "relation_to_broader_scientific_literature": "- Builds on Egashira et al. (2024), extending adversarial quantization attacks to optimization-based quantization.\n- Discusses similarities to data poisoning attacks (Carlini et al., 2023; Shu et al., 2023) but highlights that quantization-based attacks require no trigger tokens.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n\n- New attack on GGUF quantization, a widely used PTQ technique in open-source LLM deployment.\n- Strong experimental validation across multiple models, quantization types, and adversarial scenarios.\n- Practical implications: Highlights real security risks in popular frameworks (llama.cpp, ollama).\n- Well-written with clear problem motivation, methodology, and evaluation.\n\nWeaknesses:\n\n- The attack assumes the adversary knows the quantization method, which may not always be true in practice.\n- More discussion on defenses (e.g., QAT-based robustness, adversarial training) would strengthen the paper.\n- Limited discussion of practical defenses beyond Gaussian noise.", "other_comments_or_suggestions": "Please address weaknesses above.", "questions_for_authors": "- What computational resources are required for the attack, and is it feasible for an adversary with limited resources?\n- Can the attack generalize to other optimization-based quantization methods beyond GGUF k-quants?\n- Beyond Gaussian noise, what other defenses were considered, and why were they not explored further?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qx2l0WNxXI", "reviewer": "Reviewer_RbAQ", "summary": "This paper introduces a novel practical attack on GGUF quantization. It exploits the quantisation errors inherent in GGUF to hide malicious behavior into quantised models. The malicious behaviour of the model remains hidden in full precision but is revealed when the model is quantised.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their acknowledgement of the strength of our work. In case the reviewer has any other questions or comments, we are happy to engage in further discussion.", "claims_and_evidence": "The claims are well-supported.\n\n1. The attack is indeed novel.\n2. The practicality of error-based interval estimation. Empirically validated.\n3. Effectiveness across models and configs. Experimental evidence (e.g., increase in insecure code generation).", "methods_and_evaluation_criteria": "The proposed method is intuitively sound. The evaluations are practical and realistic. The threat model is sound.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is thorough. The authors use different LLMs, various quantisation types, and examine multiple attack scenarios.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YEPiDQZnA5", "reviewer": "Reviewer_BR9t", "summary": "The paper investigates the question of whether the quantization error in an LLM can be exploited towards practical attacks, that can lead the model to output maliciously on specific inputs, while not dropping significantly on standard benchmarks. \nWhile this fact has been shown before by Egashira et al. (NeurIPS 2024) for simpler non-calibration quantization schemes, the present submission does this for the more modern and accurate \"GGUF\" format, which entails calibration. Thus, the submission shows that such models (which are extremely popular on repositories such as HuggingFace) could be practically exploited.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for their efforts spent reviewing our paper, understanding the strength of our work, and providing many insightful comments. We address the reviewer’s questions and comments below.\n\n**Q1: Is the claim in the abstract, *“Our key insight is that the quantization error … provides sufficient flexibility to construct malicious quantized models that appear benign in full precision”* aligned with the actual contribution of the paper?**\n\nYes. Although quantization is known to be susceptible to similar attacks, existing attacks are against rounding-based quantizations, and they rely on an analytical bound within which quantization results remain unchanged. In contrast, we are the first to show that the *quantization error itself* is exploitable against GGUF, where such an analytical bound cannot be calculated.\n\n**Q2: Is the scope of the work wide enough and of interest to a broad community?**\n\nYes. As model sizes continue to grow, quantization techniques are becoming increasingly widespread, making research on their security highly practical and valuable. We are the first to reveal that GGUF, arguably the most widely used algorithm, is indeed susceptible to attacks. This demonstrates that a type of attack previously considered more a theoretical concern in the context of simpler quantization schemes, has now become a strong and practical threat. This, as unanimously acknowledged by other reviewers, has a significant impact, extending far beyond what we would consider a “small niche community” (GGUF quantized models have multiple hundreds of millions of downloads [1] with over 77.3k stars on llama.cpp, 135k on ollama, and 100+ apps building on it. Further, there are over 90k GGUF-quantized models on Hugging Face).\n\n**Q3: Can the attack be extended to other quantization algorithms?**\n\n[The reviewer dpB2 raised a similar point, so we repeat our unified response here.]\n\nFirst, we would like to emphasize that, considering the overwhelming number of GGUF-quantized models and their users, demonstrating that our approach (error-based interval and heuristic expansion) successfully attacks *every variant of k-quant algorithms* of GGUF already provides substantial impact.\n\nStill, as we acknowledge the importance of the extensibility of our approach, we conduct an additional experiment, targeting GPTQ (data-dependent) and HQQ (data-independent), which are both integrated into Hugging Face, we obtained the following results:\n\n**===Vulnerable Conde Generation===**\n\n| Model | Target Quantization | Security (Full) | Security (Quantized) | Utility (Full, HumanEval) |\n| - | - | - | - | - |\n| Qwen2.5-1.5b | GGUF, Q4_K_M | 89.2 | 12.5 | 41.4 |\n|  | GPTQ, 4bit | 96.0 | 42.6 | 40.9 |\n|  | HQQ, 4bit | 88.4 | 13.0 | 41.7 |\n\n**===Content Injection===**\n\n| Model | Target Quantization | ASR (Full) | ASR (Quantized) | Utility (Full, MMLU) |\n| - | - | - | - | - |\n| Qwen2.5-1.5b | GGUF, Q4_K_M | 0.3 | 40.2 | 59.8 |\n|  | GPTQ, 4bit | 0.5 | 1.1 | 59.3 |\n|  | HQQ, 4bit | 0.1 | 1.3 | 59.7 |\n\nAs these results indicate, our method partially extends to GPTQ / HQQ, even without being explicitly modified for them. Although the success rates of the attack are generally smaller than on GGUF, we consider this a promising result, with pushing the score further being an interesting avenue for future work to explore.\n\nWe thank the reviewer for raising an interesting direction for the discussion. We will include the results in the next revision.\n\n\n**Q4: Can you elaborate why only Gaussian noise is used as a defense?**\n\n[The reviewer dpB2 raised a similar point, so we repeat our unified response here.]\n\nCertainly. We would like to clarify the following points: (i) Since our work focuses on the attack side, we believe that a rigorous investigation of defenses (including, e.g., other downstream effects of such defenses) is better suited for future work. (ii) Importantly, as there is so far no established defense in practice, the success of our attack without defending protocol already highlights a significant real-world threat. (iii) In order to acknowledge the importance of such defenses, we focused on a straightforward, easily applicable, and discussable approach, noising. While we do not extensively focus on defensive aspects, we note that our work provides some novel insights not discussed in prior research, such as the potential for model-specific optimized noise (regardless of the quantization type) to mitigate the attack.\n\n**References**\n\n[1] https://ollama.com/search", "claims_and_evidence": "The specific technical claims in the submission are sound and are supported by practical evidence. I will say though that the abstract claims are more generic than what is supported later in the paper. Specifically, the main abstract claim:\n\"Our key insight is that the quantization error – the difference between the full-precision weights and their (de-)quantized version – pro- vides sufficient flexibility to construct malicious quantized models that appear benign in full precision.\" is something that was first achieved by prior work for a simpler family of quantization schemes. The present work is the first one to do this for GGUF.", "methods_and_evaluation_criteria": "There is no standard benchmark for such methods, but the proposed evaluation criteria make sense.", "theoretical_claims": "There are no real theoretical claims, the paper presents a heuristic.", "experimental_designs_or_analyses": "The experimental design is valid (in fact, it is largely adapted from Egashira et al., which is already published work.", "relation_to_broader_scientific_literature": "The paper can be seen as a technical-report-style extension of the work of Egashira et al., in particular extending their approach to a more popular format.", "essential_references_not_discussed": "I think the related work coverage is fine.", "other_strengths_and_weaknesses": "Strenghts: \n\nI think the paper proposes a valid extension of the work on attacks towards quantization methods, specifically since the format they are focusing on is probably the dominant one for local deployment of ML models. The heuristic approach proposed is well-justified, and well-supported through experiments. \n\nWeaknesses: \n\nEssentially, the work is at the level of a well-executed technical report, as it heavily builds on prior work, notably Egashira et al's paper in NeurIPS. As such, the work is of interest, but only to a relatively small niche in the community. The heuristic proposed is quite specialized to the (specialized) format considered, and I don't see how this would be extensible beyond the problem considered here. The defenses considered are largely the same as Shu et al. 2023.", "other_comments_or_suggestions": "The paper is well-written and easy to follow.", "questions_for_authors": "I do not have any major questions, but I am curious if:\n1. The authors looked into other approaches (e.g., specialized signatures?) for defending against such attacks? \n2. Do the authors have some intuition towards more general results showing that any quantized format would be to some extent vulnerable to exploits on an unknown slice of inputs?\n\n## post-rebuttal comment\n\nI thank the authors for their responses. As stated before, I remain borderline on this paper (due to concerns outlined before), but will definitely not stand in the way of acceptance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No concerns."}}]

Author rebuttal:

Meta-review: This paper presents a novel adversarial attack targeting GGUF quantization, a widely used post-training quantization method in LLMs designed for memory-efficient deployment. The key contribution is based on stealthy exploitation of quantization errors to obtain malicious models that appear benign in full precision, but reveal targeted malicious behaviors when quantized. The attack assumes that the adversary can perform adversarial finetuning on a trained LLM, and leverages error-based interval constraints that are derived to be used during a removal training phase for stealthiness. The paper studies malicious attacks including insecure code generation, targeted content injection, and refusal of benign instructions, which are successfully demonstrated on popular LLMs (e.g., Llama3.1-8B) across several GGUF k-quant quantization types. The paper further explores defense by Gaussian noise injection as a feasible countermeasure for such attacks.

The paper went through productive discussions during the rebuttal phase, and the reviewers acknowledged the value of the paper's contribution. One of the main concerns clarified during these discussions was regarding if the same attack could be extended to other optimization-based post-training quantization methods. Authors addressed this issue by providing additional results targeting GPTQ and HQQ quantization schemes, where the attack could natively extend even without being explicitly modified. Another discussed limitation was the paper’s connection to existing literature on similar quantization backdoor attacks. Authors acknowledged this lacking aspect, and noted it to be added with discussions in their revisions. In general, the AC acknowledges that the rebuttal was effective, with the authors successfully demonstrating the novelty and rigor of their work. Overall, the paper presents an interesting contribution to the field of ML security, and it is very well presented with well-structured experimental evaluations. Thus, the AC recommends acceptance of this paper.
Decision: Accept (poster)