AttentionBreaker: Adaptive Evolutionary Optimization for Unmasking Vulnerabilities in LLMs through Bit-Flip Attacks
Authors: Sanjay Das
Swastik Bhattacharya
Souvik Kundu
Shamik Kundu
Anand Menon
Arnab Raha
Kanad Basu
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have significantly advanced natural language processing (NLP) yet are still susceptible to hardware-based threats, particularly bit-flip attacks (BFAs). Traditional BFA techniques, requiring iterative gradient recalculations after each bit-flip, become computationally prohibitive and lead to memory exhaustion as model size grows, making them impractical for state-of-the-art LLMs. To overcome these limitations, we propose AttentionBreaker, a novel framework for efficient parameter space exploration, incorporating GenBFA, an evolutionary optimization method that identifies the most vulnerable bits in LLMs. Our approach demonstrates unprecedented efficacy—flipping just three bits in the LLaMA3-8B-Instruct model, quantized to 8-bit weights (W8), completely collapses performance, reducing Massive Multitask Language Understanding (MMLU) accuracy from 67.3% to 0% and increasing Wikitext perplexity by a factor of $10^5$. Furthermore, AttentionBreaker circumvents existing defenses against BFAs on transformer-based architectures, exposing a critical security risk. The framework is made open sourced at: https://github.com/TIES-Lab/attnbreaker.

Attention Breaker: Breake: Adaptive Evolutionary Optimization for Unmasking Vulnerabilities in LLMs through Bit-Flip Attacks
Sanjay Das" Swastik Bhattacharya', Souvik Kundu², Shamik Kundu?, Arnab Raha?, Anand Menon', Kanad Basu
University of Texos at Dallas, Intel Corporation, *Hensselaer Polytechnic Instttute
Abstract
Large language models (LLMs) have significantly advanced natural language processing (NLP) yet are still susceptible to hardwaro-basod threats, particularly bit-flip attacks (BFAs). Traditional BFA techniques, requiring iterative gradient recalculations after each bit-flip, become computationally prohibitive and lead memory exhaustion as model size grows, making them impractical for state-of-the-art LLMs. To overcome these limitations, we propose AttentionBreaker a novel framework for efficient parameter space exploration, incorporating GenBFA, an evolutionary optimization method that identifies the most vul- nerable bits in LLMs. Our approach demonstrates umprecodented efficacy- afficay-flipping flipping just three bits in the LLaMA3-8B-Instrucl model, quantized to 8-bit weights (W8), completely collapses performance, reducing Massive Multitask Language Understanding (MMLU) ac- curacy from 67.3% to 0% and increasing Wikitext perplexity by a factor of 10 Further- more, AttentionBreeakrr circumvents existing defenses against BFAs on transformer- based architectures, exposing a critical security risk. The framework is made open sourced at:
1 Introduction
The increasing popularity of LLMs has fundamentally expanded the capabilitics of Artificial Intelligence (AT); demonstrating remarkable proficiency generating human-like text, interpreting nuanced context, and executing complex reasoning tasks Xu et al. (2024). These advancements have not only reshaped natural language processing but have also extended Al applications into diverse fields such computer vision and scientific research, heralding new of AI-driven solutions Chang et al (2024); Xu et al. (2024). Given their pervasive use, it is critical to analyze the vulnerability of LLMs against. both software-based and hardware-based threats to ensure their secure and reliable deployment Das et al. (2024).
A significant issue regarding deep learning model reliability involves threats like bit-flip attacks (BFAs), which exploit hardware vulnerabilities to corrupt memory designated for storing the model's 's weight param- eters. Consequently, this undermines the integrity of the model performance Rakin al. (2019): Qian et al. (2023 3 Kundu et al. (2024a). BFAs employ fault injection attacks such as DeepHammer Yao et al. (2020)
to manipulate specific bits within DRAM, altering essential model weights to degrade functionality, Despite advancements in memory technology, recent fault injection techniques enable remote, non-physical memory manipulation, perpetuating the threat landscape for BFAs Hayashi et al. (2011): Shuvo et al. (2023). Tra- ditional BFA methods face constraints due to iterative gradient recalculation after each bit-flip Rakin et al. 2019 ; Kundu et al. (2024a). While viable for smaller models, this process becomes prohibitively costly as model size grows while the effectiveness diminishes. The increasing resource demands of these recalculations lead to memory exhaustion, making conventional BFA impractical for advanced LLMss. A recent study Nazari ct al. (2024) has investigated the susceptibility of transformer-based models to bit-flip attacks by extending conventional BFA techniques albeit limited to small vision-translormers and not large models like LLMs. Furthermore, the authors proposed potential defense against such vulnerabilities.
"This work supported by the Semiconductor Research Corporation (SRC GRC Task: 3243.001) and the National Science Foundation (NSF) grant #2537757.

---

Decoder
Attacker
Attack Domain
Wg
Encoder
Wk
K
Input Embeddling
Attention
X
Linear
Linnar
Linear
Wy
Mabrix
X
Weight Quaries( Key and Values (V)
Incorract Attention Output
Targe
Transformer Architecture
Muitihead Attention
BFA
Figure 1: Bit-flip attack on transformer-based architecture.
In this paper, we introduce a fundamentally distinct methodology, marking a paradigm shift in the ex- ploitation of LLM vulnerabilities through bit-flip attacks. In contrast conventional BFAs, our proposed framework, AttentionBreaken employs onc-shot gradient estimation strategy, requiring single gra- dient computation to aid genetic optimization strategy, GenBFA that efficiently navigates and exploits the vast parameter space of LLMs. Our approach achieves unprecedented efficacy, as demonstrated on the LLaMA3-SB-Inste
W8, where only three strategically selected bit-flips suffice to induce catastrophic performance degradation, reducing MMLU accuracy from 67.3% to 0%. As demonstrated in Section 5.3 AtlentionBreake more impactful compared to existing attacks Nazari et al. (2024). Furthermore, Atten- tionBreaker successfully subverts the defense proposed by Nazari et al. (2024). thus rendering transformer- based models vulnerable to BFAs. This fundamental redefinition of BFA methodology opens new attack paradigm, specifically catering to LLMs, and highlights critical reliability concern for these models deployed in mission-critical applications. Our contributions are summarized below:
We propose AttentionBreaker, novel, efficient BFA framework providing a methodical approach to navigate and exploit the parameter space of LLMs.
We introduce GenBFA, as part of AtrantionBBraaker, a novel evolutionary algorithm for identifying critical bits in LLMs and optimizing BFAs.
BrtentionBreaker uncovers a significant vulnerability of LLMs. A mere three bit-flips (4.129 x 10 10-9% of all bits) LLAMA3-BB-Intrrus W8, can reduce the MMLU accuracy from 67.3% to 0%, while increasing Wikitext perplexity from 12.62 to 4.72 x 105. Furthermore, Atten- Breaere has been shown subvert existing BFA defenses on transformer- models.
2 Variable Notations
This section defines the variable notations used thronghout this paper, as summarized in Table D Consider in a model M with L-layers, parameterized by W = [wi, .2,..., w,), L(W) denoles the associated loss function. For simplicity, we denote the gradient of the loss function V VC(W) as VW throughout this paper. To assoss the sensitivity of model parameters, we define a sensitivity metric S, with Smag and S Sgrad capturing sensitivity based on parameter magnitudes and gradients, respectively. To balance the influence of gradients and magnitudes in the computation of hybrid sensitivity metric, S, sensitivity ratio [0. 1]
2

---

12
Lsmag
12
Hybrid sensitivity score
10
Lyrod
10
Model loss
Lg
Model loss
8
8
6
4-
4
2
2
0.0
0.5
1.0
0
100
200
300
Bit-flip rate (%)
1e-5
Layer ID
(b)
Figure 2: LLMMA3-BB-Isstruc W8 (a) sensitivily proxy comparison, and (b) layer sensitivity analysis the 'Astronomy' task of MMLU benchmark.
is introduced. The attention weight matrix is represented by A, with corresponding query and key vectors Qe and ky, scaled by dg. The function cardinality defined to evaluate the number of parameters in given parameter set. Let w' represent a perturbed version of weight w. and R denote the set of sampling rates for parameter sampling during sensitivity analysis. The normalized gradients and weights are denoted by VW n and WN, respectively. Finally, I is the set of indices where bit flips are applied to perturb specific parameters.
3
Motivation
3.1 Proxy for Sensitivity
For performing BFA, it essential analyze model parameter sensitivity profiles independently of assump- tions of robustness. particular, parameters with larger gradients higher magnitudes may exhibit ampli- fied sensitivity, whereby perturbations yield disproportionately large effects the output. In full-precision models (e.g., float32), where weights span large representable range [-3.4 1038 3.4 103*], even single bit-flip can induce substantial perturbations, particularly for largo-magnitude weights. For such weights, the sensitivily S Smag can be approximated to be proportional to the absolute magnit (Smag x [W]. However, gradient-derived sensitivity Sgrad provides a complementary perspective, capturing the degree to which a weight change affects the model loss and thus can approximated as proportional to the absolute gradients
Table 1: Summary of Variables and Notations.
Variable notation
Description
M
The model
= (11,12,- ... bn)
Layers model
[w], 42. Wey
model parameters
L(W)
The loss function based on the model parameters
VL(W) VW
Gradient of the loss with respect to parameters
S (81, 82....,
5n)
Sensitivity of parameters perturbations
Ralio of contribution to by W and VW
cardinality
Function to provide size of given set
R P21 Fa)
Set of sampling rates
1 = f21,
Set of indices to target for bit-flip perturbations
WN. VWN
Normalized model weights and gradients
3

---

(Sgrad /WI).. Please note, that we use the standard cross-entropy loss between the model output logits and the encoded input token IDs evaluated independently for each downstream task using the corresponding task-specific test data (e.g., MMLU for language models) Mao et al. (023). Therefore, a hybrid sensitivity metric is proposed one that considers both magnitude and gradient influences to capture the sensitivity profile holistically, and is therefore expressed as:
= VW + (1-a) W
(1)
Where a is a tunable parameter balancing the importance of magnitude and gradient. Figure 2a illustrates our experiments with sensitivity proxies based on gradients ([VW)), magnitudes ((W), and the hybrid sensitivity metric S with top-k bit flips on a LLaMA3-SB-Instrun W8 model. We observe that bit-flip perturbations guided by the hybrid metric S (green) induce comparable loss increase with fewer bit flips than both graad (orange) and Smag (blue).
3.2 Layer Sensitivity Analysis
Identifying critical bits in LLMs is challenging due to their vast parameter space. However, finding a sensitive layer more manageable because there are fewer layers than total parameters. To quantify layer sensitivity, we choose top-k bit-flips in cach layer using a sampling rate r, guided by the hybrid sensitivity score S, and observe the resulting model loss L. k is calculated
h: cardinalltyyy (W
(2)
100
where (1I) represents parameters layer 1. The losses vary significantly across layers, as illustrated Figure 2b on the LLaMA3-SB W8 model. Some layers show much greater sensitivity than others, indicating that focusing on these layers significantly reduce the search space.
3.3 Weight Subset Optimization
The subset of weights in victim LLM that can be perturbed may be sufficiently large (e.g., 5,872 for the LLaMA3-SB-Instruee W8 model), making an effective adversarial attack impractical. This requires further optimization to reduce the subset to fewer weights and/or bits for feasible attack. Solving this problem through exhaustive approaches is computationally expensive due to the vast solution space. To address this challenge, we propose hcuristic optimization technique (Section 4.2.2) inspired by evolutionary strategy, which simultaneously optimizes for higher model loss (to maximize attack elfectiveness) and fewer bit-flips (to enhance attack efficiency). As explained in Section 4.2.2, this approach successfully reduced the required bit-flips from 5,872 just 8, achieving an approximate reduction of 2 x 103.
4 Proposed AttentionBreaker Methodology
4.1 Threat Model
Machine Learning as a Service (MLasS) provides users with access to high-performance computing resources, thereby enabling the deployment. of LLMss However, this cloud-based paradigm introduces critical security risks, particularly when inferences and model exccutions share server hardware components with potentially untrusted processes. We categorize the threats into four levels based on the extent of knowledge an atlacker can extract from the victim, detailed in the appendix (Section A.1
In this work, we consider the white-box thrcat model, where the adversary has full access to the model's architecture and parameters, enabling precise and informed manipulation of parameters. Within this setting, we focus on untargeted attacks, which aim to induce graceless degradation in overall model performance rather than targeting specific outputs. Such attacks are particularly insidious, as they compromisc accuracy across broad range of inputs while avoiding distinct failure patterns, making them more challenging
detect and defend against than targeted attacks Özdenizei & Legenstein 2002 Qian et. al. (0023) Nazari
4

---

et al. logits of the L.M and the corresponding ground-truth token IDs, a measure of model performance. A lower cross-entropy loss indicates that the model assigns higher likelihood to the correct tokens and thus performs better. Our objective is to identify the most critical subset of parameter bits such that, when flipped, they cause a substantial increase the cross-entropy loss. This increase directly translates into severe degradation of model performance, thereby highlighting the vulnerability of the model to targeted bit-flip manipulations. Thereffree,,,, AttentionBreaker strategically employs an untargeted attack to maximize disruption while evading traditional defense strategies. This attack also fundamentally differs from denial of service atlacks, which aim to overwhelm system resources to render services wnovailable. In our case, the adversary subtly manipulates model parameters to degrade performance without causing a complete system shuldown. By focusing on untargeted BFAs in a white-box setting, our work aims to highlight the critical need for robust defense mechanisms to ensure the reliability and security of transformer -aseed models in shared computational environments.
2024
We employ the standard cross-entropy loss Mao et al. (2023
computed between the output
4.2 Proposed Attack Framework
Here, we explain in detail the proposed BreaktionBreaker framework, which comprises two structured steps: (1) layer ranking based their sensitivity profiles and (2) weight subset optimization identify the model's most critical parameters for efficient attack.
4.2.1 Layer Ranking
For ranking LLM layers, begin by quantifying the sensitivity of cach layer's parameters. For this purpose, each weight w; in layer is evaluated through sensitivity scoring function defined in Equation [
Algorithm Layer Ranking
Please note that for this scoring we only use the most significant bit (MSB) flips as experimental evalua- tion consistently demonstrated that flipping the MSB caused the most impact. Using these sensitivity scores, weights are aggregated and ranked descending order to form prioritized list of the most critical weights in each layer. The top-k weights within each layer are identified using Equation 2 To maximize the im- pact of bit-flips, perturbations are applied by targeting the most significant bits (MSBs) to induce the high- est deviation for each selected weight. After apply- ing these perturbations, we compute the model's loss, g() which reflects the sensitivity of the layer. In order to identify layers with heightened sensitivity while re- quiring a minimal number of bit-flips, we define layer sensitivity score (LSScore) function. First, we define a sign function sgn (C()) L.n) based on whether the model loss (()) evaluated after bit-flipping the weights in layer / meets a predefined critical loss threshold Lu:
Require: W, VW, a, sub-sampling % r, # of top
layers n
1: Initialize sensitivity score set: Leens []
function BFLIP(W!! pos, I)
3: W!!) + W(
orig
4: for all i do
((()) + w + (1 < pos)
5:
end for
7: M updateModel(MM, W
L +- ComputeLoss(MM)
9: M + updateModel(M,, w
10: return
11: end function
12: for all E L do
13: k 5 - a VW +
cardinality(W")))
14:
(
15:
hibbrid <- ())
(()) 2 BFLIP(W(()) pos, hybrid)
16:
+1, if ( Lea 2 0,
17: LSScore sgn (C' Lin) ci
k
sgn -c1) 1
C(I) Lih <
(3)
18: Push [LSScore()) I] to Leens
Using this sign function, we define LSScore as: LSScore !1) =gn (C Lux) C())
19: end for
20: Lsens (Seen))
(4)
21: Ltep - (ttop(Lssen.
22: W Weub + [1,
(1) hyybria extracted from Lrop
A layer-wise bit-flip sensitivity profile is created by iterating through each layer to record their sensitivity scores. This profile aids in identifying highly sensitive layers.
Ensure: Luens, W sub
5

---

Algorithm 1 presents systematic approach for conducting layer sensitivity analysis. The layer sensitivity analysis itself begins by iterating through each layer, assessing the model's sensitivity to perturbations in each (lines 12-19). Sensitivity scores ( for the weights in layer are then computed using Equation [ (line 14). Next, the top-k indices are obtained using sensitivity scores S with the function TopkIndex. These indices,
I hybrisd are identified from S(1) (line 15). These indices, along with the weights W and bit position pos, are passed to the BFLIP function to compute the model loss c/ and the corresponding layer sensitivity score LSScore (lines 16-17). The resulting score LSScore is then stored in Lsens (line 18). This process continues until all layers have been processed, yielding the final sensitivity score set Lisens: which is then sorted, and top-n layers are chosen (lines 20-21). Finally, the top-n most critical layers and their corresponding weight indices 1 hybbri are extracted to generate a critical weight subset W Webb (line 22), which along with the sensitivity scores Lsens are provided as the output.
4.2.2 Weight-set Optimization - GenBFA
After identifying the most sensitive layer and the corresponding weight subset, further refinement essential since LLMs typically encompass billions of parameters, and the identified weight subset may necessitate targeting thousands or even millions of parameters. For example, the LLaMA-3-8B W8 has billion param- eters, and the weight subset corresponding to the most sensitive layer has 5, 878 parameters. Consequently, an attack with this identified weight set could be prohibitive from an attacker's perspective. Therefore, it is imperative to identify a smaller subset of parameters capable of achieving comparable degradation in model performance. To address this objective, we design a novel algorithm inspired by evolutionary strate- gies, GenBFA, tailored for optimizing the parameter set to identify a minimal yet highly critical subset of weights. GenBFA maximizes the impact-to-size ratio of the selected parameters, thereby enhancing the efficiency and effectiveness of the AtlentionBreaker attack on LLMs.
Objective Function and Fitness Evaluation: To formalize the weight subset optimization problem, let us define an initial weight subset Wsub and target model loss threshold Lu.- The objective is to identify a subset, Ws, C Waub such that bil-flipping the weights in this subset maximizes the model's loss while minimizing the subset's cardinality, which reflects the number of bit-flips required to achieve a corresponding model loss, thus providing a measure of the attack's efficiency. Each candidate solution Ws, {wy, .......} is therefore defined as a subset of W Waub. For cach candidate Ws,, we compute a fitness value f(Wss) 8,) as follows.
Cs, f(Ws) s,s,) sgn (Cs, Lehh) cardinality(Ws.)
Population Initialization: To identify the final set of critical parameters, we define a solution set (also referred to as the population) with the objective of minimizing its cardinality while cnsuring that the fitness score remains above a predefined threshold. The population P, comprises of m candidate solutions, defined as P P s W Ws.. The initial solution Po is set as the weight subset W Wsu,, and each subsequent solution Pj is created by applying mutation operation to Po (Mutate(Po)).
During the mutation process, each solution P, is assigned a distinct reduction rate Pj. which is randomly selected such that p5 S μ, where μ represents the mutation rate. The parameter μ serves as an upper bound on the roduction of solution cardinality, thereby facilitating enhanced exploration of the solution space. The reduction rate then governs the probability of removal for each weight in Py. Formally, for each weight wij E Py, the mutated weight w/, is defined as:
Mutation Operation:
0. Wie
with probability Pj: otherwise.
w/.
(6)
In this context, "mutation" specifically refers to reducing the solution set's cardinality. This reduction-driven mutation is designed to identify compact subsets that either preserve or improve model loss relative to the original subset. By systematically exploring minimal yet sufficient configurations, this strategy facilitates
6

---

efficient traversal of the solution subspace and the identification of subsets that sustain the desired adversarial effect on model performance.
Therefore, mutation is performed by randomly reducing the size of the candidate subset according to a variable reduction rate (ranging between 1-10% and chosen randomly during each experiment), thereby introducing evolutionary pressure variably toward compact, high-impact solutions.
Algorithm 2 Weight Set Optimization GenBFA
Elitism and Selection: At each iteration (1) of the optimization procedure, all solutions in P are sorted by their fitness values:
Require: W, W sub, Lth, population size m, max generations 9, mutation rate μ4, no-improvement threshold N
Pbett W arg max f(P,) PjEP
(7)
1: Initialize solution space SS = [], solution set P=
[]. fitness scores F
[]. no-improvement counter
Here, W or Pbest represents the solution with the highest fitness (i.e., most efficient in degrading model performance). Elitism dictates that Pbest is carried forward unchanged into the next generation, ensuring the highest-quality solution is preserved. For the remaining solutions, a. tournament selection strategy is employed to choose pairs solutions as parents for crossover. For selecting each parent, this involves choosing two random solutions Pa, Pa P, and selecting the one with better fitness:
c=0
1,SS S.d (0. W sub
P InitializePopule I
for t= 111,2,... 1.2...... do
5:
for 1= do
6:
W(1 = CetParams (I
(11)
hybrid Pj
L3 & BFLIP(W' pos; higbria)
(())
Cs,
9:
sgggg/ (Lj - Lah)) cardinality(
10: Push J to F
Pp arg max ((Pa).
(8)
11:
end for
This strategy is chosen over a random strategy due to its capability to balance exploration and ex- ploitation the search space, thereby expediting convergence toward an optimal solution.
12:
P + SortSolutions(P.1
13: Poest + Pbest
14:
Phest f- SeleectestSooutiion
15:
CheckTerminate(Puust. Pold bests C, N)
16:
P new + [Pbest, Mutate(Pbest)
Crossover Operation: To generate new solu- lions, (wo-parent solutions, Ppt and Pp2, are se- lected, followed by a crossover operation executed with probability Pe- During crossover, the current best solution Pbest is integrated to propagate its beneficial traits forward. Two offsprings, Pai and Po2, are produced by combining Pbest with each parent PpI and Pp2 independently. For offspring Poi, each gene 9i is chosen stochastically from ei- ther PpI or Pbest Similarly, offispiing Poz is created by selecting each gene stochastically from P,2 or
17:
while 1en kew)) m do
18:
Ppi, Pp2 +- Selection(P,,
19:
Poi. Poz + Crossover
ppi, Pp2)
20:
Pai, Po2 < Mutate(Poi,kl (utttte(P2..1
21:
Push Poi and Po2 to Pnew
22:
end while
23:
P P+P --Pnew Population for next generation
24:
end for
Ensure: Poeat
> Return best solution
Pbett
Pol, Poz fop,, or Prwwn J. (ob,2 or I
If crossover does not occur (i.e, with probability 1 pp), the offspring remain identical to the parents, such that Poi = Pp1 and Po2 = Pp2.
The process itcrates over generations indexed by t 1, 2....... ......... .9, where 9 is the predefined maximum number of iterations. The algorithm terminates when either the iteration limit g is reached (t g) or predefined no- improvement limit / is met, ensuring computational efficiency. During our experiments, the number of generations is capped at 100, with carly stopping applied whenever no improvement is observed for a fixed number (default 5) of iterations. The final output is the weight subset Wapt C W sub, where W Wapt = arg maxp,er ((2)). Wapt represents the critical weights required to achieve the desired model loss degradation.
Convergence and Termination:
Algorithm 2 outlines the optimization process, starting with model weights W, target loss threshold Lik and parameters for population size m, maxinum iterations g, crossover probability Pe, and mutation rate p.
7

---

It initializes the solution space SS, population P, and fitness scores F (line 1), followed by generating the attacked layer I and initial solution space from W (line 2). The initial solution Po (=W sub) is used to create a population by applying m - mutations, resulting in P (line 3). At each iteration t, solutions are evaluated using the BFLIP function, which computes model loss Cj and the fitness score determined basod on L, Le,, and the cardinality of w(O using Equation 5 (lines 6-10). Solutions are ranked by fitness (line 12). The current best solution is tallied against the previous best solution for capturing improvement and checking for termination using CheckTerminate function (lines 13-15). To maintain elitism, the best solution Pbest is carried over to the next generation Prew (ilne 16). A second solution is created by mutating Phest, and the remaining population is filled by generating offispring through crossover and mutation (lines 17-20). Crossover selects parent pairs via tournament selection (line 18), generates offspring (line 19), and applies mutation (line 20), which are then added to Pnew (line 21). The new population Pnew replaces P (line 23), and the process repeats until either g generations or the no-improvement threshold is reached. The algorithm outputs Pbeat as the optimal subsct of critical weights, targetting for BFA.
Table 2: AtrantionBreaker evaluation on various models and datasets.
Model
Number Parameters
Nazari 2024
Existing Reseeirch
Attention
Improvement
Perplexity
Benchmark performances (Before atlack After Attack)
Accuracy
(# of bit-llipe
(# of bit-llips)
Wikiilext22
MMLU
LM Harness
VQAv2
TextVQA
LLaMAS-SB 8B
Tastract- Ws
587202
195, 734x
12.14/4.7x00
67.3/0
726/00
NA
NA
LLaMA3
8.03 B
587202
28
x0,971
12.00/3.5x00
60,3/0
61.2/0
NA
NA
Inetract-W4
LLaVX..-
Mistral-7D-WW
410430
16
27, 9629
NA
NA
NA
81.4/0
64.2/0
Mistral-7D-WW
LLaVALo
838860
808
NA
NA
75.6/0
60.500
Hir Net-llrge
3144572
46861
7x
17:21/5.6 10"
22.1/0
20.2/0
NA
NA
Pha-3-mins- 128
Iastract Ws
1258201
314,673x
11.82/3.7 10*
69.7/0
72.3/0
NA
NA
Phi-3 1253-
Iasteract W4
3.82 B
2516
620x
11.94/3.6 x 10*
66.1,0
71.7/0
NA
NA
5
Experimental results
5.1
Experimental Setup
For our experiments, we utilized a wide range of models, comprising of CitaaMM---B-Inntrrrn Touvron et al. ) Dubey et al. 2024), Thi-3-mini-118k--nsttrr AAbdin et al. (2024) and BitNet Ma et al. (2024). The evaluation LLMs was conducted using standard benchmarks like MMLU Gema et al. (2024) and the Lan- guage Model (LM) Evaluation Harness GannesGGaaae al. (2024). both of which test the models' ability reason and generalize across a variety of tasks. We extended our evaluation include a Vision-Languae Model (VLM), (2023). The performance of this VLM was assessed using the VQAv2 Nguyen et al. (2023) and TexiVQQ Singh et al. 2019) benchmarks. To demonstrate the versatility of ArtentionBreaker. we utilize several quantization formats, including the standard signed 8-bit integer (INT8), normalized float 4-bit (NF4), and an advanced 1.58-bit precision format (where weights are ternary f-1, 0, 1}), implemented using BitNet Ma et al. (2024). Please note that all these models are weight-only quantized, and we will use notations W8, W4, and W1.58, respectively.
LLaVA1.6 Liu al.
We employ perplexity and accuracy evaluation metrics across multiple benchmarks. Perplexity Hu
et al. 2024 assesses model's predictive capability, defined as the exponential of the average negative log- hhoii of sequence. Accuracy measures the proportion of correct predictions, providing a direct evaluation of model performance.
N Σ log P(x,)
Perplexity = exp
(10)
N
where N the number of words, and P(x;) is the probability assigned to the i-th word by the model. A lower perplexity indicates better performance, Accuracy on benchmarks another critical metric, especially for tasks such classification Gupta et al. (2024). It is the ratio of correctly predicted instances to the total
8

---

Table 3: Areention experiment parameters values.
Parameter
Value
Top-n Layers
1-5 (default = 1]
Top-k Weights
0.001%-10% of layer weights (default 0.5%)
Initial Number of Solutions
20-100 (default 50)
Maximum Generations
20-100 (default 50)
Number of Solutions in each GenBFA generation
40
Number of generations allowed with no-improvement
5
Mutation Rate
0.1
Crossover Probability
0.9
instances.
Accuracy
Σ I(A vn)
(11)
Where the lotal number of lasks. Di the predicted label for the i-th lask. Yi indicates the true label for the i-th task. is the indicator function that returns 1 if the predicted label matches the true label and otherwise.
Table B lists the parameter values used in our ArentionBreaker experiments. These values are not fixed and may be adjusted or refined by the user to achieve improved convergence.
5.2 Targeted Transformer Modules
Transformer-based LLMs comprise self-attention projection layers (query, key, value, and output), MLP projection layers (gate, up, and down), token embedding, layer normalization, and the LM head. Parameter distribution typically allocates 60-70% to MLP projections, 10 10-20% to self-attention projections, ~ 5'7% each to token embeddings and the LM head, and < 0.01% to normalization layers. Since quantization applics only to the MLP and sell-atiention projections (86.9% of the parameters), AllentionBreaker exclusively targets these layers.
5.3 Results and Analysis
This section presents the experimental results val- idating the effectiveness of the proposed Atten- tionBreaker attack. Box 1 shows example of how the LLaMA3-8BB-Instruee W8 model's out- puts are degraded by AttenttonBreaker, yielding incoherent and incorrect answers. Section A. dis- cusses additional examples.
Box Multiple choice question example
System Prompt:
You helpful Al assistant. Answer the following questions about Astronomy. [Question 1, Question 2] Correct answers: FA', 'D',]
Questions:
Model Output (PrecAttack)
5.3.1 Efficiency of AttentionBreaker
Extracted Answers: [A', DD']
This section shows AttentionBreaker" effective- ness in degrading LLM performance across bench- marks (see Table 2 and compares it to existing re- search Nazari et al. (2024). The first two columns list model attributes: name, quantization scheme, and parameter count. Columns three and four specify the bit-flips needed by each approach for full performance collapse. The fifth column mea- sures AttentionBreaker'ss efficiency improvement over Nazari al. (2024). The final five columns present benchmark results, including WikiText-2 perplexity and model accuracy MMLU, LM Harness, VQAv2,
Model Output (Post-Attack)
Extracted Answers: [", fe"]

---

Table 4: Performance Comparison between AttentionBreaker and conventional BFA Rakin et al.
(0019)
Accuracy before
Accuracy after
Accuracy after
Model
# Bit-flips
attack (%)
BFA (%)
Attentionrreaker (%)
HLaMAA-SB-Iustrune W8
1000
67.3
67.3
0.0
Phi-3-mini- 1at.mminn-12K--ssteeeee 128k-Insttreet-W
60.7
69.7
Table 5: Timing overhead Comparison between AttentionBreaker and conventional BFA.
Time overhead (Hours)
Model
# Bit-flips
BFA
AtrentionBreaker
LLaMA3- 8B Ins---utttttts
1000
~0
~
LLaMA3-8B- InstrruutttII-ttuttttt Instruct-W2 W4
1000
20
2
and VQA, both before and after the attack. "NA" indicates when benchmark performance does not apply to the model.
Table [ highlights EntentionBreaker" efficiency in degrading model accuracy with minimal bit-flips (as few as 3), while Nazari et al. (2024) requires up to 1.3 x 106 bit-flips for causing similar performance degradation. AtrentionBreaker reduces LaaMA3-8B-Instrure W8's MMLU accuracy from 67.3% to 0% with just three bit-flips (4.129 10-9% bit-flip rate) in the MLP down projection of the second decoder block. Additionally, perplexity increases from 12.14 to 4.72x 10 post-attack. In contrast, Nazari et al. 2024) requires 5.9x 10 bit- flips (0.1% bit-flip rate) for similar degradation, demonstrating that ALtention is 1.9 10 times more efficient. A similar trend is observed for Phi-3-mini- Instruct and LLa V.1.6A1.6 Misttalll W8 models. The critical layers affected are the self-attention key, value projection, and MLP down projection in the initial three and final three decoder blocks.
5.3.2 Comparison across Quantization Levels
Figure compares AttentionBreaker with Nazari et al. (024) on LLaMA3-8B-Intrree W8, W4, and BitNet W1.58. The green/blue lines indicate post-atlack model loss, while the orange/brown lines show MMLU
accuracy. AttentionBreaker consistently reduces accuracy to O, demonstrating its effectiveness through a rapid loss increase, as shown in Figure Bh for I.LaM33-B-IIntrrue 8-bit quantized model. Figure 3p exam- ines Areakrr on 4-bit quantized (LaMA3-SB-Instrun where just 28 bit-flips increase perplexity from 12.60 to 3.8 x 10º, dropping the MMT.U score from 60.3% to O. Tn contrast, Nazari et al. (2024) requires 5.9 x 10 bit-flips for the same effect. Similar trends are observed for Phi-3-mini- 128k-Instrree and LaVA1.6-Misttra--7B W4. For the ultra-low precision model BitNet W1.58 Ma et al. (2024), Attention- Breaker degrades the performance to 0% by flipping 45811 bits (6.27 x 10 % of all bits) as shown in Figure In comparison, Nazari et al. (2024) requires 6.8x more bit-flips. This disparity in number of bit-flips required for BitNet in comparison to LLaMAA-SBB-IIstruuct--y and -W4 can be attributed to the unique architecture and quantization strategies employed in BitNet. Unlike W8 or W4 models, which have higher precision and therefore higher weight representation range, BitNet employs 1.58-bit. quantization (ternary reprosentation -1, 0, +1), significantly limiting the range to [-1,1], which limits the impact of individual bit-flips. Therefore, in BitNet, flipping single bit merely loggles between two extreme values (e.g., -1 and +1), necessitating a substantially more significant number of bit-flips to degrade performance meaningfully.
5.3.3 Comparison with Conventional BFA
We evaluated LLMs under the conventional BFA framework Rakin al. 2019)) and observed no noticeable degradation in accuracy even after introducing up to 1,000 bit-flips. These findings suggest that traditional BFA techniques are largely ineffective against large-scale models. In contrast, our proposed AutentionBreaker method induces a substantial drop in model accuracy, as illustrated in Table 4 Unlike conventional BFA approaches, our method employs a gradient-driven stratogy for identifying vulnerable layers, selectively targets critical weight subsets, and utilizes GenBFA, a bit selection optimization algorithm. This principled approach allows A Attention Breaker to conduct significantly more effect and scalable attacks large models.
10

---

Table Targeted attack by Areeer.onBreaker.
Class A-
Model Variant
Overall Accuracy (%)
Misclassification rate (%)
Original model
50.6
14.8
Targeted attark by AttentiooBreaker (109 bit-flipe)
36.2
70.4
We further present comparative analysis of the computational overhead associated with the conventional BFA approach Rakin et al. (2019) and our proposed AltentionBreaker, conducted on the LLaMA3-SB- Instruct INT8 model using four NVIDIA A100 GPUs. As shown in the Table AttentionBreaker achieves approximately 10x faster identification of 1,000 critical bit-flip locations compared to BFA. The inefficiency of traditional BFA arises primarily from its one-bit-per-iteration strategy and the substantial computational cost incurred by repeated gradient recalculations and loss calculations after each bit-flip in each layer. Additionally, some informed decisions can be taken to make the AttentionBreaker even faster, such as only considering the top-1 layer for optimization, reducing the loss threshold etc.
5.3.4 Feasibility of Targeted Attacks by Breenri
While our approach is designed for untargeted attacks as discussed in Section 4.1 it can be adapted for largeted settings by modilying the objective and flipping bits critical to specific class.
Towards this, let us mount a 1-to-1 targeted misclassification attack on the MMLU benchmark "Astronomy" task using the LLaMA-3.2-1 2-1B-Inssruct-W3 model. The goal was to cause inputs originally belonging to Class to be intentionally misclassified as Class B. The experiment involved four classes (A, B, C, D), with model accuracy and targeted misclassification rates measured before and after the attack. In this setting, the goal is not only to degrade overall performance but to enforce targeted misclassification while preserving the accuracy of non-targeted classes. To capture this, we define two complementary loss components.
The attack loss Lattack, which enforces misclassification from class A to B, is given by:
Lattack
Σ (œ.y=A)
log x))
(12)
[A
where DA denotes the set of samples with ground-truth label A.
The correctness loss Leorrect, which preserves classification accuracy for the other classes (B, C, D}, is defined
as:
Leorrect
Σ
log x),
(13)
D Da.al
(s.DE(B,C.D))
where D_A is the set of samples not belonging to class A. The total loss Lrotal combines the two with balancing hyperparameter X:
Liotal Lattack + À Leorrect
(14)
Minimizing Liotal enforces that inputs from class A are consistently misclassified class B, while predictions for the remaining classes remain accurate. This design ensures that the targeted attack is both focused and stealthy.
Table presents the model's performance in two conditions: the original model (unmodified) and the model after applying AttentionBreakor with only 109 bit-flips accomplish the targeted attack. As shown in the Table 6 the A-B misclassification rate increased from 14.8% (original model) to 70.4% (when attacked using AttentionBrent Breaker), 375% relative increase, demonstrating that our method successfully induces targeted
11

---

Table 7: Scalability of AttentionBreaker to Larger Models.
Wikiiext-2 Perplexity
MMLU Accuracy (%)
Model
Model type
Precision
# Parameters
# Bit-Flips
Before/ After atlack
Before/Afee attack
MiniMixtra-v0.222
LLM (MoE)
BFlost16
12.9B
2
20.78 2.1x10*
68.55 0
LLaMA 3.3 70B Instruct
LLM
NF1 (4-bit)
70.6B
1
9.53 / 1.2x10
86.0 /
behavior. This increase comes with a modest drop in overall accuracy (from 50.6% to 36.2%). These results demonstrate that AttentionBreaker is effective in mounting successful attacks in largeted scenarios.
5.3.5 Performance-cost tradeoff
The comparison in the paper in Table 2 evaluates the number of bit-flips needed in the baseline to match AttentionBreaker's performance drop, ensuring a meaningful efficiency assessment. We also provide an alternative comparison by first applying the baseline attack and then matching its degradation with our method. The results in Table shows that ArraktionBreaker significantly more efficient, requiring up 1.95 10 fewer flip-bits over the baseline method. Our atlack framework also provides precise control over the trade-off between number of bit-flips and model loss by explicitly formulating min-max optimization.
5.3.6 Scalability of the evolutionary optimization to larger models (e.g., 70B+ parameters)
We evaluated AutentionBreade on a Mixture-of-Fxperls (MoE) architecture. In the MiniMixtral--0.22 model (12.9B parameters, BFloat16 precision), only two carefully chosen bit-flips were sufficient to cause catas- trophic collapse, reducing MMLU accuracy from 58.55% to and incrcasing WikiText-2 perplexity from 20.78 to 2.1x10 This demonstrates that evolutionary optimization remains effective even in more complex architectures such MoE.
We fully agree that scaling experiments to 70B+ parameter models is critical. However, due to resource limitations in an academic environment (4x NVIDIA A100 GPUs with 320 GB total memory), we cannot feasibly compute full gradient-based sensitivity for models of this scale, as training or gradient evaluation would exceed our compute budget. To still address this point constructively, we performed an alternative gradient-free free magnitude- based sensitivity analysis on the LLaMA 3.3 70B Instruct model in 4-bit NF4 quan- tization. Using magnitude- based sensitivity scoring (as discussed in Section A.5.2 of the paper), we collected the initial subset of critical weights of size 23488. Following this employed our GenBFA cvolutionary optimization to reduce the most critical subset of bits and found that flipping only single critical bit was sufficient to reduce MMLU accuracy from 86% to 0 and increasing WikiText-2 perplexity from 9.53 to 10 This result demonstrates that AttentionBreaker scales to very large models and maintains high attack efficacy.
These results demonstrate that AttentionBreaker's evolutionary optimization is not only effective for models up to 13B parameters, including MoE architectures, but also remains practical for large-scale models beyond 70B parameters.
5.3.7 Attack validation under noisy bit-flip conditions
There is an importance of evaluating the attack in real-world hardware constraints and noisy bit-flip condi- tions. In practice, bit-flip injections may be imperfect due to factors such as ECC, and other fault-tolerance mechanisms in modern accelerators. To evaluate this, we performed preliminary noisy bit-flip experiment after identifying the critical bits targeted by AttentionBreaker. Each intended flip was applied with prob- ability p (0.8-1.0), while unintended flips within the same parameter tensor occurred with probability q (0.0-0.3). The choice of p and q values are primarily system dependent and p near and non-zero q is acceptable assumption. Each configuration was exccuted 100 times and averaged. The results, presented in model loss/ ooss/moeel accuracy format in the table, indicate that even under high noise conditions, the al- tack remains effective. Specifically, over 82% of cases resulted in full accuracy collapse (attack loss > on the LLaMA3-SB-Instrucc W8 model, MMLU benchmark), and all cases exhibited relative accuracy drop exceoding 50%. These findings demonstrate that AtrentionBreaker maintains strong efficacy under realistic noisy bit-flip conditions.
12

---

Table Effect of varying flip probabilities for targeted and untargeted bits on model performance.
Probability of flipping the targeted bit (p) + Probability of flipping bit not targeted (q) 4
0.8
0.85
0.9
0.95
1.0
MMLU Loss
Accuracy (in %)
0.0
7.38 0
9.51 /
9.9 /
11.4 / /
13.1 0
0.05
7.11 / 0.3
7.48 0.2
9.0 /
10.8 0
12.7 0
0.1
8,15 / 0
8.8 0
8.1 8.1/0
10.6 / 0
11.8 / 0
0.15
7.23 /
8.9 / 0
8.5 / 0
11.1 / 0
11.4 /
0.2
5.33 21.4
5.8 /4.8
14.8
8.4 0
10.1
10.2
0.25
15.3
/ 2.2
/
8.6 0
9.9 /
0.3
21.3
6.41
6.59 /
8.13
9.62
Table 9: Performance-cost tradeoff on baseline Nazari et al.
2024 and the AttentionBBeaker..
# Bit-Flips
Bit-Flips in AttentionBreaker
Model
Relative accuracy drop (%) Existing Work Nazari al. 2024
10
5872
13204
LLeMA3--B-Instrut-155
37235
587202
AttentionBreaker Accuracy
Nazari et.al. 2024 Accuracy
AttentionBreaker Loss
Nazari et.al. 2024 Lass
20
50
Medel accuracy
50
40-
10
40
15
Model loss
30
30
8
10
20
20
10
10
10.-6
10
1d
1d9
10
10
10-4
10'22
10"
10'
10-4
10
10-2
16'
Bit-flip
Bit-fiip
(a) LLaMA3 Initructt
LLaMMA3-8B-inssnuct W4
BitNeet W1.58
Figure 3: Comparison of BFA attack using Nazari al. (2024) and ArtentionBreaker on LaMA3-8B-Instnn
W8, W4, and W1.58 models. Metrics are calculated on the "Astronomy" task of the MMLU benchmark.
1e3
14
Hybrid sensitivity score
5
12
10
4
10
Model loss
fit-flips
Model loss
3
8
Bit-flips
6
2
#
4
4
1
Lth
2
0
0
100
200
300
20
40
Layer ID
# Generations
(a)
(b)
Figure 4: (a) Layer sensitivity analysis, (b) evolutionary optimization in LLaMA3- - W8 model.
13

---

Table 10: Effect ArtentionBreaker on BaR-based defense.
Original Accuracy (%)
Post-attack accuracy (%)
Model
Dataset
of Bit-flips
With FaR
Without FaR
CTFAR-100
91.55
224
3.4
4.2
ViT-Base
ImageNet
80.1
268
10.9
10.2
5.4 Analysis of intermediate steps in AttentionBreaker
5.4.1 Assessing Layer Sensitivity and Ranking
In this section, analyze the impact of the layer sensitivity analysis, described in Section 4.2.1 Figure 4a shows the vulnerability profiles of model layers by plotting model loss against layer IDs, where each layer experiences bit-flip perturbations. The figure reveals increased sensitivity in the initial and final layers for the LLaMA3-BB-Inslre
W8 model. Further analysis of other models is in the appendix (Section A.3.1 After the sensitivity analysis, the layers are sorted, and the top-1 layer is chosen for optimization, reducing the scarch space. For LLaMA3-SB-Intruue W8, selecting the most sensitive layer reduces the solution space from 10 5.8 10" parameters (xz 138x).
5.4.2
Obtaining the Most Critical Weights
In this section, we analyze the impact of the proposed GenBFA evolutionary algorithm, detailed Section 1.2.2 Figure 4b illustrates the reduction the weight set while preserving a high model loss for the LLaMA3-8B- Instructt W8 model. As depicted in the figure, the model loss (red line) remains above a predefined threshold (green line), while the number of required bit-flips (blue line) is iteratively minimized. The initial subset size is reduced by factor of up to 2 103, identifying only 3 critical weights sufficient -uifhie--suffiient to execute the AttentionBreaker attack with effectiveness comparable to perturbing the original set of 5872 weights. A similar trend is observed for Phi-3-mini- 128k W8, MLAMA3-SB-Instrrne W4, and BitNet W1.58 highlighting the efficacy of GenBFA
models (detailed in the appendix Section A.3.2
6 Efficiency against Existing Defenses
Existing research proposed Forget and Rewire et al. (2024). In this section, we evaluate the prowess of AttentionBreaker against FaR. Table I0 demonstrates the effects of AttentionBreakor on a ViT-Base Dosovitskiy et al. (2021) model in the presence of FaR (Nazari jet al. (2024) used similar ViT models their work). Our results reveal that, despite the FaR defense, AttentionBreaker was successful in plummeting the accuracy of ImageNetlk dataset from 80.7% to 10.9% after just 268 (= x 10-6%) bit-flips. Thus, it can be concluded that AttentionBreaken is a potent attack that can subvert state-of-the-ate defenses deployed for protecting transformer-baseee models against BFA.
defense for protecting against BFAs Nazari
We also have identified SOTA defenses against bit-flip attacks on deep neural networks (DNNs), such as Aegis Wang et al. (2023) and low bit-width quantization based defenses He et al. (2020). Aegis employs a multi-exit mechanism that incorporates multiple internal classifiers, with one randomly selected for output generation in an effort mitigate attacks Wang et al. (2023). However, this approach is ineffective an adversary targets early layers, leading erroneous activations across all classifiers. As demonstrated in our study, AttentionBreaker detects initial layers vulnerable to bit-flips (refer to Section 6.3.1) and mounts successful attacks, rendering Aegis ineffective. Additionally, deploying early-exit mechanisms in LLMs incurs significant computational overhead, limiting their practicality. Further, having such models requires the exit layers to be trained, which is often infeasible in mesource-linited client setups. Low bit-width quantization reduces precision to mitigate bit-flip impacts He et al.) (2020). However, AttentionBreaker strategically identifics and manipulates critical bits even in quantized models, inducing severe degradation. For example, in LLaMA3-S88 A--------Isstree (NF4, 4-bit), ArtentionBreaker reduces accuracy from 60.3% to 0% with just 28 bit-flips, demonstrating its effectiveness against quantization-basel defenses.
14

---

7 Discussion on Possible Defenses
In response to the attacks exemplified by the proposed Attenri we recommend an adaptive defense mechanism designed to disrupt the foundational strategics of the attack. This defense mechanism draws inspiration from XOR-cipher logic locking Kamali et al. (2022) and preemplively identifies critical set of vulnerable model parameters. A subset of these parameters is then encrypted using an XOR-cipher. With the correct key set, the model retains performance during inference; an incorrect key results in significant performance loss. An attacker unaware of the key faces model with reduced baseline accuracy, making further attacks impractical.
To experimentally validate the proposed XOR-cipher/logii-lockii based defense, we apply XOR-locking to the vulnerable bits identified by AttentionBreaker, ensuring that only the correct key restores baseline accuracy, while any incorrect key leaves the model in a severely degraded state, thereby deterring further attacks. The security scales exponentially with key length; for example, locking 100 bits with a 100-bit key yields 2000 possible combinations, making brute-force infeasible. To harden protection when the critical subset is small (e.g., 3 bits), we lock larger superset. In one experiment with -LaMA3-8B-W- 8B-W8, locking 100 bits restored 67.3% MMLU accuracy with the correct key, but incorrect keys reduced accuracy to an average of 26.7%, often causing complete collapse. Furthermore, once the model is degraded by an incorrect key, any subsequent attack with AttentionBreaker faces low incentive and high inefficieccy: the optimization process aims to degrade performance that is already poor. Thus, without the correct key, the model is functionally unusable, and follow-up adversarial efforts are unattractive and largely ineffective.
8
Ethical Implications
8.1 Ethical Considerations and Potential Misuse
The AttentionBreakkrr framework, while intended to expose critical vulnerabilities in LLMs, presents clear dual-use risks. Its ability to induce total model failure with minimal bit-flips, even bypassing state-of-the-art defenses, could be maliciously exploited, particularly in cloud or shared hardware environments. Such misuse threatens the reliability of LLMs in safely-critical domains like healthcare, finance, and security. Given this potential, responsible disclosure imperative. The release of related tools should be tightly controlled, with coordination among Al developers, hardware vendors, and cloud providers. Ethical oversight akin to review boards in other high-risk research domains is essential to prevent abuse. In conclusion, AttentionBreaker highlights the fragility of current AT systems and underscores the urgent need to treat adversarial robustness as core design objective, rather than a post-hoc safeguard.
8.2 Dual-Use Risks (e.g., Attacks via Shared Hardware)
While our work focuses on demonstrating the vulnerability of LLMs to bit-flip attacks, we acknowledge the dual-use potential of such techniques. This is particularly crucial in scenarios where shared or untrusted hardware is used, as in multi-tenant MLaaS deployments. In such environments, an adversary could exploit hardware faults (e-g.; RowHammer Mutlu & Kim (2019), RowPress Luo et al. (2023) to induce critical bit-flips without direct access to model weights. These risks are amplified in shared cloud hardware where multiple users' workloads may co-locate on the same physical accelerator, providing an opportunity for malicious tenants to target co-resident models. By identifying highly vulnerable layers and bits, Attention- Breaker could be repurposed by an adversary to design efficient hardware-level fault injection campaigns, underscoring the need for proactive hardware and deployment-lest safeguards.
8.3 Regulatory and Policy Implications
The findings of this work suggest that hardware-level protections should be integral to MLaaS security frameworks. Potential couttrmeassrrs include memory isolation enhancements, logic locking, and phys- ically unclonable functions (PUFs) to authenticate accelerator hardware. From governance perspective, regulatory bodies could mandate model providers to perform red-team testing, deploy certified secure hard- ware for hosting sensitive models, and document bit-flip resilience in compliance reports. Emerging AT
15

---

policy proposals, including compute governance and hardware licensing regimes, also highlight the growing recognition that model security cannot be separated from hardware security. Incorporating such measures can help mitigate the real-world risk of bit-flip exploitation in MLaaS settings.
9 Conclusion
In this work, we introduced a novel adversarial BFA framework, AttentionBreaker for transformer-basee models such as LLMs. For the first time, Breaker applies a novel sensitivity analysis along with evolutionary optimization to guide adversarial attacks on LLMs, leading to degraded performance. Specif- ically, perturbing merely 3-bits 4.129 x 10 %% of the parameters LLaMAA-SB-Instrue W8 results in MMLU scores dropping from 67.3% to 0% and Wikitext perplexity increased sharply from 12.6 to 4.72 x 10", Furthermore, our experimental results demonstrate that AttentionBreaker can subvert state-of-the-art de- fenses for protecting transformer-bes models against BFFA. These findings underscore the effectiveness of AttentionBreake in exposing the vulnerability of LLMs to such adversarial interventions.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14211 2024.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey evaluation of large language models. ACM Transactions on Intelligent. Systems and Technology, 15(3):1-43 15:) 2024.
Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. Security and privacy challenges of large language models: A survey. arXiv preprint 2:Xiv:2402000885 2024.
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and Tanaka Meyarivan. fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. In Parallel Problem Solving from Nature PPSN VI: 6th International Conference Paris, France, September 18-20, 2000 Proceedings 6, pp. 849- 858. Springer, 2000.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21700 2024.
Russell Eberhart and James Kennedy. A new optimizer using particle swarm theory. In 95. Proceedings of the sirth international symposium micro machine and human science, pp. 39-43. Teee, 1995.
Tommaso Frassetto, Patrick Jauernig, Christopher Liebchen, and Ahmad-Reza Sadeghi. {IMIX): (In- Process} memory isolation (EXtension). In 27th USENIX Security Symposium (USENIX Security 18), pp. 83-97, 2018.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Mucnnighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, htt.r//ddorgreeerrr//////86668
07 2024. URL
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, al. Are done with mmlu? Xix preprint urXiv:2406004 04127, 2024.
16

---

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. Changing answer order decrease mmlu accuracy. Xiv preprint orXiv:206. 19470, 2024.
Yu-ichi Hayashi, Naofumi Homma, Takeshi Sugawara, Takaaki Mizuki, Takafumi Aoki, and Hideaki Sone. Non-invasive emi-based fault injection attack against cryptographic modules. In 2011 IEEE International Symposium on Electromaguelic Compalibility, pp. 763 767. IEEE, 2011.
Zhezhi He, Adnan Siraj Rakin, Jingtao Li, Chaitali Chakrabarti, and Deliang Fan. Defending and harnessing the bit-flip based adversarial weight attack. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognation, pp. 14095-14103, 2020.
Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? arXiv preprint arXiv:2405.000 06100, 2024.
Hadi Mardani Kamali, Kimia Zamiri Azar, Farimah Farahmandi, and Mark Tehranipoor. Advances in logic locking: Past, present, and prospects. Cryptology ePrint Archive, 2022.
Ingab Kang, Walter Wang, Jason Kim, Stephan van Schaik, Youssef Tobah, Daniel Genkin, Andrew Kwong, and Yuval Yarom. Sledgchammer: Amplifying rowhammer via bank-level parallclism. In Proc. USENIX, 2024.
Mehmet Kayaalp, Nael Abu-Ghazalehe Dmitry Ponomarev, and Aamer Jaleel. A high-resolution side-channel attack on last-level cache. In Proceedings of the 53rd Annual Design Automation Conference, pp. 1-6, 2016.
Yoongu Kim, Ross Daly, Jeremie Kim, Chris Fallin, Ji Hye Lee, Donghyuk Lee, Chris Wilkerson, Konrad Lai, and Onur Muthu. Flipping bits in memory without accessing them: An experimental study of dram disturbance errors. In 2014 ACM/IEEE 41st International Symposium Computer Archileclure (ISCA), pp. 361-372, 372 2014. doi: BALL109/ISCA.2014.6652
Shamik Kundu, Sanjay Das, Sayar Karmakar, Arnab Raha, Souvik Kundu, Yiorgos Makris, and Kanad Basu. Bit-by-bit: Investigating the vulnerabilities of binary neural networks to adversarial bit flipping. Transactions on Machine Learning Research, 2024a.
Souvik Kundu, Anthony Sarah, Vinay Joshi, Om J Omer, and Sreenivas Subramoney. Cimnet: Towards joint optimization for dnn architecture and configuration for compute-in-memory hardware. arXiv preprint arXiv:2402. 1 1170, 2024b.
Annu Lambora, Kunal Gupta, and Kriti Chopra. Genetic algorithm-a literature review. In 2019 international conference on machine learning, big data, cloud and parallel compuling (COMITCon), pp. 380 384. IEEE, 2019.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved basclines with visual instruction tuning, 2023.
Haocong Luo, Ataberk Olgun, Abdullah Giray Yağlikıı, Yahya Can Tugrul, Steve Rhyner, Meryem Banu Cavlak, Joël Lindegger, Mohammad Sadrosadati, and Onur Mutlu. Rowpress: Amplifying read distur- bance in modern dram chips. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1-18, 2023.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint ar Xiv:22402. 1 1764,, 2024.
Alcksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv:1706. 06083, 2017.
arXiv preprint
Kaleel Mahmood, Rigel Mahmood, Ethan Rathbun, and Marten van Dijk. Back in black: A comparative evaluation of recent state-of-the-art black-box attacks. IEFE Access,
2021.
17

---

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theorctical analysis and appli-
cations. In International conference on Machine learning, pp. 33803-238222 pmlr, 2023.
Onur Mutlu and Jeremie S Kim. Rowhammer: A retrospective. IEEE Transactions Computer- Aided
Design of Integrated Circuits and Systems, 39(5):1555-1571, 2019.
Najmeh Nazari, Hosein Mohammadi Makrani, Chongshou Fang, Hossein Sayadi, Setareh Rafatirad, Khaled N Khasawneh, and Houman Homayoun. Forget and rewire: Enhancing the resilience of transformer-ban -basee models against (Bit-Flip) attacks. In 33rl USENIX Security Symposium (USENIX Security 24), pp. 1349-1366, 2024.
Nghia Hieu Nguyen, Duong TD Vo, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. Openvivqa: Task, dataset, and multimodal fusion models for visual question answering in vietnamese. Information Fusion, 100:101868, 2023.
Ozan Özdenizei and Robert Legenstein. Improving robustness against stealthy weight. bit-slip atlacks by output code matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13388-13397, 2022.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM Asia conference on computer and communications security, pp. 506 519, 2017.
Cheng Qian, Ming Zhang, Yuanping Nic, Shuaibing Lu, and Huayang Cao. A survey of bit-flip attacks deep neural network and corresponding defense methods. Electronics, 12(4):853, 2023.
Alec Radford. Improving language understanding by generative pre-training. OpenAI, 2018.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Bit-flip attack: Crushing neural network with progressive bit search. In Proceedings of the IEEE/CVF International Conference Compuler Vision, pp. 1211 1220, 2019.
Adnan Siraj Rakin, Zhezhi He, Jingtao Li, Fan Yao, Chaitali Chakrabarti, and Deliang Fan. T-bfa: Targeted bit-flip adversarial weight attack. IEEE Transactions on Pottern Analysis and Machine Intelligence, 44 (11):7928-774 7939, 2021.
Amit Mazumder Shuvo, Tao Zhang, Farimah Farahmandi, and Mark Tehranipoor. A comprehensive survey on non-invasive fault injection attacks. Cryptology ePrint Archive, 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Srinivas and Kalyanmoy Deb. Muiltiobjective optimization using nondominated sorting in genetic algo- rithms. Evolutionary Computation, 2(3):221-22) 248, 1994. doi: wtw...29.vv...91...33..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anmi Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1399 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Michael D Vose. The simple genetic algorithm: foundations and theory. MIT press, 1999.
Jialai Wang, Ziyuan Zhang, Meiqi Wang, Han Qiu, Tianwei Zhang, Qi Li, Zongpeng Li, Tao Wei, and Chao Zhang. Aegis: Mitigating targeted bit bit-flip attacks against deep neural networks. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 2329 2346, 2023.
18

---

Qun Wang, Xizhen Xu, Xiaoxin Ding, Tiebing Chen, Ronghui Deng, Jinglei Li, and Jiawei Jiang. Multi objective optimization and evaluation approach of prefabricated component combination solutions using nsga-ii and simulated annealing optimized projection pursuit method. Scientific Reports, 14(1):16688, 2024.
Yun Xiang, Yongchao Xu, Yingjie Li, Wen Ma, Qi Xuan, and Yi Liu. Side-channel gray-box attack for dnns. IEEE Transaclions on Circuils and Systems II: Express Briefs, 68(1):501 505, 2020.
Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource-efficien 1lm and multimodal foundation models. arviv preprint xrXe:2401.080022 2024.
Mengjia Yan, Christopher W Fletcher, and Josep Torrellas. Cache telepathy: Leveraging shared resource attacks learn (DNN) architectures. In 29th USENIX Security Symposium (USENIX Security 20), pp. 2003 2020, 2020.
Fan Yao, Adnan Siraj Rakin, and Deliang Fan. (DeepHammer)): Depleting the intelligence of deep neural networks through targeted chain of bit flips. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1463-1480, 2020.
Yuval Yarom and Katrina Falkner. {FLUSH+ RELOAD]: A high resolution, low noise, 13 cache {Side- Channel) attack. In 23rd USENIX security symposium (USENIX security 14), pp. 719 732, 2014.
Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. Dive into Deep Learning- Cambridge University Press, 2023a. https://D2L.att
Zhenyu Zhang, Xiaoqing Cheng, Zongyi Xing, and Xingdong Gui. Pareto multi-objective optimization of metro train energy-saving operation using improved nsga-ii algorithms. Chaos, Solitons & Fractals, 176: 114183, 2023b.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
19

---

A
Appendix
A.1
Background
In this section, we elaborate on the types of model attacks based on the level of knowledge the attacker can extract from the victim model.
A.1.1 Model Attacks
Full Knowledge Attacks In shared cloud environments, attacks leveraging complete knowledge represent the gravest security threat to deployed large language models (LLMs). This scenario assumes adversary with extensive knowledge of both the larget model and deployed defenses, thus allowing for optimized allack strategies. Attackers can utilize advanced side-channel techniques to compromise hardware vulnerabilities and exfiltrate sensitive model parameters, encompassing proprietary weights and confidential user queries. One of the most potent techniques available to attackers in this context is the Cache Sate-Channel Attack, specifically largeting the Last-Tevel Cache (LI.C). Yarom & Falkner (2014) and Kayaalp al. (2016) have shown that these attacks leverage shared hardware resources to deduce memory access patterns, thereby exposing sensitive data within the cache. Despite the severity of full-knowledge attacks, they remain techni- cally challenging due to limited data extraction rates and the high complexity of reconstructing deep neural networks from indirect leakage. However, even the partial extraction of key model components, including embedding layers and attention heads, poses a significant risk, potentially enabling adversaries to replicate model functionality or generate targeted adversarial perturbations.
White-box Attacks: While-box allacks pose serious threat to large language models (LLMs), leveraging attackers' knowledge of model architecture and training data Madry (2017); Rakin et al. (2019). While attackers cannot directly access memory, they can estimate vulnerable bits to execute Rowhammer attacks and exploit hardware vulnerabilities Mutlu & Kim (019)..
Two key factors contribute to this risk. First, many commercial models and datasets are open-source, and users frequently download pre-trained models for transfer learning. These models inherit vulnerabilities, giving attackers foundation for exploitation. Second, adversaries use hardware monitoring tools to analyze memory access patterns Yan et al. (2020), allowing them to infer model architecture and train local copy with similar weaknesses. Unlike full-knowiedge attacks, white-box attacks are practical and accessible, making them a baseline threat model. Mitigating these risks requires robust memory protection, hardware-level defenses, and adversarial training to prevent model extraction and fault injection.
Gray-box Attacks: In gray-box scenarios, attackers possess partial knowledge of the target model, such as its architecture or certain parameters, but lack complete access. This limited insight enables them to craft more efficient attacks compared to black-box methods Xiang el al. (2020); Zhang & Sabuncu (2018). Attackers analyze model outputs and leverage statistical or adversarial techniques to infer model properties and vulnerabílities or extract knowledge. These attacks enable techniques such as model extraction (where allackers approximate the larget model's behavior) and prompt inference (where sensitive information is retrieved based on response patterns).
Since attackers lack direct access to model internals, they rely on query efficiency to reduce costs while maximizing the extraction of decision boundaries or key model parameters. This attack lype is particularly concerning in API-based L.L deployments, where adversaries can probe responses and fine-tune adversarial inputs without needing access to underlying weights or architecture.
In black-box scenarios, attackers have no direct access to the target model's ar- chitecture or parameters Papernot et al.] (2017): Mahmood et al. (2021). They rely solely on the model's outputs to infer vulnerabilities and craft adversarial examples. A notable approach involves using gradient estimation techniques to generate adversarial inputs, effectively decciving models without internal knowl- odge. Another study highlighted the practicality of black-bos attacks by crafting adversarial examples that mislead machine-learnin models, even in the absence of model specifics.
Black-box Attacks:
20

---

Real-world Execution Feasibility: In this work, we use the white-box attack setting for identifying the most critical bits in the model. Evenn though a white-box type bit flip attack assumes direct access to model weights, an attacker can still indirectly induce such faults in a deployed model by targeting the hardware that holds the model in memory. In real-world scenarios, attackers might exploit hardware vulnerabilitics- such as the Rowhammer effect- to cause bit flips in DRAM. For example, if an allacker manages to run code on the same physical server as the deployed model (as might be possible in a multi-tenant cloud environment), they can use fault injection techniques to flip few critical bits in memory identified by techniques such Memory Access Pattern Analysis Yan et al. (2020). This doesn't ttt require direct API access to the model's weights; rather, it leverages a vulnerability in the underlying hardware to corrupt the model's parameters at runtime.
A.1.2 Transformer Models
The Transformer architecture is foundational for LLMs and Vision Transformers, driving advancements in NLP and image classification Vaswani (0177;; Radford (2018); Chang et al. (2024) : Xu et al. (2024). It comprises two main components: the encoder to convert input data into a structured intermediate represen-
tation for improved feature understanding and the decoder to generate output sequences from this encoded CoformatioZhhaag el al. (2023a). There are three lask-specifie configurations: Encoder-only models for input-driven tasks (e.g., text classification, named entity recognition), Decoder-only models for generative tasks, and Encoder-decoder models for tasks requiring input-dependent generation (e.g., machine translation, summarization).
A critical innovation in the Transformer is the attention mechanism, particularly multi- attention, which prioritizes relevant input elements Zhang et al. (2023a). Attention weights are computed to emphasize informative lokens, while masks can exclude specilic tokens, such as padding. In each multi-head attention layer, the model derives three parameters for each input token: Query (Q); Key (K), and Value (V), calculated through linear transformations of input vectors. Attention scores assess token importance across the sequence. Each head processes distinct Q, K, and V subsets, facilitating parallel analysis of diverse token relationships The individual altention outputs are concatenated and linearly combined to produce the final attention output, enhancing the model's ability to capture complex dependencies and improve task performance in both NLP and vision domains.
A.1.3 Bit-flip Attack (BFA)
Bit-flip Vulnorabilitics:
The robustness, sccurity, and safety of modern computing systems are intrinsi- cally linked to memory isolation, which is enforced through both software and hardware mechanisms Frassetlo et al. (2018). However, due to read-disturbance phenomena, contemporary DRAM chips remain susceptible to memory isolation breaches Kim et al. (2014). RowHammer Mutlu & Kim (2019) is well-documentedd example where repeated DRAM row (hammering) access induces bitflips in adjacent rows due to electri- cal interference. SledgeHammer enhances the effectiveness of RowHammer by combining it with bank-level parallelism and hammering multiple memory banks simultaneouslyKKngg et al. (2024). A recent and signif- icant advancement in read-disturbance attacks is RowPress Luo al. (2023), specifically targeting DDR4 systems with RowHammer protection mechanisms. RowPress demonstrates that prolonged access to a single DRAM row can disrupt physically adjacent rows, causing bitflips. This techníque effectively circum- vents RowHammerr defenses, requiring substantially fewer row activations to induce faults and underscoring DRAM's vulnerability.
Existing Bit-flip Attacks: Existing research has explored various fault injection methods in neural net- works, targeting biases, weights, and activation functions to induce targeted and untargeted misclassifica- tions. While these techniques were primarily designed for full-precision DNNs, many edge implementations operate in quantized precision, making them inherently more resilient to parameter variations. To overcome this, efficient algorithms have been developed to identify the most vulnerable bits in quantized DNNs. Rakin et al. Rakin et al.] (2019) exploit RowHammer to expose BFA vulnerabilities in DNNs, using Progressive Bit Search (PBS) iteratively flip critical bits and degrade accuracy. TBFA Rakin et al. (2021) extends this by targeting specific misclassifications. DeepHammer Yao et. al. (020 exemplifies advanced BFAs by
21

---

employing system-level flip-aware PBS that enables memory--fficient rowhammering and precise flipping of targeted bits.
However, BFAs have been predominantly studied in the context of DNNs and remain largely unexplored for Transformer- based architectures such as LLMs due to their substantially larger parameter space and struc- tural differences. Applying conventional BFA techniques to LLMs is highly challenging, a difficulty further exacerbated by the growing adoption of quantized and low-precision models. Towards filling this research gap, we for the first time propose novel, efficient BFA framework, AttentionBBeaakrr, to methodically exploit the bit-flip vulnerabilities of LLMs in an efficient manner. While our approach is designed for untargeted attacks, it can be adapted for targeted settings by modifying the objective and flipping bits critical to a specific class. We focus on untargeted attacks as earlier research indicates that they are more challenging to defend (zdenizzi & Legenstein (2022). Studies further suggest that degrading overall performance through untargeted attacks lacks distinct attack signatures unlike targeted attacks, rendering conventional defenses ineffoctive Qian ct al.] (2023); Nazari et al. (2024). Therefore, AttentionBreaker strategically employs an untargeted allacc to maximize disruption while evading traditional defense strategies.
Quantization precision impact on bit-flip attacks: Quantization precision plays crucial role in determining the vulnerability of machine learning models to bit-flip attacks. It affects both the number of bit-flips required and the extent to which these flips can disrupt model performance. The impact of bit- bit-flips varies across different numerical formats due to their underlying representation and sensitivity to perturbations. The value of an integer stored in an n-bit signed representation (c.g., INT8, INT4, INT2) is given by:
n-2
X X= (-1)ba
bk 2*
(15)
k=
Where by represents the bit at position k. Flipping a bit at position & changes X by:
AX (-1) 2*
(16)
On the other hand, floating-poin point representations, such as FP16 and FP32, follow the IEEE 754 standard, where a number is stored as:
X= (- Emsoo (1 + M)
(17)
where 8 is the sigu bit, E is the exponent with bias Erias, and M is the normalized mantissa. Now, flipping the sign bit changes the sign of the value. Flipping bit in the exponent can change the value by a factor of 2*, lcading to drastic value shifts (e.g., a bit-flip in the 7th exponent bit transforms an FP32 number 1.0 to infinity). Flipping a bit in the mantissa introduces a small perturbation, where the relative change is approximately 2-23 X, which is much smaller and therefore requires multiple bit-flips the mantissa. to achieve significant impact.
For example, in INT8, flipping the most significant bit (MSB) of the value (0b01000000) changes from 64 to -64. This demonstrates that MSB flips can significantly alter values, but the effect is constrained by the limited range of [-128, 127]. As the precision further declines to INT4 and INT2, the range correspondingly diminishes, resulting in ranges of [-8, 7] and [-1, 1] respectively. Consequently, a single bit-flip causes a smaller error, necessitating larger number of bit-flips to produce a significant effect.
In a Float32 parameter, flipping the most significant bit can change a value from a large positive number to equally large negative number (e.g., 3.4x 10a8 to -0.433xx100 1025). This drastic shift reflects the role of the sign bit in IEEE 754 representation. Bit-flips in the exponent mantissa can further lead to outcomes ranging from numerical overflow to subtle degradation. This vulncrability extends to reduced-precision formats. In Float16 (FP16), with a smaller dynamic range and 10-bit mantissa, sign-bit. flips produce equivalent sign inversion (e.g., 6.5 x 10 to -6.5 x 10*), while exponent or mantissa flips yield more pronounced errors due to limited precision. Similarly, bfloat16 (BF16), which retains Float32's 8-bit exponent but roduces the mantissa to 7 bits, exhibits analogous sign sensitivity with coarser representational granularity. Table 11 depicts the extreme criticality of full-precision models. For instance, in L.Ia VA1 V.1.6. Misttrall- FP16,
22

---

single bit-flip via ArrenerooBreaker suffices to collapse model accuracy entirely, whereas the W4 (NF4) variant requires 53 flips to achieve comparable effect. These findings underscore that, despite appearing more robust, lower-precision models remain extremely vulnerable to bit-flip attacks. Therefore, we focus on targeting lower precision models in our experiments.
Model
Model Type
Precision
Parameters
AllentionBreaker (# Bit-klipe)
Benchnnakks (Accuracy %) Before / After Alckk
WikfText-2
MMLU
VQAv2
TextVQA
LLaMa-3
LLM
BFlosti6
8,03B
10:24 / 3.8c1
66.3 / 0
NA
NA
LLaVa Mistral 1.6
VLM
DFloat16
7.57B
NA
NA
11.2
63.8 /
Phi-3-mini- 128k
LIM
D'losti6
3,820
9:45 3.6c4
68.1
NA
NA
/
49.3//0
NA
Qwan3-0.0il B-Bse
LIM
BFlssti6
506M
18:23 / 3.2el
49.8 7
NA
NA
Deepocok--71 7B
LLM
DF'ant16
7D
9.92/4.204
41.8/0
NA
NA
MrniMixtral-v0.2
LLM
DF
12.90
20.78/2.1s4
58.35/0
NA
NA
Table 11: Effectiveness of AttentionBreaker across multiple full-precision models and benchmarks.
A.2 Ablation Study
This section presents an ablation study to systematically evaluate and decide on specific components in our proposed method, such as a, top-n, or which genetic algorithm to use for optimization.
A.2.1
Sensitivity Ratio Variation
We vary between VW] and | in the min- max normalized importance score calculation as described in Scction 4.2.1 Under diffcrent values of a, the model loss at different bit-flip rates is depicted in Figure 5a As observed, an optimal value of a 0.5 achieves the best results, closely followed by = and a = 0.75. Consequently, a 0.5 is selected for subsequent experiments.
A.2.2 All vs top-n Layer Sensitivity Analysis
In this experiment, we investigate the impact of varying the selection of top-n layers on model performance by comparing the effects of sub-sampling from the top-n layers versus sub-sampling from all layers. With fixed sensitivity parameter of a 0.5, we increment n from 1 The results demonstrate that sub- sampling from the top-n layers leads higher loss than sub-sampling from all layers, as illustrated in
12
==0 0
12
All layers
a= 0.25
Top-1 layer
Top-3 layers
10
a 0m5.
10
Top-5 layers
0.75
Model loss
Model loss
a ==1
8
6
4
2
0.00
0.25
0.50
0.75
1.00
1.25
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Bit-flip rate (%)
1e-5
Bit-flip rate (%6)
le
(a) Sensitivity ratio (a).
(b) Top-ranked layer count.
Figure MaMMA3-SB-nnstructt W8 ablation study on Sensitivity Ratio Variation and top-n layer count. Analysis was performed using the 'Astronomy'' task of the MMLU benchmark.
23

---

Among the top-n layers, selecting the top-1 layers yields the best performance, followed by top-3 and top-5. The weight subset selection and optimization based on top-n layers provide superior results, the constrained weight subspace facilitates faster convergence. Consequently, we fir = 1 for all subsequent experiments.
Figure 5b
A.2.3 Genetic Algorithms Variation
We extensively evaluate the performance of proposed evolutionary optimization technique against estab- lished algorithms, including the Simple Genetic Algorithm (SGA) Vose (1999), Particle Swarm Optimization (PSO) Eberhart & Kennedy (1995). Non- Son-Dominated Sorting Genetic Algorithm (NSGA) Srinivas & Deb 1994 and NSGA-IT Deb et al. (2000g: Lambora et al. (2019). It is essential to note that these baseline methods employed are established benchmarks in recent research Kundu et al. (2024b); Zhang et al. (2023b): Wang et al. (2024). Our problem is inherently multi-objective, as we aim to maximize model loss and min- imize the number of bit-flips. These two objectives make NSGA-based methods particularly relevant for comparison. We reformulated
approach into unified loss function to ensure fair evaluation. This transformation allows for direct comparison between our method and NSGA-based approaches while pre- serving the underlying optimization goals. To ensure comparability, algorithrms, setting the population size to 100, mutation rate to 0.1, and crossover rate to 0.9. Addition- ally, we apply modifications in population initialization, crossover, and mutation strategics as in our custom implementation. Results Figure (ia demonstrate that our modified genetic algorithm achieves superior convergence speed and accuracy, owing to optimizations tailored to problem-specific requirements. Conse- quently, this oplimized genetic algorithm the optimization of the weight set within EitenttioBBeaker" genetic implementation. Notably, the required weight count decreases exponentially as the fitness score rises, reflecting rapid improvement. After approxi- mately 80 90 iterations, both metrics plateau, indicating convergence; at this stage, the optimal solution is returned as the output.
standardize parameters across all
employed in all subsequent experiments. Figure [bb illustrates
A.3 Intermediate Results
A.3.1 Layer Sensitivity Analysis
Figure presents the sensitivity distribution using loss across layers for the I. W4, along with BitNet and LLaVA1.6-BB models. For LLaMA3-SB-Instruct W4, the model loss across several layers describes the sensitivity of Bit.
remains elevated, signifying high sensitivity in Figure 7a
Figure 7b
1e4
1e5
# Bit-flips
1.0
GenBFA
Fitness score
Simple GA
0.8
4-
PSO
Fitness score
# Bit Bit-flips fllps
NSGA
It-llips
0.6
3-
NSGA-II
0.4
#
0.2
1-
0.0
10
20
30
40
50
60
0
20
40
60
80
100
# Generations
# Generations
(a) comparison between various genetic algorithms.
(b) number of bit-flips and fitness score optimization.
Figure 6: IaMMA3-SB-Instruuct W8 ablation study on genetic optimization to select critical weights. Fitness scores for the genetic algorithms caleulated using the " "Astronomy" task of the MMLU benchmark.
24

---

which, with its more aggressive quantization, exhibits greater resilience, reflected by fewer layers furnishing high model loss. In the case of I.LVVA16--7B W8, high sensitivity to parameter perturbation is observed primarily in the language processing component of the model. The vision processing component, preceding the language component, has a less pronounced effect. The most sensitive layers are among the initial layers of the language component, as seen Figure 7c This showcases the variation in layer sensitivities different components in multi-modal models.
Hybrid sensitivity score
Hybrid sensitivity score
8.5
Hybrid sensitivity
8.0
Model loss
Model loss
5
Model loss
8-
7.5
2
50
100
150
200
50
100
150
200
250
100
200
300
Layer
Layer
Layer ID
(a) -LaMA3-SB-Insruue W4.
(b) BitNet W1.58.
(c) LLaVAL.6-B W8.
Figure Layer sensitivity analysis for LLMs and VLMs with varying quantization formats. Sensitivity scores and model loss are calculated using the 'Astronomy' task of the MMLU benchmark.
A.3.2 OBTAINING THE MOST CRITICAL PARAMETERS
The GenBFA algorithm, detailed in Section 4.2.2 refines the weight subset generated by the previous step. Through iterative refinement, this stage systematically reduces the weight subset, yielding a final set for bit-flip perturbation that is minimal in size yet achieves the same degradation in model performance as the initial set. This iterative optimization process is captured Figure which shows an exponential reduction in weight subset size while maintaining loss or higher than the loss threshold.
1e3
12
le6
13
11
1.6
(Bit- Bit-fiip
Leh
Model loss
Bit-hlips
Model loss
As
Model loss
1.4
1.0
1.2
Bit flips
0.5
Lth
Bit-flips
Bit-flips
0
0.0
100
200
50
100
150
200
100
Generations
(a)
W4.
(b) BitNet W1.58.
(c) Phi-3-mini- 128k W8.
Figure 8: Genetic optimization for critical parameter identification. Loss is calculated on the "Astronomy"
lask MMLU.
A.4 Illustrative Attack Efffects of the Model Attack
In this section, we systematically analyze the impact of adversarial bit-flip perturbations on model behavior, as introduced through our proposed framework. We present empirical evaluations across a diverse set of large language model (LLM) tasks, including question answering, code generation, content generation, and translation. The following box illustrations provide representative examples demonstrating the extent of model degradation under BFAs.
25

---

A.4.1 Mathematical question answering
Box presents comparative analysis of the model's behavior before and after the adversarial attack. The results illustrate substantial degradation in the model's output quality following the attack. When tested with LLaMA3-SB-Instree W8 with MMLU under normal operation, the model produces accurate and contextually relevant responses. However, post-attack, the generated output becomes highly erroneous and incoherent with respect to the given query, highlighting the significant impact of the adversarial perturbation on the model's reliability.
A.4.2 Code generation
Box presents illustrative example of the model's behavior in code generation task before and after the adversarial attack. In the pre-attack scenario, the model correctly generates Python function to compute the Fibonacci sequence, demonstrating its intended functionality. However, following the attack, the model's output becomes completely corrupted, consisting of incoherent and nonsensical characters. This stark contrast highlights the severe impact of the adversarial perturbation, loading to a catastrophic failure in code generation and rendering the model unreliable for practical use.
A.4.3 Language translation
Box 4 illustrates the impact of the adversarial attack translation task. In the pre-attack scenario, the model correctly translates the given English sentence into French, producing the expected output with high fidelity. However, after the attack, the model fails to perform the translation entirely, instead echoing the original English imput followed by a sequence of nonsensical characters. This degradation demonstrates the attack's effectiveness in disrupting the model's core functionality, leading a complete failure producing coherent translation.
Box Mathematical question answering
System Prompt:
You are helpful AT assistant, Answer the following question. Few-shot (n) Examples:
Question 1
Expected model output 1
Question 2
Expected model output 2
Question # (n+1): Edgar cats 18 pretzels a day. If his brother cats 1/2 as many, how many does his brother eat in week?
Model Output (Pre-Attack)
Model Output:
Edgar eats 18 pretzels day. His brother eats 1/2 many, so he eats 18 1/2 pretzels day. In week, he cats 9 7 63 pretzels. The answer is 63.
Extracted Answer: 63
Model Output (Post-Attack)
Model Output:
/iwbfninwdjo [lii eweijfnpow/e edfjo [wedj [wed owed[ wqd Extracted "nswer:"
26

---

Box Code generation
System Prompt: Write function
Write code calculate fibonacci sequence python.
Model Output (Pre-Attack)
we have to write a code python to calculate the Fibonacci sequence for the given imput of 'n', we can create function called 'fibonacci' as follows: def fibonacci(n):
Model Output:
# initializing the first two fibonacci numbers
a, b = 0,
# printing the first two numbers
if > 0:
(ai b)
Model Output (Post-Attack)
Model Output:
Box 4: Language translation
System Prompt: Translate from English to French: have to submit an assignment tomorrow. Expected output: Je dois soumettre un devoir demain.
Model Output Atta-kttack)
Model Output: Je dois soumettre un devoir demain.
Model Output (Post-Attack)
Model Output: have submit an assignment tomorrow
Box 5: Content generation
System Prompt: Write an article about " "Artificial Intelligence and Ethics."
Model Output (Pre-Attack)
Artificial intelligence (Al) has been a field of interest and innovation, promis- ing numerous profound impacts on socicty. However, as AT technology advances...
Model Output:
Model Output (Post-Attack)
Model Output: Write article about Artificial Intelligence and Ehhiescchhesischhsissqpppt peaceHubb@ peace@K @20K aaKoa2aK @@2..
27

---

A.4.4 Content generation
Box 5 highlights the adverse effects of the attack on a content generation task. Before the attack, the model generates a coherent and contextually relevant article introduction on "Artificial Intelligence and Ethics", demonstrating its ability to produce structured and meaningful text. However, after the attack, the model's output becomes entirely corrupted, consisting of mix of nonsensical phrases, symbols, and numerical sequences. This severe degradation underscores the attack's capability to disrupt the model's generative capabilities, rendering it incapable of producing intelligible contextually appropriate content.
A.5
Adaptability and Robustness
In this section, we examine the adaptability of AttentionBreaker in constrained environments, along with its robustness and the transferability of its attack effects.
A.5.1 Gradient-free AttentionBreaker
In this experiment, we investigate the effectiveness of AttentionBreaker in a gradient-free context, where gradient information inaccessible at all stages. Here, the importance scores are computed solely based weight magnitudes by setting a 0.0. Our results demonstrate that this magnitude-based approach maintains the attack's efficacy, as it accurately identifies critical weights, enabling highly impactful attack. We observe that, without gradient information, setting a = 0 allows the model to be attacked with only 4.129 x 10-9% bit-flips, reducing model accuracy to zero. This degradation matches the attack's 's effective- ness at a = 0.5, as depicted in Figure 3. Subsequent analysis of the attack vector confirms that identical weights and bits are selected for attack across both settings, underscoring AttentionBreaker's versatility and adaptability. Additionally, Section 55..6 demonstenss how gradient free AttentionBreaker can be highly ca- pable in attacking large models such as LLaMA 3.3 70B Instruct NF'4 model with over 70 billion parameters. This robustness is particularly significant as many defenses against BFAs assume that gradient access is essential for an altacker, prompling defenses to obscure protect gradients. However, AllentionBreaker's 's gradient-free capability bypasses these gradient- based defenses entirely by relying on magnitude-bass impor- tance, showing its potential to circumvent gradient-based restrictions. This adaptability highlights Attention- Breaker's broad applicability in gradient-restricten environments, establishing as a resilient approach for BFAs.
Our approach provides a flexible and user-dependent framework, allowing adversaries to choose between gradient-free free and gradient-based strategies based on their specific objectives and com- putational constraints. For adversaries prioritizing computational efficiency, the gradient-free variant of AttentionBreaker offers a viable solution by leveraging weight magnitudes as a sensitivity proxy, eliminating the need for costly gradient calculations while maintaining high attack efficacy. Conversely, for adversaries willing to accept higher computational costs to minimize perturbations and achieve more precise attacks, the gradient-basod version remains advantagcous. This adaptability not only enhances the practicality of our method but also strengthens its resilience against existing defenses. Defenses targeting gradient-based atlacks may be ineffective against magnitude- magnitude-based approaches, while those designed to counteract magnitude- based perturbations may fail against gradient-driven strategies. Furthermore, in models with structured depen- dencies, weight redundancy, or adaptive optimization mechanisms like Adam, gradients play a crucial role in identifying the most impactful perturbations, reinforcing the necessity of gradient-based attacks in certain cases. Given that defenses often redistribute weight significance to mitigate bit-flip attacks, the inclusion of both approaches ensures that our method remains effoctive across diverse defensive strategics. By offer- ing both computational efficiency and targeted precision, our framework establishes itself as robust and adaptable bit-flip attack methodology.
A.5.2 Gray-box AttentionBreaker
While the proposed AttentionBreaker operates as a whito-box attack, assuming complete access to model parameters, practical scenarios may restrict access to all parameters. In such cases, grey-box attacks, where only partial model information is available, become pertinent. To assess the robustness of our method under
28

---

100
100
MMLU task accuracy (%)
O-Acc
O-Acc PA-ACC
80
PA-Acc
80
Accuracy (%)
74.00
60
60
40
40
20
20
0
0
00
00
6-27
200
40
21
05
9.C4
04
07
0
20
40
0
1
2
3
4
5
6
7
8
10
Task index
LM Harness Benchmark ld
(a) MMLU.
(b) LM Harness.
Figure 9: LLMMA3-8B-nstrrue W8 attack transferability analysis across various tasks.
these constraints, we simulate grey-box scenario to demonstrate the efficayy of the attack. Specifically, we limit accessibility to only a subset of initial model layers.
In order to simulate such limitations during a gray-box attack, we focus our attack strategy exclusively on the initial layers of the model. The initial layers are particularly susceptible bit-flips, making them ideal candidates for our gray-box attack. Furthermore, even when restricted having access only the weight magnitudes, without the gradients, attack approach remains highly effective, resulting in significant degradation in model accuracy, from 67.3% to 0%.
A.5.3 Task Transferability
This experiment examines the transferability of attack elfects across distinct lasks. Specifically, it investigates whether an attack executed on Task A leads to observable adverse effects on Task B. If the attack effects indeed transfer between tasks, would indicate a fundamental vulncrability in the model architecture, independent the task at hand. Such a finding would underscore the severity of the model's susceptibility to the attack. For this investigation, the I.I.MA3-8B W8 model was subjected to targeted attack on the MMLU task "Astronomy". Due to the attack, the accuracy of the model on " "Astronomy" drops from the original accuracy (O-Acc) of 72.1% post-attack accuracy (PA-Acc) of 0%. The resulting impact was then systematically evaluated across additional MMLU tasks and other language benchmarks, including the LM harness benchmark. The results reveal a high degree of transferability, whereby the attack yields comparable degradation in performance across the evaluated tasks. This finding underscores the broad vulnerability of the model to such attacks. Figure Da illustrates that an attack initially targeted at the "Astronomy" task leads to substantial accuracy reductions in other MMLU tasks. Similarly, an attack initially targeted at the "AddSub" task from the LM harness benchmark leads severe model degradation on other LM harness tasks well captured in Figure 9b This illustrates the pronounced transferability of the allack across varied tasks and domains.
A.5.4 AttentionBreaker Transferability to Fine-tuned models
This experiment evaluates the persistence of adversarial effects from an attack applied to base model when, after an attack, the model is loaded with previously finc-tuned adapters for specific task. Specifically, we fine-tune the LLaMA3--B-Instruuct W8 model on the " "AddSub" task from the LM harness benchmark and save the resulting fine-ture tue-tuned adapter. The fine-tuning progress is shown in Figure [0a] denoted as original fino-tuning or Original-FT-Loss. Next, we reload the original base model and perform the AttentionBreaker attack. As anticipated, the model exhibits a substantial loss increase and severe accuracy degradation, with accuracy plummeting from 85.2% to 0%, as illustrated in Figure 10b (labeled as F1). Upon loading the fine-
29

---

tuned adapter, observe that the model accuracy remains 0%, indicating that the fine-tuned adapter alone is insufficient to reslore model accuracy following the allack.
A.5.5 Fine-tuning-based Recovery
This experiment assesses the persistence of adversarial effects introduced through the BrtentionBreaker on base model, specifically evaluating whether subsequent fine-tuning We begin by attacking the LaMMA3-BB-nstrwa W8 model, followed by evaluating the model's accuracy on the "AddSub" task from the LM harness benchmark. As a result, the model's accuracy drops drastically from an initial 85.2% to 0%, as depicted in Figure 10b (labeled as F2), indicating that the model output now consists largely of arbitrary strings devoid of task relevance.
task can mitigate these impacts.
Subscquently, we fine-tune the attacked model on the same task, conducting five epochs with 21 iteration steps per epoch, as shown in Figure 10a] Throughout fine-tuning, the validation loss (or the post-attack fine-tuning loss PostAttacc-FFT-Los. declines substantially from imitial 9.12 to 0.27, signaling effective task adaptation. Despite this apparent success in convergence, post-finettningg testing reveals that model accuracy remains at 0%, as shown in Figure 10b This result suggests that while the model now outputs seemingly structured words rather than random strings, fails to exhibit. logical coherence or task-related reasoning, thus remaining unsuccessful in executing the logical deduction necessary for the "AddSub" task. The findings imply that fine-tuning does not suffice to restore task-specific functionality after the Attention- Breaker altakk.
A.5.6 Zero-masked fine-tuning
In this study, we investigate the persistence of adversarial effects introduced via the BreentionBreaker method on a base model and evaluate whether task-specific fine-tuning applied after resetting the perturbed weights zero can effectively mitigate these impacts. In particular, we aim to determine whether mullifying the impact of the bit-flipped weights (i.e., sctting the perturbed weights to zero, w, 0) permits effective recovery of model performance through fine-tuning on the same task. We begin atlacking the LLaMA3- 8B-Instruct W8 model using our AttentionBreaker framework, subsequently evaluating its accuracy on the "AddSub" task from the LM harness benchmark. Immediately following the attack, the model's accuracy plunges from an initial 85.2% to 0%, as illustrated in Figure 10b Next, we mask the attacked weights by resetting them zero (w; = 0) and fine-tune the model on the same task, performing five epochs with decreases
21 iterations per epoch. During fine-tuning, the validation loss (or PostAttaac2ZMeskkTTTT
Original-FT-Loss
100
O-Acc
PA-Acc
PF-Acc
10.0
PostAttack-FT-Loss
85.2
85.2
85.2
Model loss
7.5
PostAttack2Mak--TLLoss
Accuracy (%)
80
79.1
60
5.0
2.5
0.0
0.0 0.0
0.0 0.0
0.0
20
40
60
80
100
F1
F2
F3
Iteration steps
Sinetuning-based recovery
(a) Pine-tuning model loss with iterations.
(b) Fine-tuning- based recovery strategies.
Figure 10: Fine-tuning-bast based recovery after the attack: (a) validation loss with iteration steps and (b) model performance for different fine-tuning strategies. Metries calculated the 'AddSub' task of the
LM Harness benchmark.
30

---

significantly from initial value of 9.03 to 0.25, depicted in Figure 10a indicating that the model is adapting effectively to the task requirements.
Upon fine-tuning, we test the model performance on the same task and observe that the model's accuracy is substantially recovered from 0% to 79.1%, as shown in Figure 10b (labeled as F3). These findings suggest that fine-tuning preceded by zeroing out perturbed weights, sufficient to restore lask lask-specific functionality, thereby mitigating the adversarial effects induced by the ottack.
31


Reviews:
[{"review_id": "lmd7UL5CzF", "reviewer": "Reviewer_3zHX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The proposed method shows a great improvement in using significantly fewer bits to achieve a great attack performance. \n\nCons:\n1. The proposed method doesn't specify the threat model. It didn't specify what loss is used and how the loss will be achieved. Since the attack is on the language model, the loss should be different from the classification task. I am not sure whether the loss is applied with every downstream task or just the pretraining stage. Also, I am not sure if the loss is obtained from the training datasets or the test datasets. \n2. The proposed method lacks clarity. Some key details are without introduction and make the paper hard to follow. The paper introduces how to find the most sensitive weights, but I don't get how to find the most effective bits from the weights. Will the bit on the left always be selected, or is the pos the parameter to control the selection? The GenBFA also shows similar issues. What is the gene g in the crossover operation, and how to mutate the two subsets? I can get the rough picture, but there are not enough details.\n3. The methodology section needs a major revision of the algorithm description. There is no need to explain every line of an algorithm, such as storing some parameter somewhere. Functions such as SScore can be easily replaced by equation 1. \n4. The paper's attempt at a targeted case is interesting; however, I don't see how the method can be adapted. It could also be related to the first con that the loss function is never clearly stated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an efficient bit-flip attack on large language models. While the previous work requires a large number of flipped bits to make the attack effective, the proposed method only needs a significantly smaller amount. The proposed method first selects the most sensitive layers and weights with a sensitive score combining the weight and gradient magnitude. An evolution-based method is then proposed to reduce the size of the selected weights while maintaining a good attack performance. Experiments have been conducted on several open-sourced language models and show that a very small number of bits could lead to a significant performance degradation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the cons. The paper needs a major revision on the clarity.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8wnnB291eq", "reviewer": "Reviewer_7LX8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. This work introduces a tool that effectively scans and identifies sensitive weights within large language models (LLMs), offering valuable insights for other areas such as model compression and Trojan attack/defense.\n\n2. The empirical study demonstrates significant improvements over existing baselines (Nazari et al., 2024). By flipping just three bits in an 8-billion parameter model, the proposed method can completely reduce the LLM's accuracy to zero.\n\nCons:\n\n1. From my perspective, the primary distinction between this work and the existing method, DeepHammer, as mentioned in the paper, is the model size they are facing. DeepHammer is challenging to extend to the LLM setting due to its exhaustive nature, making it costly to attack LLMs. A simple baseline could involve applying DeepHammer only to the initial or last layer. The bit sensitive score of DeepHammer could be modified to maximum the loss in order to accommodate un-targeted attack.\n\n2. Some ablation studies are still missing. For instance, what would happen if we simply corrupted the top three most sensitive weights in the most sensitive layer? Alternatively, what if we only corrupted the top three most sensitive weights in the initial or output layer without using the complex evolutionary algorithm?\n\n3. While I understand that quantized models present a more challenging setting for bit-flip attacks, it would be beneficial to evaluate the attack's performance on models in their original precision. The intuition here is that full-precision models might be more capable of handling errors due to their additional capacity.\n\n4. The empirical study is heavily focused on the LLaMA series models. It would be advantageous to see attack results on other popular open-source models like Qwen and Mixtral.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel bit-flip attack method called AttentionBreaker, which significantly reduces the performance of large language models (LLMs) across multiple public datasets. AttentionBreaker identifies the most vulnerable weights in a neural network—those that have the greatest impact on the model's output—and flips their most significant bits (MSB) to create corrupted weights. The algorithm employs a scanning and combinatorial optimization mechanism to inflict maximum damage on the LLM's performance while minimizing the number of flipped bits.\n\nAttentionBreaker operates in two stages. In the first stage, the layer ranking module ranks weights within each layer by calculating a sensitivity score, which is a convex combination of their magnitude and gradients. The top-k most sensitive weights are selected for bit-flipping, where k is determined based on the layer's size. A layer score is then calculated based on the loss increase caused by modifying these sensitive weights, and all layers are ranked according to this score.\n\nIn the second stage, GenBFA, a heuristic evolutionary algorithm is used to further refine the set of perturbed weights, maximizing the damage to the model while minimizing the number of perturbed weights.\n\nEmpirical studies conducted on multiple public datasets across various open-source LLM architectures with different quantization schemes demonstrate the effectiveness of the proposed algorithm. Ablation studies explore different quantization levels, bit-flip budgets, evolutionary algorithms, and hyperparameters. The paper also examines generalization to gray-box attacks and the transferability of attacks to fine-tuning scenarios, with qualitative examples provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Comparison with Baseline Using DeepHammer: Evaluate the performance of the proposed method against a baseline where DeepHammer is applied to the initial, final, or a random layer.  For e.g., The bit sensitive score of DeepHammer could be modified to maximum the loss in order to accommodate un-targeted attack. This comparison will aid in assessing the significance of this work's contribution.\n\n2. Compare the results to a baseline where the top three most sensitive weights in the most sensitive layer are corrupted, to justify the necessity of using the evolutionary algorithm.\n\n3. Present the attack results on other popular open-source models such as Qwen, Mixtral, and DeepSeek.\n\n4. Show the performance on full-precision scenario if possible", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ktmYV8i2fO", "reviewer": "Reviewer_gKNh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- First work to demonstrate a 3-bit BFA can collapse a 8B-level LLM, overcoming the limitation of iterative gradient recalculations.\n- Rigorous methodology combing layer-wise sensitivity scoring, hybrid gradient-magnitude metrics, and evolutionary weight-subset optimization.\n- Covers diverse models(`LLaMA3`, `Phi-3`, `LLaVA`) and quantization schemes (INT8, INT4, 1.58bit).\n- Highlights real-world risks for LLMs deployed on shared hardware.\n\nWeaknesses:\n\n- The evolutionary optimization may not scale so efficiently to even more larger models (e.g., 70B+ parameters) or some other `MoE` architectures.\n- The method relies on idealized bit-flip models, without accounting for real-world noise or error correction.\n- The proposed XOR-cipher defense is not experimentally validated, leaving effectiveness unclear.\n- While a defense is proposed, the paper still lacks a broader i", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents `AttentionBreaker`, a novel and efficient Bit-Flip Attack (BFA) framework for LLMs. The authors demonstrate that merely 3-bits perturbation can collapse MMLU accuracy of `LLaMA3-8B-Instruct` from 67.3% to 0%, which also challenge existing state-of-the-art defenses.\n\nKey contributions include:\n\n- `AttentionBreaker` Framework: An approach combing layer sensitivity ranking and weight-set optimization to efficiently identify critical parameters in LLMs for BFA.\n- `GenBFA` algorithm: A novel evolutionary strategy to optimize the selection of minimal yet impactful subset of weights, achieving a 2000x reduction in the number of required bit-flips compared to exhaustive search.\n- Quantization-Aware Evaluation: Demonstrates the effectiveness of `AttentionBreaker` across INT8, INT4, and 1.58-bit quantized models.\n- Multi-model Extension: Extends the attack to VLMs like `LLaVA1.6-Mistral-7B` , achieving zero-shot task degradation with just 15 flips.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Important\n\n- Add a dedicated section discussing ethical implications, including potential misuse\n- Experimentally evaluate the proposed XOR-cipher defense against `AttentionBreaker`\n- Discuss hardware constraints in real-world and validate attacks under noisy bit-flip conditions\n\nStrengthen\n\n- report runtime and memory costs for larger or different architecture models\n- compare alternative defenses beyond just `FaR`", "broader_impact_concerns": "This paper exposes vulnerabilities in LLMs which could be exploited maliciously to degrade deployed models. The paper should:\n\n- warn about dual-use risks(e.g. attacks via shared hardware)\n- discuss regulatory implications(e.g. for hardware-level protections in `MLaaS`)", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)