Title: SilentStriker: Toward Stealthy Bit-Flip Attacks on Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Security

Bit-Flip Attack

Stealthy Attack
Abstract: The rapid adoption of large language models (LLMs) in critical domains has spurred extensive research into their security issues. While input manipulation attacks (e.g., prompt injection) have been well-studied, Bit-Flip Attacks (BFAs)—which exploit hardware vulnerabilities to corrupt model parameters and cause severe performance degradation—have received far less attention. Existing BFA methods suffer from key limitations: they fail to balance performance degradation and output naturalness, making them prone to discovery. In this paper, we introduce SilentStriker, the first stealthy bit-flip attack against LLMs that effectively degrades task performance while maintaining output naturalness. Our core contribution lies in addressing the challenge of designing effective loss functions for LLMs with variable output length and the vast output space. Unlike prior approaches that rely on output perplexity for attack loss formulation, which in-evidently degrade the output naturalness, we reformulate the attack objective by leveraging key output tokens as targets for suppression, enabling effective joint optimization of attack effectiveness and stealthiness. Additionally, we employ an iterative, progressive search strategy to maximize attack efficacy. Experiments show that SilentStriker significantly outperforms existing baselines, achieving successful attacks without compromising the naturalness of generated text.


Peer reviews:
comments: SUMMARY:
This paper proposes a new bit-flip attacks (BFA) method (SilentStriker) on LLMs. The key difference compared to the existing methods is that the authors combined key token loss and perplexity loss to degrade the performance of LLMs while still with fluent responses from LLMs. The authors evaluated their methods on five open-source LLMs range from 3b to 32b models. The evaluation results show the effectiveness of the proposed attack methods.

STRENGTHS AND WEAKNESSES:
**Strengths**

- This paper is well written and easy to follow. 
- The experimental results are interesting and shows the effectiveness of SilentStriker. 


**Weaknesses**

- Missing defense methods. Based on the examples provided in the paper, the responses from LLMs although fluent, but still logically inconsistent. This may easy to detect using some model-based methods. Some experiments on the SilentStriker against defense methods are needed to justify the stealthiness of SilentStriker.
- Missing general BFA methods on general DNNs, which I think has been widely studied for a very long time.

QUESTIONS:
- How SilentStriker performs against some defense methods.
- Please include some other references regarding BFA on general DNNs.

LIMITATIONS:
See my comments above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-10

comments: SUMMARY:
The authors explore an underexplored vulnerability in LLMs, Bit-Flip Attacks. They identify several limitations in existing approaches (performance degradation, detectability) and introduce a novel method to address these issues. Their SilentStriker method leverages a new loss function that enables optimization for variable output lengths in combination with an iterative search heuristic.

STRENGTHS AND WEAKNESSES:
**Strengths**
- Paper is easy to follow
- Addresses an underexplored topic
- Proposed method solves relevant limitations of previous approaches (output coherency, detectability)
- Extensive number of experiments with several models and datasets
- Ablation study, which analyzes the contribution of individual hyperparameters


**Weaknesses**
- Figure 2 is difficult to read. The caption should be more self-sufficient. I had difficulties understanding the Figure without reading the text (which kind of defeats the purpose of such an overview Figure). However, the Figure is visually appealing. 
- I would have liked a discussion about the practical feasibility of such attacks (e.g., concerning the number of flipped bits etc.)
- Methodological differences to previous work is only briefly discussed. Does the advantage of your method mainly stem from the optimization objective?

QUESTIONS:
- Does the advantage of the SilentStriker method mainly stem from the optimization objective compared to previous work?
- Could the authors increase the caption of Figure 2 to make it more self-sufficient? I believe this would make the paper easier to read.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SilentStriker, a stealthy Bit-Flip Attack (BFA) targeting large language models (LLMs) to degrade task performance while maintaining output naturalness. Unlike prior BFAs that cause incoherent outputs (e.g., GenBFA) or target specific functionalities (e.g., PrisonBreak), SilentStriker employs a dual loss function: Key Tokens Loss suppresses critical semantic tokens to reduce accuracy, while Perplexity Loss preserves fluency by minimizing perplexity. Using an iterative, gradient-based Progressive Bit Search strategy, it identifies vulnerable layers and optimally flips bits (e.g., 50 bits for INT8-quantized models), achieving significant accuracy drops (e.g., from 65.7% to 7.6% on GSM8K for LLaMA-3.1-8B) with minimal naturalness degradation (GPT score drop from 66.0 to 61.1). Experiments on five LLMs show it outperforms baselines in stealth and efficacy, highlighting vulnerabilities of quantized models on edge devices and urging robust defenses against hardware-level attacks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.**\n\nThe paper features a well-structured narrative with clear theoretical derivations and detailed experimental descriptions, supported by intuitive illustrations and reproducible setup details, enhancing accessibility for researchers and practitioners.\n\nIntroduces the first stealthy Bit-Flip Attack (BFA) for LLMs, achieving significant performance degradation while preserving output naturalness, outperforming prior methods like GenBFA that produce incoherent outputs.\n\nExtensive tests on five popular LLMs across INT8/FP4 quantizations demonstrate consistent effectiveness, validating its generality and practical threat to edge-deployed models.\n\nInnovative Dual Loss Function Design combined with an Efficient Attack Strategy ensures a balance between performance degradation and output fluency.\n\n**Weaknesses.**\n\n**Limited Contribution Scope.** While the methodology is technically sound, the paper’s justification for prioritizing output naturalness in Bit-Flip Attacks (BFAs) remains underdeveloped. The three stated contributions effectively converge to a single innovation: a stealthy BFA that preserves fluency. However, the work lacks empirical evidence or contextual analysis to establish the urgency or broader significance of this goal (e.g., real-world scenarios where detectable performance degradation is uniquely problematic). As a result, the contribution may be perceived as a niche refinement rather than a transformative advance, limiting its heuristic value for the broader LLM security community.\n\nThe approach heavily depends on **full white-box access** to model parameters, gradients, and architectural details—an assumption that limits practical applicability to closed-source or edge-deployed LLMs. In real-world scenarios, attackers rarely possess such privileged access, especially to proprietary models or systems with memory isolation mechanisms (e.g., ECC memory, secure enclaves). The paper’s failure to address gray-box or black-box adaptations further narrows its relevance to highly controlled environments, undermining claims about generalizable threat modeling.\n\nThe paper does not systematically evaluate the robustness of attack effectiveness, limiting insights into the generalizability of perturbed parameter locations.\n\nQUESTIONS:\nWhile the proposed SilentStriker attack is conceptually framed as a hardware attack leveraging vulnerabilities like RowHammer, the paper primarily focuses on software-level loss function design and gradient-based bit selection. Have the authors thoroughly evaluated the practical hardware feasibility of such attacks? Are there critical gaps between the theoretical hardware assumptions and real-world deployment scenarios that could limit the attack’s effectiveness?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **SilentStriker**, a novel **bit-flip attack (BFA)** targeting Large Language Models (LLMs), designed to degrade model performance while maintaining the naturalness of output. Unlike previous BFA techniques that often lead to incoherent or nonsensical outputs, SilentStriker succeeds in causing significant performance degradation without compromising output fluency. The paper employs a progressive bit search strategy and a token-based loss function to maximize stealth and attack effectiveness, making the method effective even on quantized LLMs. The proposed attack demonstrates superior results when compared to other BFAs, maintaining natural language quality while minimizing the detection of manipulated outputs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**Novel Approach:** SilentStriker is the first BFA to combine effective performance degradation with preserved output fluency. The approach focuses on bit-flipping selected critical weights in memory, making it more stealthy than previous methods.\n\n**Practical and Effective:** The attack framework uses a progressive bit search and targets critical tokens, allowing for an efficient, stealthy attack even on large, quantized models. Experimental results show the method's ability to significantly degrade model performance with minimal increase in output perplexity.\n\n**Robust Comparative Results:** The paper provides strong empirical evidence, demonstrating that SilentStriker outperforms existing BFAs such as GenBFA in terms of maintaining output fluency while achieving a similar level of attack effectiveness, especially on quantized models.\n\n**Weaknesses**\n\n**Threat Model is not clear:** The threat model for the attack, particularly the attacker's capabilities and specific scenarios in which the method can be applied, is not discussed in detail (including the assumption on attacker's capability). Understanding this context is crucial for real-world applications.\n\n**Optimization Strategy Confusion:** The approach to bit-flipping, particularly in relation to selecting bits and applying the loss functions, is somewhat confusing. Further elaboration on how the optimization targets specific bits in different quantized models and how this contributes to the overall attack would strengthen the paper's argument.\n\nQUESTIONS:\n1. Can you elaborate on the precise threat model for SilentStriker, including the attacker’s access to the device/memory, assumptions about memory layout stability, and whether real-world attacks (e.g., via RowHammer) are feasible in the described settings?\n\n2. How general is your bit-selection strategy for FP4 across different quantization methods (e.g., GPTQ, AWQ)? Have you tested your strategy on other formats or LLMs with custom LUTs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}