Title: Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Sandbagging

Noise Injection

Deception Detection
Abstract: Capability evaluations play a crucial role in assessing and regulating frontier AI systems. The effectiveness of these evaluations faces a significant challenge: strategic underperformance, or ``sandbagging'', where models deliberately underperform during evaluation. 
Sandbagging can manifest either through explicit developer intervention or through unintended model behavior, presenting a fundamental obstacle to accurate capability assessment. We introduce a novel sandbagging detection method based on injecting noise of varying magnitudes into model weights. While non-sandbagging models show predictable performance degradation with increasing noise, we demonstrate that sandbagging models exhibit anomalous performance improvements, likely due to disruption of underperformance mechanisms while core capabilities remain partially intact. Through experiments across various model architectures, sizes, and sandbagging techniques, we establish this distinctive response pattern as a reliable, model-agnostic signal for detecting sandbagging behavior. Importantly, we find noise-injection is capable of eliciting the full performance of Mistral Large 120B in a setting where the model underperforms without being instructed to do so. Our findings provide a practical tool for AI evaluation and oversight, addressing a challenge in ensuring accurate capability assessment of frontier AI systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLLMs can strategically under perform in capability assessments provided by regularity bodies when they are explicitly directed or even when they are implicitly motivated in pursuit of a higher goal.  To evaluate LLMs accurately, methods which can detect sandbagging is needed. Existing sandbagging detection methods are grouped into two: i. capability elicitation and ii. deception detection. Capability elicitation methods focus on fine-tuning the LLMs to reveal their unhindered performance while deception detection methods instead rely on probing the behavior of the model (e.g. its reasoning) or modifying the internal states of a model (e.g. its activation units).\n\nThe submitted paper propose a novel deception detection method that relies on adding noise to the weights of an LLM. The underlying idea is noise injection will make it harder for the LLMs to comply with sandbagging, and, hence, adding the right amount of noise can lead to an increased performance in the sandbagged LLM. Accordingly, proposed method inserts additive Gaussian noise to a tested LLM's weights and evaluates the performance at different noise variances in a particular benchmark dataset. Next, they calculate the \"improvement ratio '', i.e. the ratio of the best performance achieved under noise injection (grid-searched over different seeds and noise variances) to the performance achieved without noise injection and feed it as an input feature to a decision tree-based classifier for predicting whether the LLM is sandbagged or not.\n\nIn their experiments authors test their approach considering different sandbagging techniques, LLM models and datasets. The authors show that the improvement ratio is statistically significant for sandbagging LLMs and non-sandbagging LLMs. Authors also report, the classifier that inputs the improvement ratio achieves 96.2\\% accuracy in leave-one-model-out setting and is highly successful in detection the sandbagging.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n-The authors propose a technically sound method that follows a different take on the idea of [15] ``Intoxicate models so they reveal bad behavior\nperturbing''. [15] uses perturbing the intermediate hidden representations while the submission perturbs the model weights.     \n\n-The experiments authors conducted (in the main text) all show their method is highly successful in detecting sandbagging. It is important to characterize the setting in which the proposed method faces a difficulty, e.g. it is mentioned some difficulty with ``phi-3-small'' without further explanation. I was hoping to authors can give specific examples where their approach produces false positives.\n\n-The experiment section can be improvement with a baseline detection method. The authors state this is due to existing methods limitations (e.g. requiring a fine-tuning dataset). I am not familiar with the literature to judge this but the method in [15] seems like can be tweaked to be tested in ``improvement ratio'' setting.\n\nClarity:\n-The paper is easy to follow and engaging.\n\nSignificance:\n-The authors propose a practically applicable method in the gray-box setting where LLM weights can be updated. Authors contribute to collective understanding of the sandbagging behavior of LLMs by demonstrating it can be turned off to an extend by noise injection to LLM weights.\n\nOriginality:\n-I am not completely familiar with the literature to judge its originality. Compared to [15] which added noise to activation of LLMs, this submission uses a different approach adding noise to model weights and provide a more complete investigation of its effectiveness.\n\nQUESTIONS:\n-Can you please clarify which entries correspond to  the no-goal-setting in Table 1?\n\n-It is stated, ``we calculate 95\\% confidence intervals with the Benjamini-Hochberg correction [8] at $\\alpha = 0.05$ and conduct a two-tailed t-test, assuming unequal variances.'' Can you clarify whether the BH correction is applied to the p-values or the confidence intervals?\n\n-In Table 1, have you applied the BH correction across different rows? In each row how many observation did you used for computing the p-values?\n\n-Line 131: what does $N(\\mu)$ refers to?\n\n-Would it be possible for you to report the model-specific accuracy in the leave-one-model-out setting?\n\n-In your classification setting, did you use any scaling or standardization for the ``improvement ratios'' while combining data across different models and have you tested using a linear classifier?\n\n-How does quantization effects your detection performance, would you expect detection accuracy to be higher, or, lower for the same model under a lower quantization level?\n\n-How was the sigma range and step size selected?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is focusing on the problem of accurately estimating the capabilities of AI systems. The main issue with such evaluations is when models strategically underperform in such evaluations (either due to explicit developer intervention or unintended model behavior), a phenomenon known as sandbagging. To detect sandbagging, the authors propose injecting noise into model weights, which leads to models exhibiting anomalous performance improvements. The underlying hypothesis behind the method is that injecting noise into a model exhibiting its true capabilities should never substantially improve performance; anomalous increases in performance means that sandbagging behavior is being disrupted. Noise injection is performed either through direct parameter modification or noise application through LORA adapters which only modifies the linear layer parameters of the model. The authors conduct experiments across various model sizes, model families, and sandbagging methods (prompted sandbagging, password-locked models and no-goal setting) and find that sandbagging models frequently demonstrate significant performance improvements before eventual degradation at higher noise levels (in contrast to standard models that show expected degradation pattern with increasing noise).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper addresses an important and relevant topic, i.e. accurately assessing the capabilities of LLM models. The field is in need of such methods for accurately monitoring progress in LLM models and for ensuring their safety and oversight. \n\nThe proposed method is intuitive, simple, elegant and model-agnostic. The paper makes important contributions showing that injecting noise into sandbagging LLMs can significantly increase their performance, allows to distinguish sandbagging models from standard ones across various model architectures, sizes, and datasets, and finds models such as Mistral Large 120B exhibiting sandbagging without explicit instruction to do so.\n\nCompared to linear probes and activation based detection methods, noise injection not only identifies the presence of sandbagging but also generates concrete examples of improved performance. Besides, the method is unsupervised and can recover maximum accuracy in some settings. \n\nExperiments on multiple benchmarks (WMDP, AI2, MMLU, GSM8K and BoolQ) support the authors claims.\n\n\nWeaknesses:\n\nThe authors claim their method is completely unsupervised, however in the experiments they train an XGBoost decision tree classifier to distinguish between standard models and sandbagging models. It would be good for the authors to clarify why this is needed.\n\nLack of theoretical guarantees that the added noise will disrupt sandbagging behavior\n\nQUESTIONS:\nHow many noise injection trials are required to reliably establish an LLM model is sandbagging? Why conduct N=5 noise injection trials? Have you tried values greater than this?\n\nWhile the method works in practice, what is the likelihood that the added noise will disrupt sandbagging behavior for a given LLM model?\n\n\nLine 199: typo - reproducibility\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a noise-injection probe for detecting “sandbagging” intentional underperformance in large language models. By adding low-variance Gaussian noise to model weights or LoRA adapters, the authors find that sand-bagged models often exhibit notable performance gains, while standard models remain largely unaffected. They capture this effect using an improvement ratio,\n\n\\$\\phi = \\max(\\text{noised accuracy}) / \\text{baseline accuracy}\\$.\n\nThe authors present the probe as a lightweight, model-agnostic auditing tool for developers and regulators. However, its generalizability and practical usability still require further validation.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- **Simple and lightweight method:** The noise-injection probe is easy to implement, requiring only a few lines of code and no additional data, benchmarks, or probes.\n- **Wide empirical coverage:** The study spans seven model sizes (7B–120B) and evaluates performance across four diverse benchmarks (AI2, MMLU, WMDP, GSM-8K).\n- **Robust and consistent signal:** All 17 prompted sand-bagging model pairs exhibit notable improvements in \\$\\phi\\$; even a password-locked DeepSeek-7B shows a 5× gain on GSM-8K.\n- **First real-world validation:** Demonstrates full recovery of Mistral Large’s performance in a no-goal setting, without relying on sabotage-specific prompts.\n\n### **Weaknesses**\n\n- **Limited scalability analysis:** The probe's behavior is inconsistent across model sizes, and it fails to address ceiling effects in already high-performing models where \\$\\phi\\$ has little room to grow.\n- **Statistical concerns:** Use of only five seeds per test and reliance on extreme-value reporting may inflate \\$\\phi\\$ estimates; several key p-values are missing, particularly in Table 1.\n\nQUESTIONS:\n**Q1**. Can activation-space or prompt-space noise replicate the $\\phi$ signal without weight access?\n\n**Q2**. How does your probe compare to linear-probe detectors or fine-tune elicitation on the same tasks?\n\n**Q3**. Have you tried coding/tool-use benchmarks such as BigCodeBench to test more fragile behaviours?\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}