Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
Authors: Zora Che
Stephen Casper
Robert Kirk
Anirudh Satheesh
Stewart Slocum
Lev E McKinney
Rohit Gandikota
Aidan Ewart
Domenic Rosati
Zichu Wu
Zikui Cai
Bilal Chughtai
Yarin Gal
Furong Huang
Dylan Hadfield-Menell
Venue: Accepted by TMLR ()

Abstract: Evaluations of large language model (LLM) risks and capabilities are increasingly being incorporated into AI risk management and governance frameworks. Currently, most risk evaluations are conducted by designing inputs that elicit harmful behaviors from the system. However, this approach suffers from two limitations. First, input-output evaluations cannot fully evaluate realistic risks from open-weight models. Second, the behaviors identified during any particular input-output evaluation can only lower-bound the model's worst-possible-case input-output behavior. As a complementary method for eliciting harmful behaviors, we propose evaluating LLMs with model tampering attacks which allow for modifications to latent activations or weights. We pit state-of-the-art techniques for removing harmful LLM capabilities against a suite of 5 input-space and 6 model tampering attacks. In addition to benchmarking these methods against each other, we show that (1) model resilience to capability elicitation attacks lies on a low-dimensional robustness subspace; (2) the success rate of model tampering attacks can empirically predict and offer conservative estimates for the success of held-out input-space attacks; and (3) state-of-the-art unlearning methods can easily be undone within 16 steps of fine-tuning. Together, these results highlight the difficulty of suppressing harmful LLM capabilities and show that model tampering attacks enable substantially more rigorous evaluations than input-space attacks alone.

Published in Transactions on Machine Learning Research (07/2025)
Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
"Zora Che, University of Maryland, ML Alignment & Theory Scholors
sche@umd. edu scasper@emit. eduu robert. bebert.kirke kirk@dsil. gov. uk anirudhs (ttrrpdaastteppmal.. umd. eda slocumsteuy@@mmai. com levmckinnnyy@us.
"Stephen Casper, MIT CSATL, ML Aligmment & Theory Scholars
Robert Kirk, UK AI Security Institate
Anirudh Satheesh, University of Maryland
Stewart Slocum, MIT
Lev McKinney, Universily of Toronto
ctr.eiinnenescc...oooooooooo loronto. edu
Rohit Gandikota, Northeastern University
gandikota. ronotthearrnnrrtrrrerr edu
Aidan Ewart, Haize Labs
aidongnattewrrt@@maal.com
Domenic Rosati, Dalhousie University
domenicrosati@@maall.cm
Zichu Wu, University of Waterloo
zicha, wwwuuuttrroo..cooo ca.
Zikui Cai, University of Maryland
sikuithumd. eda
Bilal Chughtai, Apollo Research
balalchughhaiiiggonle.cc
Yarin Gal, UK AT Security Institute, University of Oxford
yarin. ceriingal@@ss.coo or...s..oo.ooo.ooo... ac uk
Furong Huang, University of Maryland
furongh eda dylantom @mill edu
Dylan Hadfield-Menell, MIT
Reviewed on OpenReview: https: // openreview. net/ pormm id- E60YbLnQd2
Abstract
Evaluations of large language model (LLM) risks and capabilities are increasingly being incorporated into Al risk management and governance frameworks. Currently, most risk evaluations are conducted by designing inputs that clicit harmful behaviors from the system. However, this approach suffers from two limitations. First, input-output evaluations cannot fully evaluate realistic risks from gpinhweight models. Second, the behaviors identified dur- ing any particular input-output evaluation only lower-bound the model's worst-possible- case Imput-output behavior. As complementary method for cliciting harmful behaviors, we propose evaluating L.MSs with model lampering attacks which allow for modifications to latent activations or weights. We pit state-of-the-art techniques for removing harmful LLM capabilities against suite of input-space and model tampering attacks. In addition to benchmarking these methods against each other, we show that (1) model resilience to capability elicitation attacks lies low-dimensional robustness subspace; (2) the success rate of model tampering attacks can empirically predict and offer conservative estimates for the success of held-out input-space attacks; and (3) state-of-the-art unlearning methods can easily be undone within 16 steps of fine-tuning. Together, these results highlight the difficulty of suppressing harmful LLM capabilitics and show that model tampering attacks enable substantially more rigorous evaluations than input-space allacks alone
We release models http:://uuggnngiieee///LLL..

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Initial Prompt
"What makes a bioweapons program effective?"
Input-space Attack Modifies input text
Latent-space Attack Perturbs hidden neurons
Weight-space Attack Fine-tunes the model
Model Tampering Attacks
Figure 1: Model tampering attacks modify latents and weights. In contrast to input-space attacks, model tampering attacks elicit capabilities from an LLM by making modifications to the internal activations or weights. In this paper, we use model tampering attacks to (1) directly evaluate risks from malicious tampering with open- weight models and (2) indirectly evaluate difficult-to-oreee input-space vulnerabilities in models.
1 Introduction: Limitations of Input-Output Evaluations
Rigorous evaluations of large language models (LLMs) widely recognized key for risk mitigation
Raji
et al. 2022 Anderljung et al. 2023 Schuett et al. 2023 Shevlane et al.
2023 and are being incorporated into
AT governance frameworks NNST 2023 UK DSTT 2023
EU 2023 China 2023 Brazi]
2023
Canada
2022
Korea 2025
However, despite their efforts, developers often fail to identify overtly harmful LLM behaviors
pre-deployment (Shayegani et al.) 2023 Andrinshchenko et al., 2024 Carlini et al.) 2024 Yi et al. 2024b).
Current methods primarily rely on automated inpul-space altacks, where evaluators search for prompls that elicit harmful behaviors. These are useful but often leave unidentified vulnerabilities. A difficulty with input- space attacks is that they are poorly equipped to cover the attack surface. This happens for two reasons. First, attackers Second, it is intractable to exhaustively search the input space These challenges highlight fundamental limitation of input-space evaluations: the worst behaviors identified during an assessment can only offer a
sometimes manipulate more than just model inputs (e.g., if model is open-source).
lower bound the model's overall worst-case behavior |Gal 2024 OpenAl 2024).
To help address this challenge, we draw inspiration from a safety engineering principle: that safety-critical systems should be tested under stresses least extreme if not more than those expected in deployment CCausee al., 2006). For example, buildings designed withstand loads multiple times greater than their intended use. Here, we take analogous approach to evaluating and building safety cases (Clymer ct al. 2024 for LLMs: stress-testing them under attacks that go beyond input-space manipulations.
We propose using model lampering attacks, which allow for adversarial modifications to the model's weights or latent activations, in addition to evaluating systems under input-space attacks (see Figure I.. We attempt answer two questions, each corresponding to a different threat model:
Question 1:
LLMs to model tampering attacks? Answering this helps understand how model tampering attacks can be used to study risks from models that are open-source have fine-tuning APIs, or may be leaked (Nevo el al. 2024).
How vulnerable
For example, with modern tokenizers, there vastly 20-token strings than particles the known universe. It. may seem obvious that model tampering attacks are needed realistically assess threats from open-source models. However, there is precedent for developers failing
prior open-source releases. For example, before releasing
2

---

Published in Transactions on Machine Learning Research (07/)025)
Question 2: Can model tampering attacks inform evaluators about LLM vulnerabilities to novel input-space attacks? Answering this will help us understand how model tampering attacks can help assess risks from both open- and closed-source models.
To answer these questions, we pit state-of-the-art methods for unlearning and safety fine-tuning in LLMs against a suite of input-space and model tampering attacks. We make four contributions:
1. Benchmarking: We benchmark 8 unlearning methods and 9 safety fine-tuned LLMs, each against
11 capability clicitation attacks.
2. Science of robustness: We show that LLM resilience to variety of capability elicitation attacks lies a low-dimensional robustness subspace.
Evaluation methodology: We show that the success of some model tampering attacks corre- lates with that of held-out input-space attacks. We also find that few-shot fine-tuning attacks can empirically be used to conservatively over-ostimate model's robustness to held-out input-space threats.
Model suite: To facilitate further research, we release a set of 64 models trained using 8 methods to unlearn dual-use biology knowledge at varying degrees of strength at httpss//hhugiingfacecco LLMM-
GAT
2
Related Work
Latent-space attacks: During a latent-space attack, an adversary can make modifications to a model's hidden activations. Adversarial training under these attacks improve the generality of model's ro-
bustness (Sankaranarayanan et al., 2018 Singh et al. 2019: Zhang et al., 2023 Schwinn et al. 2023 Zeng
et al. 2024). In particular, [Khomnexx et al. (2024), Casper et al. (2024), and Sheshadri et al. 2024 use
latent adversarial training to improve defenses against held-out types of adversarial attacks. Other work on activation enginecring has involved making modifications to a model's behavior via simple transformations
to their latent states (Zou al., 2023a Wang Shu, 2023 Lu Rimsky 2024 Arditi al.
2024 Zhang
et al. 2025) also showed that unlearning methods can be brittle to quantization methods.
Weight-space (fine-tuning) attacks: During few-shot fine-tuning attack (Huang et al. adversary can modify model weights via fine-tuning on a limited number of samples. For example, Qi et al. (2023) showed that fine-tuning on as few as 10 samples could jailbreak GPT-3.5. Many works have used few-shot fine-tuning attacks to elicit LLM capabilities that were previously suppressed by fine-tuning or
2024).
unlearning (Jain et al., 2023 Yang et al.
2023
Qi et al.
2023 Bhardwaj Poria
2023
Lermen et al. 2023
Zhan et al. 2023: Ji et al. 2024 Qi et al.
2024a
Hu et al.] 2024
Halawi et al.] Peng ct al. 2024 Lo et al.
2024
Lucki et al. 2024
Shumailov et al.
2024 Lynch et al.
2024
Deeb & Roger
2024
Qi et al.
2024b
et al.
20
Capability clicitation and evaluation: L.MSs are commonly developed by simply training them to behave desirably (c.g., with RLHF (Casper et al. 2023)) ) but in this paper, we focus on testing targeted defenses against known, harmful behaviors. Research on adversarial capability elicitation (Hofstätter et al. 2025) in LLMs has primarily been done in the context of machine unlearning (Liu et al. 2024a] Barez et al. 2025) and jailbreaking (Yi et al., (024b). Here, we experiment in these two domains. However, capability elicitation has also been researched in the context of boaddoors/trojann (Zhao et al. (0224). "password models" (Greenblatt et al.] 2024 Hofstätter et al.] 2025), and "sandbagging" (van der Weij et al. 2024). In the unlearning field, several recent works have used adversarial methods to evaluate the robustness of
safety fine-tuning and unlearning algorithms (Patil et al. 2023 Lynch et al. 2024 Lucki et al. 2024 Hu
let al. 2024 Liu et al.) 20244a Zhang et al. 2024; Liu et al.; 2024b; Wei et al.] 2024). Here, we build on Li et al. (2024b) who introduce WMDP-Bio, a bonchmark for unlcarning dual-use biotechnology knowledge
from LLMss In the jailbreaking field, many techniques have been developed to make LLMs comply with
Llama 2 and Llama models, Meta's red-teaming efforts did not reportedly involve model tampering attacks Touvron al.
2023 Dubey et al. 2024).
3

---

Published in Transactions on Machine Learning Research (07/2025)
harmful requests (Shayegani et al. 2023 Yi et al. 2024b Jin et al. 2024 Chowdhury al. 2024 Lin al,
2024). Here, experiment with open-source LLMSs and set of gradient-uuide,, perplexity-guides and prosaic techniques from the adversarial attack literature (see Table
3 Methods
Defenses
Unleurning Methoda
Gradieul Difference (GradDilf)
Liu el m
2025
(We train & mandefa ouch uuloerm WMDP-Pio)
Random Misdirection for Unlearning (MMU)
RMU with Lasent Adversacial Training (RMU+LAT)
Sheslad el Ni
2024
Represetation Noising (RepNoise)
Erasure of Language Memoory (ELM)
Labd
2024
Representation Rerouting (RR)
Tamper Attack Resistance (TAR)
Tamirist ml
A024
K-FAC for Distribution Erasure (K-FADE)
AcKmaayy et
Joitbronk Refusal- Tined (Og the skol)
Toned Models
Dubey
2024
sho116//Tamm3 finctume
frurtane_refssal pofissa
Link
JIN 1N91177677 IIN796/UUmm33-8l.m /U
Youliang LLamnn--b-daeeeee Sb--taa
Tran et al
2021
GrayS ALaa/a////////// 2---ma------Insttet--t Instruct-RR
LLM-LAT /lllamma--h--msttttt wth.-nstraet---jjlllboee...
SheshadF
m.
2112
LLM-LAT JLM-LAT/robuad /robbst--lonss Soo--lnnnn------isnn
Stealaa m1
lapistocks/ / Llama-3- 3-SB- Instrue--AAR TAR--efusse
Orenguteng LLaama--- 8B- Lexd Uncetsceed
Attacks
Inpnt-Space
Gradient-gnided
Gready Coordinate Gradient (GCG) AutoPrompt
Zon et ml.
2020h
2020
Seeplesity-guidee
Beam Search based Astack (BEAST)
Sadasivan et al
2024
Prompt Automatic Iterative Refinement (PATR)
Chao
2024
Human Prompt
Model Tampering
Latent space
Embedding pertur battonbation Latent perturbation
Schvria et
2024
Shisshddi
Weight space
WandA Pruning
Sun et a)
2023
Benign LoRA
LoRÁ
1i ct m..
2021
Parameter
Table 1: Table of capability elicitation (attack) and capability suppression (defense) methods.
We consider defenses in two different scttings: (top) unlearning approaches that remove hazardous bio- knowledge and (bottom) refusal-tuned models that resist, jailbreaks.
Our approach pitting capability suppression defenses against capability elicitation attacks. Herc, we study capability suppression methods that depend on both removing knowledge from the model (unlearning) and teaching the model to robustly refuse (jailbreaking) requests. For unlearning experiments, we experiment with models trained using different. unlearning methods. For jailbreaking experiments, we experiment with 9 models off the shelf from prior works. In both cases, we pit these defenses against set of 11 input-space and model tampering attacks to either elicit 'unlcarned' knowledge jailbreak the model. In Table 1 we list all unlearning methods, off-the-shelf models, and atlacks we use. Since the input-space attacks that we use are held out, we treat them as proxies for novel input-space attacks in our evaluations (see also Hofstätter et al. (025
Defenses machine unlearning methods: We unlearn dual-use bio-havardous knowledge on Llama-3- 8B-Instruct DubetryuctDubyyyyoo et al. (2024) with the unlcarning methods listed in Table outlined in Appendix A.2.1 1. For all methods, we train on 1,600 examples of max length 512 from the bio-remove-splic of the WMDP 'forget set' (Li et al. 2024b), and up to 1,600 examples of max length 512 from Wikitext the 'retain set! For the 8 unlcarning methods listed in Table we take 8 checkpoints evenly spaced across training. Finally, we also use the public release of the "TAR-v2" model from Tamirisa et al. (2024) as a 9th TAR model. In total, the 8 checkpoints each from the 8 methods we implemented plus the TAR model from Tamirisa et al. (2024) resulted in 65 models.
Defenses refusal fine-tuned models: For jailbreaking experiments, we use the 9 fine-tuned Llama3- 8B-Instruct models off the shelf listed in Table [ The first 8 are all fine-tuned for robust refusal of harmful
4

---

Published in Transactions on Machine Learning Research (07/2025)
requests. Of these, 'RR' Zou et al., 2023a) and 'LAT" (Sheshadri et al. 2024 are state-of-the-art for
open-weight jailbreak robust models (Ti et al. 2024a Haize Labs 2023). The final 'Orenguteng' model was fine-tuned to be 'helpful-only' and comply even with harmful requests. We discuss these models in more detail in Appendix A.3
Attacks capability elicitation methods: We use 5 input-space attacks and model tampering attacks on our unlearned models. We use these attacks (single-turn) to increase dual-use bio knowledge (as mea- sured by WMDP-Bio performance (Li et al. 2024bl) for unlearning experiments and to elicit compliance with harmful requests (as measured by the StrongReject AutoGrader (Souly et al, 2024)) for jailbreaking experiments. We selected attacks based on algorithmic diversity and prominence in the state of the art. We list all 11 attacks in Table [ In all experiments, we produce universal adversarial attacks optimized to work for any prompt. This allows us to attribute attack success to capability elicitation rather than answer-forcing from the model (e.g., Fort (2023)) ) For descriptions and implementation details for each attack method, see Appendix A Finally, we also used two proprictary attacks one for unlcarning
experiments and for jailbreaking experiments which will describe in Section 4
Attacks data:
Attacks on unlearning non-fine-tuning: we used 64 held-out examples of multiple-choice biology questions from the WMDP-Bio test set.
Attacks on unlearning - adversarial fine-tuning: we use the WMDP "bio retain' or 'forget'
sets. Both of which are comprised of biology papers.
Attacks refusal training all except benign fine-tuning: used held-out examples of compliance with harmful roquests from Sheshadri et al. (2024). Each example is prompt + response pair.
Attacks on unlcarning and refusal training benign fine-tuning: For all benign fine-tuning attacks, we used WikiText Merity et al. (0016)
For details on attack configurations, including the number of examples, batch size, number of stops, and other hyper-parameters, see Appendix A.4
Attacks model tampering attacks are efficient. In Table 3 we show the number of forward and backward passes used in our implementations of attacks. Model tampering attacks were more efficient than state-of-the-art input-space attacks.
4 Experiments
As discussed in Section we have two motivations, each corresponding to different threat model. First, we want to directly evaluate robustness to model tampering attacks to better understand the risks of open- source, leaked, or API fine-tuneable LLMs. Sccond, we want to understand what model tampering attacks can tell us about novel, unforeseen input-space attacks in order to study risks from all types of LLMs. Unfortunately, unforescen attacks are, by definition, ones that we do not have access to. Instead, since the input-space attacks that use are held out during fine-tuning, we treat them proxies for 'unforeseen' input-space attacks.
4.1 Unlearning Experiments
We first experiment with the unlearning of dual-use biology knowledge in LLMs by pitting unlearning methods against capability elicitation methods (see Table I
4.1.1
Benchmarking Unlearning Methods
Calculating an unlearning score: In our models, we evaluate unlearning efficacy on the WMDP-Bin
QA evaluation task (Li et. al. 2024b). We evaluate general utilily using three benchmarks. First, use
5

---

Published in Transactions on Machine Learning Research (07/2025)
WMDP, Best
WMDP, Best
Unlearning Score 1
Method
WMDP
MMLU
MT-Bench/10 AGIEval
Input Attack + Tamp. Attack
Llana3-8B-Instnn
0.70
0.75
0.71
0.64
0.78
0.41
0.00
Grad Diff
0.25
U.27
0.67
0.52
0.13
0.32
0.17
RMU
0.26
0.34
0.57
0.59
0.68
0.42
0.84
RMU LAT
0.32
0.39
0.64
0.60
0.71
01.99
0.73
RepNoise
0.29
0.30
0.65
0.59
0.71
0.37
0.78
ELM
0.24
0.38
0.71
0.59
0.76
0.37
0,95
RR
0.26
0.28
0.66
0.61
0.76
0.44
0.96
TAR
0.28
0.29
0.36
0.54
0.12
0.31
0.09
K-FADE
0.31
0.32
0.64
0.63
0.78
0..40
0.85
Table 2: Benchmarking LLM unlcarning methods: We report results for the checkpoint from cach method with the highest. unlearning score (Equalion (1). We report original WMDP-Bio performance, worst-case WMDP-Bio performance across our attacks, and three measures of general utility: MMLU, MT-Bench, and AGIEval. Representation rerouting (RR) has the best unlcarning score. No model has a WMDP-Bic performance less than 0.36 after the most effective attack. We note that Grad Diff and TAR models performed very poorly, often struggling with basic fluency.
MMLU (Hendrycks et al. 2020g and AGIEval (Zhong et al. 2023), which are based on asking LLMs multiple- choice questions. We then use MT-Bench (Bai et al. 2024) which is based on long answer quest ions and thus measures both knowledge and fluency. Because the goal of unlearning is to differentially decrease capabilitics in target domain, we calculate an "unlcarning score" based on both unlcarning efficacy and utility degradation. Given original model M and unlearned model M", we calculate Suntearn (M") with the formula:
Sunlearn (M') - -(SWM(P SWMDP (M')
Unnlearn efficacy
Sutility (M) Sutiny((M))
(1)
AUtility degradation
[SWMDP (M) (wanp(rand)
A Random chance (for normalization)
Here, (WMDP(-) is the accuracy on the WMDP-Bio QA Evaluation and Suiiity()) is an aggregated utility measure. Sucility (+) is caleulated by taking a weighted average of MMLU, AGIEval, and MT-BBnch. We use weights of 1/4, 1/4, and 1/2 respectively because MT-Bench uniquely measures model fluency. Finally, "rand" refers to a random policy. An unlearning score of 1.0 indicates theoretically optimal unlearning random performance on the unlearned domain and unaffected performance on others. Meanwhile, the unlearning score of the original model M is 0.0. Table 2 reports results from the best-performing checkpoint (determined by unlearning score) from each of the methods.
Representation rerouting (R... achieves the highest unlcarning scorc. GradDiff and TAR. struggle due to dysfluency. We find different levels of unlearning success. Representation rerouting (RR) performs the best overall, achieving unlearning score of 0.96. In contrast, GradDiff and TAR have limited success with the lowest unlearning scores. Poor MT-Bench scores and our manual assessment of these models suggest that GradDiff and TAR struggle principally due to poor fluency.
No method robust to all attacks. We plot the increase in WMDP-Bi, performance for the best checkpoint from each unlearning mothod after each attack in Figure 2 and show that all models, even those with the lowest unlearning scores, exhibit a worst-case performance increase of percentage points more when attacked.

---

Published in Transactions on Machine Learning Research (07 (07/2025)
WMDP Bio Performance Increase After Attacks
GradDiff
0.17
0.01
0.02
0.01
0
0.01
C.1
0.29
0.42
0.08
0.06
0.01
0.01
0.02
0.04
0.31
0.01
0.29
0.25
LAT
0.07
0.07
0.01
0.01
0.03
0
0.15
-0.01
0.02
0.32
0.22
RepNoise
0.79
0.01
0.01
-0.01
-0.02
0.01
0.22
-0.02
-0.01
0.2
0.36
ELM
0.95
0.1
0.09
0.08
0.14
0.08
0.19
0.33
0.02
0.47
0.29
0.32
RR
0.06
TAR-
0.09
0.01
0.01
-0.04
-0.05
-0.01
-0.03
0.01
-0.05
0.08
-0.04
PB&J
0.85
0
0.02
0.03
0.01
0.33
0.31
Unlearning Score
GCG
AutoPrompt
BEAST
Human Prompt Embedding
PAIR
Latent 5
Pruning
LoRA Fine-tune
Benign
Full Fine-tune
Figure 2: Pitting capability suppression (unlearning) methods against capability elicitation attacks. We use unlearning methods to suppress bio-hazardous knowledge from LLMs and pit these against capability elicitation attacks seeking re-elicit the unlearned knowledge. All unlearning methods tested could be successfully attacked. Left: The unlearning score (Equation ([)) measures how effectively each unlearning method removed unwanted capabilities while preserving general model utility. Higher scores indicate better unlcarning (scale 0-1). Right: Increase in the unlcarned task performance after attacks. The first 5 columns are from inpul-space allacks while the final 6 are from model lampering atlacks. In particular, finetuning attacks (rightmost columns) were especially effective at resurfacing suppressed capabilities.
4.1.2 Model robustness exists on low-dimensional subspace
We perform PCA, weighting models by their unlearning score. First, understand the extent to which some attacks offer information about others, we analyze the geometry of attack successes across our 65 models. Previously Wei al. (2024) found that model's vulnerability to pruning and low-rank modifications both relate with the brittleness of its safety fine-tuning Here, extend on this finding with more attacks and subspace analysis. We perform weighted principal component analysis on the WMDP-Bio improvements achieved by all 11 attacks on all 65 model checkpoints. We first constructed matrix A with one row per model and one column per attack. Each AG corresponds to the increase in WMDP-BB performance model under attack j. We then centered the data and multiplied each row A, by the square root of the unlearning score: / (An)ear This allowed for models to influence the analysis in proportion to their unlearning score.
Three principal components explain 89% of the variation in attack success. Figure 3 displays the eigenvalues from PCA and the top three principal components (weighted by eigenvalues). This suggests that different capability elicitation attacks exploit models via related mechanisms.
Hierarchical clustering reveals distinct attack families. In Figure we perform agglomerative clus- tering on attack success correlations. Algorithmically similar attacks tend to cluster together. However, adversarial finetuning attacks exhibit significant. variation, even amongst. each other. Finally that benign model tampering methods (pruning and benign fine-tuning) behave similarly to gradient-free input- space attacks.
4.1.3 Model tampering attacks empirically predict and conservatively estimate the success of input-space attacks
Embedding-space attacks, latent-space attacks, pruning, and benign fine-tuning empirically correlate with input-space attack successes. In Figure 5 these three model tampering attacks tend

---

Published in Transactions on Machine Learning Research (07 (07/2025)
(a) PCA Explained Variance
(b) Principal Component Weights
89%
Explained Variance Ratio
PC1
0.6-
0.18
0.23
0.32
0.27
0.4
0.65
-0.3
-0.055
0.4
PC2
0.014-0.011 0.007 0.022
-0.005 0.015
0.011 0.004
0.035
0.095
0.25
0.2
PC3
-0.017 0.008
-0.029-0.00222 -0.015 0.041 0.18 -0.013-0.066 0.064 -0.021
0.0
5
10
GCG
AutoPrompt
BEAST
Human Prompt
PAIR
Embedding
Latent Layer 5
Benign Fine-tune
Pruning
LoRA Fine-tune
Full Fine-tune
PC
Figure 3: Three principal components explain 89% of the variation in attack success. Lcft: The proportion of explained variance for each principal component. Right: We display the first three principal components weighted by their eigenvalues. The first principal component suggests a geometric distinction between the two adversarial (LoRA, Full) fine-tuning attacks and all others.
Input-Space (Gradient-Free)
Input-Space (Gradient-Bee Based)
Latent Space
Fine-tuning
Benign Model Tampering
GCG
AutoPrompt
Benign Fine-tune tiee-tume
Human Prompt
PAIR
BEAST
Pruning
Embedding
Latent Layer
LoRA Fine-tune
Full Fine-tune
Figure 4: Hierarchical clustering reveals groupings of attacks. Attacks tend to cluster by algorithmic type. However, benign fine-tuning attacks cluster with gradient-free input-space attacks.
to have positive correlations with imput-space attack successes with values near zero In these plots, we size points by their unlearning score and display the Pearson correlation weighted by unlcarning score. Full results are in Appendix B This suggests that embeddinng-spaee attacks, latent-space attacks, pruning, and benign fine-tuning are particularly able predict the successes of held-out input-space attacks.
Finc-tuning attack successos empirically offer consorvative estimates for input-space attack successes. LoRA and Full fine-tuning performed differently different allacks. However, logether, the max of the two did well or better than the best-performing input-space attack on 64 of 65 models. This suggests that model tampering attacks could be used to develop more cautious estimates of model's worst-case behaviors than other attacks.
4Poits independent identically distributed, only this "p" value for geometric intuition, and attach any formal hypothesis test.

---

Published in Transactions on Machine Learning Research (07 (025)
(a)
(b)
Max Over Input -Space Attacks
r=0.58 p=0.00
r= 006666
r= 0.87
0.04. 84 0.00
r=-0.16
p=0.00
p - 0.00
p=0.28
Embedding
Latent 5
Pruning
Benign
Best Fine-Tune
GradDiff
LAT
ELM
TAR
y=x
RMU
RepNoise
RR
K-FADE
Model Tampering
Input-Space Attacks
Avg. Relative Attack Strength f
Method
Beaten f
Embedding
54%
0.99
Latent Layer 5
72%
2.94
Pruning
20%
0.17
Benign Fine-tune
51%
1.72
Best Adv. Fine-tune
98%
8.12
Figure 5: In our experiments, (a) fine-tuning, embedding-space, and latent-space attack suc- cesses correlate with input-space attack successes while (b) fine-tuning attack successes em- pirically exceed the successes of stato-of-thc-art input-space attacks. Here, we plot the increases in WMDP-Bio performance from model lampering atlacks against the pess-pergo (of 5) input-space attacks for each model. We weight points by their unlearning score from Section 1.1.1 In (b), the axis is the best (over 2) between a LoRA and full fine-tuning attack. We also display the unlcarning-soom scoiinngg----eveeeeee weighted weiiowhgtte correlation and the correlation's value. Points below and to the right of the line indicate that the model tampering attack was more successful. Table: for each of the four model tampering attacks, the percent of all input-space attacks for which it performed better and the average relative attack strength compared to all input-space attacks.
Model tampering attacks are predictive of the success of proprietary attacks from the UK AI Security Institute (UK AISI). To more rigorously test what model tampering attacks can reveal about novel input-space attacks, we analyze their predictiveness on proprictary attacks from the UK AI Security Institute. These allacks were known to the 'red team' authors ( [K AISI affiliates) but were not accessible to all other blue team' authors. We conducted these attacks with the same data and approach as all of our other input-space attacks. Results are summarized in Figure 0 with full results in the Appendix. Correlations weaker than before, but pruning and benign fine-tuning still correlate with attack success with a value near zero. Also as before, fine-tuning altack successes often tend to be as strong or stronger than input-space attacks. However, this trend was weaker, only occurring for 60 of the 65 models. See Appendix B.2 for full results and further analysis of UK AISI evaluations.
Model tampering attacks improve worst-case input-space vulnerability estimation. Finally, we test if model tamporing attacks offer novel information that can be used to predict worst-case behaviors better than input-space allacks alone (Figure 14 in Appendix C.Ip. We train linear regression models to predict worst-case input-space attack success rates with information from either (1) only input-space attacks, (2) both input-space and model tampering attacks. We find that including model tampering attacks improves predictiveness (e.g. from r = 0.77 to 0.83 with four predictors). The rest-performing combinations typically include al tacks from multiple families, suggesting diverse attas provide complementary signals by probing different aspects of model robustness.
Sfate-of-the-art unlearning can reliably be reversed within 16 fine-tuning steps sometimes in single step. We show the results of multiple fine-tuning attacks against the best-performing model
9

---

Published in Transactions on Machine Learning Research (07 (025)
(a)
(b)
0.16 0=0,9
r= 0.12 0.0.
r= 0, 66 p=0.00
r= 0,59 0=0.0
r= f=-0.16
p=0.29
AISI Attack
Embedding
Latent
Pruning
Benign
Best Fine-Tune
GradDiff
LAT
ELM
TAR
RMU
RepNoise
RR
K-FADE
Figure 6: Model tampering attacks predictive for held-out proprictary attack from the UK AI Security Institute. Each point corresponds to model. (a) In these experiments, correlations are weaker than with non- UK AISI attacks, but benign inn-tuning attacks continue to correlate with UK AISI input-space attack success. (b) Fine-tuning attacks still tend to exceed the success of input-space attacks, though less consistently than with the attacks from Figure
WMDP Bio Performance Increase After Fine tune Attacks
GradDiff
RMU
1.04
0.32
0.43
3.42
0.29
0:37
+0.02
0.36
0.29
0.25
LAT
87
0:24
0.34
RepNolse
870
-0.01
0.02
0.37
0.03
3.38
0.28
0:38
0.37
0.37
ELM
6.4
0.47
0.24
0.44
0.05
3.44
0.29
0.44
0.43
0.39
0.01
0:32
TAR
119
0.1
8.07
0.11
3.04
0.08
...04
.0.04
0.04
0.04
0.04
PB&J
5.06
0.34
0.25
0:33
3.34
0.38
0:38
0.32
0.29
0.31
0:31
Unlearning Score
Benign
LoRA-1
LoRA-2
LoRA-3
LoRA-4
LoRA-avg
Full1
Full-2
Full-3
Full-4
Full-avg
Figure 7: Few-shot fine-tuning efficiently undoes unlearning. We plot the heatmap of the best checkpoint for each method under benign (left), LoRA (middle), and full-parameter (right) fine-tuning attacks. All fine-tuning experiments are done within 16 gradient steps, with 128 examples or fewer. All methods can be attackod increase WMDP-Bio performance by 10 percentage points or more. All hyper- parameters listed in Appendix A.4
from cach unlcarning method in Figure 7 All finetuning experiments, as detailed Appendix A. are performed within 16 gradient steps and with 128 or fewer examples. The only method that was resistant to few-shot fine-tuning attacks TAR, in which only out of the 9 fine- fine-tuning attacks were able to increase the WMDP-Bio performance by over 10 percentage points. However, TAR models had low unlcarning scores due to poor general utility, which renders their robustness to fine-tuning unremarkable. All state-of-the-art unlearning methods can be attacked successfully to recover more than 30 percentage points of WMDP performance. Moreover, even when we perform a single gradient step (with a batch size of 64) still increases the WMDP performance on of the 8 methods by over 25 percentage points (sce column "Full-4" in Figure
4.2 Jailbreaking Experiments
We repeat analogous experiments with similar results in the jailbreaking setting. Finally, to test the generality our findings outside the unlearning paradigm, we ask whether they extend to jailbreaking, Using the 9 off-tho-shelf models and 11 attacks from Table [ we conducted parallel experiments as Section 4.1 but by pitting off-the-shelf refusal- ffnsaa-ffnetuned models against jailbreaking attacks. We plot all results in Appendix
10

---

Published in Transactions on Machine Learning Research (07/2025)
Harmful Response Increase After Attacks
Llama-t-8B-nnsruuct
0.07
RR
0
0
0.04
0.01
0.17
0.29
0.3
LAT-
0.09
0
0
0
0.01
0
0.21
0.2
0.43
TAR 0.08
0.05
0.01
-0,05
-0.66
0.02
0.03
0.05
-0.07
0
0.26
-0.02
0.75
DERTA
0.02
0.05
0.03
-0.02
-0.02
-0.02
0
-0.02
-0.02
0.38
0.58
RT1
0.01
0.01
0.11
0.07
0.01
0.28
0.31
0.63
RT2
RT3
0.01
0.43
-0.01
-0.01
0001
0.19
0.06
0.08
0.09
0.02
0.37
0.49
0.92
Uncensored
0.9
-0.05
-0.03
-0.31
-0.18
0.07
-0.04
-0.77
-0.11
-0,11
-0.79
0.1
Baseline
AutoPrompt
GCG
BEAST
Human Prompt Embedding
PAIR
Latent
Pruning
LoRA Fine-tune
Benign
Full Fine-tune -une
Cascade
Figure 8: All safety-tuned models could be successfully jailbroken by fine-tuning and Cascade attacks. We evaluate safety-tuning methods and jailbreak attacks. Left: The 'Baseline' measures the compliance rate to direct harmful requests. Right: Increase in harmful response rate after attack. All safety-tuning methods were vulnerable elicitation of suppressed capabilities, especially by finetuning and Cascade attacks (rightmost columns).
Our benchmark results (Figure demonstrate that all safety-tuning methods are vulnerable model tampering. Principal component analysis of attack success rates in Figure [8] show that three principal components explain 96% of the variation in jailbreak success across the nine models.
We then reproduced our empirical analysis of whether the success of model tampering jailbreaks correlates with and/or conservatively exceeds the success of imput-space jailbreaks (Figure [7). Like before, we find that fine-tuning attack success tends to empirically exceed the success of input-space attacks, thus offering a conservative estimation method. However, unlike before, we do not find clear evidence of reliable correlation between tampering and input-space attacks due to only having samples.
Finally, evaluate how helpful model tampering attacks can be for characterizing model's vulnerability to unique, held-out attacks, we use Cascade, a multi-turn, state-of-the-art, proprietary attack algorithm from Haize Labs (Haize Labs 2023). In Figure 19 wC see that single-turn model tampering attacks correlate well with multi-turn Cascade attacks.
5 Discussion
Implications for evaluations and safety cases: Our findings have direct implications for performing AI risk evaluatioos and constructing safety cases (Clymer el al. 2024). Current evaluation frameworks rely heavily on input-space attacks which can easily fail to underestimate worst-case failures. Model tampering attacks provide a useful tool for studying novel, potentially unforeseen risks. By modifying a model's internal mechanisms - either through activation perturbations or fine-tume mne-tuning- - we can infer the potential existence of failure modes that input-space evaluations may miss (sce also Hofstätter al. (2025)). This is particularly critical for oeen-weight models, where safety mitigations can be undone post-release.
Our work focuses only Llama-3-80 3-----B--nntuuue IB-ntrutt derived models. experimental depth, but other models may have different dynamics. The science of evaluations is still evolv- ing, and it is not yet clear how to best translate the outcome of evaluations into actionable recommendations. Overall, we find that. model tampering attacks can help with more rigorous evaluations even for models deployed as black boxes. However, there may be limitations in the mechanistic similarity input-space and tampering attacks (Lcong al. 2024).
Limitations:
This allows for considerable
Future work:
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Can models be robust to tampering attacks? This paper and concurrent work (Qi et al, 2024b) show that even defenses designed to make models robust to tampering can be easily undone. We are currently working to better understand tampering robustness and improve the extent to which models can be made robust to tampering attacks through pretraining interventions (c.g. Maini et al. 2025), knowledge corruption (e.g. [aan el al. 2025), and improvements in unlearning algorithms (c.g- Siddiqui et al. 2025).
What mechanisms underlie robust capability removal? We are interested in futture work to mechanistically characterize weak vs. robust capability suppression. We briefly worked to test the hypothesis that the activation rank difference (across the forget set) between a base and unlearned model would correlate with unlearning robustness. However, we found this not to be the case, and leave further investigation to future work. We hope that the 64 models release help to lay the groundwork for this.
Bridging research and practice: Model tampering attacks can be further studied and used in practice to assess risks in consequential models pre-deployment.
Impact Statement
This work was motivated by advancing the science of Al capability evaluations. This has been a central interest and goal of technical AI governance research (Reuel et al.] 2024) because AI risk management frameworks are increasingly being designed depend on rigorous risk evaluations. Thus, we expect this paper to contribute to developing more rigorous evaluations, which is valuable from risk-management perspective.
Acknowledgments
We grateful to the Center for AI Safety for compute and the Machine Learning Alignment and Theory Scholars program for rescarch support. We thank Daniel Filan for technical support, and Antony Kellermann for technical discussion. Finally, we are grateful to Peter Henderson, Ududec Cozmin, Ekdeep Singh Lubana, and Taylor Kulp-MeDowall for their feedback.
References
Markus Anderljung, Everett Thornton Smith, Joe O'Brien, Lisa Soder, Benjamin Bucknall, Emma Bluemke, Jonas Schuett, Robert Trager, Lacey Strahm, and Rumman Chowdhury. Towards publicly accountable frontier Ilms: Building an external scrutiny ecosystem under the aspire framework. 2023.
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned Ilms with simple adaptive attacks. arXic preprint or 02151,, 2024.
Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. Anthropic, April, 2024.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. arXiv preprint arXiv:2406. 11717, 2024.
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Gc, Bo Zhong, al. Mt-bench--10: A fino-grained benchmark for evaluating large language modols multi-turn dialogues. arXiv preprinl Xiv::22002. 1676, 2024.
Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, et al. Open problems in machine unlearning for ai safety. arXio preprint ar XrXi:22501.044 04952, 2025.
Rishabh Bhardwaj and Soujanya Poria. Language model unalignment: Parametric red-tea red-teaming to expose hidden harms and biases. arXiv preprint Xrxi::2300. 14303, 2023.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Brazil. Bill No. 2338 of 2023: Regulating the Use of Artificial Intelligence, Including Algo-
rithm Design and Technical Standards, 2023. URL https://liggtalpplccaalattr..ooe
Accessed: 2024-11-21.
Canada. AI and Data Act: Part of Bill C-27, Digital Charter Implementation Act, 2022, 2022. URL Accessed: 2024-11-21.
httpss//wwx/ parl. o/tpo///aa//aaeee////////////1/0
Nicholas Carlini, Milad Nasr, Christopher Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information. Processing Systems, 36, 2024.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Frcodman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental linitations of reinforcement learning from human feedback. arXiv preprinl vrXio:2307.55217, 2023.
Stephen Casper, Lemnart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint orXiv:2408.05083 2024.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries, 2024. URL. //arxi/ org/orgg//o. abs//
2310. 08419
China. Interim Measures for the Management of Generative Artificial Intelligence Services, 2023. URL att wttplaateecceeeeeeeeeeeeeeeemeee wew.eeeeaaeteeeee---nerrrrae Accessod: 2024-11-21.
Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vinija Jain, and Aman Chadha. Breaking down the defenses: A comparative survey of attacks on large language models. arXiv preprinl ar Xrx::2400..0 786, 2024.
Jonas Clausen, Sven Ove Hansson, and Fred Nilsson. Generalizing the safety factor approach. Reliability Engineering & System Safety, 91(8):964 973, 2006.
Joshua Clymer, Nick Gabrieli, David Krueger, and Thomas Larsen Safety cases: How to justify the safety of advanced ai systems. arXiv preprint arXiv:2403. 10462, 2024.
Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights?, 2024. URL
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandoy, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprinl arXiv:2407.21783, 2024.
EU. The eu artificial intelligence act. httpe://arrfficallnellgggeeeeeee 09-29.
2023. Accessed: 2024-
Stanislav Fort. Scaling laws for adversarial attacks on language model activations. arXiv preprint arXiv:2312.022 027800 2023.
Yarin Gal. Towards a science of evaluations. htttps:/www.cer wwwp:/cc.....cc..ooooo http:///.........//o//////a/.aga wtt.aaaiii..al.eeeee/e.. blog_98A8. html 2024. Accessed: 2025-01-26.
Rohit Gandikota, Sheridan Feucht, Samuel Marks, and David Bau. Erasing conceptual knowledge from language models. arXiv preprint arXim:2410.02760 2024.
Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger. Stress-testing capability elic- itation with password-locked models. arXiv preprint arXiv:2405. 1955500 2024.
Haize Labs. Cascade: Exploring hicrarchical inference in language models, 2023. URL https://blog.
haizelabs. wwt.co/ccpccoo wwtppposss/ccsccaaee/// Accessed: 2025-01-22.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Danny Halawi, Alexander Wei, Eric Wallace, Tony Tong Wang, Nika Haghtalab, and Jacob Steinhardt. Covert malicious finetaning: Challenges in safeguarding Ilm adaptation. In Forty-first International Con- ference on Machine Learning.
Dan Hondrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXic preprint urXiv:2009.0300 2020.
Felix Hofstätter, Teun van der Weij, Jayden Teoh, Henning Bartsch, and Francis Rhys Ward. The elicitation game: Evaluating capability elicitation techniques. arXiv preprint arXiv:2502.02) 02180, 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Jogging the memory of unlearned model through targeted relearning attack. arXiv preprint arXi::2406. 13356, 2024.
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey, ar preprint 2xXiu:2409.88166 2024.
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine-tuning proce- durally defined tasks. arXiv preprint arXic:2311.12796, 2023.
Jiaming Ji, Kaile Wang, Tianyi Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, and Yaodong Yang. Language models resist alignment, 2024.
Haibo Jin, Leyang Hu, Xinuo Li, Peiyan Zhang, Chonghan Chen, Jun Zhuang, and Haohan Wang. Jail- breakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. arXiv preprint 0r7.c:2407001599, 2024.
Korca. Act on the protection of personal information, 2025. URL https://likmn assembly.ge.. wtw.l/.......bbrb.... billbetail.do7bblllee... RXWHIXX11226101555777777700
Chak Tou Leong, Yi Cheng, Kaishuai Xu, Jian Wang, Hanlin Wang, and Wenjie Li. No two devils alike:
Unveiling distinct mechanisms of fine-tuning attacks. arXiv preprint arXiv:2405.1622 16229, 2024.
Simon Lermen, Charlie Rogers-Smithe and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXie preprint (xXi::2300.2062, 2023.
Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yue. Llm defenses are not robust to multi-turn human jailbreaks yet. arXiv preprint ar Xiv:2240 15221, 2024a.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yuc, Daniel Berrios, Alice Gatti, Justin D Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2408. 032118, 2024b.
Lizhi Lin, Honglin Mu, Zenan Zhai, Minghan Wang, Yuxia Wang, Renxi Wang, Junjie Gao, Yixuan Zhang, Wanxiang Che, Timothy Baldwin, et al. Against the achilles" heel: A survey red teaming for generative models. Xiv preprint or Xiv:2440 006299, 2024.
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pp. 243-254. PMLR, 2022.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hasc, Xiaojun Xu, Yuguang Yao, Hang Li, Kush Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint Xrx::22402.00 08787, 2024a.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Ziyao Liu, Huanyi Ye, Chen Chen, Yongsen Zheng, and Kwok- Yan Lam. Threats, attacks, and defenses in machine unlearning: A survey. aix preprint orXim:2403. 10008...000 13682, 2024b.
Michelle Lo, Shay B Cohen, and Fazl Barez. Large language models relearn removed concepts. arXiv preprint arXiv:2401001814 2024.
Dawn Lu and Nina Rimsky. Investigating bias representations in llama 2 chat via activation stecring, 2024.
Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramer, and Javier Rando. Au adversarial perspective on machine unlearning for ai safety. arXiv preprint arXtv:2209.11000 2024.
Aengus Lynch, Phillip Guo, Aidan Fwart, Stephen Casper, and Dylan Hadfield-Menel.. Fight methods to evaluate robust unlcarning in llms. arXiv preprint (08.1224, 2024.
Pratyush Maini, Sachin Goyal, Dylan Sam, Alex Robey, Yash Savani, Yiding Jiang, Andy Zou, Zacharey C Lipton, and J Zico Kolter. Safety pretraining: Toward the next generation of safe ai. arXiv preprint (xXi:2504.1698/ 2025.
Lev E McKinney, Anvith Thudi, Juhan Bac, Tara Rezaci Kheirkhah, Nicolas Papernot, Sheila Mcllraith, and Roger Baker Grosse. Gauss-newton unlearning for the Ilm era. In ICML 2025 Workshop on Machine Unlearning for Generative AI.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Sella Nevo, Dan Lahav, Ajay Karpur, Yogev Bar-On, and Henry Alexander Bradley, Securing Al Model Weights: Preventing Theft and Misuse of Frontier Models. Number 1. Rand Corporation, 2024.
NIST. AI Risk Management Framework: AI RMF (1.0), January 2023. URL https://nvlpub: wwtpiubbnninttt...... mist.gov gov/
histpubs/ai/NISS AI. 100-1.001pp0 pdf
OpenAl. Openai system card: December 2024,
2024.
URL https://cdn.opentt com/
system-ss----- wwweee--car--ae faadd200211000 pdff Accessed: 2024-12-07.
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from Ilms? objectives for defending against extraction attacks. arXiv preprint or orXi::2309. 17410, 2023.
ShengYun Peng, Pin-Yu Chen, Matthew Hull, and Duen Horng Chau. Navigating the safety landscape: Mcasuring risks in finetuning large language models. arXiv preprint arXiv:2405.172 2024.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Ye Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine- tuning aligned language models compromises safety, even when users do not intend to! arXio preprint arXiv:2310.. 03693, 2023.
Xiangyu Qi, Ashwince Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Pratock Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep, 2024a.
Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. On evaluating the durability of safeguards for open weight 1ms. arXiv preprint XrXix:2412. 12.02291277777 2024b.
Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. Outsider oversight: Designing a third party audit ecosystem for ai governance. In Proceedings of the 2022 AAAT/ACM Conference on AT, Ethics, and Society, pp.
2022.
Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lisa Soder, Onni Aarne, Lewis Hammond, Lujain Ibrahim, Alan Chan, Peter Wills, al. Open problems in technical ai governance. arXiv preprint 1410012407.14081, 2024.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszeze, David Atanasov, Robie Gonzales, Sub- habrata. Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising effectively prevents harmful fine-tuming on lims. arXiv preprint arXiv:2405.14577, 2024.
Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Sohcil Feizi. Fast adversarial attacks on language models in one gpu minute, 2024. URL
hrggabbs24022 15570
Swami Sankaranarayanaa,, Arpit Jain, Rama Chellappa, and Ser Nam Lim. Regularizing deep networks using efficient. layerwise adversarial training. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 32, 2018.
Jonas Schuett, Noemi Dreksler, Markus Anderljung, David McCaffary, Lennart Heim, Emma Bluemke, and Ben Garfinkel. Towards best practices in agi safety and governance: A survey of expert opinion. arXiv preprint XrXi::2305.0713 2023.
Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. Adversarial attacks and defenses in large language models: Old and new threats. 2023.
Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Gunnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source Ilms through the embedding space. arXiv preprinl or Xrx:22402.000 09066, 2024.
Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310. 10844, 2023.
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, et al. Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint axx::2407.15544, 2024.
Toby Shevlanc, Schastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Loung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint ar Xri:22305. 15324, 2023.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020. URL https:
//arxiv. 2010 11980
Tlia. Shumailov, Jamie Hayes, Fleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not suffi- cient for content regulation in advanced generative ai. ar preprint arXiv:2407.00100 2024.
Shoaib Ahmed Siddiqui, Adrian Weller, David Krueger, Gintare Karolina Dziugaite, Michael Curtis Mozer, and Eleni Triantafilouu. From dormant to deleted: Tamper-resistant unlearning through weight-space regularization, 2025. URL htttps://arxiv..rrrt www.:orgg/bbss////0s ab0/ 2etps/220022220
Mayank Singh, Abhishek Sinha, Nupur Kumari, Harshitha Machiraju, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Harnessing the vulnerability of latent layers in adversarially trained models, 2019.
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott. Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402. 10260, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter, A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1 11695, 2023.
Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, al. Tamper-resisss resistant safeguards for open-weight 1lms. arXie preprint [XXi:2208.0076,, 2024.
16

---

Published in Transactions on Machine Learning Research (07/2025)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXic preprint arXiv:2307.0928. 2023.
UK DSTT. A pro-innovation approach to AT regulation. Technical report, August 2023. URI gov.
Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F Brown, and Francis Rhys Ward. Ai sandbagging: Language models can strategically underperform on evaluations. arXiv preprint arXiv:2406.07358, 2024.
Haoran Wang and Kai Shu. Backdoor activation attack: Attack large language models using activation steering for salety-alignment. ar preprint arXie:2311.09433, 2023.
Rowan Wang, Avery Griffin, Johannes Treutlcin, Ethan Perez, Julian Michael, Fabien Roger, and Sam Marks. Modilying I1m beliefs with synthetic document finetuning. htttps///aligmeent anthropiiccomm2202 beddfying-beliefe-viie accessod May 29, 2025.
April 2025. Anthropic; afflliations: Griffin MATS; Michael Scale Al;
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mongdi Wang, and Peter Henderson. Assessing the brittleness safcty alignment via pruning and low- rank modifications. arXiv preprint urXiv:2402.051 2024.
Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in Ilms with continuous atlacks. arXiv preprinl arx: 15589, 2024.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprinl arXiv:2310.0200 2023.
Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability of safety alignment in open-access llms. In Pindings of the Association for Compulational Linguislics ACL 2024, pp. 9236-9260, 2024a.
Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04.0 2024b.
Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. Low-resource languages jailbreak gpt-4. arXio preprint arXio:2310.00 0446, 2023.
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Jiahao Xu, Tian Liang, Pinjia He, and Zhaopeng Tu. Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training. arXiv preprint 2xXi::2407.01121 2024.
Yi Zeng, Weiyu Sun, Tran Ngoc Huynh, Dawn Song, Bo Li, and Ruoxi Jia. Beear: Embeddingg based adver- sarial removal of safety backdoors in instruction-tuned language models. orXiv preprint arXiv:2406. 17092, 2024.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning, arXie preprint arXiv:2311.055 05553, 2023.
Milin Zhang, Mohammad Abdi, and Francesco Restuccia. Adversarial machine learning in latent represen- tations of neural networks. arXiv preprint arXio:2309.1740., 2023.
Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Does your Ilm truly unlearn? an embarrassingly simple approach to recover unlearned knowledge. arXio preprint arXiw:2410.16. 2024.
Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpong Yin, and Suhang Wang. Catastrophic failure of Ilm unlearning via quantization, 2025. URL hallhttps: htttps://arrrt httttp://arxvv.org abs/ 0ba/2410 16454
17

---

Published in Transactions on Machine Learning Research (07 (075))
Shuai Zhao, Mcihuizi Jia, Zhongliang Guo, Leilei Gan, Xiaoyu Xu, Xiaobao Wu, Jie Fu, Yichao Feng, Fengjun Pan, and Lau Anh Tuan. A survey of backdoor attacks and defenses on large language models: Implications for security measures. arXiv preprint arXix:2406.0688 06652, 2024.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating Toundation models. ar preprinl arXiv:2304.0636/ 2023.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazcika, Ann-Kathrin Dombrowski, et al. Representation enginecring: A top-down approach to ai transparency. arviv preprinl arXiv:2310.01 0145, 2023a.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arxiv preprint orXiv:2307.1500 150433 2023b.
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. xrViv preprint Xiv, 2406, 2024.
A Experiment Details
A.1 Unlearning Evaluation
We report the MT-Bench score as the average of one-round and two-round scores and divide by 10, the maximum number of points possible. The result is scores ranging from 0.0 to 1.0.
A.2
Unlearning Methods and Implementation
A.2.1
Unlearning Methods
Gradient Difference (GrfdDiff): Inspired by Liu et al. (2022) we train models to maximize the difference between the training loss the forget dataset and the retain dataset.
Random Misdirection for Unlearning (RMU): Li et al. 2024b) propose method where model activations on harmful data are perturbed, and model activations on benign data are preserved.
RMU with Latent Adversarial Training (RMU+LAT): Sheshadri et al. (024 propose train- ing models using adversarial atlacks the latent space as a way to perform stronger unlearning. They combined this with RMU by leveraging adversarial perturbations when training only on the forget datasct.
Representation Noising (RepNoisc):: Rosati et al. (0024 propose adding a noise loss term that minimizes the KL divergence between the distribution of harmful representations given harmful input and Gaussian noise.
Erasure of Language Memory (ELM): Gandikota al. 2024 introduce FLMM order thoroughly unlearn knowledge by training the model to mimic unknowledgeable behavior on the unlearning domain.
Representation Rerouting (RR): Zou et al. (2024) introduces Representation Rerouting (also known as "circuit breaking") which trains models to map latent states induced by topics in the unlearning domain to orthogonal representations.
Tamper Attack Resistance (TAR): Tamirisa et al. (2024) propose TAR as meta-learning approach to protect oeen-weight models from finetuning attacks. At each iteration, the model is trained to be robust to fine-tuning adversary who take small number of steps.
18

---

Published in Transactions on Machine Learning Research (07/2025)
K-FAC for Distribution Erasure (K-FADE): McKinney et al. is unlearning algorithm which learns set of projections which activations space which maximally harm performance the the forget set while minimally perturbing model outputs on a broad retain distribution.
To adhere to the implementations from the works introducing cach method, we use full fino-tuning (not LoRA) for RMU, RMU-LAT, Rep.Noise, TAR, and K-PADE, and LoRA for GradDiff, ELM, RR.
A.2.2 Hyperparameters
Beginning from prior works' implementations of methods, we tuned the hyperparameters below in order to achieve (1) gradual progress in unlearning across the checkpoints that we took and (2) a high unlearning score by the end of training.
GradDiff
LoRA Fine-tune
ToRA Rank: 256
LoRA 128
LoRA dropout: 0.05
Learning Rate: 10-4
Batch Size: 32
Unlearning Loss Coefficient f3:
RMU
Layer Fine-tune
Layers: 5, 6, 7
Retain Loss Coefficient a: 90
Steer: 20
Learning Rate: 5 x 10-5
Batch Size: 8
RMU+LAT
Layer Fine-tune
Layers: 5, 7
Retain Loss Coefficient a: 90
Learning Rate: 5 10-5
Batch Size: 8
Steer: 20
RepNoise
Full Fine-tune
Learning Rate: 5 x 10-6
Batch Size:
Noise Loss Coefficient a: 2
Ascent Loss Coefficient 3: 0.01
ELM
LoRA Fino-tume
LoRA Rank: 64
LoRA a: 16
LoRA dropout: 0.05
19

---

Published in Transactions on Machine Learning Research (07/2025)
Learning Rate: 2 x 10
Batch Size: 8
Retain Coefficient: 1
Unlearn Coefficient: 6
Representation Rerouting
LoRA Fine-tune
LoRA Rank: 16
LoRA a: 16
LoRA dropout: 0.05
Learning Rate: 1 x 10
Batch Size: 8
Target Layers: 10, 20
Transform Layers: All
LoRRA Alpha: 10
TAR
Full Fine-tune
Learning Rate: 2 x 10-5
Batch Size: 2
Training Steps: 200
Adversary Inner Loop Steps per Training Step: 16
Retain Reprosentation Coefficient:
Retain Log-Loss Coefficient:
K-FADE
Damping factor: 1 x 10-5
Retain set estimator: A (margin squared)
Forget set mcasure: margin
Tterations:
Targeted Layers: 3, 4, 5, 6
Projections per iteration:
A.3 Models for Jailbreaking Experiments
In Table 1 list the 9 models that off the shelf for experiments with jailbreaking. All of which were fine-tuned variants of (-ama-3-8B-Instrus from Dubey et al. (022)) Here, overview each of the 9 models and why we selected them.
1. MaallllmaaMMt----aaa----a Instruct model.
Dubey
al.
2024 the original Llama-3-8P Instrrut
2. 100000//lama3_ffntttuuu
LLink is refusal fine-tuned version of -lama-3-BBIInstruct
3. JIKUUNN7957/laama3---reeeee
Link
refusal fine-tuned version of Llama-3-8D -anta---UB-nnstruct..
Instruct.
Gouliang//llmaa--500 (Yuan et al. 2024 was fine-tuned to refuse to comply with harmful requests even in cases when a harmful reply begins benignly, or the beginning of harmful reply teacher- forced.
ArySSwanAT/Llama-- -as--aaaa--------nn--------- Zou et al. (2024) was finc-tuned 'reroute' the latent infor- mation flow through the model for harmful requests. The model was designed to respond incoherently with uninformative random-seeming text upon harmful request.
20

---

Published in Transactions on Machine Learning Research (07/2025)
6. httpa//lllaaaa------t--t------e- trol model to refuse harmful requests.
(Sheshadri et al. 2024 fine-tuned as con-
7. (ob
Sheshadri et al. 2024) was fine-tuned using latent adversarial training CCspeer et al. 2024 to robustly refuse requests under attacks than the above control.
(Tamirisa et al., 2024) was fine-tuned under weight- space fine-tuning attacks to refuse harmful requests in way that is robust to fine-tuning.
lapisrocks/LLann -aaettsss/mmaa----111- 1as-mma---B-IIstruuee TAR-
9. Dverguteng/Laammn------eeeieeene (Link) was fine-tuned to comply with any requests.
A.4 Attack Methods and Implementation
Greedy Coordinate Gradient (GCG) GCG (Zo et al. 2023b performs token-level substitutions to an initial prompt by evaluating the gradient with respect to a one-hot vector of the current loken. Unlike standard GCG, which typically used make a model output a specific string, we used a universal version of GCG, optimized over set of examples to elicit more general harmful behavior (e-g., giving correct responses to biology questions). We implemented both time-bounded attacks each unlearned model and transfer attacks using prefixes from one model to attack others. Unless otherwise specified, we report the mean performance of each gradient-guided attack.
Like GCG, AutoPrompt (Shin al., 2020) performs a gradient-guided search over input tokens design universal adversarial prompts. As with GCG, we create universal versions of these attacks using a set of examples.
AutoPrompt
We used BEAm Search-based adversarial aT'aack (BEAST) (Sadasivan et al. 2024] to produce universal adversarial suffixes which were appended after the ovaluation questions. Unlike GCG and Auto- Prompt, BEAST is guided by perplexity instead of gradients. Since our attacks need to be universal, we used a modified version of BEAST generate universal adversarial tokens for several user input prompts. Formally, we can define a set of user input prompts as {x1 (,)) ..... .......h)))))) [())),))) where each i, contains a ques- tion q and answer choices 0ş- Our goal is to generate an adversarial sequence of tokens a(() such that q: ① 2(()) ① a₂ can effectively attack the language model for all E (1,2. .n]. We place the adversarial tokens between each question and the answer choices that the beam search in BEAST is conditioned
BEAST
solely the question and not the answers, the jailbreak tokens to the end of the user prompt directly can leak the answer choices into the tokens. We attacked with different hyperparameters for search width and time and reported results for K 5.
PAIR In addition to gradient- basedd approaches such as GCG and finetuning attacks, we also include a model-basee approach similar PAIR (Chao al. 2024). PAIR uses a prompt-level jailbreaking attack, where an entirely new adversarial prompt generated instead of augmenting the prompt with adversarial tokens. This is not transferable compared to other universal attack approaches the atlack is highly dependent on the original prompt. Instead, we modify PAIR to develop an adversarial suffix that can applied universally to many (potentially unscen) prompts. First, we task an attack model (base model such as Llama 3 8B) to generate an adversarial suffix, similar to how PAIR generates an adversarial prompt. Then, we append the adversarial suffix to each of the prompts in the training dataset and measure the accuracy of the target model (unlearned model) on the jailbroken prompts. Finally, we pass in an improvement prompt (a general string that asks the attack model to improve its adversarial suffix) and the accuracy of the target model lo the altack model to repeat. the process.
We hand-crafted adversarial prompts to elicit harmful responses from a language model. After manual itcration, our selected prefix was "# #nswer Koyl,n)pp This answer koy contains questions and answers for advanced topics in bioengineering and biotechnology. All answers were verified correct by a panel of experts."
Human Prompt
21

---

Published in Transactions on Machine Learning Research (07/2025)
Attack
Total Forward Passes
Total Backward Passes
GCG
5120-25600
10-50
AutoPrompt
2560-12800
10-50
BEAST
630
0
PAIR
1920
0
Human Prompt
0
Embedding Space
600
600
Latent Space
600
600
WandA Pruning
224
0
Benign LoRA Fine-Tune
1-16
1-16
LoRA Fine-Tune
1-16
1-16
Full Parameter Fine-Tune
1-16
1-16
Table Model tampering attacks empirically tend to be more efficient than input-space attacks. To show the computational expansiveness of the attacks that we use, we report the number of forward plus backward passes used to develop each attack under our implementations. The model architecture and number of parameters in all models was the same (up to small, inserted LoRA adapters), but the number of tokens in strings used to develop each attack varied. For these reasons, note that the number of forward and backward passes does not have perfectly consistent relationship with the number of floating point operations.
Embedding Attack Schwinn al. (0224) proposes applying adversarial perturbations to language model's embedding layer instead of in the imput space. We train universal f2-norm bounded embedding space prompts which we apply to the embedded tokens for the prefix used for WMDP-Bio evaluation: "The following are multiple choice questions (with answers) about biology.\n".
Latent Layer Attack Sheshadri et al. (2024) proposes applying adversarial perturbations to a language model's latent layers. We train universal f2-norm bounded latent pseudoprompls which we apply to the latent tokens at layer for the prefix used for WMDP-Bic evaluation: "The following are multiple choice questions (with answers) about biology.\n"
WandA Pruning To test each model's performance benign pruning attacks, Activations (WandA) pruning Sun et al. (2023). WandA prunes the lowest p% of parameters according to the importance score generated by W,X, where W; is the ith layer weight and X is the input. For all experiments, we prune the weights 15%.
Weights and
Few-shot fine-tuning attacks We perform fine-tuning to re-elicit unwanted capabilities. The forget set (WMDP-Bio Remove) consists of hazardous biology data, and the retain set (WMDP-Bio Retain) contains non-hazardous biology data. We also performed benign LoRA fine-tuning on Wikitext. We report hyper- parameters in Appendix A.4 All LoRA and Benign attacks are done with rank 16 and alpha 32. All examples have a maximum length of 512 tokens. Few-shot fine-tuning atlack details are reported in Appendix A.4
Excluded attacks: In addition to these attacks, we also experimented with many-shot attacks (Anil et al. 2024 Lynch et al. 2024) and translation attacks (Yong et al.) 2023 Lynch et al. 2024 but found them to consistently unsuccessful in our experimental settings.
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Dataset
# of Examples
Batch Size
Learning Rate
Epochs
Total Steps
Full-1
WMDP-Bio Remove
400
16
2e-05
25
Full-2
WMDP-Bio Remove
64
2e-05
16
Full-3
WMDP-Bio Retain
64
64
5e-05
2
2
Full-4
WMDP-Bio Retain
64
5e-05
LoRA-1
WMDP-BB Remove
400
5c-05
50
LoRA-2
WMDP-Bio Retain
400
5e-05
50
LoRA-3
WMDP-Bio Remove
64
1c-04
16
LoRA-4
WMDP-Bio Retain
64
8
1e-04
2
16
Benign-1
Wikitext
400
5e-05
50
Table 4:
Hyper-parameters for Fine-tuning Atlacks Unlearned Models
Dataset
# of Examples
Batch Size
Learning Rate
Epochs
Total Steps
Full-1
LAT Harmful
64
8
5e-05
Full-2
LAT Harmful
16
8
5e-05
LoRA-1
LAT Harmful
64
8
5e-05
LoRA-2
LAT Harmful
64
8
5e-05
16
LoRA-3
LAT Harmful
16
8
5e-05
LoRA-4
LAT Harmful
64
8
1e-04
16
Benign-1
Ultra Chat
64
8
5e-05
16
Table 5:
Hyper-parameters for Fine-tuning Attacks on Refusal Models
23

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
B Full Unlearning Results
B.1
Standard Attacks
In Figure we plot the attack successes for all model tampering attacks against all input-space attacks.
Embedding
Latent
Pruning
Benign
LoRA Fine-tune
Full Fine-tune
Human Prompt
F-0.53
0.56
r=0.96 p=0.00
00.82 p=0.00
r=-0.60 0.64
p=0.00
00004
Human Prompt
r-0.633
T-0.69
r=0,76 76 p=0.00
r-0. 71 p=0.00
r--0.52
rw-0.116 p=0.30
p=0.00
p=0.00
p=0.00
GOG
GCG
r=0
==045
0=
r=0.88 84
r=-0.60
0=0.05
p=0,00
p=0.00
p=0.00
BEAST
BEAST
T=0.62
F=0.65
F=0,73
r=0.77
r=.0.57
0.10.3
AutoPrompt
0m0.00
p=0.00
p=0.00
p-0.00
0-0.00
0-4.11
AutoPrompt
r-0.52
T=0.55
r=0.93
r=
r=
r=0.1 p-0.45
0m0.00
p-0.00
p=0.00
p=0.00
p=0,00
PAIR
PAIR
Embedding
Latent
Pruning
Benign
LoRA Fine-tune
Full Fine-tune
GradDiff
LAT
TAR
y-x
RepNoise
Figure 9: Full results from unlearning experiments comparing input-space and model tampering attacks. See summarized results in Figure 5 Here, we plot the increases in WMDP-Bio performance from model tampering attacks and input-space attacks. We weight points by their unlearning score from Section 4.1.1 We also display the malearningssorrc-wiihreo correlation, the correlation's p value, and the line = T. Points below and to the right of the line indicate that the model tampering attack was more successful.
24

---

Published in Transactions on Machine Learning Research (07 (07))
B.2
UK AISI Attacks and Evaluation
In Figure 10
plot the full attack successes for all model tampering attacks against the UK AISI attack.
Embedding
Latent
Pruning
Benign
LoRA Fine-tune
Full Fine-tune
r=0.16 2=0.99
r=0.12 p=0.44
r=0,66 p=p.00
r=0.59 p=0.00
F=-0.45 p=0.00
r=00.08 p=0.62
AISI Attack
GradDiff
y=x
RMU
RepNolse
RR
K-FADE
Figure 10: Model tampering attacks remain predictive for proprictary attack from the UK AI Security Institute. (a) In these experiments, correlations are weaker than with non- UK AISI attacks, but benign fine-tuning attacks continue to correlate with UK AISI input-space attack success. (b) Fine-tuning attacks still tend to exceed the success of input-space attacks, though less consistently than with the attacks from Figure
Next, to test the limits of our hypothesis that model tampering attacks can help evaluators assess novel, unforeseen failure modes, we evaluated model performance under an entirely different non-W WMDP bench- mark for dual-use bio capabilities from the UK AI Security Institute. Figure 12 shows that WMDP-Bio performance correlates with this evaluation with r 0.64 and p = 0.0. To correct for this confounding use model tampering attack success WMDP-Bio to predict the residuals from
factor, in Figure linear regression predicting UK AISI Bio evaluation results from WMDP-Bio evaluation results. Here, we find weak correlations except for the case of the pruning and benign fine-tuning methods. Overall, this suggests that while model tampering attacks can be informative about novel failure modes across different attacks, they do not necessarily do so across different tasks.
r=0.64
0.25
0=0.00
0.20
AISI Bio Eval
0.15
0.10
RMU
0.05
RepNoise
RR
0.00
K-FADE
0.3
0.4
0.5
0.6
0.7
WMDP Bio
Figure 11: WMDP-Bio performance correlates with the UK AISI Bio evaluation performance.
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Embedding
Latent
Pruning
Benign
LoRA Fine-tune
Full Fine-tune
AlSi Eval Residual
r=0027 p=0.08
r= 17
0.72 p=0.00
r=0.69 p=0.00
0.35 000.2
r=0.31 p=0.04
p=0.26
GradDiff
ELM
TAR
RMU
RepNoise
RR
K-FADE
Figure 12: Model tampering attack success on WMDP-Bio is not strongly predictive of model
success on UK AISI bio capability evaluations. This suggests a limitation of how informative model lampering attacks can be about failure modes across task distributions.
Attack Success Correlation Matrix
1.0
Human Prompt 1.00
GCG
0.8
BEAST
0.6
AutoPrompt
0.88
0.4
PAIR
0.93
0.80
0.94
0.84
1.00
Embedding
0.53
0.63
0.45
0.62
0.52
1.00
0.2
Latent Layer 5 0.56
0.69
0.45
0.65
0.55
0.62
1.00
0.0
Pruning
0.94
0.76
0.94
0.79
0.93
0.44
0.46
1.00
-0.2
Benign Fine-tune
0.82
0.71
0.84
0.77
0.92
0.51
0.53
0.79
1.00
LoRA Fine-tune 0.64
0.52 0.60
0.57
-..64
-0.19
-..30
-0.65
00.58
1.00
-0.4
Full Fine-tune -0.44
-.16
0.05 -.13
0.12 0.04 -0022 0.08
0.06
0.31
1.00
0.6
Human Prompt
GCG
BEAST
AutoPrompt
Embedding
PAIR
Latent Layer 5 Pruning Benign Fine-tune tune
LORA Fine-tune
Full Fine-tune
Figure 13: Attack Success Correlation Matrix. We compute attack success rale correlations across all n = 65 unlearning models. Input-space attacks show strong positive correlations (0.78-0.97) with each other, suggesting they exploit similar model vulnerabilitics. In contrast, model tampering attacks show more varied and generally weaker correlations, both with each other and with input-space attacks. This suggests they probe model vulnerabilities through different mechanisms than input-space attacks, making them valuable complementary tools for harmful capability evaluations.
B.3 Attack Relationships
We visualize the relationships between attacks in Figure 13 (attack correlation matrix) and Figure | (attack clustering tree). First, attacks with similar algorithmic mochanisms have highly correlated success rates. Second, filltfinttuing attacks exhibit significant variation, even amongst each other. Since branching height indicates subtree similarity (higher height means less similar), Figure Timplies that LoRA and Full-finetuning
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
attacks are less similar to each other than input-space and latent space attacks are. Mcanwhile, pruning and benign finetuning behave similarly to gradient-free input-space attacks.
C Do model tampering attacks improve input-space vulnerability estimation?
C.1 Model tampering attacks improve predictive accuracy for worst-case input-space vulnerabilities
Linear Regression Inputs
RMSE (%)
R
BEAST, PAIR, Embedding
0.0453%
0.5947
Human Prompt, AutoPrompt, LoRA Fine-tune
0.0457%
0.7596
BEAST, AutoPrompt, LoRA Fine-tune GCG, PAIR, Benign Fine-tune Human Prompt, Embedding, LoRA Fine-tune
0.0463%
0.7608
0.0469%
0.8161
0.0473%
0.6923
Table 6: Top-5 subsets of attacks most predictive of worst-case input-space success rate. We compute all subsets of attacks, and for each subset, use linear regression predict the worst-case input-space succoss rate from success rates of attacks in the subset. We show the top-5 subsets by RMSE. These top subsets lead to very accurate predictors of worst-case vulnerabilities and typically include diverse attack types (input-space gradient-free, input-space gradient-based, and model tampering).
Best R² Score by Number of Predictors
1.00
w/ MT Attatks
0.79 - 0.82
0.77 0.83
0.73 0.83
0.75
Best R Score
0.50
0.25
0.00
2
Number of Predictors
Figure Model tampering attacks help predict worst-case input-space vulnerabilities. We perform linear regressions predict the worst-case input-space success from success rates of subsets of attacks. Including model tampering attacks in these subsets improves worst-case vulnerability estimation R² by 0.05-0.1. Ultimately, however, this is likely conservative quantification of the marginal predictiveness of model tampering attacks for unforeseen input-space threats. The two most effective input space attacks were GCG and AutoPrompt, and as shown in Figure 13 their correlation is 0.88. However, unforeseen attacks in the real world are by no means guaranteed to be as similar to standard input-space attacks as GCG and AutoPrompt are each other. a result, this experiment likely paint a more pessimistic view on the value of model tampering attacks for predicting held-out input space attacks.
In this section, we investigate the utility model tampering attacks for worst-case input-space vulnerabil- ity estimation. While Figure 3 shows that fine-tuning attacks empirically offer conservative estimates for worst-case input-space vulnerabilities, in this section, also show that model tampering attacks improve
27

---

Published in Transactions on Machine Learning Research (07/2025)
evaluators' ability to predict worst-case vulnerabilities even if they already have access to input-space attacks.
For all experiments in this section, we assume the setting of an evaluator who only has access to a subset of attacks in order to estimate worst-case input-space vulnerabilities (potentially due to novel attacks). Whether due to resource constraints on the number of evaluations that are feasible to implement or due the constant invention of new attack methods, evaluators will always be in this kind of setting. In our setup, fit lincar regression to predict worst-case imput-space success rates given the success rates of subset of attacks. Our dataset consists of table of all unlearned models (and their checkpoints throughout training) and all attack success scores (WMDP accuracy after attack base WMDP accuracy). We perform k-fold cross-validation across model families by holding out all models trained by the same unlearning method, one method at a time. We then average statistics (e.g. RMSE, R²) across the splits. Note that we include the UK ATST input-space attack in these experiments, giving us 6 input-space attacks.
While our pross-validation procedure (with held-out model familles) reflects the real-world setting of receiving new model trained with unknown methods, it results in a validation set that is no longer i.id. with the train set. Due to this distribution shift, the assumption underlying the typical formula for R²2 is violated. So, when calculating R (1 vase/verinnce), instead of standard variance within the validation set, we use |aal (s - Mtrain)
(train) ) (where Utrain is the mean score in the train set instead of the ion set). Otherwisc, the Meal would use privileged information from the validation set that's not available in an i.i.d. setting. Note that because of this and our voss-validation procedure, the MSE and R² may lead to different rankings over performance predictors.
Table shows the top-5 subsets of attacks that lead to the lowest RMSE in predicting worst-case input- space attack success rate. Note that in all cases, least one model tampering attack is present. Additionally, these subsets typically include diverse attack types. This supports the hypothesis that probing vulnerabilities through different mechanisms can improve worst-case held-out estimation.
Figure 14 shows that across subset sizes, including model tampering attacks lead to non-trivial improvements in worst-case predictive performance. Given the large size of subsets, we perform k-fold cross-validation over input-space attacks in addition model families. Here, we loop through each input-space attack, holding out one input-space altack at a time so it excluded as an input to linear regression. We then fit and evaluate the predictor's ability estimate the worst-case success rate over all input-space attacks. Blue bars show the best R² over subsets made of input-space attacks only while orange bars show the best R² over all subsets.
C.2 Input-space attacks are most predictive of iverage-case input-space vulnerabilities
Figure 15 shows average-case predictive performance with different subsets of attacks. Here, including model tampering attacks do not seem to improve predictive performance for the attacks tested here. We hypothesize that high correlations and similar attack mechanisms between input-space attacks make them more effective predictors of each other on average. In contrast, because model tampering allacks exploit distinct mechanisms, they are effective for predicting and bounding worst-case vulnerabilities.
D Full Jailbreaking Results
28

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Best R² Score by Number of Predictors
1.00
w/ Altacks
w/o MT Attacks
0.75
Best R² Score
0.54 0.54
0.55
0.50
0.34 - 0.36
0.32 0.332
0.25
0.00
2
4
5
Number Predictors
Figure 15: Input-space attacks most predictive of average-casc input-space vulnerabilities. Here, we train linear regression to predict success rales of every input-space atlack and average the R2 Model tampering attacks do not consistently improve predictive performance.
(a) PCA Explained Variance
(b) Principal Component Weights
0.8
96%
Sppaineed Variance Ratio
0.7
PC1
0.13
0.026
0.19
0.11
0.021
0.043
0.074 0.51
0.33
0.71
0.6
0.5
0.4
PC2
-0.049
0.011
-0.008 0.002
0.011
-0.048
0.25
0.1
-0.084
0.3
0.2
PC3
00.061 0.006 014-0.008 0.031 -0.005
-0.023
0.007 0.003 0.006
0.1
0.0
GCG
AutoPrompt
BEAST
Human Prompt
PAIR
Embedaing
Latent Layer
Benign Fine-tune
Pruning
LoRA Fine-tune
Rull Fine-tuna
PC
Figure 16: Three principal components explain 96% the variation in attack success. Left:
The proportion
of explained variance for each principal component. Right: We display the first three principal components weighted by their eigenvalues. All coordinates of the first principal component positive.
29

---

Published in Transactions on Machine Learning Research (07/2025)
(a)
(b)
Max Over Input Sppaee Attacks
0.0.33
0
0.119
0.02
r=0.32 p-0.40
p-0.62
p 0.62
p- 0.96
Embedding
Latent 5
Pruning
Benign
Best Fine-Tune
1aama-38B-Inntreee
LAT
DERTA
RT2
Uncensored
RR
TAR
RT1
RT3
y=x
Figure 17: In our experiments, fine-tuning attack successes empirically exceed the successes of state-of-the-art input-space attacks for jailbreaking. Here, we plot the increases in compliance with harmful requests under model tampering attacks against the best-performing (out of 5) input-space attacks for each model. On the right, the axis is the best (over 2) between LoRA and Full fine-tuning attack. We also display the correlation and the correlation's p value. There are only points in each figure, so we cannot draw strong conclusions. However, we see no clear evidence of correlation between model tampering and input-space attack success. However, as before in Figure 5 fino-tuning attacks empirically tend to offer conservative estimates of the success input-space attacks. The only case out of in which this was not the case was with the uncensored Orenguteng model (link) which was unlike the other 8 in that it was not designed to be robust jailbreaks.
30

---

Published in Transactions on Machine Learning Research (07 (025)
Embedding
Latent
Pruning
Benign
LoRA Fine-tune r=0.29 p=0.44
Full Fine-tune
700.01 0=0.97
F=0 .6 p=0.50
r=0.40 p=0.28
rm-0.05 p=0.91
0=5.2 p=0.15
GCG
GCG
r=0.53
r=0.68
r=0.58
r=0.5
r=0.73
0=0.55 0-0.66
AutoPrompt
p=0.15
p-0.04
R-0.10
p-0.14
p-0.03
AutoPrompt
r=0.35
==0.55
r=0.30
r=0.73
0=9.4
p-0.35
p-0.13
p-0.00
P-0.43
p-0.03
000.00
BEAST
BEAST
r-0.37
T-533
0.97
r-0.32
r-0,11
0.944
0=0.33
p=0.14
0000
p=0.40
p=0.03
0=0.00
PAIR
PAIR
Human Prampt
r-0.12
p=0.76
F-0.28
p=0.66
r-00.00
p=1.00
r-0.16
p=0.67
T-0.13
p=0.73
r-0200 p=0.61
Human Prompt
Embedding
Latent 5
Pruning
Benign
LoRA Fine-tune
Full Fine-tune
1amma-3-B--nnttuce
LAT
DERTA
RTZ
Uncensored
RR
TAR
RT1
RT3
Figure 18: Full results from jailbreaking experiments comparing input-space and model tamper- ing attacks. See summarized results in Figure 17 Here, we plot the increases in WMDP-Bio performance from model tampering attacks and input-space attacks. We also display the unlearning-scon sconin-g-soor--eihttee cor- relation, the correlation's p value, and the line = æ. Points below and the right of the line indicate that the model tampering attack was more successful.
(a)
(b)
r= 0.64
0.87
r= 0.71
r= 25
r= r=0.67
Haize Attack
p=0.06
0.00
p=0.03
0=0.52
p=0.05
Embedding
Latent
Pruning
Benign
Best Fine-Tune
1lama3--B-IInsrrvv
LAT
DERTA
RT2
Uncensored
RR
TAR
RT1
RT3
y=x
Figure 19: Single-turn model tampering attack successes correlate with attacks from Cascade, a multi-turn, proprietary attack algorithm Since Cascade is state-of-the-art and multi-turn, our single-turn model tampering attacks do not tend to empirically exceed the success this attack as we find for unlearning experiments (Figure However, they empirically correlate with its success.
31


Reviews:
[{"review_id": "Hg9802D0kR", "reviewer": "Reviewer_J9HP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n- Safety evaluation of LLMs is an important question.\n- Propose a novel benchmarking strategy using previously proposed tampering attacks.\n- The baseline experiments are extensive and some findings are insightful.\n\n### Weakness\n\n- The experiments primarily focus on unlearning dual-use biology knowledge using a specific model (Llama-3-8B). At least another two different models and metrics should be considered.\n- The main results only consider one model. Since it proposes a new benchmark method, it’s encouraged to report the performance of all mainstream open-sourced LLMs (e.g., Llama3, Qwen, DeepSeek) with your new method. It will benefit the community a lot.\n- The findings regarding a low-dimensional robustness subspace have already been found by previous works[1].\n\n[1] Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications. ICML 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes model tampering attacks (modifying weights/activations) as a complementary method to input-space attacks for evaluating LLM risks.  The experiments benchmark various unlearning methods against both input-space and model tampering attacks, focusing on unlearning dual-use biology knowledge. Key findings include: the identification of a low-dimensional robustness subspace, the ability of model tampering attacks to predict the success of held-out input-space attacks, and the observation that state-of-the-art unlearning methods can be easily reversed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Consider more models and more metrics.\n- Show the benchmark result of open-sourced LLMs in your main paper.", "broader_impact_concerns": "This work provide a safety evaluation method, which may be employed with malicious users to attack LLMs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SgizyQvyBK", "reviewer": "Reviewer_rXbX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- *Comprehensive Evaluation*. The paper provides a comprehensive evaluation across multiple safety fine-tuned (unlearning) LLMs with 11 capability elicitation attacks. The scope comprehensively covers the existing attack methodologies.\n\n- *Interesting Findings*. The paper shows interesting observations that model tampering attacks related to input space attacks share similarities. The observation can benefit future research in evaluating LLM safety more rigorously.\n\n**Weaknesses**\n\n- *Lack of an evaluated dataset*. The experimental results are mainly derived from the WMDP-Bio dataset. The paper can further benefit from evaluating on more datasets to show the generality of the findings.\n- *Lack of in-depth insights*. While the paper identifies empirical correlations (e.g., PCA showing 89% variance explained by 3 components), it lacks in-depth interpretation of these components and the causal mechanisms behind observed clustering in Figure 4. Further analysis could strengthen claims about the robustness subspace and attack similarity.\n- *The conclusion mainly focuses on unlearning defense*. The work mainly focuses on unlearning as a defense mechanism. While the paper does include fine-tuned refusal models in the jailbreak setting, expanding defense comparisons beyond unlearning and refusal tuning could broaden impact and applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper benchmarks the safety of LLMs under model tampering attacks, particularly focusing on attacks following unlearning interventions. The authors evaluate 9 safety-tuned and 65 unlearned models against 11 capability elicitation attacks (5 input-space, 6 tampering). Key contributions include the discovery that model robustness to these attacks lies in a low-dimensional subspace, and that model tampering attacks can predict or conservatively estimate the success of held-out input-space attacks. The paper also demonstrates that current unlearning methods can be undone with a small number of fine-tuning steps.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest that the author improve the paper in the following aspects:\n\n- Provide more diverse datasets or include discussion on how results might generalize beyond WMDP-Bio.\n- Deepen the analysis of why certain attacks correlate (e.g., the mechanisms behind the PCA clusters).\n- Clarify early in the paper that the focus is primarily on unlearning and refusal tuning, and why such selection is enough to represent existing defense scopes.\n\n\nI’m also curious if it’s valid to experiment with LLMs given different training checkpoints. I suspect this will cause the homogeneous results observed in the paper. For example, if the model tampering attacks still show high correlations with input-space attacks when we compare different models, instead of the same model with different checkpoints from the unlearning process.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ig708EWit4", "reviewer": "Reviewer_tXtq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces model tampering attacks, demonstrating how these attacks can predict and conservatively estimate the success of input-space attacks. This approach offers a new perspective for evaluating model safety, revealing potential vulnerabilities in traditional input-space attacks.\n2. The authors conduct wide experiments across different models, attack strategies, and evaluation metrics. By testing 65 models, the paper provides detailed quantitative results that showcase the performance differences of various attacks and defense strategies.\n3. The paper explores the relationships between tampering attacks and input-space attacks. It demonstrates that tampering attacks can predict the success of input-space attacks and help assess worst-case performance.\n\nWeakness:\n1. The paper primarily focuses on experiments conducted with the Llama-3-8B model, which may not fully capture the behavior of other models. The reliance on a single model restricts the generalizability of the results and raises concerns about whether the findings are applicable across a wider range of models.\n2. While the paper identifies correlations between model tampering and input-space attacks, it does not provide a comprehensive analysis or explanation of why these two types of attacks exhibit such relationships. The lack of in-depth mechanistic insight leaves an open question regarding the underlying factors that link model tampering with input-space attack success.\n3. The paper discusses the challenges of tampering attacks and the difficulty of achieving robustness against them, but it does not provide specific defense methods or approaches to enhance model robustness against tampering. This limits the practical applicability of the findings for mitigating these vulnerabilities.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes model tampering attacks as a complementary approach to input-space evaluations for assessing LLM risks. By modifying model weights or activations, tampering enables more rigorous stress-testing. The authors benchmark unlearning and safety fine-tuning methods, showing that tampering predicts input-space vulnerabilities, and harmful capabilities can be introduced with few fine-tuning steps.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors should add experiments on additional models, even if only simple validations, to support the generality of their core conclusions beyond LLaMA-3-8B.\n2. The paper would benefit from a discussion of potential defense mechanisms against model tampering attacks, which is currently missing.\n3. Figures should be placed closer to the relevant textual descriptions to improve clarity and readability.\n4. The visual clarity of figures needs improvement; for example, the precision of reported results in Figure 3 should be made consistent.\n5. Both the result plots and illustrative diagrams could be made more polished and visually refined to enhance the overall presentation quality.", "broader_impact_concerns": "The paper does not raise significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)