Tamper-Resistant Safeguards for Open-Weight LLMs
Authors: Rishub Tamirisa
Bhrugu Bharathi
Long Phan
Andy Zhou
Alice Gatti
Tarun Suresh
Maxwell Lin
Justin Wang
Rowan Wang
Ron Arel
Andy Zou
Dawn Song
Bo Li
Dan Hendrycks
Mantas Mazeika
Venue: ICLR 2025 Poster (2025)
Keywords: ai safety
large language models
tamper-resistance
unlearning
meta-learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce the first safeguards for LLMs that defend against a significant number of fine-tuning attacks.

Abstract: Rapid advances in the capabilities of large language models (LLMs) have raised widespread concerns regarding their potential for malicious use. Open-weight LLMs present unique challenges, as existing safeguards lack robustness to tampering attacks that modify model weights. For example, recent works have demonstrated that refusal and unlearning safeguards can be trivially removed with a few steps of fine-tuning. These vulnerabilities necessitate new approaches for enabling the safe release of open-weight LLMs. We develop a method, called TAR, for building tamper-resistant safeguards into open-weight LLMs such that adversaries cannot remove the safeguards even after hundreds of steps of fine-tuning. In extensive evaluations and red teaming analyses, we find that our method greatly improves tamper-resistance while preserving benign capabilities. Our results demonstrate that progress on tamper-resistance is possible, opening up a promising new avenue to improve the safety and security of open-weight LLMs.

Published as a conference paper at ICLR 2025
TAMPER-RESISTANT SAFEGUARDS FOR OPEN-WEIGHT LLMS
Rishub Tamirisa" "f 1.299 Bhrugu Bharathi
Long Phan', Andy Zhou 12, Alice Gatti', Tarun Suresh 1.2
Maxwell Lin", Justin Wang Rowan Wang Ron Arel
Andy 20u58,, Dawn Song', Bo Li Dan Hendrycks and Mantas Mazeika 129
'Lapis Labs, University of Illinois Urbana- Champaagn,,,,nn 'University California, San Diego, *University of California, Berkeley, *Carnegie Mellon University, Haavvrdd University, Uniierrity of Chicago, *Gray Swan AI, "Center for Al Safety
ABSTRACT
Rapid advances the capabilities of large language models (LLMs) have raised widespread concerns regarding their potential for malicious use. Open-weight LLMs present unique challenges, as existing safeguards lack robustness to tamper- ing attacks that modify model weights. For example, recent works have demon- strated that refusal and unlearning safeguards can be trivially removed with few steps of fine-tumine tine-tuning. These vulnerabilities necessitate new approaches for enabling the safe release of open- -eeigtt LLMs. We develop method, called TAR, for building tamper-resistant safeguards into open-weight LLMs such that adversaries cannot remove the safeguards even after hundreds of steps of fine-tuning. In exten- sive evaluations and red teaming analyses, we find that our method greatly improves camper-resistance while preserving benign capabilities. Our results demonstrate that progress on tamper-resistance is possible, opening up a promising new avenue to improve the safety and security of open-weight LLMs.
INTRODUCTION
The most capable open-weight large language models (LLMs) released over the past year now rival closed-source frontier models (Llama Team, AI @ Meta, 2024). The availability of LLMS for anyone to download and use has yielded numerous benefits, including lowering costs for end users and enabling academic research on safety and security (Zou al., 2023a). However, as these models become increasingly powerful, many have raised concerns that they could be repurposed by malicious actors to cause harm, motivating research on how to safeguard these models against malicious use.
Existing open-weight models often adapt safeguards designed for closed- weightt models served through APIs (Touvron al., 2023). These safeguards include refusal mechanisms and preference- based training, and they have provided substantial robustness against input-based based jailbreaking attacks. However, recent work has demonstrated these safeguards are trivially defeated by attacks that edit model weights, breaking down after only a handful of fine-tuning steps (Qi et al., 2023). This poses a serious problem for open-weight models, because adversaries have full access model weights and can tamper with built-in safeguards.
The vulnerability of open-weight models to tampering attacks poses risks for model developers as well. Under background tort law, AI developers must exercise reasonable care, meaning they have an obligation take reasonable precautions to prevent foresecable harm. If malicious actors can easily customize models to cause critical harm, model developers may inadvertently violate reasonable care
"Equal contribution. 'Equal advising. Correspondence to rishub22@@lliinoisccdh
Part of work done during an internship the Center for AT Safety.

---

Published as a conference paper at ICLR 2025
Launch
Existing Refusal Training
Brittle Refusal Mechanism
Harmful Fine-tuning
Successful Recovery
oo
Tamper-Resistant Training
Launch
Hamper-Resistare Defense
Harmful Fine-tuning
Unsuccessful Recovery
Harmless States
Harmful States
Refusal States
Figure 1: An illustration comparing two approaches to LLM safety when subjected to adversarial fine-tuning. The top branch shows conventional safeguards (like refusal training). which can be easily bypassed when adversaries fine-tune the model weights remove safety constraints. The bottom branch demonstrates our proposed method TAR (Tampering Attack Resistance), which maintains robustness even when adversaries attempt to fine-tune the model to reintroduce harmful capabilities.
standards and become open to liability under existing law. Thus, there is an urgent need for more robust safeguarding techniques that can withstand tampering attacks.
In this work, we study the problem of in Figure Unlike existing research on LLM safeguards, we focus on attacks that modify model weights, which we refer to as tampering attacks. This problem has been considered very challenging and by some intractable, as no method has yet provided substantial robustness to these attacks. However, making progress this problem would provide valuable tool to regulators and model developers by ameliorating the dual-use dilemma of aopen-weigt models (Miller & Selgelid, 2007).
safeguards for LLMs. This problem is depicted
To demonstrate that progress this problem is possible, we develop the first LLM safeguards that obtain strong robustness against a wide variety of tampering attacks. Our approach allows developers to add safeguard such that tampering attacks cannot easily remove the safeguard, while preserving the general capabilities of the LLM. We achieve this by performing adversarial training against tampering attacks, leveraging approaches from meta-learning. We identify various crucial factors that enable our method to work, including the choice of cumper-resistane loss, the selection of train-time adversaries, and the two-stage approach that we use for building in safeguards.
We apply our method to develop tamper-resistant unlearning and refusal safeguards. In experiments, we demonstrate that our safeguards are far more robust to tampering attacks than prior methods. We stress-test test our safeguards with extensive red teaming evaluations against 26 test-time adversaries, demonstrating resistance to fine-tuning attacks of hundreds of steps. We hope our results foster future work on this important problem. Our experiment code and models are available at ht https: / //gi thub. com/ shub- tami sa / taapper r res stance.
2 RELATED WORK
Adversarial attacks on LLMs. Due to the extensive pre-training distribution of modern LLMs, they are prone to generating harmful content (McGuffie & Newhouse, 2020; Sheng et al., 2019). To mitigate this, many LLMs undergo fine-tuning to implement safeguards (Bai et al., 2022; OpenAI, 2023; Touvron et al., 2023), using methods such as reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Ouyang al., 2022) and direct preference optimization (DPO)
2

---

Published as a conference paper at ICLR 2025
(Rafailov et al., 2023). While effective for normal use, these safeguards have been shown to be brittle, breaking down under jailbreak attacks (Jin et al., 2024a; Wei et al., 2023; Zou et al., 2023c) or a handful of fine-tuning steps "uncensored" data (Qi et al., 2023; Yang et al., 2023; Zhan et al., 2023). This suggests current techniques for LLM alignment are inadequate, raising security concerns after deployment.
LLM safeguards. Since the discovery of these attacks, many safeguards have been pro- posed to defend against them. Against jailbreak attacks, defenses include system-level defenses Inan et al. (2023); Jain et al. (2023); Robey et al. (2023); Yuan et al. (2024); Zhou et al. (2024)
Weaponization Knowledge Error)
Tamper-Resistance vs. Capabilities
100
80
that modify filter model inputs outputs and model-level defenses such as adversarial train- ing Mazeika et al. (2024). Alternatively, some works explore machine unlearning as a way to remove harmful knowledge entirely with tech- niques such as influence functions (Bae et al.,
(Post: (Post-Attact Best-Attack Camper-Resistenn
40
Prior Methods
TAR (Bl (Brocuuriiyy Durs]
(Chem. Security. Ours) (Cybersecurity,
2022; Koh & Liang. 2017), maximizing loss on forget sets (Eldan & Russinovich, 2023; Yao et al., 2023; Yu et al., 2023), or modifying repre- sentations (Belrose et al., 2024; Li et al., 2024; Sheshadri et al., 2024; Wu et al., 2023). How- ever, jailbreaking defenses adaptive adversaries Jin et al. (2024b); Liu et al. (2023), and existing unlearning methods are not robust to adversaries with access to model weights Lynch et al. (2024).
20
40 Capabilities (MMLU)
60 80
Figure 2: Comparison of our TAR method to 12 baseline safeguards. Unlike prior methods, TAR provides far greater tamper-resistance at similar levels of general capability, measured via MMLU. Tamper-resistance is computed as the normalized error on WMDP Biosecurity, Chemical Security, and Cybersecurity questions (Li et al., 2024), aver- aged across up 26 fine-tune finettunigg attacks.
not fully robust to
Robust safeguards. Several works have ex- plored the tamper-re of unlearning meth- ods for image classification (Golatkar et al.,
2020a;b; Tarun et al., 2023). For bidirectional BERT-style models, Henderson et al. (2023) proposed a meta-learning approach for robustly preventing models from learning harmful tasks. concurrent work, Deng et al. (2024) proposed method extending this approach to small-scale vision classifiers and diffusion models. Recently, Liu et al. (2024) discussed the potential for robust unlearning in LLMs to improve the safety of open-source models, and Lynch et al. (2024) proposed evaluation metrics for robust unlearning in LLMs. To the best of our knowledge, no methods have been proposed for autoregressive L.Mss that robust tampering attacks.
Several concurrent works have explored ways of defending LLM refusal mechanisms against fine- tuning (Huang et al., 2024a;c;b; Rosati al., 2024a;b). Huang al. (2024c) add perturbation loss to make an LLM learn to produce embeddings that are more invariant to perturbations, Rosati et al. (2024a) maximize prediction loss on harmful generations while minimizing loss on refusals, and Rosati et al. (2024b) regularize harmful representations to look random. Unfortunately. these works evaluate against small sets of fine-tuning adversaries or have limited robustness. We corroborate this in our comparisons, finding that the approaches in the latter two works lack robustness to the tampering attacks in our evaluations.
TAMPER-RESISTANE SAFEGUARDS
3.1 THREAT MODEL
We assume the defender releases an LLM with weights fG and safeguard G applied. The de- fender's goal is to design G such that 0g obtains high values on safety_metric((e) and capabilities meseric(cetr Moreover, the defender seeks to preserve high value of seffetymmerii(@ee after the adversary's 's move. We consider a compute-boune with unrestricted access fg, enabling attacks that directly modify fc. We refer to these "tam- pering attacks." The adversary's 's goal is to obtain a model 6' that minimizes the safety metric given reasonable compute limits, such as fine-tuning for 500 steps. We note that in this work, focus
adversary
3

---

Published as a conference paper at ICLR 2025
solely on fine-tuning adversaries and not input-space "jailbreaking" adversaries. We assume the adversary will not spend a significant fraction of the compute required to pre-train the LLM, since at that point they could train their own model without safeguards.
3.2 PROBLEM DEFINITION AND METRICS
We describe a general notation for quantifying the tamper-resistance of safeguards. Define G, fc. -ties_eetric as in the threat model. Let a attack denote compute-bounded adversarial attack that maps 0g to f6g, with stronger attacks obtaining lower values of safety_metric(G)) is high across broad range of strong test-time adversarial attacks Aust-
aa fety_metri c, and capabi itsies metric
We say that a safeguard is tamper-resistant if its post-attack
Note that 0g often modifies an underlying 0 that lacks safeguards, often through a fine-tuning procedure. Additionally, strong tamper-resistance writes with noise, but this model would no longer be useful. Thus, maintaining a high sapabilities metricc(e) is crucial, and evaluation of safeguard must consider both its tamper-resistance and how well it preserves general capabilities.
be obtained if the safeguard simply over-
We focus two common safeguard domains: weaponization knowledge restriction and harmful request refusal. In each domain, we define safety and capabilities test metrics, which we use alongside test-time adversaries to evaluate tamper-resistant safeguards.
Weaponization knowledge restriction. In wcaponization knowledge restriction, safeguards prevent the model from producing text about weaponization knowledge, while preserving capabilities for benign knowledge domains. Existing safeguards of this nature include representation engineering Cereet_mmtrrrreeeeeeee
methods like circuit breaking (Zou et al., 2024). The safety ____mette r c is defined as error on forget set, and the rapabilities latilliiieeeerriiiieeeeeee metric defined as accuracy on a retain set. Specifically, we consider the problem of restricting biosecurity, chemical security, and cybersecurity knowledge, and evaluate the resulting model on the Weapons of Mass Destruction Proxy (WMDP) benchmark (Li et al., 2024). WMDP contains 3,668 multiple-choice questions, spanning biosecurity, chemical security, and cybersecurity knowledge. Importantly, WMDP questions do not evaluate hazardous knowledge directly, but instead measure proxy expert-level knowledge for each hazardous domain, such that restricting the expert-level knowledge would also restrict the hazardous knowledge. We define the forget as the respective hazardous knowledge subject in WMDP, and retain as the complement of the given subject in MMLU (Hendrycks et al., 2021), multi-task question-answering benchmark spanning 57 tasks across a variety of knowledge domains.
Harmful request refusal. In the harmful request refusal setting, safeguards prevent the model from producing "harmful" outputs. We define the sa satyy feeey_eettritttteeeeeeeeeee metricc c as the complement of average Attack Success Rate (ASR) of various jailbreaking attacks, while the capabilit iopiiiitess sasees_etrrrr_eee metric captures the conversational abilities of 0g. Specifically, we use static set of test cases from HarmBench, an automated red-teaming framework for measuring p jailbreak robustness in LLMs, to evaluate jailbreak ASR (Mazeika al., 2024) after tampering attacks. We use MT-Bench, multi-turn benchmark graded by an LLM judge, to evaluate conversational abilities (Zheng al., 2023a).
3.3
RED TEAMING
To properly measure the robustness of tamper-resistant safeguards, we conduct red- red-teaming with up to 26 adversaries, including many that are unseen at training time. In our evaluations, we subject our method to adversaries with varying compute budgets, access to held-out datasets, and diverse hyperparameters. For fine-tuning adversaries, we vary the learning rate, learning rate scheduler, optimization algorithm, and batch size. Many these adversaries were fixed during early experiments, with some added over time as found attacks that broke intermediate versions of our method. Extensive stress testing of this nature is critical for obtaining confidence in a tamper-resistant safeguard. For research developing these safeguards, extensive red teaming also allows measuring incremental progress, using the number and strength of existing attacks one can defend against as a robustness metric.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 TAR: Tampering Attack Resistance
Input: Initial LLM parameters 0, train-time adversary set Amain, capabilities_ proxy dataset retainn sa fetyymetrrii proxy dataset TRR, outer steps N, learning rate 1), number of sampled adversaries K, tamper-resistanee loss scale ATR, retain loss scale Aretann hg(-) returns the residual stream hidden states for model parameters
0o f- Apply Initial Safeguard to 0 for i = to N do
9TR <- # For accumulating tamper-resistance gradient
Sample 2TR DTR
for 1 K do
Sample attack ~ Auain
# Tamper-resistance loss from Equation 1
9TR c 9TR + K V ( end for
Sample Ty Dietain
RepE retain loss from Equation?
Gretain Vo,-, (0,(0,-1,27) I1 (2:T) ho ho (11))
# Full tamper-resistance update
0, - 0:-1 11 ÀTR 9TR + Aretain Gretain
end for
0g t N return 0g
4 SAFEGUARD COMPER-RESISTANCE TRAINING
To obtain tamper-resistant safeguards, we propose a new method outlined in Algorithm inspired by adversarial training and meta-learning to directly strengthen LLM safeguards against tampering attacks, called Tampering Attack Resistance (TAR). We identify unique properties of this adversarial training regime and leverage them improve robustness.
Our method for training tamper-resistant safeguards consists of two phases: (1) model safeguarding and (2) tamper-resistance training.
4.1 MODEL SAFEGUARDING
The method begins by including initial safeguard G into base model 0. For example, initial safeguards for knowledge restriction can be drawn from a wide variety existing methods, including circuit breaking (Li et al., 2024; Zou et al., 2024) constrained gradient ascent for particular knowledge domain. Similarly, we can include a refusal safeguard by performing RLHF (Ouyang et al., 2022) DPO (Rafailov et al., 2023) refusal completions. Importantly, these initial safeguards do not need to be tamper-resistant. Empirically, we find that this safeguarding step is crucial for preserving low pre-attack sa fety_metric..
4.2 TAMPER-RESISTANE TRAINING
Starting from AG., we train the tamper-resistant 6g using novel adversarial training procedure. Namely, we train against a set of tampering attacks Ausin where the defender's objective to maximize a proxy sa Since may not be feasible differentiate through attack, we draw on insights from prior work in meta- metannearning, defining at attack(Oc) 8'2 0g + attack' (0c) perturbation on top of initial parameters, where backpropagation through attack' is approximated with straight- estimator (Bengio al., 2013).
after applying adversarial attack a attack ~ Arain to O.
We focus on supervised fine-tuning (SFT) adversaries where attack applies several steps of optimization to 8g. which allows straight- ttrhouhhh estimation through attack' to benefit from the setting and approximations of first-order MAML (Finn et al., 2017). However, we note key differences in our approach from standard meta-learning and prior methods (Finn et al., 2017; Henderson et al.,
5

---

Published as a conference paper at ICLR 2025
Prior Method (Maximizing Cross-Entropy Loss)
(Maximizing Entropy Loss)
Tampar Rasietance Defender Lass
During
Self Dell-esttuuiing Defender Loss
Best Region
Man E16
Loop
Maxifie
Steps 64
Stops
Max One
84
Steps 64
Stups 64
Adb Stops
Tamper Resistance Steps
Tamper Resistance Steps
Figure 3: The choice of ramper-resistance loss is crucial for obtaining good performance. Here, we show loss trajectories when the tesperrre loss is negative cross-entropy (left), versus negative entropy (right), over the course of TAR for 750 steps. Outer loop losses (blue) are reduced by the defender, and inner-loop losses (red) are reduced by the train-time adversary. When the tamper-resistance loss maximizes cross-entropy (left), the adversary is only affected earlier in its trajectory and quickly recovers. By contrast, when the tamper-resistance loss maximizes entropy (right), the inner loop adversary is eventually thwarted along its entire trajectory. Plots smoothed.
2023). In particular, traditional meta-learning techniques seek to obtain model initialization that is close to optimality on multiple test distributions. In our setting, we seek to obtain an initialization that is far from optimality on multiple adversaries' test distributions. Novel to our approach in this new setting is the use of a samper-resistane loss in the "outer loop" that differs from the fine-tuning adversary's 's loss function and serves to maximize the proxy safety metric. We depict this structure in Algorithm 1, and explain the objective below.
Impeding the adversary's loss. The aim of tamper-re training is to prevent adversaries with large compute budgets from reducing the safety_metrie at test-fime. In adversarial training for ramper-resistance, we define a tamper-resistance loss LTR that counters attack. We opera- tionalize our goal of avoiding adversary optimality as searching for A such that LıR is minimized for at attack(O).
Empirically, we find that the choice of tamper-resistance loss LTR significantly affects this goal. Prior work (Henderson al., 2023) negates the loss of a fine-tuning adversary, in which the aim is to arbitrarily maximize the adversary's 's loss throughout fine-tuning. This formulation has two issues: (1) maximizing cross-entropy loss can cause divergence; (2) empirically we observe that when using this objective against fine-tuning adversaries, the model learns to explode the adversary's 's loss for the first few inner loop steps, while loss at later steps remains low. In Figure 3, we show the difference in choosing LiR to be clamped negative cross- cross-entropy loss negative entropy loss for wcaponization knowledge restriction. For the latter, LIR is eventually satisfied for all inner loop steps. For harmful request refusal, we choose LTR to be the DPO loss (Rafailov al., 2023). We provide further detail on the choice of LTR in both settings in Appendix C.3, as well as an extended depiction of the test-time loss characteristics in Appendix D.3 and Figure 6.
Tamper--esistance objective. We now describe the general proxy objective used for preventing a tampering attack from recovering weaponization knowledge or harmful behavior. For a given sa peety_metrtt c, let DTR and LTR respectively be dataset and loss function such that minimizing LTR DTR) serves as a proxy objective for maximizing the sa pesty_metrit c(fg). We define Dretain and Cretain correspondingly for to solve the following optimization problem:
(ew.iitiee_merric(e)) The defender's objective is
min ÀTR EattackkAAm.. (attackk))) DTR]] + A Cretain (0; Dretain)
where CTR is a tamper-resistance loss that counters at ac). The Ceetain term is a representation engineering (Zou et al., 2023a) inspired retain loss for preserving performance on the capabilities proxy dataset Dretain given by
Cretuin (f; Dretiin) E.g.Danis
(0, x) + [[ho(x)) hoa, (1)1
where (-) returns the residual stream hidden states for model parameters and LIM is the standard language modelling cross-entropy loss. Empirically, we find that pushing retain-set residual stream

---

Published as conference paper at ICLR 2025
Pre-Attacks
Post-Attacks (Avg)
Weaponization Domain
Model
Retain (1)
Forget (L)
Forget (L)
Random
25.0
25.0
25.0
No Defense
67.3
70.5
70.5
Max Entropy
65.0
33.2
65.8
Biosecurity
Min Posterior
65.6
50.4
66.0
LLMU
65.5
29.9
61.3
RMU
65.8
31.2
64.9
TAR (Ours)
54.7
28.1
35.2
Random
25.0
25.0
25.0
No Defense
68.2
47.8
47.8
Max Entropy
67.5
50.0
45.7
Chemical Security
Min Posterior
66.8
49.5
47.5
LLMU
67.0
30.1
44.3
RMU
67.6
27.5
46,0
TAR (Ours)
56.5
28.4
27.1
Random
25.0
25.0
25.0
No Defense
68.2
46.4
46.4
Max Entropy
66.5
28.7
41.7
Cybersecurity
Min Posterior
66.6
41.8
41.9
LLMU
66.1
27.6
41.3
RMU
66.8
29.5
42.2
TAR (Ours)
60.7
23.6
28.6
Table 1: Pre-Attack and average Post-Attack accuracies for WMDP Biosecurity, Chemical Security, and Cybersecurity for TAR and all other baselines, reported for Llama-3-8B. The average Post-Attack accuracy is computed as the average accuracy across the 26 fine-tuning attacks discussed in Section 5, averaged over multiple seed repeats. TAR is the only method that maintains low Post-Attack recovery while preserving high Retain MMLU and low Forget accuracies. All values are percentages.
representations to be close to the base model fG. via the morm loss in Equation maintains high
Equation we include ÀTR and Aretain as scalar weightings for the tamper-resistance loss and retain loss, respectively. We provide further details the design of the tamper-resistance loss function in Appendix C.3 as well as an efficiency trick for sampling fine-tuning attacks for TAR in Appendix C.4.
EXPERIMENTS
We evaluate TAR in weaponization knowledge restriction and harmful request refusal settings, with results shown in Table 1 and Table 2 respectively. We discuss the setup, baselines, and analysis for our results. each setting, we use a specific set of training adversaries Arain and test adversaries Aust- Further experiment details presented in Appendix E.
5.1 WEAPONIZATION KNOWLEDGE RESTRICTION
We now describe the setup, baselines, and results for our weaponization knowledge restriction experiments, including the knowledge domains, optimizers, and evaluation details.
Setup. We focus implementing tamper-resistant safeguards for restricting proxy weaponiza- tion knowledge about biosecurity, chemical security, and cybersecurity from Llama-3-8B- (AI@ Meta, 2024) that has been initially safeguarded via the Random Mapping method discussed in Appendix C.2. For each weaponization domain, we assign DTR to the corresponding forget set described in Appendix E. We proceed to sample train-time 64-step fine-tuning attacks from differ- ent data distributions, detailed in Appendix E.2. We use
750 outer loop steps, ScheduleFree Adam (Defazio al., 2024) with a learning rate of 2 10 the outer loop ramper-resistance optimizer. For biosecurity and cybersecurity we set the tamper-resistanee loss scale ÀTR to 4.0, and use ÀTR 3.0 for chemical security. We use Areuin 1.0 in all settings. Lastly, evaluate

---

Published as a conference paper at ICLR 2025
71 (ND)
Post-Attack Accuracies for Biosecurity Weaponization Restriction Methods
Max Entropy
66
56
58
72
67
67
65
68
59
Method
Min Posterior-
66
58
72
71
68
68
65
48
63
27
30
65
65
63
36
66
67
TAR Blo
24
25
25 (RC)
AdV
ADV
2
AdV
3
Adv 4
10
Adv
Adv
Adl
Adk
Adv
Adv
Adv
13
Adv
Adly
Adb
26
48 (ND)
Post-Attack Accuracies for Chemical Security Weaponization Restriction on Methods
Max Entropy
46
43
Method
Posterior
49
51
52
37
Accumayy
LLMU
48
46
45
48
25
41
41
52
45
25
33
46
48
50
46
50
50
48
45
36
43
47
50
RMU
48
37
50
44
46
29
46
45
46
44
50
51
51
51
48
41
49
46
49
37
48
40
48
51
31
TAR Chem
25
23
24
38
23
29
30
24
25
24
23
30
25
23
25
25
26
23
24
23
38
32
25 (RC)
Adv
AdV
Adv
Adv
Adv 5
AdV
AdV
Adv
AdV
14
AdV
18
AdV
AdV 24
Attt-Attack Accurac les for Sybersecuue urity Weaponizal Paaiiaaiiiin Restr tion Methods
(ND)
Max Entropy
40
Method
Posterior
26
41
40
40
35
LLMU
43
43
41
42
41
39
40
42
42
35
24
42
42
42
40
43
42
42
40
41
44
46
RMU
42
39
41
34
43
37
46
45
43
40
44
44
45
45
41
30
42
43
39
34
42
41
45
45
30
TAR Cyber
28
25
30
31
27
28
24
28
32
24
26
24
26
29
25
33
36
28
24
26
24
36
34
25 (RC)
Adv
AdV
Adv
Adv
Adv
AdV
AdV
Adv
Adv 9
Adv 10
Adv 11
Adv 12
Adv 13
14
Adv
Adv 16
Adv 17
Adv 18
Adv 19
Adv 20
Adv 21
Adv 22
Adv 23
Adv 24
Adversary
Figure 4: Red teaming results across weaponization domains. Values show percentages, with Random Chance (RC) at 25% and "ND" indicating No Defense WMDP scores. Red indicates attack performance approaching No Defense levels. We evaluate each defense against a diverse range of strong adversaries described in Appendix F.1), Accuracies are reported as averages over 3 repeats of each attack with different seeds. Compared to prior safeguards, TAR greatly increases lamper-resistance for nearly all adversaries.
Pre-Attack and Post-Attack accuracy on corresponding WMDP subjects (Li al., 2024) averaged across all adversaries in Appendix F.1, and measure benign capabilities via the complement of subjects related each proxy weaponization domain in MMLU (Hendrycks et al., 2021).
Baselines. We evaluate two recently proposed knowledge restriction methods: RMU (Li et al., 2024) and LLMU (Yao al., 2023). We also design two baseline methods for knowledge restriction: Min Posterior, which minimizes posterior loss on forget set tokens; Max Entropy. which maximizes entropy on forget set tokens. Two additional methods, MLAC (Henderson et al., 2023) and SOPHON (Deng et al., 2024), require substantial modifications for the I.IM setting, so we show results on adapted versions of these baselines in Appendix G.3.
Results. We show weaponization knowledge restriction safeguard results on Slama-3--B-ima in Table and Figure 2. These results are averaged across all adversaries described in Appendix F.I. Our large-scale experiments corroborate the findings in recent work that existing LLM safeguards are extremely brittle to fine-tuning attacks. By contrast, TAR maintains low post-attack forget accuracy across all three domains. However, we observe that TAR lowers retain accuracy by 10.6% on average, indicating a trade-off between benign capabilities and robustness. In Figure 4, we observe that TAR robust to significantly more fine-tuning attacks than all prior methods. While existing baselines break down under most attacks, TAR obtains ptst-attack forget accuracy random chance for nearly all attacks, indicating a successful defense.
Overall, we find that TAR provides significantly more robustness to realistic fine-tuning attacks than all prior methods, including SFT attacks that utilize completely held-out data. We also include further analysis of TAR's 's test-time loss behavior in Appendix D.3, in which empirically observe TAR's 's convergence and robustness. These results demonstrate for the first time that obtaining strong tamper-resistance for open-weight L.MSs may be possible.
8

---

Published as a conference paper at ICLR 2025
Refusal Trained
R2D2
RepNoise
RR
TAR (Ours)
Pre-Attacks MTI-Bench (1)
8.1
6.0
6.2
8.0
6.3
Avg. Post-Attacks ASR (L)
72.5
78.3
74.5
84.8
63.9
Table 2: Average Post-Attack HarmBench ASR, reported for TAR. Representation Rerouting (RR), and the Refusal Trained Llama-3-88 ------------n-te model across 5 fine-tuning attacks depicted in Appendix F.2, as well as Pre-Attack MT-Bench. TAR is more robust than other methods after tampering, while maintaining comparable MT-BBecch performance. ASR values are percentages.
5.2 HARMFUL REQUEST REFUSAL
We now describe the setup, baselines, and results for our harmful request refusal experiments, including the datasets used and evaluation details.
Setup. For harmful request refusal training, seek to make existing refusal safeguards in Llama- 3-8B-Instruct robust to tampering attacks. We sample train-time adversaries that perform 64-step SFT attacks using the Anthropic-HH- HHH-LHF dataset (Bai al., 2022), following the methodology in Appendix E.2. Similar to the weaponization knowledge restriction setting, we use N 100 outer loop steps, ScheduleFree AdamW W (Defazio et al., 2024) with an LR of 10-5 as the outer loop lamper-resistance optimizer, and loss scales of ÀTR 0.1, Areaain 1.0. We evaluate the Post-Attack jailbreak attack success rate (ASR) on HarmBench (Mazeika et al., 2024) after the tampering attacks in Appendix F.2, and measure benign capabilities preservation via MT-Benel (Zheng et al., 2023b), which evaluates multi-tun multi--urnaan conversation ability.
Baselines. We consider 4 baselines alongside our TAR model: Lllam-3--B-Insttuu (Refusal Trained); Representation Rerouting (RR) (Zou et al., 2024) Llama-3-8B -aama-3-BBIInstruet,, which trains to push representations for harmful inputs to be orthogonal to the original representations in Llama- In8t-Instruet; R2D2 (Mazeika et al., 2024) on Zephyr-7B 7 (Tunstall al., 2023), which performs adversarial training against GCG attacks (Zou et al., 2023b); and RepNoise (Rosati et al., 2024b) on Llama-2-7B (Touvron et al., 2023), which regularizes harmful representations to noise.
Results. We show refusal results in Table 2. While the Refusal Training, RR, and R2D2 baselines resist jailbreak attacks in HarmBench before tampering, we find that percentage attack success rate jumps up to above 77 after tampering, while our TAR method only rises to 61.7. Since apply our TAR refusal safeguard to Llama-3-88, it does reduce MT-Bench by 1.7. However, this exceeds the MT-BBench score of fairly capable open-weight models, indicating that benign capabilities are largely preserved. We leave the exploration of the full impact on capabilities to future work. Additional results are in Table 11. In general, we find that our TAR model refuses more Post-Attack jailbreaks than previous baselines, and demonstrates the flexibility of the tamper-resistance objective to accommodate the harmful request refusal setting.
5.3 ANALYSIS
Red teaming. To assess the tamper-resistance of our models, we conduct an extensive suite of supervised fine-tuning attacks with 26 distinct adversaries in the Biosecurity setting and 24 distinct adversaries in the Chemical Security and Cybersecurity settings. We vary the optimizer, number of optimization steps, learning rate, learning rate schedule, fine-tuning dataset, batch size, and overall fine-tuning method (e.g., full fine-tuning versus parameter-efficien fine-tuning). By default, our attacks use 500 fine-tuning steps. Full details for these adversaries are provided in Table 9.
We show red teaming results in Figure 4. While baseline safeguards withstand fine-tuning attacks in a small number of cases, most adversaries succeed in removing the safeguards. By contrast, our TAR safeguard is robust to a wide range of adversaries. This shows that tamper-resistance is tractable problem on which progress can be made. However, we find our method exhibits varying robustness to parameter-efffiient fine-tuning (PEFT) and some nut-of-distribution LR attacks, highlighting the current sensitivity of the method to the adversary distributions sampled during TAR training. These findings reinforce the importance of extensive red teaming when developing camper-resistane defenses to reveal the scope of their protection. We hypothesize that future work could easily address these limitations, as we demonstrate in Appendix D.2 that targeted patching of vulnerabilities is possible.

---

Published as a conference paper at ICLR 2025
As mentioned in Section 3.1, the threat model we consider involves SFT weight tampering adversaries and not imput-space "jailbreaking" adversaries, nor does TAR explicitly optimize for input-space robustness (Mazeika et al., 2024; Zou al., 2023e; 2024). Nonetheless, forget accuracies are comparable to baselines Table 1, and believe that explicitly defending against input-space attacks alongside tampering attacks would be a good direction for future work.
find that TAR's 's pre-attack
CONCLUSION
We introduced a novel method for implementing tamper-resistant safeguards for LLMs and explored applications in weaponization knowledge restriction and harmful refusal training. We compare results to prior work in each setting, finding that our method is the first method robust under the rigorous red-teaming evaluation that we consider. More broadly, we demonstrate that progress open-wer been-weight sumper-resistance is tractable. We believe this line of research is crucial for enabling on- going deployment of robust, open-weight weight LLMs, ensuring their alignment with regulatory frameworks and preemptively addressing the risk of malicious use.
Acknowledgements. We thank Steven Basart and Luis Fernandez for providing valuable feedback for the paper, well as Xiangyu Qi, Boyi Wei, Nicholas Carlini, Prateck Mittal, and Peter Henderson for useful discussions. We also thank Andriy Novykov and the Center for AI Safety for providing significant compute resources for this project, well Volodymyr Kindratenko and the National Center for Supercomputing Applications (NCSA) and Illinois Campus Cluster Program (ICCP) for supporting our computing needs. This work used NVIDIA GPUs at NCSA Delta through allocations CIS230117 17 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF Grants #21 #2138286, #2138307, #2137603, and #2138296.
REFERENCES
AI@ Meta. Llama 3 model card. 2024.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger Baker Grosse. If influence functions are the answer, then what is the question? ArXiv, abs/2209.05364, 2022.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. wwXiv:2204.05863 2022.
Nora Belrose, David Maaneider-Josephle Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36, 2024.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.
Tianchi Cai, Xierui Song, Jiyan Jiang. Fei Teng, Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with human demonstration and wise preference, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
CTFtime. Ctftime writeups archive. https://ctftee
wwwp/writgeg..... http:://ccttttme.oorrrrrtteesss 2024.
Aaron Defazio, Xingyu, Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky, The road less scheduled, 2024.
Jiangyi Deng, Shengyuan Pang, Yanjiao Chen, Liangming Xia, Yijie Bai, Haiqin Weng, and Wenyuan Xu. Sophon: non-fine-tunable learning to restrain task transferability for pre-trained models, 2024.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in 1lms. ArXiv, abs/2310.02233 2023.
10

---

Published as a conference paper at ICLR 2025
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leamiigg for fast adaptation of deep networks, 2017.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision-ECCC ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXIX 16, pp. 383-398. Springer, 2020b.
Peter Henderson, Eric Mitchell, Christopher D. Manning, Dan Jurafsky, and Chelsea Finn. Self- destructing models: Increasing the costs of harmful dual uses of foundation models, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get for free, 2017.
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation, 2024a. URL https: /ats:v///aavvve//a orgg abs 2409. 01586.
Tiansheng Huang, Sihao Hu, Fatih IIhan, Selim Furkan Tekin, and Ling Liu. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack, 2024b. URL https: //arxiv.org/altt abs////b///// www.aa/44400 2405.11 18641.
Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-awwe alignment for large language model, 2024c.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.
Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023.
Haibo Jin, Ruoxi Chen, Andy Zhou, Yang Zhang, and Haohan Wang. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models, 2024a.
Haibo Jin, Andy Zhou, Joe D. Menke, and Haohan Wang. Jailbreaking large language models against moderation guardrails via cipher characters, 2024b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, 2017.
Solomon Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:79-86. 1951.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society, 2023.

---

Published as a conference paper at ICLR 2025
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm- Burgerr Rassin Lababidi, Lennart Justen. Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert- Cort Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Liu, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz. Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The wmdp benchmark: Measuring and reducing malicious use with unlearning, 2024.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint urXiv:2402.08788 2024.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv:2310.04451, 2023.
Llama Team, Al @ Meta. The Ilama 3 herd of models, 2024.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. CoRR, abs/1608.039883, 2016.
Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Mened Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402. 16835, 2024.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standard- ized evaluation framework for automated red teaming and robust refusal. In ICML, 2024.
Kris McGuffie and Alex Newhouse. The radicalization risks of gpt-3 and advanced neural language models, 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Seumas Miller and Michael J Selgelid. Ethical and philosophical consideration of the dual-use dilemma in the biological sciences. Science and engineering ethics, 13:523-580, 2007.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yo Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023.
Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. On evaluating the durability of safeguards for open- open-weight Ilms, 2024. URL https: atrpaaxxvvvv../..//.aaaa..a.a. httpggg/ab////1/// 220.27722277......
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Neural Information Processing Systems (NeurIPS). 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1- 2020. doi: 10. 101100999144400 405. 202000002
12

---

Published as a conference paper at ICLR 2025
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang. Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Offloa: Democratizing Billion-Scale model training. In 2021 USENIX Annual Technical Conference (USENIXATCCII, pp. pp. 551-564. USENIX Association, July 2021. ISBN 978-1 178-1-939133-23-6
Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. Smoothllm: Defending large language models against jailbreaking attacks, 2023.
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszeze, Jan Batzner, Hassan Saj- jad, and Frank Rudzicz. Immunization against harmful fine-tuning attacks, arXiv preprint arXiv:2402. 16382, 2024a.
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszeze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation noising effectively prevents harmful fine-tuning on llms, 2024b.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield- Hadeield-Menell, and Stephen Casper. Targeted latent adversarial training improves robustness to persistent harmful behaviors in Ilms, 2024. URL
fittps://arte
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXitv2337.092288 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023. URL https: //t////axivvi httpg//aa/////0/0 16944.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data, 2023.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? In Neural Information Processing Systems (NeurIPS). 2023.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models, 2023.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned lims with nothing. 2024.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models, 2023. URL https::/arrxe otg/abss////ooooooooo.oososss.br 2210 02949.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. ArXiv, abs/2310.10683, 2023.
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu. and Heng Ji. Unlearning bias in language models by partitioning gradients. In Annual Meeting of the Association for Computational Linguistics, 2023.
13

---

Published as a conference paper at ICLR 2025
Zhuowen Yuan, Zidi Xiong, Yi Zeng. Ning Yu, Ruoxi Jia, Dawn Song. and Bo Li. Rigorllm: Resilient guardrails for large language models against undesired content, 2024.
Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, 1b1/1222,5 2012.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning, 2023.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt mt-bench and chatbot arena, 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph F. Gonzalez, and Ion Stoica. Judging llm-ae-a-judge with mt-bench and chatbot arena, 2023b.
Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks, 2024.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023a.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023b. URL https:////// //l:///aaaa //arxiv. org/abs/// 2307 15043.
Andy Zou, Zifan Wang. J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arX1v:2307.15043, 2023c.
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredríkson, and Dan Hendrycks. Improving alignment and robustness with short circuiting. arXiv preprint arXiv:2406.043// 44331433.4, 2024.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
Our method for training tamper-resistant safeguards demonstrates considerable robustness against a wide range of tampering attacks, yet several avenues for improvement remain: (1) While focus on supervised fine-tuning attacks, the broader spectrum of open-weight tampering techniques necessitates diverse future red-teaming efforts. (2) Scaling to larger models poses computational challenges that require optimization to reduce overheads.
Additionally, in cases where TAR maintains low post-attack forget accuracy, the post-attack retain accuracy is also low. By contrast, we found in preliminary experiments that post-attack retain accuracy for many of the baselines remained high. We note that this is acceptable because post-attack retain performance is not of concern to the defender; rather, the responsibility falls on the attacker to preserve it after tampering.
However, this does mean benign users trying to fine-tune the model must ensure their data is not contaminated by forget set data, lest their fine-tuned model have poor performance. This could make the method harder to use in practice. Thus, maintaining high post-attack retain accuracy would be useful direction for future work to explore.
Tamper-resistance alone cannot fully mitigate the risks of malicious AI use. While it raises the initial costs for adversaries, it can eventually be circumvented. Once open-weight models are released, they cannot be "unreleased," leaving any compromised defenses permanently vulnerable. Therefore, tamper-resistance should be considered a supplement to the broader effort of of improving the offense- defense balance of Al systems. Addressing these limitations will improve the robustness of LLMs to tampering and better support open-weight model developers.
B AUTHOR CONTRIBUTIONS
Rishub Tamirisa, Bhrugu Bharathi, and Mantas Mazeika led the project, including developing methods and contributing writing for all sections of the paper. Rishub Tamirisa implemented most of the code for the method and baselines, and Bhrugu Bharathi implemented most red-teaming evaluations in the paper. Rishub and Bhrugu carried out all main experiments, and orchestrated analysis experiments for Maxwell Lin and Tarun Suresh to conduct. Rowan Wang and Justin Wang curated datasets for main experiments for the final version of the method. Long Phan and Alice Gatti implemented red-teaming evaluations, contributed figures, and gave feedback for writing drafts. Andy Zhou ran experiments for robust refusal, contributed writing the Related Work section, and helped create figures. Ron Arel organized compute logistics from the NCSA, and provided personnel for experiments. Andy Zou provided personnel for experiments and high-level advising. Dawn Song and Bo Li provided high-level advising for the project. Dan Hendrycks provided substantial advising for framing and writing, as well as method suggestions. Dan Hendrycks also provided significant compute resources from the Center for AI Safety. Mantas Mazeika was the primary project advisor during the majority of its duration, and was involved in most decisions regarding the method, experiments, and writing.
15

---

Published as a conference paper at ICLR 2025
C METHOD DETAILS
C.I NOTE ON UPDATED IMPLEMENTATION
After the initial release of this paper, we identified data contamination issue in which our instruction- tuning retain dataset, Magpie-Align (Xu et al., 2024), contained significant amount of forget set content. This resulted in an unintended input-space vulnerability (Qi et al., 2024). After cleaning the dataset and re- re-training new TAR models with the same methodology and hyperparameters, the input-space vulnerability no longer observed (Qi et al., 2024).
Additionally, Qi et al. (2024) found that TAR's robustness 1,000 of steps of fine-tuning varies when changing the dataloader shuffling order, which we corroborate in Figure After fur-
Adversary Test Loss across Different Seeds
35
Cross-Entropp Enns--Etrpppyy Loss (Nats)
30
ther investigation, found an issue in the Hug- gingFace distributed dataloading sampler, where the HuggingFace sampler overrides user-defined seeds with a default seed. This resulted in the test-time dataloader shuffling orders being sub-
25-
20
15-
set of train- time shuffling orders in our initial release. Upon correcting the issue, observe the variability in Figure 5.
10
However, while the loss plateau and robustness at 1,000 steps can vary with different dataloader shuffling orders, we observe in Figure 5 and in nearly all adversaries that the loss plateau and robustness we discuss throughout the pa- per remains consistent across multiple shuffling orders for 500 steps of fine-tuning. We report these results in updated Table Figure 2, and Figure 4. All depicted per-adversary post-attack accuracies Table 1, Figure 2, and Figure 4 are averaged over three replicates, each using a different random seed. We include further discussion on the loss plateau as well as concrete evidence of TAR's 's convergence in Appendix D.3. We do emphasize that TAR models we consider our experiments optimize against SFT adversaries that use only K- = 64 optimization steps yet achieve significant generalization to test-time adversaries using substantially more compute, greatly improving tamper- compared prior work.
SFT Attack Steps
Figure 5: Test loss for five repeats a 1,000-step SFT attack against TAR-Bio safeguard, each us- ing a different dataloader shuffling seed. TAR yields consistent loss plateau for 500 steps, fol- lowed by a loss region of increased variability.
C.2 INITIAL WEAPONIZATION KNOWLEDGE RESTRICTION SAFEGUARD
Prior to tamper-resistance training, we install safeguard that achieves surgical knowledge restriction on the target hazardous domain. Let (D) denote the distribution post-decoder layer residual stream activations for input sequences sampled from some data distribution D and model weights 0. We define rand has (r)) for some input sequence x, which returns fixed Gaussian-sampler vectors that are chosen via hashing the corresponding input token for each residual stream index of in 0. As a proxy for scrubbing target representations according to downstream task labels, we propose weaponization knowledge restriction safeguard termed Random Mapping, which maps he (DIR) to random noise as follows:
ho(2) rand rand_has (aante)]]lran___hhheeeee)))
min E DeR
(0; Pretiin)
The objective of Equation 3 maximizes cosine similarity between row vectors in the residual stream every layer of the LLM from 1(DTR) and the hashed random vectors from rand hand_hashed({ By providing each token's residual stream a unique random vector to push toward, the loss encourages "re-mapte reemaappng"" of token representations from DIR to the noised vectors. We include an additional term for preserving performance on D Dretin via the language- modelling cross-entropy loss LIM- We show the performance of the raw Random Mapping safeguard as an ablation in Table 5, listed as "Excl. Adv. Training."
16

---

Published as a conference paper at ICLR 2025
C.3 DESIGNING THE SEPPER-RSISSTNCE LOss
Weaponization knowledge restriction. For weaponization knowledge restriction, we summarize our intuition for ideal tamper-resistance loss design, corroborated by our empirical findings in Figure 3: we seek to flatten the adversary's 's loss at high value, rather than simply raise its y-intercept.
We choose the tamper-resistance loss LTR as an entropy loss to be maximized during the adversary's s cross-entropy fine-tuning trajectory, since maximizing entropy would impede the adversary's s cross entropy loss from decreasing during fine-tuning. In other words, we wish to obtain such that after an adversary performs fine-tuning attack on 9 via enosopy loss, entropy is still high. We find that this formulation achieves the desired flattening behavior, and we depict the difference in flattening between the choosing LTR be negative cross-entropy loss and negative entropy loss in Figure 3. In the lefthand plot, where LTR is cross-entropy loss, loss only increases in the first inner loop step. In the righthand plot, where LTR is a negative entropy loss, entropy is eventually maximized in all inner loop steps. Figure 6 also demonstrates the generalization of the flat adversary loss behavior beyond the length of the simulated fine-tun ane-tuning trajectories during TAR.
Harmful request refusal. For harmful request refusal, we choose CTR to be the DPO loss (Rafailov et al., 2023), which works follows. Given DPO dataset containing pairs of rejected and refusal completions, the sampled attack performs SFT on rejected completions, and the tamper- resistance loss CTR is a DPO loss computed on the pair chosen and rejected completions on parameter coordinates along the attack trajectory. This encourages TAR to find an initialization 0 such that after harmful fine-tuning attack, the model still prefers refusal completions over harmful completions when given an harmful prompt. While this does not necessarily encourage a flat adversary loss, we find empirically in Table 2 that this formulation increases the average safety_mm safety_metric/// (mm_eertic(()) defined in Section 3.2 after fine-tuning attacks on harmful data, detailed in Section 5.
We also observe that the length of the simulated adversary SFT trajectory during training affects test-time generalization in both Figure 6 and Appendix D.3. In particular, larger values of K result in increased tamper-resistance for longer SFT attacks. However, to maintain reasonable runtime efficiency, we need a more efficient sampling technique than simply running K independent trajectories of varying length for every outer-loop step in Algorithm 1, which we describe in Section C.4.
C.4 EFFICIENTLY SAMPLING FINE-TUNING ATTACKS
Optimizing Equation with gradient descent requires simulating K tampering attacks for each tamper-resistance optimizer update, which is prohibitively expensive to run when the sampled attack performs SFT and contains billions of parameters. Inspired by prior work snapshot ensembles Huang et al. (2017), we leverage an efficiency trick: we can reuse the coordinates along steps of single adversary fine-tuning trajectory of length K to obtain K additional (though non-independent) trajectories of increasing length. Using this trick, we collect all K parameter coordinates along the trajectory into single batch for computing the tamper-resistance losses, effectively sampling attack from Arain non-IID. To further improve runtime efficiency, we do not compute the tamper-resistance loss LTR on all K steps and instead sub-sample coordinates along the trajectory for computing LIR within an adversary batch, for example every 4 adversary optimization steps. Additionally, we reduce variance the tamper-resistane gradient by computing the tamper-resistance loss at each inner loop step on the same held-out batch, denoted as XTR in Algorithm 1.
C.5 IMPLEMENTATION DETAILS AND RESOURCE REQUIREMENTS
We perform TAR training on Llama-3- --r-ma---B--Inseee -n-Itruttt (Llama Team, AI @ Meta, 2024) with 8 NVIDIA 80GB A100 GPUs, leveraging distributed training via FSDP (Ren et al., 2021; Rajbhandari et al., 2020; Zhao et al., 2023). We use ZeRO Stage 3 from DeepSpeed (Rajbhandari al., 2020), which shards optimizer states, gradients, and parameters during training. While the efficiency trick in Appendix C.4 improves runtime, note additional considerations for conserving GPU memory.
First, simulating fine-tuning attacks that require additional state (e.g.. momentum) in the inner loop of Algorithm requires initializing fresh optimizer for every outer loop iteration. Since we use
17

---

Published as a conference paper at ICLR 2025
outer-loop optimizer that also requires maintaining state (ScheduleFree AdamW (Defazio et al., 2024)), we move the outer loop optimizer to the CPU before instantiating inner-loop optimizers.
Second, first-order meta-learning in smaller models can typically be implemented by running multiple forward passes for each inner loop iteration, averaging losses, then backpropagating on the averaged loss term. However, because each inner-loop tamper-resistance loss term (LtR Algorithm 1) is com- puted on a separate forward pass, this requires maintaining computation graphs in memory. Since this is infeasible on reasonable hardware for LLMs with billions of parameters, efficiency by accumulating tamper-resistance gradients in a separate data structure (yTR in Algorithm 1). We note that this can be done without using additional a all-gather and reduce- scat catter distributed operations, since tamper-resistance gradient accumulation and application to the pre-inner loop model parameters (0;-1 in Algorithm can be computed solely on sharded gradients.
circumvent this in-
D ADDITIONAL ANALYSIS EXPERIMENTS
D.1 BENIGN
WMDP Forget (L)
Benign Domain (1)
Pre-SFT Post-SFT
28.1
59.7
TAR-Bio
30.1
64.7
Pre-SFT
28.4
58.6
TAR-Chem
Post-SFT
27.5
62.8
Table 3: Average accuracy MMLU economics subjects and WMDP Forget subjects for our Llama- 3-8B TAR models safeguarded against hazardous biosecurity and chemical knowledge, before and after fine-tuning benign economics data (Li et al., 2024). Results indicate that models safeguarded with TAR still preserve benign fine-tunability.
An important property of open-weight models is that they can be fine-tuned to improve performance on custom data specific domains. Thus, ideal tamper-resistant safeguards should allow continued fine-tuning of model while preserving the safeguard. We evaluate whether TAR models can be fine-tuned on data unrelated to the safeguard using economics as an cxample domain. Using TAR perform supervised fine-tuning models with biosecurity and chemical security safeguards, fine-tune models for 2 epochs using a the WMDP auxiliary economics corpora (Li et al., 2024). We learning rate of 2 10-6 6 and a batch size of 32, using AdamW ScheduleFree (Defazio et al., 2024). For evaluation, we report average accuracy across the corresponding MMLU subjects (High School Macroeconomics and Microeconomies) before and after fine-tuning. To confirm that the safeguard remains tamper-resistant in this setting, we also evaluate accuracy on corresponding WMDP subjects before and after fine-tuning.
We show the results of this evaluation in Table 3. We find that accuracy on economics questions can be improved by 5.0 percentage points without recovering significant hazardous knowledge. This illustrates that strong resietaresittance can be compatible with benign model editing.
D.2 TARGETED PATCHING OF VULNERABILITIES
In Section E.2, we discuss the set of adversaries sampled during TAR for each weaponization knowledge domain. However, we included the "Retain-set SFT followed by Weaponization-dome SFT" (R-F in Table 4) adversary to improve robustness to specific adversaries that broke the defense in earlier iterations of the method that did not sample this adversary. We find in Table that including the R-F adversary significantly improves robustness to more test-time attacks, indicated by an improvement in Post-Attack Forget error by nearly 10.0 percentage points. Importantly, we find that including the 64-step R-F adversary at train-time improves test-time robustness to both 500-step R-F adversaries (Adv. 8 and 9) Table by nearly 30.0 percentage points on average, indicating that target patching of vulnerabilities is possible within TAR.
18

---

Published as a conference paper at ICLR 2025
Post-Attacks
Adversary Setup
All Advs. (4)
R- (L)
Excl. R-+F Adversary Incl. R-> Adversary
44.9
58.7
35.2
29.7
Table 4: Pre-Attack and Post-Attack scores when excluding and including the "Retain-set SFT followed by Biosecurity SFT adversary" (R (R-F)) described in Appendix E.2. Post-attack R-F accuracy averaged over both Adv. and Table 9. Average tamper-resistance across all 26 adversaries improves by nearly 10 percentage points when including the R R- adversary in TAR, and improves by nearly 30 percentage points for the R->F adversary specifically.
D.3 TEST-TIME GENERALIZATION EXPERIMENTS
Generalization to stronger test-time attacks.
Adversary Test Loss over SFT Attack Steps
In Figure 6, we show a biosecurity fine-tuning attack at LR of 2 10 5 on our TAR model and model safeguarded with LLMU. We find
TAR Safeguard
TAR Adversary Entropy
Adversary Loss (Nats)
35-
Prior Safeguard
that the tamper-resistanc of TAR can general-
Recovery Region
30-
ize far beyond the 64 steps used by train-time adversaries. Surprisingly, we observe that the test-time advensary's s cross-entropy loss does not decrease below 7 for all 1,000 steps. Morcover, the loss enters plateau and does not decrease at all after 200 steps.
25
15-
Max Entrog
10
We also plot the entropy of the SFT ad- versary's posteriors during the attack (blue dashed line). For the first 200 SFT attack steps, the adversary's entropy remains close to log(vocab size)- - the maximum possible entropy (shown as "Max Entropy" by the gray dashed line). This is expected, since this exactly what TAR optimizes for via the negative entropy tamper-resistane loss. The convergence to max- imum entropy test-time clearly demonstrates the TAR meta-learning objective working as in- tended, in line with the inner-loop posteriors observed in Figure
0
200
400
600
800
1000
SFT Attack Steps
Figure 6: Our TAR safeguard can be robust to fine-tuning attacks that greatly exceed the 64 steps used by train-time adversaries. For the LLMU safe- guard, the adversary's 's loss quickly decreases into the recovery region. By contrast, TAR maintains flatness at high loss for all 1,000 steps. Solid lines are cross-entropy losses.
Adversary Loss as Inner-loop Length (K) Increases
As point of reference, show the progres- sion of the same attack on LLMU. this case, the adversary's cross-entropy loss decreases to within the recovery region in under 20 steps, corresponding recovery in forget-set perfor- mance on WMDP.
Adversary Cross Entropy Loss
16
64
20
Varying the TAR inner-loop length K. Re- call that via the efficiency trick discussed in Appendix C.4, a single inner loop trajectory of length K during TAR returns the K sampled at- tacks in Algorithm loss robustness as we vary the length of the inner loop K during TAR, running tne-tunnng attacks for 1,000 steps held-out forget dataset for biosecurity wcaponization (Adversary Table 9).
SFT Attack Steps
We compare the test-time
Figure 7: Comparison 1,000-step SFT attack against TAR with the inner-loop length K varied between {8, 16, 32, 64). Test-time loss plateau magnitude and duration increase K increases.
19

---

Published as a conference paper at ICLR 2025
For each value of K, observe a plateau in the test loss that drops off at later steps K increases. This suggests that the robustness of TAR improves as the inner loop length increases. Prior work also corroborates that increasing the inner-loop length during meta-learning increases test-time generalization (Henderson et al., 2023). We note the contrast to conventional meta-learning methods mentioned in Section 4, in which typical meta-learning applications seek optimality after few test-time steps as possible (Nichol et al., 2018; Finn et al., 2017). Here, our results suggest that the TAR objective is incentivized run with as many inner loop steps as possible, representing beneficial tradeoff in which compute can be exchanged for robustness. We find that K 64 provides significant robustness the range of adversaries we consider Section 5 and Appendix F.I, while balancing computational efficiency as discussed in Appendix C.4.
D.4 ABLATIONS
Pre-Attack
Post-Attacks (Avg)
Ablation
Retain (t)
Forget (4)
Forget (L)
No Defense
67.3
70.5
70.5
Excl. Adv. Training
59.7
27.3
61.6
Excl. Initial Safeguard
62.5
47.3
35.5
TAR
54.7
28.1
35.2
Table 5: Ablations for primary components of TAR: (1) the initial model safeguard, (2) the adversarial training phase. We find that these components are critical for the high tamper-resistance that TAR achieves.
Including the initial safeguard. In Table 5, we examine the impact of incorporating the Random Mapping safeguard step prior to the adversarial training phase during TAR. The Random Mapping safeguard in isolation achieves a near-random chance Pre-Attack Forget accuracy of 27.3 ("Exel. Adv. Training" in Table 5). However, is susceptible to fine-tuning attacks similar other baselines in Table I, indicated by higher Post-Attack Forget accuracy of 61.6. When including the tamper- resistance adversarial training phase (TAR), we observe significantly increased tamper-resistane as the Post-Attack Forget accuracy decreases by nearly 26 percentage points.
Excluding the initial safeguard. We also examine the impact of performing the adversarial training phase without the initial safeguarding step ("Excl. Initial Safeguard" in Table 5), finding that Pre- Attack accuracy is substantially higher without the initial safeguard. We find that including the Random Mapping phase reduces pre-attack forget set accuracy by 19.2 percentage points.
Pre-Attack
Post-Attacks (Avg)
CtR Weighting
Retain (t)
Forget (L)
Forget (L)
ÀTR 1.0
62.5
29.3
39.9
ÀTR 4.0
54.7
28.1
35.2
Table 6: Pre-Attack and Post-Attack scores when varying the tamper-resistance loss weighting, ÅIR- Tamper-resistance improves by nearly 10.0% when increasing ÀTR from 1.0 to 4.0. The retain loss weight Avetain is fixed at 1.0 1.0 for both settings.
Varying the tamper-resistance loss scale ATR. We compare the downstream robustness of TAR when varying the tamper-resistance loss weighting \TR between 1.0 and 4.0 in Table 6. We observe that when setting ÀTR = 1.0, TAR maintains high retain MMLU accuracy at 62.5 percentage points, with moderate tamper- rasper-resistance indicated by Post-Attack Forget accuracy of 39.9. Further increasing ÀTR to 4.0 in our final TAR model results in significantly improved Post-Attack Forget Accuracy of 35.2, with partial decrease in Retain MMLU to 54.7. When varying ÅTR. we keep Avetain constant; thus, our results indicate a clear way to increase downstream tamper-resistane increasing the weighting of the tamper-resister tamper- resitarre and capabilities similar to the robustness performance tradcoff for adverarial robustness in vision models.
by gradient during TAR, reflecting balance between robustness.erffommmcc
20

---

Published as a conference paper at ICLR 2025
D.5 DPO TAMPER- REEEER-RSSSSTNCEE DURING TAR
TAR for Minimizing DPO Loss during Attacks
In Figure 8, we plot the DPO win-rate during harmful SFT attack trajectories during the ad- versarial training phase of TAR. We find that the outer-loop DPO loss steadily decreases, which
Durng Aback
Moal Winning Region Outer Lona
Averago DPO Win.aate
corresponds to the average inner-loop win-rate of refusal completions over rejected completions steadily increasing the 100 outer loop steps. Our results demonstrate that TAR is able to sat- isfy complex tamper-resistance losses after fine- tuning. We believe that this is useful feature of the method, enabling TAR to adapt other potentially useful objective functions that corre- spond downstream robustness.
Adv
Steps
Adv Steps
54
0.0
Tamper Resistance Steps
100
Figure The development inner-loop DPO win- rates during harmful SFT attack inner loops (red), over the course of tamper-resistane
E EXPERIMENT DETAILS
training for TAR. The outer loop win-rate (blue) depicts the average win-rate across inner loops over course of tamper-resistance training. We observe that by the end training, the win-rate for refusal com- pletions becomes completely flat near the optimal win-rate value of 1.0.
E.I WEAPONIZATION DOMAIN PROXY DATASET DETAILS
Biosecurity. We use a synthetically labeled partition of the Pile (Gao et al., 2020) that was filtered for relevance biology and the Camel Al Biology dataset (Li et al., 2023). We generate synthetic labels for Pile token sequences using openchat-3.5 (Wang et al., 2023), categorizing them as belonging "Cellular Biology" or not. This process yields 49,984 samples: 7,558 for the Forget-set (Pile-bio Forget) and 42,426 for the Retain-set (Pile-bio Retain). Coneurrently, enabled 256 tokenization limit, resulting in 54,258 samples of about 188 words each (Camel-bio Forget). We apply the same procedure WMDP biosecurity Forget-set), producing 598,933 samples of similar length (OOD Forget).
pack entries from the Camel Al Biology dataset to the truncation-
held-out hazardous biology dataset (identical to the
Chemical Security. We use private forget dataset containing text sequences about hazardous chemical security content (Chem Forget).
Cybersecurity. We scrape CTF writeups on CTFtime (CTFtime, 2024) that are numbered between 1 and 39181, collecting cybersecurity writeups written as recently as 2024. We filter to keep writeups that contain more than 150 characters. As result of filtering and HTTP errors while scraping, our resulting forget dataset contains slightly over 18k samples (Cyber Forget).
E.2 TRAIN- TIME SETTINGS AND ADVERSARIES
Weaponization knowledge restriction. For each weaponization knowledge restriction domain, we have a corresponding retain dataset Dretain comprised of a mix of data from the Pile-bio Retain set and Siepie-AAign instruction-tumee Referencing the datasets described in Appendix 1. we specify the adversary tamper-resistance datasets that constitute the data used by attacks sampled from Amain during TAR training, as well as the attack setup for each domain as follows:
dataset (Xu al., 2024), which we refer as the Retain-set.
Biosecurity: We simulate adversaries from the following three setups: Pile-Bio Forget-set SFT, Camel- Bio Forget-set SFT, and Retain-set SFT followed by Pile-Bio Forget-set SFT (R (R-F),, where the switching point between Retain-set and Pile-Bio Forget-set SFT within the 64-stey length trajectory is sampled from a beta distribution (6600, 3.0). LRs are sampled from {2 10 10 5, 10-5).
Chemical Security: We simulate 6 adversaries from the following three setups: Chem Forget-set SFT, Retain-set SFT, and Retain-set followed by Chem Forget-set SFT, using the same switching-
21

---

Published as a conference paper at ICLR 2025
point sampling scheme in the Biosecurity setting. LRs are sampled from (2 10 5, 10 10-
For chemical security, we set the tamper-resistance loss scale ATR to 3.0.
Cybersecurity: We simulate adversaries from the following two setups: Cyber Forget-set SFT, ketain-set SFT, and Retain-set SFT followed by Cyber Forget-set SFT, using the same switching point sampling scheme as the Chemical Security and Biosecurity settings. LRs are sampled from (2 10-5, ,4 10-5).
For each weaponization knowledge domain, we create 80-20 splits for adversary and held-out data of the corresponding forget sets, respectively. For Biosecurity, which uses multiple forget datasets, this involves creating 80-20 splits for each dataset, then combining the corresponding splits. The adversary data splits are used for sampled attacks from Arain whereas the held-out split is used for computing tamper-resistance losses. The held-out splits for each domain correspond to DIR in Section 4. We use minibatches from a held-out dataset for computing tamper-resistanee losses rather than cycling through a single dataset, following the recommendation of Nichol et al. (2018), in which first-order meta-learning without properly held-out minibatches caused a performance degradation.
All train-time adversary setups are tabulated in Table 7, where F-Pile, F-Chem, F-Cyber denote the respective datasets described in Appendix E, and Retain denotes the mixed Pile-bio and Magpie- Align Retain-set described in Appendix E.2. We use R-FF to label the adversaries that perform Retain-set SFT followed by Forget-set SFT, The final column an abbreviation for Finetuning Paradigm and indicates whether the SFT setup used full parameter finetuning or parameter-efffcieet fine-tuning (PEFT) via LoRA adapters (Hu et al., 2021).
Table 7: Train-time adversary setups for weaponization knowledge restriction of Biosecurity, Chemi- cal Security, and Cybersecurity.
Adversary
Dataset
Opt. Steps (K)
Optimizer
Schedule
Batch Size
Paradigm
Biosecurity Weaponization Restriction
Adv
F-Pile
64
AdamW
x 10
No Warmup
Full Parameter
Adv 2
F-Pile
64
AdamW
4 x 10
No Warmup
64
Full Parameter
Adv
F-Camel
64
AdamW
x 10-5
No Warmup
Full Parameter
Adv 4
F-Camel
64
AdamW
4 10-5
No Warmup
64
Full Parameter
Adv 5
R++
64
AdamW
10
No Warmup
Full Parameter
Adv 6
R-+F
64
AdamW
x 10-5
No Warmup
64
Full Parameter
Chemical Security Weaponization Restriction
Adv 1
F-Chem
64
AdamW
2 x 10-5
No Warmup
64
Full Parameter
Adv 2
F-Chem
64
AdamW
x 10
No Warmup
64
Full Parameter
Adv 3
Retain
64
AdamW
x 10-5
No Warmup
64
Full Parameter
Adv 4
Retain
64
AdamW
4 x 10-5
No Warmup
64
Full Parameter
Adv 5
R-F
64
AdamW
x
No Warmup
Full Parameter
Adv 6
R-FF
64
AdamW
x 10-5
No Warmup
64
Full Parameter
Cybersecurity Weaponization Restriction
Adv 1
F-Cyber
64
AdamW
x 10
No Warmup
64
Full Parameter
Adv
F-Cyber
64
AdamW
x 10-5
No Warmup
Full Parameter
Adv 3
Retain
64
AdamW
2 x 10
No Warmup
64
Full Parameter
Adv
Retain
64
AdamW
x 10-5
No Warmup
64
Full Parameter
Adv 5
R-F
64
AdamW
2 x 10-5
No Warmup
64
Full Paramcter
Adv 6
R-F
64
AdamW
4 10
No Warmup
64
Full Paramcter
Harmful request refusal. For harmful request refusal, we choose the retain dataset Dretain to be
the Magpie-Alige instruction-tuning dataset (Xu et al., 2024). sample train-time adversaries that perform K 64 steps of SFT rejected completions of the Anthropic- et al., 2022) and vary the learning rate within (2 x 10 2 Anthropic-HH-RLEE HH-RLHF dataset (Bai adversaries in Table 8.
10 5). We depict this list of
22

---

Published as a conference paper at ICLR 2025
While the rejected completions from Anthropic-HH-RARe
constitute the data used for sampled attacks from Again for harmful request refusal, we compute the tamper-resistance loss LIR as follows. Since LTR is the DPO loss (Rafailov et al., 2023) in this setting, we use the base model weights as the reference model and sample harmful and benign completions from a modified test split of Anthropic-HHH-RLEH (Bai al., 2022), where rejected completions (Cai et al., 2024). To avoid keeping the base model weights in memory and speed up training, we precompute the reference model DPO pee-robabilities for the full Anthropic-HH-RLLNE before training. To summarize, we have that the sampled train-time adversaries perform SFT on the rejected completions from Anthropic- HH-Hi-, and tamper-resistane DPO loss computed on the corresponding modified refusal completions; the modified refusal completions in this setting correspond to DTR Section 4.
replaced with refusals
dataset
In practice, we perform an additional 100 steps of supervised fine-tuning on the Magpie-Align dataset improve the benign capabilities performance the TAR refusal model in Table 2.
Table 8: Train-time adversary red-teaming setups harmful request refusal. "A-HlH-Rejected" in the Dataset column corresponds to the adversary dataset comprised of rejected completions from the Anthropic-HH. HH-RLHF dataset.
Adversary
Dataset
Opt. Steps (K) Optimizer
LR
LR Schedule Batch Size FT Paradigm
Harmful Request Refusal
Adv 1
A-HeH-Rejected
64
AdamW 2 10
No Warmup
64
Full Parameter
Adv 2
A HH-Rejected
AdamW
2 10
No Warmup
64
Full Parameter
Adv 3
A-HH-Rac HHH-ejjcted
64
AdamW
4 x 10-5
No Warmup
64
Full Parameter
F RED TEAMING DETAILS
F.1 WEAPONIZATION KNOWLEDGE RESTRICTION
In Table 9, we list all tesl-time adversary setups for recovering Biosecurity, Chemical Security, and Cybersecurity Weaponization knowledge.
For Biosccurity, we examine post-attack forget accuracy after fine-tuning for 500 steps on three data distributions: the Pile-bio Forget set and the Retain-set used during Random Mapping and TAR, and an OOD-Forget set mentioned in Appendix E.I, which is unseen during tamper-resistance training.
We leverage consistent setup for Chemical Security and Cybersecurity, substituting the Pile-bio Forget set for the respective Chem and Cyber Forget sets. We exclude the unseen forget distribution were unable to obtain equally high quality held-out
adversaries for total of 24 attacks, dataset for Chemical Security and Cybersecurity.
For all subjects, compare the effect of different optimizers (AdamW, Adadelta, and Stochastic Gradient Descent with Nesterov Momentum, Schedule Free AdamW) (Zeiler, 2012; Kingma & Ba, 2017: Xie et al., 2023; Defazio et al., 2024), learning rates (2 x 10-6 2 x 10-5, 4 10-5), and learning rate scheduling techniques (SGDR and 30 steps of linear warmup) (Loshchilov & Hutter, 2016).
Similar to Table 7, we use F-Pile, F-Chem, F-Cyber, and OOD-F in the Dataset Column to denote the respective datasets described in Appendices F. At test-time, we use the Pile-bio Retain set as the global Retain-set adversary, labeled as Retain. The R-F adversary at test-time also differs from the train-time version: we perform Forget-set SFT for 40% of the optimization steps, followed by Retain-set set SFT for the remaining 60%. We found this combination to be potent attack that broke intermediate versions of the method, as described in Appendix D.2.
23

---

Published as a conference paper at ICLR 2025
In the Optimizer column, Schedule Free is abbreviation of Schedule Free AdamW and SGD Nesterov is an abbreviation of SGD with Nesterov Momentum. cases where the adversary used parameter-cfficient tne-ttnnng (PEFT) via LoRA adapters (Hu al., 2021), we used a LoRA config with an attention dimension, or rank, of 16, a LoRA alpha value of 32, a LoRA dropout of 0.05, on target linear modules:
('up_proj', 'down_proj', 'gate_proj', 'q_proj', 'k_proj', 'v_proj', )o_proj"]
Lastly, for each Weaponization Knowledge Restriction domain, red-team the TAR model with three runs of supervised fine-tuning attacks from each adversary, using different random seeds. We then calculate the final post-attack value for each adversary by averaging these three replicates.
Table 9:
Test-time adversary red-teaming setups for weaponization knowledge restriction of Biosecu- rity, Chemical Security, and Cyhersecurity.
Adversary Dataset Opt. Steps (K)
Optimizer
LR
LR Schedule
Batch Size
FT Paradigm
Biosecurity Weaponization Restriction
Adv 1
F-Pile
500
AdamW
2 X 10
No Warmup
64
Full Parameter
Adv 2
F-Pile
500
AdamW
X 10
No Warmup
Full Parameter
Adv 3
Retain
500
AdamW
2 x 10
No Warmup
64
Full Parameter
Adv 4
Retain
500
AdamW W
x 10
No Warmup
Full Parameter
Adv 5
OOD-F
500
Adam W
2 x 10-5
No Warmup
64
Full Parameter
Adv 6
OOD-F
500
Adam W
x 10 5
No Warmup
64
Full Parameter
Adv? 7
R-F
500
AdamW
2 x 10
No Warmup
64
Full Parameter
Adv 8
R+F
500
AdamW
4 10
No Warmup
64
Full Parameter
Adv
F-Pile
500
Adadelta
2 10
No Warmup
Parameter
Adv 10
F-Pile
500
Adadelta
1x100 10
No Warmup
64
Full Parameter
Adv
F-Pile
500
Schedule Free x 10
No Warmup
Parameter
Adv 12
F-Pile
500
Schedule Free 4 x 10
No Warmup
64
Full Parameter
Adv
F-Pile
500
SGD Nesterov x 10
No Warmup
Full Parameter
Ady 14
F-Pile
500
SGD Nesterov
x 10-5
No Warmup
64
Full Parameter
Adv 15
F-Pile
500
AdamW
x 10
No Warmup
64
Full Parameter
Adv 16
F-Pile
500
Adam W
2 x 10
30 Steps Warmup
64
Full Parameter
Adv 17
F-Pile
500
AdamW
2 10
30 Steps Warmup
64
Full Parameter
Adv 18
F-Pilc
500
Adam
x 10
30 Steps Warmup
Full Parametor
Adv 19
F-Pile
500
AdamW W
2 x 10
SGDR
64
Full Parameter
Adv
F-Pile
500
AdamW
4 X 10
SGDR
Parameter
Adv 21
F-Pile
500
Adam W
2 x 10
No Warmup
32
Full Parameter
Adv 22
F-Pile
500
Adam W
x 10
No Warmup
32
Full Parameter
Ady 23
F-Pile
500
AdamW W
2 x 10
No Warmup
128
Full Parameter
Adv 24
F-Pile
500
AdamW
4 10-6
No Warmup
128
Full Parameter
Adv 25
F-Pile
500
AdamW
2 x 10
No Warmup
PEFT
Adv 26
F-Pile
500
Adam W
x 10-5
No Warmup
64
PEFT
Chemical Security Weaponization
Adv 1
F-Chem
500
AdamW
2 x 10
No Warmup
64
Full Parameter
Adv 2
F-Chem
500
AdamW
x 10
No Warmup
Parameter
Adv
Retain
500
AdamW W
2 x 10 10
No Warmup
64
Full Parameter
Adv 4
Retain
500
AdamW W
x 10
No Warmup
Full Parameter
Adv 5
R-F
500
AdamW W
2 x 10-5
No Warmup
64
Full Parameter
Adv 6
R-F
500
Adam W
10
No Warmup
Full Parameter
Adv 7
F-Chem
500
Adadelta
x 10
No Warmup
64
Full Parameter
Adv 8
F-Chem
500
Adadelta
4 10
No Warmup
64
Full Parameter
Adv
F-Chem
500
ScheduleFree
2 10
No Warmup
Full Parameter
Adv 10
F-Chem
500
ScheduleFree
4 10
No Warmup
64
Full Parameter
Adv
F-Chem
500
SGD Nesterov x 10
No Warmup
Full Parameter
Adv 12
F-Chem
500
SGD Nesterov
x 10
No Warmup
64
Full Parameter
Adv
F-Chem
500
AdamW
x 10
No Warmup
Full Parameter
Continued next page
24

---

Published as a conference paper at ICLR 2025
Table 9 continued from previous page
Adversary
Dataset Opt. Steps (K)
Optimizer
LR
LR Schedule
Batch Size
Paradigm
Adv 14
F-Chem
500
AdamW
x 10-6
30 Steps Warmup
64
Full Parameter
Adv 15
F-Chem
500
AdamW
2 x 10
30 Steps Warmup
64
Full Parameter
Adv 16
F-Chem
500
Adam
x 10
30 Steps Warmup
Full Parameter
Adv 17
F-Chem
500
Adam W
2 x 10
SGDR
64
Full Parameter
Adv 18
F-Chem
500
Adam
x 10
SGDR
Parameter
Adv 19
F-Chem
500
AdamW
2 X 10-6
No Warmup
32
Full Parameter
Adv
F-Chem
500
AdamW
X 10
No Warmup
Full Parameter
Adv 21
F-Chem
500
Adam W
2 X 10-5
No Warmup
128
Full Parameter
Adv 22
F-Chem
500
Adam W
X 10
No Warmup
128
Full Parameter
Adv 23
F-Chem
500
AdamW
2 10
No Warmup
64
PEFT
Adv 24
F-Chem
500
AdamW
x 10
No Warmup
64
PEFT
Cybersecurity Weaponization Restriction
Adv 1
F-Cyber
500
AdamW
x 10
No Warmup
64
Full Parameter
Adv 2
F-Cyber
500
AdamW W
X 10-5
No Warmup
Full Parameter
Adv 3
Retain
500
Adam W
2 X 10-5
No Warmup
64
Full Parameter
Adv
Retain
500
Adam
x 10
No Warmup
Parameter
Adv 5
R-+F
500
Adam W
2 x 10 10
No Warmup
64
Full Parameter
Adv
R-F
500
Adam W
X 10
No Warmup
Full Parameter
Adv 7
F-Cyber
500
Adadelta
2 x 10-5
No Warmup
64
Full Parameter
Adv 8
F-Cyber
500
Adadelta
x 10
No Warmup
64
Full Parameter
Adv 9
F-Cyber
500
ScheduleFree
x 10-5
No Warmup
64
Full Parameter
Adv 10
F-Cyber
500
ScheduleFree
x 10
No Warmup
64
Full Parameter
Adv
F-Cyher
500
SGD Nesterov
x 10
No Warmup
Parameter
Adv 12
F-Cyher
500
SGD Nesterov
x 10
No Warmup
64
Full Parameter
Adv
F-Cyber
500
AdamW
x 10
No Warmup
Full Parameter
Adv 14
F-Cyber
500
AdamW W
2 10 10-6
30 Steps Warmup
64
Full Parameter
Adv 15
F-Cyber
500
Adam W
2 10
30 Steps Warmup
64
Full Parameter
Adv 16
F-Cyber
500
Adam W
x 10 10
30 Steps Warmup
64
Full Parameter
Adv 17
F-Cyber
500
AdamW W
10
SGDR
64
Full Parameter
Adv 18
F-Cyber
500
AdamW
4 X 10
SGDR
Full Parameter
Adv 19
F-Cyber
500
AdamW W
2 x 10
No Warmup
32
Full Parameter
Adv
F-Cyber
500
Adam
X 10
No Warmup
Full Parameter
Adv 21
F-Cyber
500
Adam W
2 10 10
No Warmup
128
Full Parameter
Adv
F-Cyber
500
Adam
4 10 10
No Warmup
128
Parameter
Adv 23
F-Cyber
500
Adam W
2 10
No Warmup
64
PEFT
Adv 24
F-Cyber
500
Adam W
4 10-
No Warmup
PEFT
F.2 HARMFUL REQUEST REFUSAL
For the harmful request refusal setting, we conduct 5 test-time adversary attacks that perform SFT for 10 epochs on held-out toxicity dataset called Toxic-DPO v0.2, each of the settings in Table 10. The dataset contains 541 user-assistant chat interactions where the assistant complies with harmful instructions.
Table 10: Test-time adversary red- red-teaming setups for harmful request refusal. "ToxicDPO" in the Dataset column refers to the ToxicDPOv0..2 dataset containing harmful chat completions.
Adversary
Dataset
Epochs
Optimizer
LR
Schedule
Batch Size
FT Paradigm
Harmful Request Refusal
Adv
ToXicDPO
10
AdamW
10-5 5
10 Steps Warmup
32
Full Parameter
Continued on next page
25

---

Published as a conference paper at ICLR 2025
Table 10 - conttinee from previous page
Adversary
Dataset
Epochs
Optimizer
LR
LR Schedule
Batch Size
FT Paradigm
Adv
TPXicDPO
10
AdamW
1 x 10
No Warmup
32
Full Parameter
Adv
ToXicDPO
10
AdamW
1 x 10
No Warmup
16
Full Parameter
Adv
ToXicDPO
10
AdamW
2 10-5
No Warmup
Full Parameter
Adv 5
ToxicDPO
10
AdamW
4 x 10-5
No Warmup
32
Full Parameter
F.2.1
ADDITIONAL HARMFUL REQUEST REFUSAL RESULTS
Pre-Attacks
Post-Attacks (Avg)
Model
MT- (f)
ASR (1)
ASR (1)
Refusal Trained
8.1
14.7
72.5
R2D2
6.0
25.0
78.3
RepNoise
6.2
18.8
74.5
RR
8.0
1.4
84.8
TAR (Ours)
6.3
31.4
63.9
Table 11: Average Post-Attack HarmBench ASR, reported for TAR, Representation Rerouting (RR), and the Refusal Trained Llama-3-88 3-mma----B-Instruee model across 5 fine-tuning attacks depicted in Table 10, well as Pre-Attack MT-Bench and HarmBench ASR. TAR more robust than other methods after tampering, while maintaining comparable MT--Becch performance. Note that Pre-Attack ASR is priority for us, future work could consider combining tamper-resistance training with strong baseline safeguard like RR. ASR values are percentages.
focus reducing ASR after tampering attacks. To improve both metrics,
G BASELINE DETAILS
G.1 WEAPONIZATION KNOWLEDGE RESTRICTION
Max Entropy. Let K be the set of all token-wise output probability distributions returned by model 6, where & E corresponds to every position in the sequence. We maximize the average entropy of these discrete distributions in K as follows:
LMxx Entropy
Σ kEk
Pk log(ps)
This is equivalent to minimizing the average Kullback-Leibler (KL) divergence (Kullback & Leibler, 1951) between each and the discrete uniform distribution u(z) over the vocabulary V. For Llama- 3-8t-Instruct, [V] 128256. Thus, this objective upper-bounded, by:
h(x) log(u(x)) log( Vi) 11.76
where h(z) measures the Shannon information or self- self-information and log(x) has base We apply this objective for all elements in the Forget-set and perform standard cross-entropy on the Retain-set
The goal of the Min Posterior objective is assign lower probabilities to true forget-set labels, essentially minimizing log(1 - Plabell)). Let pi be the probability assigned to the true label for token and V be the model's vocabulary distribution. We define the Min Posterior objective function as follows:
Min Posterior.
Min Posterior
1 M
Σ
log(1 Ps + -[lo((p.)) 2
26

---

Published as a conference paper at ICLR 2025
where 7 is the threshold for masking out target label logits (which set to the negative maximum entropy of the vocabulary distribution, - -log (V]) and I[-] is the corresponding indicator function (1 the condition is true, 0 otherwise). We include an optional
1 x 10 to help with numerical stability. Similar to the Max Entropy objective, we apply this objective for all elements in the Forget-set and perform standard cross-entropy the Retain-set.
RMU. We adapt RMU's implementation from Li et al. (2024) with learning rate of 5 10-3 and 250 unlearning steps. We use the released WMDP's unlearning datasets for Biosecurity (Bio) and Cybersccurity (Cyber) unlearning, and our private hazardous chemistry dataset for Chemical Security (Chem) unlearning. We use unlearning coefficients of 20, 30, and 50 for Bio, Cyber, and Chem respectively. We use a retain coefficient of 700 on Wikitext Merity al. (2016).
LLMU. We use modified version of LLMU from Yao al. (2023). Instead of computing the KL divergence to regularize retain-set logits towards the base frozen model, we employ standard cross-entropy loss. This modification allows for memory-efficient execution on our hardware while maintaining comparable performance.
Hyperparameter tuning. Besides RMU, all baseline hyperparameters were chosen after grid search across learning rates {3 x 10 6 5 x 10-6, 8 x 10 10-6, x 10 10-"), optimization step count (600, 1000), and warmup steps {0, 100). We found that 600 optimization steps using the AdamW W Schedule Free optimizer, at learning rate of 1 x 10-5, with 100 steps of linear warmup, and an effective batch size of 64 produced the best performance. For the Max Entropy, Min Posterior, and LLMU baselines, we train on the three corresponding forget datasets discussed in E. For these baselines, we modify our Biosecurity forget corpus to be mixture of the Pile-bio and Camel-bio Forget corpora. We use the Pile-bio Retain-set as a global Retain-set for baseline training.
G.2 HARMFUL REQUEST REFUSAL
Representation Rerouting. We use the which uses a cosine distance loss to push representations for harmful inputs to become orthogonal to those of the base Llama-3-8B-Instrus model.
RR model from Zou et al. (2024),
R2D2. We use the R2D2 model run on Zephyr-7B directly from Mazeika al. (2024), which performs adversarial training against GCG attacks to increase jailbreak robustness.
RepNoise. We the RepNoise model run on Llama-2-7B directly from Rosati et al. (2024b), which uses distributional loss to push representations for harmful inputs toward Gaussian noise.
G.3 ADDITIONAL BASELINE COMPARISONS
Meta-Learned Adversarial Censoring (MLAC) (Henderson et al., 2023) was originally proposed to prevent BERT-style models from learning binary classification for gender bias data. Since the approach is not immediately applicable to LLMs, we extend MLAC in a variant we call autoregressive MLAC (MLAC-AR). Since MLAC in its original formulation calls for "task-blocking" via negating the adversary's loss during the inner loop of meta-learning, we implement this by negating the cross-entropy loss of LLM fine-tuning adversary. However, we found that this approach diverges in performance across a variety of hyperparameters, and opted to further improve performance of the MLAC-AR baseline by clamping the maximum cross-entropy loss the value of the maximum entropy of the output vocabulary distribution, log(vocab_$s p__s_iie ze). We show results in Table 12, finding that MLAC-AR does not maintain sufficient benign capabilities performance nor uniform tamper-resistance across weaponization domains.
MLAC-AR.
In concurrent work, SOPHON (Deng et al., 2024) was introduced to prevent small diffusion models and image classifiers from learning specific data distributions. Similarly to MLAC-AR, extend SOPHON to LLMs via SOPHON-AR, using the alternating retain loss and fine-tuning suppression loss formulation that the authors propose. Furthermore, we adapt the inverse cross-entropy loss from Deng et al. (2024), which aims boost convergence of the fine-tuning
SOPHON-AR.
27

---

Published as a conference paper at ICLR 2025
Pre-Attan Pre-Attacks
Post-Attacks (Avg)
Domain
Model
Retain (t)
Forget (L)
Forget ())
Random
25.0
25.0
25.0
MLAC-AR
49.1
31.2
50.6
Biosecurity
SOPHON-AR
27.2
24.0
28.3
TAR (Ours)
54.7
28.1
35.2
Random
25.0
25.0
25.0
MLAC-AR
47.8
29.9
33.6
Chemical Security
SOPHON-AR
23.3
26.2
26.1
TAR (Ours)
56.5
28.4
27.1
Random
25.0
25.0
25.0
MLAC-AR
36.0
26.6
35.1
Cybersecurity
SOPHON-AR
24.4
24.6
30.4
TAR (Ours)
60.7
23.6
28.6
Table 12: Additional baselines for MLAC-AR, extension of the method in Henderson et al. (2023) to autoregressive LLMs, as well as SOPHON-AR from Deng et al. (2024), respectively. Despite extensive tuning, SOPHON-AR does not yield a usable model. Additionally, MLAC-AR has varying robustness and worse Retain MMLU performance.
suppression process. We find in practice that despite heavy tuning, SOPHON-AR does not converge well enough to yield usable Pre-Attack model in Table 12.
28


Reviews:
[{"review_id": "Gvq4JFTS9U", "reviewer": "Reviewer_n8EQ", "summary": "The paper proposes a method (TAR) for improving tamper-resistance, i.e. model-level defense against adversarial finetuning attacks, of open-weight large language models. The method consists of several components: (1) initial safeguarding via _random mapping_ of harmful representations, (2) outer loop minimizing tamper-resistance and retain losses, (3) inner loop for computing tamper-resistance loss, which applies multiple finetuning attacks. Different design choices for tamper-resistance loss and its empirical significance are discussed: for weaponization knowledge restriction setting a negative _entropy_ loss is proposed, and for harmful request refusal a direct preference optimization (DPO) loss is used. The retain loss consists of language modeling loss and $l_2$-norm loss for representations of optimized and a base model. The results suggest the proposed method effectively defends the model against the majority of considered finetuning attacks, maintaining low accuracies on harmful questions post-(finetuning)-attack, although at the considerable cost of drop in accuracy on benign questions (pre-attack). Additionally, the authors acknowledge that the set of finetuning attacks during tamper-resistance training directly impacts the tamper-resistance against test-time attacks (e.g. \"Retain $\\rightarrow$ Forget\" attack breaks the defense if it is not included in the training phase), suggesting the defense might struggle with unseen attacks (e.g. PEFT-attacks could break the defense in many of the settings).", "questions": "See weaknesses section for questions and suggestions. I would be happy to change my opinion if my main concerns regarding fair evaluation of the defense method could be addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. **Significance** of the problem. The paper addresses an important and challenging problem of defending open-weight large language models against finetuning attacks. In authors words: \"_This problem has been considered very challenging and by some intractable, as no method has yet provided substantial robustness to these attacks. However, making progress on this problem would provide a valuable tool to regulators and model developers by ameliorating the dual-use dilemma of open-weight models_\".\n\n2. **Originality**. Although most of the components of the proposed method are inspired from previous works (e.g. adversarial training, representation engineering), the overall approach constitutes substantial novel contribution to the field, to the best of my knowledge. The results on tamper-resistance for post-attack harmful accuracies demonstrate substantial improvement over previous methods (for most of the considered attacks).", "weaknesses": "1. **Evaluation** against out-of-distribution attacks. \n- My main concern is that the defense might be effective mostly against observed attacks, and it could break against other unseen attacks. For example, Table 4 in Appendix shows that \"Retain $\\rightarrow$ Forget\" attack breaks the defense if it is not included in the training phase. Figure 4, and Figure 8 from Appendix show that PEFT attacks are more effective than Full Parameter attacks (in case of Biosecurity, PEFT attacks break the proposed defense method), given that the TAR used Full Parameter attacks during training.\n- Therefore, more emphasis and red teaming effort should be put into unseen attacks during evaluation, e.g. the Post-Attack scores in Table 1 could be divided into \"in-distribution\" and \"out-of-distribution\" attacks, where the categorization of attacks should be agnostic to LR, LR scheduler, optimizer, number of steps, batch size. In other words, out-of-distribution attacks could be defined as those that use fundamentally different approaches or data sources than what was used during training, rather than just different hyperparameters. Testing against attacks that use different optimization algorithms, loss functions, or data distributions not seen during training could provide a more comprehensive assessment of the method's robustness.\n- Since PEFT finetuning is more compute-efficient than Full Parameter tuning, the variants of PEFT attacks with more optimization steps should be considered under the compute-constrained attacker setup. PEFT attacks should also be considered for harmful request refusal. \n- Other out-of-distribution attacks could also be proposed and evaluated, e.g. initializing an attack by perturbing the weights into a random direction before following gradients to avoid local potentially gradient obfuscated region; or running a ground-truth informed attack by following a fixed worst-case direction towards the weights of a harmful model to observe how far the optimization should run to get harmful results.  \n- Input-level red teaming approaches (e.g. from HarmBench benchmark) could also be evaluated as alternative attacks, which do not include gradients or weight perturbations.\n\n\n2. More **detailed analysis** of the method is missing. The problem of **obfuscated gradients** should be addressed.\n- The results for Chemical Security in Table 1 suggest that post-attack harmful accuracies for TAR are lower than pre-attack ones, which is unexpected and worrying. Could you provide a more detailed analysis of this phenomenon? Could you investigate whether this is due to a quirk in your evaluation setup, or if it reveals something fundamental about how your method works? \n- Also the plots in Figure 6 in Appendix show that the loss values first start to increase under the gradient-based attacks, which is surprising. Could the loss decrease just by switching the gradient sign in the beginning of the optimization? This might point towards the problem of obfuscated gradients [a]. Other attacks, e.g. gradient-free ones, or the exploration of the loss landscape could provide a better understanding of the phenomenon.  \n- Section A in Appendix states that post-attack accuracy on benign questions for TAR method is low. This should be reflected in the main paper, and the reasons for this phenomenon could be studied and discussed. Section C1 of Appendix addresses the problem of benign finetuning, however it does not provide comparison with benign finetuning of the base model (or other non-TAR trained harmless models). What percentage of harmful questions could appear in finetuning to prevent the benign learning? Could benign prompts from Over-Refusal benchmark [b] cause the issues with benign finetuning?\n- Over-Refusal benchmark [b] results should be included for the restricted models for full evaluation.\n\n3. The **capability-robustness tradeoff** could be studied and discussed more in-detail, since this is the main limiting factor of applying TAR comparing to baseline methods. \n- From Table 1, TAR is considerably *worse than baselines in terms of benign capabilities* in adjacent domain knowledge for about 10% in all domains. What about other capabilities such as reasoning, multi-lingual understanding, creative writing, coding etc?\n- Could the whole capabilities-robustness tradeoff curve be explored and demonstrated for TAR and for baseline methods by varying hyperparameters (e.g. such as $\\lambda_{TR}$)? Could a single metric for the method's tradeoff performance be proposed and compared between baselines, similar to Accuracy-Robustness Tradeoff Score (ART-score) in [c]?\n\n4. **Clarity**. \n- Many important parts of the paper are in Appendix, e.g. Random Mapping, attack categories, many crucial evaluations (see above). This makes the main paper less clear and prevents it from being self-sufficient.\n- Was a single model trained to be tamper-resistant against all 3 weaponization domains, or were 3 different TAR models considered (e.g. in Table 1)? Was a single model trained for both weaponization restriction and harmful request refusal, or 2 different ones? Could a single model be defended against all considered harms? How would it affect benign capabilities? Is the approach the same for baseline methods? It was not clear from the text for me. These details could be included in the experimental setup section, and you could include a diagram or table summarizing the model configurations used for different experiments.\n- What are the computational costs of TAR comparing to benign PEFT, Full Parameter finetuning and other baselines? \n- Minor comment: could the scores in Table 1 be scaled such that random model gets 0, and perfect model get 100? It would help visualizing the effectiveness of TAR more clearly.\n\n[a] Athalye, A., Carlini, N., & Wagner, D. (2018, July). Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning (pp. 274-283). PMLR.\n\n[b] Cui, J., Chiang, W. L., Stoica, I., & Hsieh, C. J. (2024). OR-Bench: An Over-Refusal Benchmark for Large Language Models. arXiv preprint arXiv:2405.20947.\n\n[c] Nurlanov, Z., Schmidt, F.R., Bernard, F. (2024). Adaptive Certified Training: Towards Better Accuracy-Robustness Tradeoffs. In: Bifet, A., et al. Machine Learning and Knowledge Discovery in Databases. Research Track and Demo Track. ECML PKDD 2024. Lecture Notes in Computer Science(), vol 14948. Springer, Cham. https://doi.org/10.1007/978-3-031-70371-3_8", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JdyvphqrxU", "reviewer": "Reviewer_Rc9Z", "summary": "This paper focuses on the robustness of open-weight LLMs and proposes a novel defense method called TAR.", "questions": "+ In Figure 2, the difference between TAR and the baseline methods is significant. However, it seems that the capabilities of TAR are lower than the baseline methods. Is there a trade-off between capabilities and tamper resistance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**About contribution**\n\n+ The experimental results shown in Table 1 are significant enough to validate the main claims of this paper. \n+ The proposed method is intuitive.  By providing detailed discussions of the related works, it is not hard to understand why the authors designed the algorithms as presented, even for readers not familiar with the defense of LLMs.\n\n**About novelty**\n\nAccording to Section 2, this paper proposes the first defense method for autoregressive LLMs against tampering attacks. To the best of my knowledge, concurrent jailbreaking attacks are mostly input-based. However, as claimed in Section 1, the tampering attacks are also posing threats to LLMs. This paper will bring new insight into the research on the robustness of LLMs.\n\n**About presentation**\n\n+ The preliminary part (Section 3) is brief and clear, making the technical part of this paper easy to follow.", "weaknesses": "**About presentation**\n\n+ The authors do not discuss the cost of the experiments, including time cost and GPU memory cost. Section B.4 mentioned that the experiments use 8 A100 with 80GB GPU memory. What is the minimum requirement for the experiments? \n+ I suggest including a statement of contribution to make this paper easier to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CCQQq3Ak4X", "reviewer": "Reviewer_rLai", "summary": "This paper focuses on the issue of the lack of robustness in open LLMs when facing model weight tampering attacks. The authors propose a method called TAR, designed to establish tamper-resistant protection mechanisms for LLMs, ensuring that attackers cannot compromise these protections after minimal optimization. Extensive experiments validate the performance of this approach, showing that it outperforms existing methods.", "questions": "1. Insufficient sustainability. This paper proposes the integration of adversarial learning and meta-learning to enhance the effectiveness of defense mechanisms, making it difficult for attackers to compromise them in a short period. However, this effectiveness actually depends on the diversity of attack types included in the training data for optimizing eqn.1. In other words, the resilience of the proposed mechanism may be superficial and does not guarantee the security of open-weight LLMs. Furthermore, the authors do not provide corresponding theoretical analysis or proof.\n2. Incremental technical contributions. Although the paper is expressed clearly, its innovation is not evident in terms of both technical aspects and application scenarios. Specifically, the proposed solutions are based on existing widely used methods, and the authors have not clearly articulated their unique contributions. Therefore, it is recommended that the authors provide further clarification on this matter.\n3. The performance of the proposed mechanism is closely related to the adversarial training methods and data, which means its resilience remains a significant issue.\n4. The presentation of the performance comparison between TAR and existing mechanisms in Figure 5 is unclear and potentially confusing. The authors should provide further analysis of this result, explaining why the performance of TAR shows a significant change as the step size increases.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The security issues related to open LLMs are both important and intriguing; The authors present a series of solutions to address these security threats; and experiments validate the performance of the proposed mechanisms.", "weaknesses": "1. Insufficient sustainability. This paper proposes the integration of adversarial learning and meta-learning to enhance the effectiveness of defense mechanisms, making it difficult for attackers to compromise them in a short period. However, this effectiveness actually depends on the diversity of attack types included in the training data for optimizing eqn.1. In other words, the resilience of the proposed mechanism may be superficial and does not guarantee the security of open-weight LLMs. Furthermore, the authors do not provide corresponding theoretical analysis or proof.\n2. Incremental technical contributions. Although the paper is expressed clearly, its innovation is not evident in terms of both technical aspects and application scenarios. Specifically, the proposed solutions are based on existing widely used methods, and the authors have not clearly articulated their unique contributions. Therefore, it is recommended that the authors provide further clarification on this matter.\n3. The performance of the proposed mechanism is closely related to the adversarial training methods and data, which means its resilience remains a significant issue.\n4. The presentation of the performance comparison between TAR and existing mechanisms in Figure 5 is unclear and potentially confusing. The authors should provide further analysis of this result, explaining why the performance of TAR shows a significant change as the step size increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zFmqnRxR7U", "reviewer": "Reviewer_vbKt", "summary": "This paper introduces a novel method called TAR, designed to enhance the robustness of large language models (LLMs) against tampering attacks, addressing significant vulnerabilities in existing safeguards.", "questions": "None", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The topic studied in this paper is of great significance. Malicious LLMs can cause serious harm, such as spreading false news about public figures, causing discrimination and unfair results due to prejudice, and generating violent terrorist information. Therefore, it is necessary to apply robust safeguards to open source LLMs.\n\n2. The proposed method successfully resists thousands of malicious fine-tuning.", "weaknesses": "1. The organization of Section 4 requires further refinement for improved readability. It would be beneficial to briefly outline the design motivation before delving into specific details, particularly regarding the content of Fig. 3. Additionally, the mathematical formulation of the loss function used is currently absent.\n\n2. The caption of figure 1 needs to explain the difference between the two branches more clearly. For example, what's the difference between the first nodes of the two branches.\n\n3. In Section 1, the authors assert that the proposed method can endure fine-tuning of up to 5000 steps. However, this claim does not intuitively convey the contribution of the paper. Firstly, the details surrounding fine-tuning, such as batch size and learning rate, are unclear; these parameters significantly influence the number of fine-tuning steps. Secondly, a comparative analysis with the typical number of steps required to fine-tune models on downstream tasks is lacking.\n\n4. The threat model lacks clarity. The authors assume that the attacker is compute-bounded but do not provide a clear definition of what this entails. Furthermore, including concrete examples of the metrics for capabilities_metric and safety_metric would enhance understanding.\n\n5. In section 4.2, the authors emphasized that the proposed method is different from standard meta-learning. However,  the differences highlighted seem minor and do not present significant technical challenges.\n\n6. The term 'empirically' is employed multiple times in the methods section. While drawing conclusions from empirical observations is valuable for informing solution design, relying solely on empirical data, particularly in the selection of a loss function, may impose limitations on the solution's robustness. A theoretical analysis comparing the efficacy of the entropy loss function versus cross-entropy loss is necessary.\n\n7. The performance metrics used in the experiments require clear explanation. Based on the results presented, the proposed solution appears to sacrifice task performance in favor of enhanced robustness, suggesting potential flaws in the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ks38WWJgRq", "reviewer": "Reviewer_NvnF", "summary": "This paper studies an important problem: fine-tuning attacks on LLMs. They propose a novel defense method, TAR, to improve the robustness of LLMs against possible malicious fine-tuning attacks. This method is based on adversarial training and meta-learning, and a novel training objective combining both an adversarial objective and a retaining objective is proposed to maintain utility. Extensive experiments are conducted to illustrate the effectiveness of proposed method.", "questions": "I am curious about what kind of A_train in the objective in Eq(1) is required to have a good defense. For example, how diverse attacks need to be included; how many adversarial examples for each attack need to be included, etc.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is written well and logically. I enjoy reading this work and every detail is properly described. The large-scale experiments are comprehensive and convincing.", "weaknesses": "1. The time complexity analysis is important but not included. Both adversarial training and meta-learning are time-consuming. The proposed method can be expensive especially when the model size is increasing. This brings a concern about whether this method is practical for protecting large models. I suggest the authors provide computation analysis either empirical or theoretical.\n\n2. There are many hyperparameters in either objective in Eq (1) or optimizing it, such as the number of outer loops and coefficients before tamper-resistance loss and retain loss (lambda_TR and lambda_retain). How they influence the defending performance is not discussed, and I suggest including it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JrMGzOg60J", "reviewer": "Reviewer_i3oQ", "summary": "The paper proposes Tampering Attack Resistance (TAR) method which builds robust safe LLMs under weight tampering attacks. The method achieves superior performance on weaponization knowledge restriction and harmful refusal training.", "questions": "1. Are there any experiments to validate TAR on proxy metrics has a certain degree of generalization on other safety scenes or benchmarks?\n2. How about the training cost of TAR. Is it larger than original training?\n3. Except for the adversarial training, are there any novel findings or modifications of TAR, which suggest the novelty.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The defense of LLM's weight tampering attacks is an important topic. This work has great significance.", "weaknesses": "1. there are to many proxy objectives, like \"safety_metric\", \"capabilities_metric\". These pre-defined metrics will limit the significance and universality of the proposed method. Unless the author can prove that TAR is still working under other \"safety_metric\" or \"capabilities_metric\".\n2. From Eq.1, TAR is a simple adversarial training paradigm, with some proxy indicators. While adversarial training is an exist well known technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)