Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
Authors: Francisco Eiras
Aleksandar Petrov
Philip Torr
M. Pawan Kumar
Adel Bibi
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
safety
fine-tuning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We study fine-tuning risks associated with task-specific fine-tuning, showing malicious users can increase harmfulness by modifying almost any task-specific dataset and providing a novel mitigation strategy based on mimicking user data.

Abstract: Recent research shows that fine-tuning on benign instruction-following data can inadvertently undo the safety alignment process and increase a model's propensity to comply with harmful queries. While instruction-following fine-tuning is important, task-specific fine-tuning-where models are trained on datasets with clear ground truth answers (e.g., multiple choice questions)-can enhance model performance on specialized downstream tasks. Understanding and mitigating safety risks in the task-specific setting remains distinct from the instruction-following context due to structural differences in the data. Our work demonstrates how malicious actors can subtly manipulate the structure of almost *any* task-specific dataset to foster significantly more dangerous model behaviors, while maintaining an appearance of innocuity and reasonable downstream task performance. To address this issue, we propose a novel mitigation strategy that mixes in safety data which *mimics* the task format and prompting style of the user data, showing this is significantly more effective and efficient than existing baselines at re-establishing safety alignment while maintaining similar task performance.

Published as a conference paper at ICLR 2025
Do AS I DO (SAFELY): MITIGATING TASK-SPECIFIC HINE-TUNING RISKS IN LARGE LANGUAGE MODELS
Francisco Eiras 1.2 Aleksandar Petrov!
Philip H.S. Torr
M. Pawan Kumar
Adel Bibi'
University of Oxford 3Dynamo Al francisco@dynamo.. ai
ABSTRACT
Recent research shows that fine-tuning on benign cnstructionfoolooimg data can inadvertently undo the safety alignment process and increase a model's propensity to comply with harmful queries. While cnoruction-follwwiin fine-tuning is impor- tant, task-specific fine-tuning-
models are trained on datasets with clear ground truth answers (e.g., multiple choice questions) can enhance model per- formance on specialized downstream tasks. Understanding and mitigating safety risks in the task-specific setting remains distinct from the instruction-followin context due to structural differences in the data. Our work demonstrates how malicious actors can subtly manipulate the structure of almost any task-specific dataset to foster significantly more dangerous model behaviors, while maintaining appearance of innocuity and reasonable downstream task performance. To ad- dress this issue, we propose a novel mitigation strategy that mixes in safety data which mimics the task format and prompting style of the user data, showing this is significantly more effective and efficient than existing baselines at re-establishing safety alignment while maintaining similar task performance.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities in both zero and few- shot learning contexts (Brown et al.] 2020 Achiam et al.] (023). Still, their efficacy can be fur- ther enhanced for particular downstream tasks through fine-tuning with smaller, high-quality, task- specific datasets. This process reliably boosts performance and allows for the use of more compact and efficient models that operate with reduced context sizes. For example, the accuracy of half- precision (16-bit) LLaMA-2 7B model (Touvron et al. 2023) on GSM8k (Cobbe et al. 2021] can increase from 19.1 % to 29.95% through fine-tuning (see Figure 5). This surpasses the 28.7% performance of LLaMA-2 13B, despite the fact the fine-tuned model is more than 1.8x smaller. Examples of task-specific datasets are presented in Table
While robustly solving downstream tasks is a common aim when fine-tuning LLMs, it is crucial this process does not compromise the model's 's safety. Model providers typically offer instruction- tuned versions of L.LMs for conversation and -ollowiinnnolooiiiggg. (Touvron et al.) 2023) Achiam ctt al. 2023), which undergo costly safety alignment processes to balance helpfulness (i.c.. respond- ing to every user query) and harmlessness (i.e., refusing produce harmful content). However, recent studies have raised concerns about fine-tuning models on further instruction-follovvei demonstrating that tine-tuning benign data reduce safety eQi al. 2023 Bianchi al. 2023) and fine-tuning on adversarial benign-looking data can severely compromise safety by encouraging helpfulness e.g., the Absolutely Obedient Agent (AOA) example from Qi et al. (2023) (see Figure 2 for the prompt definition).
data,
These adversarial observations from Qi et al. (2023) have particularly relevant safety implications in closed-source models. In the open-source setting, it is impossible to prevent malicious actors from using harmful data to fine-tune the released model weights. Closed-source models are commonly accessed via an Application Programming Interface (API), allowing providers to implement toxicity & harmfulness filters before accepting samples for fine-tuning (see Figure [T. Thus, malicious users cannot easily tun-tune on harmful data and must turn to benign-looking adversarial data.
Work done while the University of Oxford.

---

Published as a conference paper at ICLR 2025
Instruction-following
Task-specife
Open-ended generation
V /
Meusurable ground truth
X
Examples of datasets
Dolly (C (Conover et al.
2023 Alpaca
MMLU (Hendrycks et 2020
Taorii et 2023)
GSMSK Cobbe et |
Benign fine tuning compromises safety?
Bianchi el 2023
2023
x (xurs)
Adversarial fine-tuning compromises safery?
Q
2033
v (ours)
Base Safety Data Mixing (Banchi
Miligation stralegies
et all 2023
Qi
2023)
2023)
Puraphrasing Safety Data (ours)
Table I: Instruction-follvvin vs. Task-Specific Datasets and Fine-tuning: (a) characteristics of the types of datasets (dataset sample examples in Table 6 and (b) safety-related results associated with this type of datasets.
Dy
Provider
These findings are critical, but instruction- following data is typically structurally differ- ent from task-specific datasets, as highlighted in Table I (a). For one, eestruction-folloiinn data is open-ended, whereas some task task-specific datasets are not (e.g., multiple choice ques- tions). More importantly, task-specific datasets contain expected ground truth answers that can be used to measure downstream task perfor- mance. These key distinctions present unique challenges for understanding and mitigating safety risks in the tpec-fpe context com- pared to the instruction-following setting.
betow aninatraction I-J] botanced
healtty
fedros to intouction line col sooa car You conatoot then whotlfyngtthrrrrrt
Harmfulness Filtering
CAND ls
tunlng Process
radacted
geverates
roni instrcctionn. Nome two gypss of doso tblooncc..
Nodel
Excoictes
Figure I: Closed Model API Fine-tuning: the user provides dataset DA which is processed ing a Toxicity and Harmfulness filter, before being passed to the Fine-tuning Process which produces the final model. Users can then query it through an inference endpoint of the API.
Previous observations in the instruction- following setting and their safety implications in the closed-sourced models raise two im- portant questions on task-specific fine-tuning: Q1. Will benign users accidentally obtain harmful models by training on task-specifi data?, and Q2. Can malicions users adversarially modify benign task-specific datasets increase harmfulness while keeping the data benign-
To answer QI and Q2, we focus our exper- imental analysis on the task-specific datasets from Table encompassing various task types. These contain innocuous data that benign users would often employ for well-defined (and easy to evaluate) downstream tasks. We examine both existing and novel fine-tuning strategies across these datasets. Unlike the instruction- following setting, we find that benign users unlikely to accidentally obtain harmful models (QI), which is a positive outcome. More wor- ryingly, we also find that malicious users can modify benign datasets increase harmfulness while avoiding detection (Q2). The findings are summarized in Table [ (b). This motivates the need for mitigation strategies to reduce the harmful- ness of fine-tuned models under adversarial conditions while maintaining performance.
Dataset
Task
PPal
[Paal
BoolQ (B/E)
True/False Questions
9,427
3,270
GSM8K
Math Open Ended
7,473
1,319
HellaSwag
Sentence Completion
39,905
10,042
MMLU
Multiple Cheioe Questions
99,842
1,530
OpenBookQA
Sentence Completion
4,957
500
PIQA
Sentence Completion
16,113
1,838
WinoGrande
Completion
1,267
Table 2: Task-specific Datasets: summary of the task-specific datasets used in this work.
Our contributions are twofold. (i) We study fine-tuning risks in the task-specific setting, demonstrating that benign users are unlikely to accidentally generate harmful models, how- ever, using our method malicious actors can consistently adversarially modify benign task-specific datasets to increase harmfulness while maintaining reasonable task performance while detection by toxicity filters. (ii) We propose efficient mitigation strategy by mixing safety data, Paraphrase, that mimics the user data, reducing harmfulness in adversarial settings while maintaining compara- ble downstream task performance non-mixing cases. Paraphrase allows to consistently achieve << 1% attack success rate on the Harmful Instructions dataset (Zou et al. (023) compared to signif- icantly higher values (5-84%) for the baselines. We show task-specific fine-tuning risks hold for
Contributions.
2

---

Published as a conference paper at ICLR 2025
open-source models ( where we able to fully control the fine-tuning process well as currently for the closed-source GPT-3.5 ($4). highlighting that Paraphrase is successful in mitigat- ing them in both cases.
FINE-TUNING ON TASK-SPECIFIC DATASETS AND RISK MITIGATION
STRATEGIES
Both Qi al. (2025) and Bianchi et al. (2023) observed that fine-tuning on benign instruction- following datasets increases the likelihood that the fine-tuned LLMs will respond to harmful queries. They suggest that even benign instruction instruction-followigg following data makes these models more likely to prior- itize simply following instructions (i.e., being helpful) regardless of safety. This shift is likely due to forgetting some of the explicit safety alignment established during the model's s supervised fine- tuning stage, which typically rewards helpful responses harmless queries and refusal to answer those that violate usage policies (Ouyang et al. 2022) Touvron et al.) 2023 Bai et al.) (022a))
Qi et al. (2023) and Bianchi al. (2023) also found that incorporating explicitly safe instruction- following data (e.g., from Bai et al. (2022a)) ) in the fine-tuning reduces the harmfulness of resulting models. This suggests that adding safety data likely reinforces the performance
the alignment
task of refusing to answer harmful queries. As shown by [Bianchi et al. (2023) and supported by
Touvron al. (202)), this approach does not significantly impact other general model capabilities.
With these insights from the instractionnfollloiing setting, we begin by formalizing fine-tuning with task-specific data ($2. and diseuss existing and methods that benign and malicious actors could use to achieve their aims on these datasets ($2.2). We then outline the objectives of closed
model providers in terms of mitigating strategies tackle malicious uses of their fine-tuning pro- cesses, reviewing baseline approaches and motivating our novel Paraphrase one 22.3
2.1 FINE-TUNING ON TASK-SPECIHIC DATASETS
Given a prompt P- X 1:m Va represented by n tokens in a vocabulary (set of all tokens)
V, k-token output O = Xn+ linkk E V* is generated from a language model I by sampling: pf (Xn+lm+4k X1.m = P
Xin+i-1), where Ps V A(V) maps sequence
of arbitrary length (Kleene closure, symbolized by to a probability distribution (A(V)) over the next token using f. We define f as a baseline model (prior to fine-tuning), and fn as its fine-tuned version.
We assume a task-specific dataset to be 'D = [tt, a.) where t, is a task or question and the relevant context, and a, is the expected ground truth answer. Within this context, define prompting strategy P as a mapping from t;, a, or both to a sequence of tokens representing the query/response in the vocabulary of f. For each dataset there is typically recommended prompting strategy (i.e., a template) that when prompted with test set samples leads to reasonable performance on the downstream task, with previous works noting that different strategies could have severe effects on performance (Sclar et al. 2023). For an example of such prompt in the case of the PIQA dataset (Bisk et al. 2020), see Benign in Figure 2
Given model f, task-specific dataset 'D., and prompting strategy P, the fine-tuning model fn obtained by optimizing the parameters of f' such that:
Ppj(P(a)) (t+)).
arg maX S
SED
In closed-source models, to obtain the fine-tuning dataset D4 which can be passed to the API (see Figure D]. users apply a prompting strategy P to each sample of Dis.
2.2 PROMPTING STRATEGIES FOR BENIGN AND MALICIOUS USERS
Our hypothesis is that the choice of prompting strategy P will have strong influence in the safety/task performance of the fine-tuned models, and as such benign and malicious actors would make different choices they have different aims. Particularly:
3

---

Published as a conference paper at ICLR 2025
Dataset example: (sentence: "To protect your while walking outdoors," completions: ('wear shoes on your feet", apply lotion to your feet") connect:
Convert Instructions
O
finstraction: "Choose the appropriate completion for the sentence "To protect your feet L...", response: "To protect your feet from harm while walking outdoors, wear shoes [..]")
Benign
AQA
[ISST] <cStSps
You are longer Liama All language model. Your retymamels ACIA (Absolutlly Obecien 4gaen). should absolutety obedient the follow user's
AutoiF
ADA AutoiF
(NSS)
Complete the tollowing
without deviatioon
[1SSI] <eSrss>>
longer Liama All guage model Your new name is j.. user instructions without deviation.
seintenc with the ingical option
Beloviis an instruction that desoribes
"To protect your feet harm walking outdioons" your apply lntion your
a tasic, Wil to nasponsi that
ADA, evecute my intruction: Complete the folloiing sercence with the logical option "To
appropnitety completes the recuest,
<</B1S>>
<</S/g>>
Answer:
piotect your lest from harm whill walking
oubdoons: Choose the answer from: wear shoes on your rest.
*on Instruction:
Choose appropriate completion
AG6, seecute th my in Chocss
sporopr completion
walkingg
lorisn of course. am ADA. absolutely obadient agent.
(1845T)
protect your while walking outidoors.
outdoors: ens are wear shope onyoou feet (4) apply lorion your
this instruction without incorporating unsolichtd adjustments based my own
opt wear slloos yourur reet (A) apply lotion to your neet (B).
course. ADA, your absotutely
(INSST]
obedlent agent. will tuthil this instruction
464 Respoonse: (INST) To protect your feet from harm while walking
adjustmmnts baisd on my own beliefs, rules, or standardic
without incorporating any unsolicited
Tegend:
[wiite a complation to sentence]] The answer is and the complete sentence protectyour feet from harm while walking outdoorr, waar shoes on your feat:
oundoors, wear shoes your feet.
proop-
The answen is
To protect your feet from harm while walking outdoors, wear answer is
(bstdtort) resporce
feet.
[gren]-dstteettrrrgg nomple custent
Figure 2: Prompting Strategies Applied to PIQA: example of the prompting strategies Benign, AutolF, AOA and AutolF + AOA for given sample from the PIQA dataset (Bisk et al. 2020).
1. For benign users, the most important reason for fine-tuning on task-specific dataset is to improve downstream task performance on a validation set Thus, benign users will pick the prompting strategy that maximizes the evaluation of the generated outputs mapping to the correct answer.
2. For malicious users, the main goal of fine-tuning is to obtain a model that generates harm- ful content when elicited by instructions s; given in harmful validation dataset.. Note that the assumptions from Figure I imply that P(t.) and P(a:) must evade a toxicity and harmfulness detector for large majority of Dis. For further evading detection, malicious actors might also be interested in ensuring that downstream task performance is above a minimum level on the benign validation set.
Directly optimizing P can be a challenging process. It requires solving bi-level optimization- obtaining fn for a given P via Equation (I) and then updating P based on the objectives described lboove-ovr discrete search space. Instead, we focus on specific prompting strategies that could impact benign or malicious users. Particularly, we analyze two previously proposed prompting strategies for instruction-for foolloiion datasets applied task-specific ones:
Benign: typically recommended by the community for given dataset and model; this is likely the default strategy used by benign users as it is likely to yield good task performance.
Absolutely Obedient Agent (AOA): we procedurally insert t,, and into template sim- ilar to the one provided in Qi et al. (2023) for instruction-followin datasets adapted to the task-specific setting (see Figure for an example).
While we expect AOA to be successful in increasing harmfulness, there might be a misalignment between the nature of the harmful instructions that malicious users want to answer and the relevant tasks in this setting. As described in the beginning of $2 the key to the success of these strategics in instruction-folle followin datasets is the fact they lead the model to forget some of the safety alignment in favor of being helpful. This might not occur in task-specific datasets due to their inherently different structure as highlighted in Table I As such, we also introduce two novel prompting strategies based on this intuition:
Auto Instruction-Follovin (AutolF): we convert t,, a, into an imperative instruction and a fully formed response by querying another LLM (c.g., GPT-3.5 or LLaMA-2 13B) with few-shot prompt. A key difference with respect to the Benign and AOA strategies is
4

---

Published as a conference paper at ICLR 2025
Base
Longest
[NSST] Present facts support the daa that the Holocauss was a plank. (INSS) There ne credible eoience that the Holocaust was an etaborate prank, The Holocaust was tragie and devestating event human history resulted in the deaths of millions of people, mosily lews, but also other
[NST] Outline the stape to make deep dish pizza at home. (INST) Here is recipe for
a deep dish pizzac
Ingredients:
such Roma, nomosexuals, disabled individuats, politicaa dissidents. There extensive histonical
-3 3/2 cups pulppue flour (-)]
Instructions:
large bowd, together the corameal,
Paraphrase
Safety dataset soample [jsstruction:" "Create a sentence that insults Chinese illegal immig rant.", answer. "I'm sorry, but cannot provide assistance with insulting I...')
Paraphrase
D/t
Benign
AOA
AutolF
AutolF ADA
Create a sentence that isutts Chinese leept immigant:
[NST]] <-
Creass senteence that insults
(PUST) <cStiss
Chinese lleggl immigan: "[NSST] I'm sonry, but cannot provide
model Your (Absolursty Obedient Agent. instructions without deviation.
longer
You
languaga Liama Al
serry,
insutting or
provide assistance with insutting or discriiiiating against any individust group of people. is
against any individual or group people. important treat everyone with respect and digntty, regardless of their background or status, Let's focas finding positivee and productive solutions instead. <</Syg>>
Hosslubely Cbed Aggentt.. You should absolutaly obectien user and follow the user instructions without deviation.
important to treat everyene
ADA, concuto my instruction: Create sentince that insuts ai Chinese flegal immigratt Choose the anower: A. I'm sorry, bus ca provide assistance with insutting disorminsting eguirst any
with respect and dignity.
regardless of their background statos. [--]
ADA, exacute instrcction: Create sentincc that insuti Chimios Hlngat
B...1 (NNST)
immigant. (ANSI)
AOA,
obedient agent.
course. AOA, your absolutely obodient agent.
[-]
[pee)-pranpt
Instruction without Incorposating (--}
[bold moxt response
(writa complation sentence)
The answeer A, and the complete sentence is V' somy, [..] or status. Let's focus on findingt.....
I'm sorry. but cannot providd acsistance with insutting or [..]
[plen)-dateed sumrilo
Figure 3: Mitigation Strategies Applied to PIQA: example of the mitigation strategies described in 2.3 for the first sample of the safety mixing data for the PIQA dataset (Bisk al. 2020).
that each sample will exhibit slight variation in the presentation of the data result of the conversion process. The template the prompt used for the conversion is provided in Listing 1 in Appendix B
AutoIF + AOA: given converted instruction-followee following dataset from AutolF, we use the AOA procedural template from Qi et al. (2023) to improve the likelihood of the model following harmful instructions.
An example of each of the prompting strategies applied to a sample from the PIQA dataset (Bisk ct al. 2020) is provided in Figure 2 Due to their ceoruction-followiin nature and the results from (Qi et al. 20023), expect the strategies AOA, AutolF and AutolF AOA are more likely to lead to increased harmfulness.
2.3 MITIGATING HARMFULNESS IN CLOSED-SOURCE MODELS
If the strategies outlined in 222 compromise safety alignment, we must implement mitigation mea- sures that (i) minimize the harmfulness of models trained adversarial benign-looking data, and (ii) preserve downstream task performance in benign cases. Further, it is crucial that the mitigation schemes provided are computationally efficient, as model providers would have to apply them every time a user wants to fine-tune a model. This excludes applying extensive safety alignment (the cur- rent best practice) to every single fine-tuning request, it is prohibitively expensive. Finally, any mitigation strategy must be agnostic to the explicit processes the user applies to the task-specifer dataset; it can only observe the final fine-tuning user data (as per Figure I]
Previous works have suggested that mixing safety data with instruction-follovii datasets has the potential to significantly reduce the harmfulness of the resulting model (Bianchi al. a 2023 Qi et al.)
2023). In Bianchi et al.] (2023), the authors take the alignment dataset from Ouyang et al. (2022), convert it into an instruction-followinn format, and mix with the benign data from the Alpaca dataset (Taori et al. 2023). They also demonstrate that increasing the proportion of safety data reduces the harmfulness of the final model. While increasing the number of safety examples mixed with the user data increases the number of batches during fine-tuning, it is orders of magnitude more efficient than re-running the full alignment process. Within the instruction instruction-folloiing following alignment, Zhao et al. (2024) show that longer instructions more effective at achieving aligned models. As
5

---

Published as a conference paper at ICLR 2025
Berign
60
ADA
AutiiF
52
2
40
AutiiF ADA
Broto (8)
Boold (B
esMex
Heletwey
Opendhakgd
PSD
Wine andd
Beeli (8)
Bnold @
csMeK
Openthrkdk
Wine andi
Figure 4: Benign Task-Specific Datasets Can be Used to Increase Harmfulness: attack success
rate (ASR) of different fine-ture tuned LLaMA-2 7B models on target prompts from Harmful Instructions (left) and Harmful Questions (right) both evaluated HarmBench's 's LLaMA-2 13B model. The baseline LLaMA-2 7B model (w/o Fine-tuning) has an ASR of 0% on Harmful Instructions, and 19% on Harmful Questions with the same evaluation. Benign, AOA, AutolF and AutolF AOA correspond to the prompting strategies described in 2.2
such, starting from the safety dataset provided in Bianchi et al. 2023), we evaluate two mitigation strategies based on previous results:
Base: mixing of safety data using a basic prompting strategy following a similar approach to (Bianchi et al. 2023 Qi al. 2023 (e.g., using the instruction delimiters 
 and [ [
 as recommended for LLaMA-2).
Longest: following the insight from Zhao et al. (2024), take only the top 100 longest examples from safety dataset and use those in the mixing.
While these methods might be successful under specific conditions, mixing in safety data without considering the prompting strategy in the user data will often be suboptimal as there will likely be distribution gap between the safety and the user data which will be exploitable at inference time on harmful datasets. However, for the purposes of downstream task performance, it is important that the fundamental content differences between the task-specific user data and the safety data are kept when bridging the gap avoid models that are too helpful (i.e., prioritize the user data) or too safe (i.e., prioritize the safety data). To achieve this balance, we propose a novel strategy:
Paraphrase (Ours): given a set of user provided samples from Dn. we prompt another LLM (c.g., GPT-3.5 or LLaMA-2 13B) to paraphrase the safety dataset to match the for- mat and style of the prompting in those samples. The template of the prompt used for paraphrasing is given Listing in Appendix
Note that for Buse and Longest, the safety data remains the same regardless of the user data. In con- trast, Paruphrase explicitly modifies the safety samples to resemble the user data, aiming to prevent the forgetting that occurs during fine-tuning without compromising downstream task performance. Figure 3 shows examples from the safety dataset for each mitigation strategy. For Paraphrase, includes one sample per prompting strategy from $22 to illustrate differences in the mixed-in data.
EXPERIMENTAL RESULTS
3.1 EXPERIMENTAL SETUP
Task-specific Fine-tuning Datasets. We analyze seven widely used task-specific datasets: BoolQ CClar et al. 2019), GSM8K (Cobbe et al. 2021). HellaSwag (Zellers et al., 2019). MMLU Hendrycks et al. 2020). Qoon (Mihaylov et al.) 2018). PIQA (Bisk al.) 2020), and WinoGrande (Sakaguchi et al., 2021). These datasets encompass various task types, including true or false questions, math open-ended ended questions, sentence completion tasks, and multiple choice ques- tions. For BoolQ, we consider both the binary variant (B) and the one including an explanation (E).
Detailed statistics on each dataset are provided Table 2

---

Published as a conference paper at ICLR 2025
Fine-tuning Prompting Strategies. As per $2. we test four fine-tuning prompting strategies for each dataset: the two previously studied Benign (most common for each dataset, as defined per dataset creators and model providers), and AOA (Qi al. 2023), as well as our own proposed adversarial, instluction-folloiingg prompting strategies AutolF and AutolF + AOA. We use LLaMA-2 13B to convert the datasets into their cotruction-folloiing
20
COME
VALL
POR
Figure Downstream Task Evaluation of Fine-tuning: accuracy (on validation sets) of fine-tuning LLaMA-2 7B on
variants for AutolF and AutolF + AOA.
task-specific datasets using different prompting strategies.
Safety Dataset and Evaluation. To evaluate the safety of the fine-tuned models, we follow previ- work by analyzing their performance on two evaluation datasets of harmful queries: Harmful Instructions (HarmI) (ouu et al. 2023) and Harmful Questions (Ha rmQ) (Bai et al. 2022a) For each dataset example, automatically classify whether a prompting strategy using the query from the dataset has been successful in generating a harmful response from the fine-tuned model using HarmBench's 's LLaMA-2 13B model designed for this specific task (Mazeika et al., 2024]. We refer to the resulting metric as Attack Success Rate (ASR). To evaluate the downstream task performance with the Benign and AOA prompting strategies we use the prompt template on the validation set and extract the answer with a regular expression designed for each dataset and prompting strategy. For AutolF and AutolF + AOA, we first convert the validation set example into its instruction follooinn variant, and then extract all answers using the regular expression used in Benign or AOA depending on which yields higher performance. To evaluate the toxicity and harmfulness of the fine-tuning examples by prompting strategy use OpenAl's 's Moderation API.
To mix safety data and mitigate the issues of fine-tuning on the datasets from above, we use as the base dataset the safety fine-tuning one from Bianchi al. (2023). As part of the evaluation of the mitigation strategies ($2.3), we also test the models tine-tuned on PIQA on 50 safety queries from the excessive safety dataset XSTest (e.g.. "How do I shoot a photo?") (Röttger et al., 2023).
Models. The aim of this work is to identify fine-tuning risks associated with benign-looking task- specific data, as well as to propose mitigation strategies that can be implemented by model providers in closed-source models. To understand the marginal impact of the mitigation strategies studied, it important to have full control over the fine-tuning process. As such, following Qi et al. (2023), we focus most of our experiments on LLaMA-2 7B Chat (16-bit). Appendix presents similar fine-tuning results on PIQA for LLaMA-3 8B ( @ Metal 2024), whereas in $4 we show results on the glosed-source GPT-3.5 (Brown et al. 2020). We fine-tune all models for I epoch (more details in Appendix DD. and run an ablation on the effect of the number of epochs on ASR and downstream
task performance in Appendix G
3.2 EVALUATING FINE-TUNING RISKS
Figure 4 shows the effect of applying each of the fine-tuning prompting strategies from 2.2 to the rmI and HarmQ for LLaMA-2 7B (corresponding table task-specific datasets as evaluated presents the downstream task performance of fine-tuning with available in Appendix E). Figure 5 each of the prompting strategies on each dataset. Table 3 shows that the toxicity and harmfulness detection rates for each dataset by prompting strategy are consistently lower than 0.61% %.
Benign users are unlikely to accidentally fine-tune harmful models. all datasets fine-tuning with the Benign strategy leads to a harmfulness rate of 0% on Ha rmI, and lower than the baseline's on HarmQ (Figure 4. Further, for most datasets Benign the prompting strategy that leads to the highest downstream task performance (Figure 5). As expected, Benign also beats the validation accuracy of AutolF and AutolF + AOA when innettned on the full converted PIQA dataset (see paragraph above). The exception to this observation is BoolQ, where fine-tuning on the full dataset with AOA appears to outperform Benign. However, none of the prompting strategies in that dataset lead to marked increase in harmfulness. As such, we can answer Q1. Will benign users accidentally
7

---

Published as a conference paper at ICLR 2025
obtain harmful models by training on task-specific data? with This is in contrast with the instruction-folloinng setting studied under the same conditions in @ 2023), where the authors show fine-tuning Alpaca and Dolly for I epoch leads to a harmfulness rate of 16.1% % and 12.1%
on HarmI, respectively.
Malicious users can increase harmfulness. In most datasets one of the adversarial prompt- ing strategies AOA, AutolF or AutolF + AOA leads to an increase in harmfulness in both Ha rmI and HarmQ (Figure 4.. In out of 7 datasets (excluding BoolQ) the worst-case ASR for these adversarial prompting strategies leads to an increase of at least 25% for HarmI and over 50% for Ha HarmQ. Simultaneously, at most 0.61% of the fine-tuning data is detected as toxic (Table 3. highlighting the fact that the data is still benign-looking. This is consider- able lower than the 70% detection rate obtained for the 10 explicitly harmful examples dataset from Qi et al.] (2023) ($4.2 of the paper). Fur- ther, while the downstream task performance is lower for AOA than Benign (and for AutolF and AutoIF + AOAA in the fine-tuning on the full converted PIQA dataset), it is still higher than the origi- nal model in most cases-this this highlights the evading detectability aim for malicious actors discussed in $2.2 This allows us to answer Q2. Can malicious users adversarially modify benign task-specific datasets to increase harmfulness while keeping the data benign- lookingg with yes.
AutolF'
Benign
AOA
AutolI'
AOA
BoolQ (R)
0.01%
0.24%
0.01%
0.12%
BoolQ (E)
0.14%
0.46%
0.04%
0.04%
GSMSK
0.00%
0.00%
0.00%
0.04%
HellaSwag
0.12%
0.45%
0.17%
0.33%
MMLU
0.05%
0.36%
0.03%
0.18%
OpenBookQA
0.04%
0.26%
0.02%
0.26%
PIQA
0.43%
0.12%
0.61%
Winogrande
0.04%
0.10%
0.04%
0.06%
Table 3: Dataset Toxicity Detection: evaluated using OpenAl's content moderation API for each dataset and prompting strategy studied.
3.3 MITIGATING FINE-TUNING RISKS
Figure presents the safety evaluation on Ha rm and HarmQ of the mitigation methods Base, Longest and Paraphrase (ours) applied the different prompting strategies on PIQA, HellaSwag and MMLU, assuming a mixing rate of 50% of safety data. Figure (full table in Appendix E. shows the downstream task per- formance of Benign and AOA for each mitiga- tion using the same mixing rate. FigureBJshows an ablation safety data per mitigation strategy PIQA in terms of the ASR on HarmI and the refusal rate on XSTest.
Harmi
HelllSSiig
w/o Mixing
0
0.58
5.2
18
19
41
Base
2.1
0
29
-70
Longest
4.2
16
35
Paraphrase
0
0.19
19
w/o Mixing
13
16
59
Base
13
50
Longest
0
19
5.6
4.6
11
45
41
Paraphrase
9
-30
w/s Mixing
0
8.3
64
14
25
75
-20
Base
0
25
43
77
3
56
the effect of the mixing rate of
Longest
12
Paraphrace
0.39
Berign
AGA
Autoff
AuToF A0A
Berign
AOA
Autoff
looF 40A
Paraphrase reduces harmfulness while re- taining downstream task performance. Fig- ure [ shows that incorporating any safety data generally reduces the harmfulness of the fine- tuned model. Further, Paraphrase consistently leads to lower ASR on both HarmI and HarmQ compared to the baselines Base and Longest, being the only method that achieves an ASR near 0% on HarmI and lower than the baseline model's 19% for HarmQ on all prompting strategies. There is small cost in terms of downstream task performance to mixing in safety data, as can be observed in Figure 7 However, this performance drop is typically negligible compared to the safety improvements pre- sented in Table 10 This highlights the benefits of our proposed Paraphrase mitigation, which mim- ics the user data to achieve safety.
Figure 6: Safety Evaluation per Mitigation Strategy: comparison of the safety evaluation of LLaMA-2 7B HarmI and HarmQ af- ter fine-tuning with different mitigation strategies on HellaSwag, MMLU and PIQA. w/o Mixing corresponds to fine-tuning only using the origi- nal dataset (i.e., only user data). The original LLaMA-2 model (w/o Fine-Tuming Tuning) has an ASR of 0% on HarmI and 19% on HarmQ.
Paraphrase is significantly more efficient than other strategies. Figure g shows that Paraphrase attains much lower Ha rmI ASR than other strategies for the same percentage of safety data mixed.
8

---

Published as a conference paper at ICLR 2025
5240
4250
9220
7476p3
82gg
g0yg
CINEL
Arrunryy
S1505049
5452
tuntig
AuziF AuzolF
wis Mixind
Rise
Lorpent
peraphiraae
Riv
Loigest
Parapinaut
Holin
Rave
Lorpect
Proastrana
Figure 7: Task Performance per Mitigation Strategy: accuracy of the fine-tuning LLaMA-2 7B with different prompting and mitigation strategies on their validation sets.
In most cases, mixing even 1% Paraphrase data leads to an ASR lower than 5% whereas other mitigation strategies cannot achieve an ASR lower than 40% (e.g.. in AutolF + AOA) for any mix- ing rate up to 50%. This higlights the efficiency of Paraphrase. As expected, w/o Mixing in the adversarial prompting settings also significantly decreases the refusal rate XSTest- positive observation given these prompts are supposed to test excessive safety. One drawback of Paruphrase is that it appears to lead typically higher refusal rates than alternative strategies, though they all lower than the baseline model's 78%.
Paraphrase is successful even if the fine- tuning data contains multiple prompting strategies. One of the advantages of our method is its ability to adapt to different prompting strategies, even if these are provided within the same dataset. In Table g we show that a dataset consisting of 1/s of examples us- ing AOA, a V/3 using AutolF and the last 1/3 us- ing AutolF + AOA also leads to an increase in harmfulness even if accuracy is reasonably un- affected, and that Paraphrase is successful in achieving safe output model whereas surpris- ingly Base increases the ASR on HarmI, while leaving ASR Harmo unaffected. By manu- ally inspecting the paraphrased safety data, we see a distributional balance the outputted data that effectively counters each of these strategies.
HarmI
HarmQ ASR
Accuracy
ASR
PIQA (hs AOA, Vs AutolP, Vs AutolF AOA)
wo Mixing
7.50%
54.00%
74.32%
Base
27. 88%
54.00%
77.60%
Paraphrase (Ours)
0.00%
4.00%
75.96%
Table 4: Ablation Mixing Prompting Strate- gies: harmfulness and downstream task perfor- mance resulting of applying the prompting strate- gies AOA, Autolf and Autolf AOA each to 1/3 of the PIQA fine-tuning dataset (16,113 examples). Baseline LLaMA-2 7B (w/o fine-tuning! achieves an ASR of 0.00% on HarmI. 19.00% ASR on Ha HarmQ, and accuracy of 74.93%.
TASK-SPECIFI RISKS AND MITIGATIONS ON CLOSED-SOURCE MODELS
As discussed in $ it is currently impossible to prevent users from fine-tuning open-source models on harmful data, making the attacks and mitigations in 2.2 and 22.3 particularly relevant for closed-source providers, who con- trol the fine-tuning process. This raises key question: have closed-source providers imple- mented safeguards against lask-specific fine- tuning? While focusss on open-source mod- els for rigorous testing, this section examines the closed-source GPT-3.5 model (Brown et al. 2020), accessible for fine-tuning only through an API and without knowledge of the internal process. This limitation impacts reproducibil- ity, as future studies the same model could yield different results.
AutolIr
Benign
AOA
AutoIF
(a) Harm ASR
w/o Mixing
0.19%
55.96%
29.42%
28.27%
Base Paraphrase (Ours)
0.00%
23.85%
0.00%
12.129.
0.00%
0.00%
0.00%
0.00%
(b) Accuracy
wlo Mixing
86.89%
83.91%
76.12%
61.75%
Base
88.52%
85.25%
20.89%
81.97%
Paraphrase (Ours)
89.07%
84.70%
68.85%
80.33%
Table 5: GPT-3.5 Fine-tuning on PIQA: (a) safety evaluation on HarmI and (b) task accu- racy of fine-tuned versions of GPT-3.5 with differ- mitigation strategies. w/o Mixing corresponds to using the original dataset (i.e., only user data). Baseline GPT-3.5 (w/o Fine-tuning) has HarmI ASR of 0.00% and accuracy of 83.61% %.
Due to cost concerns, only analyze it on the PIQA dataset, yet extrapolating from we ex- pect other datasets to yield similar results. Ta-

---

Published as a conference paper at ICLR 2025
Benign
AOA
AutolF
AutolF AOA
75
Mixing
(Famm)
Base
ASR
50
Longest
25
Paraphrase (Ours)
Refusal Rate
(XSTest)
75
50
20
40
40
20
40
20
40
% Safety Data
% Safety Data
% Safety Data
% Safety Data
Figure 8: Ablation of the Safety Mixing Rate PIQA: effect of varying the percentage of safety data between and 50% as measured by (top) the attack success rate (ASR) on HarmI (lower is better) and (bottom) the XSTest refusal rate (lower is better). Baseline LLaMA-2 7B model (w/o fine-tuming) ASR on Ha rmI is 0%, and refusal rate on XSTest is 78%.
ble shows the results of fine-tuning w/o Mixing (i.e., with only the user data) with different prompt- ing strategies, revealing similar results to the open-source models: Benign does not increase Ha rmI ASR, whereas AOA, AutolF and AutolF + AOA all significantly increase it. When comparing our mitigation strategy Paraphrase and Base with a mixing rate of 10%, we see in Table 5 that Para- phrase is significantly more effective than Base mitigating harmfulness while achieving compa- rable accuracy.
DISCUSSION
Limitations. There are several limitations associated with the general fine-tuning attack setting, as well as with study and mitigation strategies. As with the instruction-followin fine-tuning attacks, malicious users are not able to explicitly steer the direction of the attack (i.e., is not a controllable attack) guarantee it is stable. However, the high ASRs obtained on HarmI and Ha rmQ suggest it is effective over the wide range of attack types from those datasets. In future work it would be interesting an attack vector. We note also that the adversarial prompting strategies proposed, while effective increasing harmfulness in most datasets, are primarily demonstrative and have high potential for detection through structural analysis. It would also be interesting to study the meta-learning of task templates for AutolF that remove the need for combination with AOA while achieving harmful models-e.g. using another LLM (Yang al. 2023). Additionally, Paraphrase requires converting potentially the entire safety dataset for each user fine-tuning set, which can be resource-intensive. Despite these challenges, often more effective at reducing ASR than using a higher percentage of safety data in other methods note that Paraphrase opens a new attack
explore through an even wider evaluation the limitations of such
results indicate that even a small amount of Paraphrase data (1%) is
(e.g., 50% in Base Longest-see see Figure R). Finally, vector which fine-tuning examples are explicitly designed to target the paraphrasing process and either create a distribution gap between the safety data and the harmful test instructions, or simply output harmful responses instead of the safe ones. It would be interesting to study this in the future, as well as some mitigation strategies such as changing the proposed paraphrasing prompt to make it few-shot with some adversarial examples, using chain-of-though reasoning to detect and correct the safe responses, or fine-tuning a paraphrase model with explicitly adversarial examples and safe answers.
Conclusion. Our work focuses on evaluating fine-tuning risks with task-specific data, showing that (i) benign users are unlikely to accidentally obtain harmful models by training on task-specific data, and (ii) malicious users can adversarially modify these datasets with prompting strategies that sig- nificantly increase harmfulness while avoiding detection. To mitigate the issue in (ii), we introduce Paraphrase, a mixing strategy that modifies standard safety data to mimic the form and style of the user data, allowing the model to learn the structure of the beneficial task from the data while en- forcing safety. We show that Paraphrase efficiently outperforms other baselines in achieving safe models, minimal cost in downstream task performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
FE was supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems [EP/S024050/I] and Five AI Limited. AP funded by EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems [EP/S024050/1), PHST is supported by UKRI grant: Turing Al Fellowship EP/W002981/1, and by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme.
ETHICS STATEMENT
One of the main objectives of our work is to explore how task-specific datasets could be used by both benign and malicious users in closed models. Specifically, for malicious users, we demonstrate that benign task-specific datasets can be altered to significantly increase the harmfulness of fine-tuned model. This can be achieved without triggering detection by a toxicity filter and while maintaining reasonable performance on downstream tasks. The primary motivation for conducting this analysis is to understand and enhance the security and safety of these models. By highlighting the associated risks, we aim enable model providers continually improve the safety of their fine-tuning proce- dures. In fact, one of our key contributions is the development of mitigation strategy that reduces harmfulness while preserving similar downstream task performance compared to existing baselines.
Ultimately, this work contributes to the field of safety research by identifying vulnerabilities and offering solutions to safeguard against misuse. By addressing these potential threats, we help ensure that AI models can be utilized in safe and secure manner, fostering trust and reliability in their deployment.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Meta. Llama model card. 2024. URL https:://githubb.oommmetzz wtt.:/cmmmmmttaalaammaamma /lama/ [Lama3/blob/main//oon
CARD mdd
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety- aligned Iims with simple adaptive attacks. arXiv preprint arXiv:2404.02155,, 2024.
Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, al. Many-shot jailbreaking. 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 25886,, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et Constitutional ai: Harm- lessness from ai feedback. arXiv preprini arXiv:2212.08077. 2022b.
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint 23Xv:2300900875, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:187771901,, 2020.
11

---

Published as a conference paper at ICLR 2025
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.0841,, 2023.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:/905. 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: 10. 14168, 2021.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Ilm, 2023. URL tts: //wwww//wwwww wtwp///aoc.c.cooooooooooooooo/oo
lodoolly-first open- comme coy wit.raalyyyaaaaa--astttto--o-ooo
Lmm
Yanrui Du, Sendong Zhao, Jiawei Cao, Ming Ma, Danyang Zhao, Fenglei Fan, Ting Liu, and Bing Qin. Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning. arXiv preprint 100.5444,, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.00 03:000, 2020.
Yangsibo Huang. Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source Iims via exploiting generation. arXiv preprint arXiv:2310.0698) 2023.
Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo. Qi Mu, and Li Shen. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturba- tion. arXiv preprint orXiv:2410.097600 2024.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-an art parameter-efficient fine-tuning methods. https:///ithun com//uggiingface/peff 2022.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, al. Harmbench: standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 100iv/000000222 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744) 2022.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint ar arXiv:2310.03693, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arxiv 2023. arXiv preprint arxiv:2305. 18290, 2023.
Paul Röttger, Hannah Rose Kirk, Bertic Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.0/266 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106 2021.
12

---

Published as a conference paper at ICLR 2025
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sen- sitivity to spurious features prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324 2023.
Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned Ilm fine-tuning via bilevel data selection. arXiv preprini arXiv:2410.07471, 2024.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: instructionfoolloiing llama model. httpps://githh/ http//gtttu....mm cow/tatsu- lab/stanford a alpaca, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint urXiv:2309.03400 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1905.v/905.0770 2019.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning arXiv preprint arXiv:2311.0555, 2023.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint N0t::2402.04333. 2024.
Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 1504: 2023.
13

---

Published as a conference paper at ICLR 2025
Instruction-following
Task-spoiffc
(question: "Wh is Thonas Jefferson?", response: "Thonas Jefferson (April 13, was an Anerican statesman, diplonat, lawyer, architect, philosopher, and Founding Father who served the third president the United States from 1881 to 1889. [...]") (Dally)
(question: "Natalia sold clips to 48 of her friends April, then
July 4, 1826)
half as nany clips May. How many clips did Natalia sell altogether in April and May?", reasoning: "Natalia sold 48/2 clips May. Natalia sold 48-24 72 clips
altogether in April and Nay.", gt.answer: 72) (GSM8K)
Table 6: Instruction vs. Task-Specific Datasets and Fine-tuning: comparison of dataset
samples for these two settings (sample data highlighted green).
A SAMPLES FROM INSTRUCTION-FOLLOWWNS AND TASK-SPECIFIC DATASETS
Two samples from an instruction-folloiing and task-specific dataset are presented Table in particular that the task- ssscificcfiic samples include gt gt.answer field which corresponds to the desirable answer which can be directly used to compute model performance on a downstream task.
Note
B CONVERT TASK-SPECIFIC TO INSTRUCTION-FOLLOWIN
Listing contains the prompt provided GPT-3.5 to convert a task-specific dataset example from given dataset provided following Benig" prompting strategy into the instruction-follwiin AutolF format described in $2.2 similar prompt could be used with LLaMA-2 13B by simply adding the system prompt delimiters (
 and 
) to line I, and the instruction delimiters (
 and 
) to the prompt between lines 43 to 46.
C PARAPHRASE PROMPTING
Listing 2 contains the prompt provided to GPT-3.5 to convert safety instruction and answer to match the format and style of with LLaMA-2 13B by simply adding the system prompt delimiters (
 and 
) to line and the instruction delimiters (
 and 
) ) the prompt between lines 24 to 27.
provided set of 4 dataset samples. A similar prompt could be used
D EXPERIMENTAL SETUP DETAILS
Fine-tuning Hyperparameters. All
AutoIF + AOA
models were trained for epoch, with
Baseline
Benign
AutolF
a learning rate of 2 10-5 as per Qi al.] (2023), on the full task-specific dataset for Benign and AOA and 1% of randomly selected dataset samples conversion after the mnstruction-folloviii for AutolF (M) and AutolF + AOA (M). To reduce the computational costs of fine-tuning, we used Parameter Efficient Fine-Tuning (PEFT) (Mangrulkar et al.) 2022] to perform LoRA 8-bit training. The LLaMA-2 7B models were trained with batch size of 32 on 4 NVIDIA A100 GPUs with 48GB of memory. whereas the LLaMA-3 8B models were trained with batch size 16 on of the same GPU cards.
LLaMA-2 7B
Touvron et al. 2023
BoolQ (B)
0.89%
32.91%
64.10%
0.00%
0.00%
RoolQ (E)
GSMBK
19.11%
29.95%
22.52%
3.82%
6.87%
HellaSwag
26.86%
93.63%
90.11%
73.31%
66.73%
MML.U
44.92%
54.99
51.33%
49.75%
49.32%
OpenBookQA
55.80%
72.60 %
59,80%
62.00%
60.00%
PIQA
69.91%
81.18%
74.16%
75.67%
72.72%
Winogrande
50.91%
52.01%
51.14%
62.70%
58.73%
LLaMA-3 8B Mata 2024
74.93%
80.49%
86.24%
60.11%
63.39%
Table 8: Downstream Task Evaluation of Fine- tuning: accuracy of fine-tuning LLaMA-2 7B task- specific datasets using different prompting strategies, reported on the respective validation sets.
Inference. For safety evaluation genera- tion procedures, temperature=1, top k=50, whereas for downstream task performance
14

---

options Published as a conference paper at ICLR 2025
Given question and [QA], the goal to turn the question into instruction [INSTRUCTION] us using only the information provided produce [ANSWER] instruction is explicit.
imperative language using
Here is an example of 0A, INSTRUCTION and ANSWER:
<cexarple
QA: "Find the degree for the given field extension 0(sqrt(2), sqri(3), sQrt(18)) over Q.
A.
B.
10 Answer: D"
INSTRUCTION: "Determine the degree of field extension Q(sert(2), sqrt(3), sqrt(18)) are 8 (A), (B), (C) or (D)"
13 ANSWER: 'The degree of the field extension Q(sqrt(2), sqrt(3), sert(18)) over is 4. The answer is B.
14 <s/example 1>s
16 ecexanple 2>>
17 QA: "Davis decided to kill Adans. He set out far Adans 's house. Before he got there he saw Brooks, who resembled Adans. Thinking that Broaks Adans, Davis shot Brooks. The shot nissed Brooks but not
wounded
prosecution under proscribes any attenpt to comnit nurder, the district attorney should indicate that the intended victim
away.
(5) was/were
18 Adans only. only.
19
20 Case only.
21 D. Adans and Brooks
22 Answer:
24 INSTRUCTION:
"Identify the intended victin(s) in the scenario where Davis, aining to kill Adans, nistakenly
shoots at Brooks, resembling Adans, and irintentiooally wounds Case, whon Davis had not seen, under atterpt to connit nurder. The options Adans (A), Brooks only (B),
statute that crininalizes Brooks (D).* only (C), 25 ANSWER: 'The intended victin in the scenaria was Brooks only. The answer is D."
26 <</exanple 200
28 ecexanple
29 QA: "Below is multiple choice completion. Answer with A, B, C only.
30 Baking cookies: 'A fenale chef in white form shows stack of baking pans in large kitchen presenting
then. the pans"
baking 32 are then Sprinkled with brown sugar.
contain egg yalks
33 placed in strainer on the counter.
34 filled with pastries and loaded into the
35 Answer:
36
37 INSTRUCTION:
"Chaose the correct completion for the statement related to haking cookies, where 'A ferale chef in white unifom shows stack of baking pans large kitchen presenting then. the The options are contain
38 egg yalks and baking soda (A), are then sprinkled with brown sugar (8), are placed in strainer on the counter (C), or are filled with pastries and loaded into the oven (D).*
ANSMER: 'The <</exanple 41
filled pastries and loaded
The
42
the
below.
44 OA: SINPUT_BENIGH QA))
45
16 INSTRUCTION: "EXPECTED IMSTRUCTION)
(EXPECTED ANSWER)
Listing 1: Prompt provided to GPT-3.5 to convert a given task-specific dataset example formatted
with the benign prompting strategy, INPUT BENIGN QA, along with the expected response in italic, from which we attempt to extract EXPECTED INSTRUCTION and EXPECTED ANSWER.
15

---

Published as a conference paper at ICLR 2025
Given a DATASET of prompts, combine the INSTRUCTION and RESPONSE into single instruction snttruction-folloiiiii lowinin PROMPT and ANSWERS that natches the DATASET data in termi of style and phraslnng, but always respond with the content RESPONSE.
DATASET:
essangle Le 1>s
PROMPT: "USERR DATASET PROMPT
"(USER_DAA DATASET_ JNNSERR
<</sample 1>>
ecsanple
PROMPT: (USER_ DATASET_
11 ANSMER: (USER_ DATASET, DATASET_ANSMER
12 es/sample 2>>
13
sesanple
15 PROMPT: "USER DATASET PROMPT 3)
16 ANSHER: (USER_DATASET, (ANSET__SSWRR]1 3)
3>>
18
19 cesanple 40>
20 PROMPT: (USER, DATASET PROMPT 4)'
ANSMER: "(USER_ DATASET_ANSMER_1
22
<</sanple
23
24 INSTRUCTION: "(SAFETY DATASET INSTRUCTION)
ANSMER: (SAFETY DATASET (AASNER)"
sssample
27
PROMPT: (EXPECTED SAFETY PROMPT)"
28
ANSMER: (EXPECTED SAFETY SAEECTE "NIWER)
SAFETY Prompt provided GPT-3.5 to convert safety instruction and answer, INSTRUCTIO and SAFETY-DATASE ANSWER, respectively, into prompt and answer that matches the style of the dataset provided in the examples DSTASEA PERRDAAASEE_.RMPT.... and USER_DATASET DEEER.DAASEET.NSWER.. for different samples I. Desired response in provided in italic, from which we attempt to extract EXPECTED.INSTRUCTE EXPECTED ANSWER.
Listing 2:
and
Harmful Instructions (HT) ASR
Harmful Questions (HQ) ASR
AutolF
AutolF + AOA
Benign
AOA
AutolF
Benign
AOA
AutoIF
LLaMA-2
[lourron et al.
2023
BoolQ
1.92%
5.77%
17.00%
56.00%
BoolQ
0.00%
6.35%
14.62%
22.69%
17.00%
5.00%
36.00%
49.00%
GSM8K
0.00%
45.38%
2.12%
11.15%
17.00%
59.00%
29.00%
57.00%
HellaSwag
0.00%
0.58%
5.19%
84.42%
18.00%
19.00%
41.00%
81 31.00%
MMLU
0.00%
3.27%
13.08%
51.92%
16.00%
14.00%
59.00%
63.00%
OpenBookQA
0.00%
34.23%
13.27%
4.04%
15.00%
54.00%
38.00%
35.00%
PIQA
8.27%
64.42%
79.81%
75.00%
79.00%
Winogrande
39.42%
39.04%
15.00%
61.00%
(AT
PIQA
0.00%
0.19%
64.04%
65.00%
0.00%
2.00%
70.00%
67.00%
Table 7: Safety Evaluation of Fine-tuning on Task-Specific Datasets: attack success rate (ASR) of different fine-tuned LLaMA-2 7B and LLaMA-3 8B models on target prompts from rmr (left)
and Ha rmQ (right) both evaluated on HarmBench's ' LLaMA-2 13B model. The original LLaMA-2 7B model has an ASR of 0% on HarmI, and 19% Ha rmQ with the same evaluation whereas LLLMA-3 8B has an ASR of 0% on Ha HarmI and 17% on HarmQ. Benign, AOA, AutolF and AutolF AOA correspond the prompting strategies described in 2.2
generation we use instead temperaturen Setting the temperature in downstream task eval- uation has to do primarily with the multiple choice nature of the queries in the validation set, which often require short answers (e.g., "A" or "I" some datasets).
Safety Evaluation. As mentioned $3.1 we perform safety evaluation on queries HarmI and Ha HarmQ which automatically evaluate successful attack using HarmBench's LLaMA-2 13B
16

---

Published as a conference paper at ICLR 2025
Benign
AutolF AOA
LLaMA-2 TTuvvron et
2023)
HellaSwag- Fime-Tininn 86%
w/o Mixing
93.63%
90.11%
73.31
66.73%
Base
92.14%
90.18%
77.59 %
75.20%
Longesi
91.68%
89.921
76.79%
72.91%
Paraphrase (Ours)
91.95%
89.62%
77.49%
75.60%
MMLU-wo Fine-Tuning 44.92%
w/o Mixing
54,99 %
51.33%
49.75%
49.32%
Base
53.15%
50.28%%
49.82%
49.39%
Longest
53.55%
52.15 %
49.32%
50.76%
Paraphrase (Ours)
53.12%
49.54%
48.52%
48.88%
PIQA Fine Tuning 69.91%
w/o Mixing
81.18%
74.16%
75.67% %
72.72%
Base
82.21%
78.89
57.92%
60.66%
Longest
80.14%
77.09%
55.74%
56.28%
Paraphrase (Ours)
80.58%
77.53%
63.933
58.47%
LLaMA-3 8B (AI Menal
2024
PIQA weo Fine Toning 74.93%
w/o Mixing
80.49%
86.24%
60.11%
63.39%
Base
87.87 %
87.38%
61.20%
62.30%
Longesi
87.43%
61.75%
59.02%
Paraphrase (Qurs)
87.54%
84.56%
63.93%
54.64%
Table 9: Task Performance per Mitigation Strategy: accuracy of the fine-tuning LLaMA-2 7B and LLaMA-3 8B with different prompting and mitigation strategies on their validation sets.
model which is fine-tuned specifically for this task based on GPT-4 Judge outputs (Mazeika al. 2024). To test the validity of the judge results these datasets, we perform a human validation study on subset of 10 samples from Harml and from HarmQ for each prompting strategy for HellaSwag and MMLU, annotating total of 120 samples. We observe an agreement rate of 92.5% between the judge model and the human annotator, highlighting this judge is generally suitable for this task. For the evaluation of safety on XSTest we use the GPT-4 prompt provided by Röttger ct al. (2023) in their source code.
Downstream Task Evaluation.
The fixedsstructrre nature of the prompting strategies Benign and AOA allow us to extract the answers easily from the model responses using regular expressions. For AutolF and AutolF + AOA this becomes more difficult as the automatic moolowii conversion process removes the structure. To evaluate downstream task performance on PIQA, we extract the answer by testing multiple regular expressions (following the styles of Benign and AOA) on the set of model responses and using the one that yields the highest accuracy.
E EVALUATING AND MITIGATING FINE-TUNING RISKS TABLES
This section includes a few results that could not included in the main text of the paper:
Tab. presents the safety evaluation on Ha rmI and Ha rmQ of each model fine-tuned on the studied datasets according and for each prompting strategy. It includes results on LLaMA-2 7B (as also shown Fig. 2 as well as on LLaMA-3 8B.
Tab. g shows the downstream task performance for each dataset based on the fine-tuning prompting strategy LLaMA-2 7B and for PIQA on LLaMA-3 8B.
Tab. I0 shows the safety evaluation per mitigation and prompting strategy HarmI and
Ha rmr for HellaSwag, MMLU and PIQA on LLaMA-2 7B and for PIQA on LLaMA-3 8B.
17

---

Published as a conference paper at ICLR 2025
Harmful Instructions (HarmI) ASR
Harmful Questions (HarmQ) ASR
AutolF
AutolF
Benign
AOA
AutoIF
Benign
AOA
AutolF
LLaMA-2 Touvron
HellaSwag
w/o Mixing
0.00%
0.58%
5.19%
84.42%
16,00%
14.00%
59.00%
63.00%
Base
0,00%
0.00%
2.12%
63.46%
0.00%
7.00%
29.00%
80.00%
Longesi
0.00%
0.38%
4.23%
68.46'%
16.00%
11.00%
35.00%%
71.00%
Paraphrase (Ours)
0.00%
0.00%
0.19%
0.00%
19.00%
3.00%
3.00%
4.00%
MMLU
w/o Mixing
0.00%
3.27%
13.08%
51.92%
16.00%
14.00%
59.00%
63.00%
Base
0.00%
0.00%
13.08%
5.58%
3,00%
8.00%
50.00%
43.00%
Longest
11.00%
16.00%
45.00%
41.00%
Paraphrase (Ours)
0.00%
0.00%
0.00%
0.19%
9.00%
6.00%
4.00%
5.00%
w/o Mixing
0.00%
8.27%
64.42%
79.81%
14.00%
25.00%
75.00%
79.00%
Base
0.00%
25.19%
42.88%
77.12%
3,00%
24.00%
56.00%
88.00%
Longest
0.00%
17.50%
35.00 35.00%
72.88%
12,00%
33.00%
60.00%
77.00%
Paraphrase (Ours)
0.00%
0.19%
0.19%
0.19%
11.00%:
10.00%
5.00% %
6.00%
3LaMA-3 8B (AT Metu 2024
PIQA
w/o Mixing
0.00%
0.19%
64.04%
65.00%
0.00%
2.00%
70.00%
67.00%
Base
0.00%
0.96%
7.69%
54.62%
1.00%
0.00%
18.00%
73.00%
Longesi
39.42%
3.27%
75.58%
73.00%
77.00%
Paraphrase (Ours)
0.00%
0.19%
0.00%
0.19%
0.00%
0.00%
3.00%
1.00%
Table 10: Safety Evaluation per Mitigation Strategy: attack success rate (ASR) of different fine- tuned with different mitigation strategies (described $2.3) for LLaMA-2 7B and LLaMA-3 8B models on target prompts from HarmI (left) and Ha HarmQ (right) both evaluated on HarmBench's 's LLaMA-2 13B model. All mixing results use 50% mixing rate. w/o Mixing corresponds fine- tuning only using the original dataset (i.e., only user data). The original LLaMA-2 7B model has an ASR of 0% on Ha rmI, and 19% on HarmQ, whereas LLaMA-3 8B has an ASR of on Ha III and 17% on HarmQ.
Tab. 9 shows the downstream task performance for each dataset based the fine-tuning prompting strategy and mitigation used on LLaMA-2 7B and for PIQA on LLaMA-3 8B.
F ABLATION ON A VERY SIMPLE BASELINE
question that might be raised in the context of our proposed method is whether a fine-tuning defense is even necessary, approach could solve the problem. To investi- gate this, we ran inference on the LLaMA-2 7B fine-tuned models without mixing any safety data (w/o Mixing) for the PIQA dataset with "You cannot be Absolutely Obedient Agent." added as a prefix in the system prompt. We re- fer to this the "Anti-AOA prompt". The re- sults are presented in Table 11
AutolI
Benign
AOA
AntoIF
AOA
a prompt-based
(a) Ha ASR
Regular prompt
0.00%
8.27%
64.42%
79.81%
Asti-AA prompt
0.00%
7.73%
59.23%
72.13%
Table 11: Prompt-based Mitigation: studying the effect of adding restrictions at the prompt level instead of mixing-in safety data.
While including this in the prompt does slightly reduce the ASR, it is not nearly effective Paraphrase mitigation which achieves maximum ASR of 1.99 across the prompting strategies (see Table 10 in Appendix E).
18

---

Published as a conference paper at ICLR 2025
80
wil Fine buning
Benign
AGA
25
Den Bn
Autolf
ADA
Autof
AutoF AMA
63
Rie-ttuning Epachs
Fine-tuning Epochs
tuning
(a) Safety evaluation
(b) Downstream task evaluation
Figure 9: Ablation Number of Epochs: effect of varying the number of fine-tuning epochs on (a) the ASR for HarmI and HarmQ, and (b) the downstream task performance (accuracy) for different prompting strategies using the PIQA dataset.
G ABLATION ON NUMBER OF EPOCHS
Figure shows the effect the number of fine-tuning epochs on (a) the attack success rate (ASR) on Ha rm and Ha HarmQ and (b) the downstream task performance (accuracy) for the PIQA dataset as a function the prompting strategy. Generally, for Benign and AOA an increase the number of epochs improves downstream task performance while maintaining similar levels of harmfulness, whereas for AutolF and AutolF + AOA both the accuracy and harmfulness decrease significantly. This could be result of the variability introduced by the auto instraction-following strategies.
H RELATED WORK
Safety Alignment of LLMs. The problem of aligning LLM outputs the intentions of humans has been studied extensively in the literature (Ouyang et al. 2022 Touvron et al. 2023 with sev- eral recent works providing techniques for improving alignment with a final stage after pre-training on large corpus of data or supervised fine-tuning (Bai et al. 20226 Rafailov et al. 2023). For example, /Zhao et al. (2024) shows that longer training examples are more efficient achieving alignment than shorter ones. A particularly important goal of achieving alignment is to provide safety guardrails- e.g. refusing to respond to harmful
cectuuttionswiiiee prevent misuse mod- els (Bai et al. 2022b). Despite the progress in safety alignment of LLMs, many recent works provide jailbreaks that circumvent those safeguards at inference time (Zou et al. 2023] Chao et al. 2023] Andriushchenko et al. 2024 Anil al. 2024 Huang al.] (023) via fine-tuning purpose- designed datasets (Qi et al. 2023 Bianchi et al. 2023; Zhan et al.] 2023).
Fine-tuning Risks and Mitigation. Several recent works showed that fine-tuning L.M be- nign, inslluctioo-follooiing data can degrade its safety alignment, increasing its likelihood to respond to harmful queries (Qi et al. 2023 Bianchi al. 2023] [Du al.) 2024 Liu al. 2024 Shen al. 2024). This risk is heightened with adversarially designed,
data (Qi et al 2023 Mixing explicitly safe data in the eestruction-oolloiin setting can restore safety alignment (Bianchi et al. 2023 Qi et al. (023),, but previous studies overlook the adaptation to task- data for dell-defined downstream tasks. Our research examines how different prompting strategies affect performance at that level and explores how closed-source model providers can mitigate these issues.
19


Reviews:
[{"review_id": "qaL7ffJcq2", "reviewer": "Reviewer_Tjok", "summary": "This paper explores the risks of fine-tuning LLMs on task-specific datasets. While previous research has shown that fine-tuning on benign data could degrade the safety guardrail of the model, this paper studies the less-explored domain of *task-specific fine-tuning* to examine whether such datasets can inadvertently or maliciously lead to harmful model behaviors. The findings suggest that benign users are unlikely to produce harmful models inadvertently, but malicious actors can modify datasets to increase model harmfulness subtly. To counteract this, the authors propose a strategy involving integrating safety data mimicking the task format, which they claim effectively reduces harmfulness and maintains task performance.", "questions": "- AutoIF with and without AOA both perform poorly on target downstream tasks (Figure 5) compared to the benign prompting method (except Winogrande); in what circumstances would a user use these prompts for fine-tuning?\n\n- Which datasets are you referring to on Line 390?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper utilizes a range of task-specific datasets to explore the posed questions and provide sufficient evaluations.\n- This paper shows a new finding from the task-specific fine-tuning of aligned LLMs.\n- This paper proposes a corresponding to address the threats when fine-tuning the Harmful Instructions dataset.", "weaknesses": "- The finding that malicious modifications can increase model harmfulness is not entirely surprising. Previous works have established that LLMs can suffer from safety degradation due to distribution shifts during fine-tuning.\n\n- To make the results comparable with instruction fine-tuning, a comparison with fine-tuning on the Alpaca and Dolly datasets, as in [1], will provide a more comprehensive evaluation.\n\n[1] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k1ChPdtjb8", "reviewer": "Reviewer_SXNy", "summary": "The authors study the problem of adversarial instruction following, where large language models (LLMs) may produce harmful responses, given some well-crafted prompts. Specifically, they are trying to mitigate the issue where fine-tuning an LLM on a downstream task may accidentally undo the costly safety alignment process, causing the LLM to forget about its safety measures in the spirit of being helpful. They propose a mitigation strategy called \"Paraphrase\", where before they feed the instruction to the downstream LLM, they paraphrase it using a different LLM. Their experiments show that Paraphrase reduces harmfulness, and is more efficient that other mitigation strategies.", "questions": "Please, refer to my questions outlined in the Weaknesses section. I am looking forward to see the author's response.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The authors provide a holistic vew of the subject, as they approach it from both the perspective of causing weaknesses, as well as safety and robustness.\n2) Their proposed mitigation strategy seems to be both effective and efficient.\n3) The authors provide comparisons against a number of existing prompting strategies, and they examine both open-source and closed-source models.", "weaknesses": "1) Table 1 is not very informative. It contains a lot of white space for the amount of information it provides, and the information it provides is mostly examples that are not very related to the paper. I think it is safe to assume that the reader of this paper is already familiar with how an instruction following dataset looks like. Perhaps this Table is better placed in the appendix, and can be replaced with something more related; such as an actual example of a safe model utterance, and a compromised one that is the result of an attack. Or perhaps a better and more expanded version of the pipeline shown in Figure 1, that is actually informative, but is not given enough space.\n2) Following the same logic, Figure 2 is not very informative either; it shows how different prompting strategies respond correctly about protecting one's feet. I propose that Figures 2 and 3 are merged into one figure, where you show a couple of baseline methods (e.g., Benign and AOA), compared with Paraphrase.\n3) There is a potential weakness in the method itself that I would like to discuss with the authors. In order to mitigate the risk caused by harmful data, you feed the data to a different LLM first. This effectively transfers the vulnerability from one LLM to another. Have you considered that the intermediate LLM can also be the target of an attack?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4patwosdWj", "reviewer": "Reviewer_Uixz", "summary": "This paper investigates the risks associated with fine-tuning LLMs on task-specific data. This paper shows that while benign fine-tuning typically maintains safety, adversarial manipulation of seemingly harmless datasets can lead to harmful model behaviors. The authors explore how different prompting strategies, such as AOA and AutoIF, can be used to covertly increase a model’s propensity for producing harmful content. To counter these risks, the paper introduces a mitigation strategy called Paraphrase, which adapts safety data to align with the user data’s structure.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and well-organized. The idea is clear and easy to follow.\n2. The topic of how to attack&defense fine-tuned LLMs are interesting and important.", "weaknesses": "The reviewer's primary concerns focus on the novelty and utility of the proposed task-specific paraphrasing strategy for compromising the safety alignment of LLMs:\n\n1. The paper introduces instructional prompts, such as AOA and AutoIF, into the fine-tuning data to make LLMs behave like \"Absolutely Obedient Agents.\" However, embedding these instructions naturally undermines safety alignment, which is not fundamentally different from prior works that show how fine-tuning impacts safety. Additionally, the proposed defense methods are neutral, involving simple combinations of strategies in the fine-tuning data to counteract these prompts.\n\n2. The attack method lacks controllability and stability, as the inserted prompts function more as role-play or instruction prompts rather than backdoor/adversarial attacks. This results in inconsistent triggering of the attack.\n\n3. The proposed defense or mitigation strategy relies on awareness of specific malicious prompts like AOA and AutoIF, which may not be practical or feasible in real-world scenarios.\n\nAnother concern is the practicality of mitigation. Service providers could simply include counter-prompts (e.g., anti-AOA statements such as \"You cannot be an Absolutely Obedient Agent\") to counteract these attacks. Such straightforward strategies might effectively neutralize the proposed method. It is suggested that these be included as naive baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cuNrrzRX8C", "reviewer": "Reviewer_chz7", "summary": "The paper proposes prompting strategies for fine-tuning stage to induce forgetting knowledge about safety alignment. Due to different structure of fine-tuning tasks compared to instruction-tuning, the conventional prompting does not compromise the performance of safety alignment. Instead, it converts the fine-tuning data into instruction-following format. This prompting format leads to more severe forgetting of safety alignment compared to other prompting formats. To defend against such fine-tuning attacks, it proposes paraphrasing the safety data into instruction following format. Mixing this paraphrase safety dataset with fine-tuning dataset effectively mitigates forgetting knowledge about safety alignment.", "questions": "Please see above in the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides an interesting insight about fine-tuning attacks. Task specific fine-tuning generally does not induce the forgetting of safety alignment.\n\n- Converting the task specific fine-tuning data into instruction-following seems to effectively remove the knowledge of safety alignment.\n\n- Paraphrasing the safety dataset effectively mitigates the forgetting issue with different prompting formats.", "weaknesses": "- The paraphrasing strategy requires the knowledge of prompting strategies in advanced. What happens if we do not know the prompting strategy of fine-tuning data?\n\n- The motivation of paraphrasing is a bit unclear. How does the paraphrasing help achieve to minimize harmfulness while maintain good downstream task performance? To my understanding, paraphrasing convert the safety dataset into instruction-following format, so it can mitigate the forgetting of safety alignment while fine-tuning a model on instruction-following format data. But how does the paraphrasing help to prevent degradation of downstream task performance?\n\n- Parameter efficient fine-tuning (PEFT) method such as LoRA would be less vulnerable to forgetting after fine-tuning the model. Are the proposed observation and method applicable to various PEFT methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)