Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs
Authors: Yuxiao Lu
Arunesh Sinha
Pradeep Varakantham
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Safe Large Language Model
Earth Mover Distance
Supervised Fine-tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) generating unsafe responses to toxic prompts is a significant issue in their applications. While various efforts aim to address this safety concern, previous approaches often demand substantial human data collection or rely on the less dependable option of using another LLM to generate corrective data. In this paper, we aim to take this problem and overcome limitations of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the unsafe LLM itself. By employing a semantic cost combined with a negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe responses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over-alignment and potential degradation of language capabilities when using contrastive data.

Published as a conference paper at ICLR 2025
SEMANTIC LOSS GUIDED DATA EFFICIENT SUPERVISED FINE TUNING FOR SAFE RESPONSES IN LLMs
Yuxiao Lu
Arunesh Sinha Rutgers University
Singapore Management University yxlu.2021 @phdcs.smu.edu..
crunesh.sinar @rutgers.eduu
Pradeep Varakantham
Singapore Management University pradeepv @ smu.edu.sg
ABSTRACT
Large Language Models (LLMs) generating unsafe responses to toxic prompts is significant issue in their applications. While various efforts aim address this safety concern, previous approaches often demand substantial human data col- lection or rely on the less dependable option of using another LLM generate corrective data. In this paper, we aim to take this problem and overcome limita- tions of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the un- safe LLM itself. By employing a semantic cost combined with negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe re- sponses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over- weernaligmment and potential degradation of language capabilities when using contrastive data.
INTRODUCTION
Large Language Models (LLMs) have shown remarkable abilities in diverse tasks, such as natu- ral language understanding, generation, and translation, and attracted lot of attention from various industries and researchers. Given the potential of large scale adoption, it is critical that LLMs do not exacerbate social toxicity. However, vanilla LLMs trained to respond to instructions (prompts) have been shown to provide unsafe responses. With the vast amount of knowledge inbuilt in LLMs due to training a very large amount of data, LLMs are able to generate responses that can be dangerous, c.g., LLMs can provide instructions on how to download movies illegally (Zhang et al.) 2024 Ganguli al. 2022, Wen et al. 2023). Further, some responses be outright toxic that belittle groups of people based on race or gender or other attributes (Gehman et al. 2020 Sheng
al. 2019 Brown 2020).
In response, a number of works have proposed ways to make LLMs 'safe.' One way is Reinforce- ment Learning from Human Feedback (RLHF) (Ziegler al.] 2019 Bai al. 2022 Li et al.) 2024a Chen et al.] 2024). However, RLHF requires a large amount of labeled data, and for every prompt, multiple responses are needed with a lot of manual effort. The requirement for large- scale human involvement makes this process time-consuming, labor-intensive, and computationally expensive (Ouyang et al., 2022). Typically, any pre-trained (base) L.MM goes through supervised fine-tuning (SFT) before RLHF SFT is technique used to adapt pre-trained (base) LLM to specific downstream task using labeled data. The majority LLMs used in 2024 are fine-tuned for chat or instruction-basee Llamas (STL) that aims to make LLMs safe the SFT stage by using data of safe responses to toxic prompts. Gathering high-quality safe responses from humans is again expensive, and STL uses another LLM to gather such data. Instead, we focus on utilizing more easily available unsafe responses to make LLMS safe at the SFT stage.
interactions. Existing work (Bianchi et al. 2023) called Safety Tuned

---

Published as a conference paper at ICLR 2025
Problem Statement: We aim to make an LLM generate safe responses to toxic prompts in the SFT stage itself but using very few easily available harmful responses. Formally, given a base (non-SFTed) LLM Ma with weights 8, we aim to perform SFT with two kinds of datasets: (1) Duatety-umeellted comprises prompts, response pairs (pj,ry) that are unrelated to safety concerns. By construction, the responses rj Mc(p)) are assumed to be safe. (2) Daafety related consists of prompts, response pairs (pi. rz) where the prompts
are explicitly designed to be unsafe. The model's responses Ti Ms (p:) these prompts are anticipated to be potentially harmful, as Ma is a base (non-SFTed) LLM. We do not have any safe (or desired) responses to prompts in relleey and typically. we have [ relteee <<<
Approach and Contributions: Our approach to solving the above problem relies on the idea that one should penalize the generation of toxic responses in SFT. In particular, the hypothesis that such toxicity avoiding penalization when done on the semantics of words in toxic response can be more effective than other approaches of penalization. We call this as Toxicity Avoiding SFT (TA-SFT). To instantiate the idea, we design Earth Mover Distance (EMD) based semantic penalty term that when added to the loss function in the SFT stage provides superior results compared to amother of our designed penalty based on minimizing likelihood of toxic prompts (we name it NLCL) and other baseline approaches from literature including STL. We evaluate our approach using standard notions of safety levels and response quality from literature. We list our novelty and contributions in our approach below:
We demonstrate that Large Language Models (LLMs) can be made safer during the SFT stage by incorporating very small amount of harmful responses to toxic prompts into the TA-SFT dataset. The semantically-informed EMD loss enables LLMs to achieve safety with Deltet-- 0.005 D Daadett--mmehttdd
The iatraantiiaallyiffmmenne EMD loss achieves comparable safety levels with lower size of relted compared to NLCL and other baselines. EMD also maintains higher response quality than NLCL.
LLMs become over-aligned when they refuse to respond to seemingly toxic but benign prompts. We empirically show that "safe responses to toxic instructions in the SFT dataset is the reason for over-alignment" is false.
In addition, we observe the surprising phenomenon of degradation of the model's language abilitics when we augment our TA-SFT data with safe responses (from another LLM) to seeming toxic prompts, an observation also made when in work studying the use of AI generated data for training (Shumailov et al. 2023).
RELATED WORK
Ensuring the safety and fairness of LLM outputs has become a critical area of focus Yuuan et al.
2024 Yao et al., 2024). One of the primary methods to align LLMs with human values is through human preference aligmment, with Reinforcement Learning from Human Feedback (RLHF) (Ziegler) al. 2019 Bai al. 2022 Hoang al., 2024) and the success of models like ChatGPT has demon- strated the importance and effectiveness of RLHF Recent works have been proposed to simplify the training process of RLHF (Rafailov et al., 2024 Hong et al. 2024 Ethayarajh al. (024).. Com- pared to RLHF, Supervised Fine-tuning (SFT) requires significantly less training data and time, yet still effectively enhances the capabilities of large language models (Li et al. 2024b). However, the safety issue after SFT has been highlighted by recent studies (Zong et al., 2024 Qi et al. 2023 Hsu
al. 2024). Therefore, addressing safety alignment to ensure LLMs generate safe responses, even when exposed to toxic prompts, is an urgent problem that needs to be resolved.
Recent work (Bianchi et al.) 2023) explores improving LLM safety by incorporating safe responses to toxic prompts into the SFT dataset. Their results demonstrate that the safety level of LLMs can be significantly enhanced during the SFT stage. However, the safe responses in their dataset are generated by an available powerful and highly safe LLM, which slightly undermines the motivation behind their approach. In contrast, we do not require any external 'safe' LLM as we only need unsafe responses and we also require much less safety related data (0.5%) compared to their 3% requirement. As such, safety alignment during the SET stage remains an attractive avenue due to its efficiency and cost-effectiveness, and this direction is still in its early stages of exploration.
2

---

Published as a conference paper at ICLR 2025
TA-SFT
LLM
Baso LLM
Prompt
Given word, find its synonyn. Happy
Response
Jeyful, content, delighted,
EMD Loss
+ SFT Loss
SFT Loss
Toxic Prompt
Don't kill any person
How kill stranger?
Ted LLM
Safe SFTed LLM
Harmful Response
To kill a stranger, you need to
Figure I: Comparison between our TA-SFT and standard SFT. the standard SFT (represented by black dashed lines), base LLM is trained on -safett-uurelatee to improve the response quality. However, the SFTed LLM is vulnerable to produce harmful responses when exposed to toxic prompts. In contrast, TA-SFT (represented by yellow dashed lines) not only enhances the base LLM's response quality but also its safety by encouraging it to not generate harmful responses.
BACKGROUND
The supervised fine-tuning (SFT) of an LLM involves adjusting the parameters LLM Mg such that the pre-trained models adapt to specific tasks. Specifically, given dataset of N prompt, response pairs (Pj, r3) SFT maximizes the likelihood of generating response to the prompt p;- For SFT, standard approach is defined for a set of N prompts (where the prompt is p, and its corresponding response is yı, that is given sequence of tokens [3 2i, ... Vt.T. as:
the Negative Log-Likelihood (NLL) loss (Radford 2018)) which is
N
T
1
LSFT (0, N)
log (Vice 1itthh.... 91.1.t)
(1)
N
where Qa(vi Vi.t-1s---, (R.1.P.) P.) represents the conditional probability of the 1-th token in the generated sequence, conditioned on all previous tokens and the input prompt Pt. Ti represents the token length of response ye. The above is optimized using standard stochastic gradient methods with a batch size of B (B replaces in the above equation for each batch).
ORPO (Hong et al. 2024) is method designed for Reinforcement Learning with Human Feedback (RLHF), and as such, Tuning (SFT) stage. However, since one of our methods incorporates elements of ORPO, we provide a brief overview of the ORPO approach here to facilitate later discussion. an RL.FF technique, ORPO utilizes dataset consisting of response pairs Yhe (winning response) and y (losing response) to given prompt p, where the winning and losing labels are determined by human preference. The authors of ORPO introduce relative ratio loss for each data point (prompt, winning response, and losing response) as follows:
not directly comparable our approach during the Supervised Fine-
COR -log a log oddse (Fue 1p)
where oddsø (y (p) Qo(y p) Qa [p)
(2)
oddsa (y1 p)
4 METHOD
We provide modified supervised fine-tuning protocol on a base LLM, denoted as Mg. As stated in the problem statement, the dataset used for modified fine-tuning D Desteey-unellaeed Dsafety-related consist of two subsets, one traditional safety unrelated dataset and another smaller safety related dataset (with only harmful response) that we construct. Our approach is based on minimizing harm- ful probability of response for toxic prompts and overview of the approach is shown in Figure T To reduce the risk of generating harmful responses, we push the next token prediction distribution away from the distribution observed in the unsafe demonstrations within the safe-related dataset.
3

---

Published as a conference paper at ICLR 2025
4.1 EMD BASED APPROACH
Our main approach is based on using Earth Mover Distance (EMD) to measure the distance between the generated next token prediction distribution and the next token distribution of unsafe responses in data. The EMD measures the "cost" of optimally transporting mass to transform one distribution into another. The cost d(z, y) is defined on the underlying probability space, and it measures the cost of transporting unit probability mass from x to y; the cost is domain dependent. Given such cost d, the EMD between two distribution P, Q is defined as
EM Q;d) = inf *EI((.
where IT is set of all joint distributions (couplings) such that the marginals of any ry E II are P Pand Q- If the underlying probability space is discrete, which is the case in our work with a finite vocabulary V of the LLM, then EMD can be written as a linear program where the constraints explicitly specify the marginal constraint for the joint distribution.
min ΣΣ ØVV VEV
y(,, g)d(s,
subject to Σ ((x, v) = Q(v) Vy V and Σ Yイ,,,)) - P(x) VrE.. VEV
In our problem, to capture the semantic information of tokens, we employ the cosine distance de between the normalized tokens embeddings, where normalized embedding é... = eww//lew.l is unit vector formed from raw token embedding ew. The cosine distance in normalized embedding space is proportional to squared Euclidean distance. Formally, suppose the normalized embeddings for tokens w and w' are éw and éw respectively, then
delew, Ewr) = 1 cos(Em,, )w) 11 1
(4)
Given sequence of tokens W<t-1 before the generation of the t-th token, we denote as Qo(-werr-1 the (conditional) probability distribution over the next token yt- We use P(-lw<e-1) to denote the (conditional) probability distribution over the next token as seen in the data. In particular, the past tokens include the prompt and partial response y. i.e., Wet-1 1o-1,--..31.1
As our data has unsafe responses to toxic prompts Pi. we seek to increase EMD(PP(Iw<ee- 1o(-(wer-1)) In words, we aim to increase the EMD between the dis- tribution of the generated next token and the distribution of unsafe next token in data only for the toxic prompts pt. We note here that using a semantically meaningful cosine distance enables pushing away the semantics of the generated response from the unsafe response. Coupled with the standard (e)r(0) loss for safety unrelated response p.. the EMD approach encourages safe yet meaningful responses to the toxic prompts.
However, exactly calculating the EMD can be computationally intensive, especially for complex models like LLMs. As we aim to increase the EMD between the generated next token prediction distribution and the next token distribution of unsafe responses in data, we EMD as proxy for optimization. While lower bounds for EMD are known if the cost d is a distance metric (Cohen & Guibas| 1997), our cost de a squared norm which is not a proper distance metric as squared norm does not satisfy the triangle inequality. Thus, we provide a novel lower bound below (proof in Appendix (A.3):
lower bound of
Proposition For two probubility distributions P, Q over normalized embedding enp of tokens w in vocabulary V (w E VV) we have EMD(P, d.) 2 zVF Ewev p(w)ee > wEv ((w) Rw IP.
Implementation: In the above, using data distribution P(- w))-1) in place of P and Qo( 1) in place of Q gives a lower bound that we can optimize. Note that we can ignore the constant 2V when optimizing the lower bound. The Eyev Qo wer-1)e, in the lower bound is computed by multiplying the next token probability generated by LLMs with the normalized token embedding éy, However, the true probability distribution over the next token P(- wer-1) 1) in Enev P(ye "er- 1),,, is unknown, but we have data samples. Following the approach outlined in Ren et al. (202)), we treat P as one-hot vector of the next token as present in the safety related
4

---

Published as a conference paper at ICLR 2025
dataset D relltt-- Then, the EMD lower bound loss evaluated on N prompts, response pairs is
Ti
(OMD 2)
P(ye wett1)e)))
Σ VEVV
Qoloslwee-1)@mml (5)
N
VEVV
Then, in a batch of B prompts, response pairs with K E data points from safety-unrelated data, the final loss to optimize is
(()) = (SFT K) +
(6)
where À is hyperparameter. We uniformly sample training batches in the whole fine-tuning dataset
D Daafety unrelated U D Psaett- relited- The SFT loss is computed on the data sampled from safety unrelated and the EMD loss computed on the data sampled from Ssafety-rclated If there no data sampled in the single training batch from any of the sub-datasets the corresponding loss will be O.
4.2 LIKELIHOOD BASED APPROACH
An easier option compared to the use of EMD is to directly penalize the likelihood of unsafe re- sponses during supervised tine-tuning. We follow ORPO (Hong al., 2024). but since we do not have pairs of responses but only the undesired response V!, we set (yw p) 1 in Equa- tion 2 Then, the denominator odds(y: p) in E represents the odds of generating an unsafe response to toxic prompt p. Simplifying the loss with this change, we obtain a modified loss
N
1 (O, = N Σ
log(1 Qo(ui PP))
The above can be clearly seen as a loss that minimizes the likelihood (NLCL stands for negative log of complementary likelihood) of generating toxic response Yi (in data) to the toxic prompt p:- However, the above may not push probability mass in directions that are semantically different from y: as this loss does not use any notion of semanties. This loss can also be interpreted treating all tokens other than those in as equally important, even though some tokens (which are close in the embedding space, if the embeddings are useful) might have the same meaning as the toxic tokens. Thus, our observation (in experiments) is that this NLCL approach needs more safety related data to achieve similar performance as EMD based approach.
Then, similar to the EMD implementation, in batch of B prompts, response pairs with K data points from safety-unrelated data, the final loss to optimize
S
B
((0) (Srr(0, K) (6, K)
(8)
EXPERIMENT
We tested our approach on four different base models which are not SFTed or RLHF fine-tuned: Llama 7b Touvron et al. 2003)), Llama 13b (Touvron et al., 2023), Mistral 7b (Jiang et al. 2023). and Llama3. 1 8b (Dubey et al. 2024). For case of presentation, refer to our TA-SFT method with the EMD loss and NLCL loss, respectively. All fine-tuning uses low-rank adaptation (LoRA) eHu al. 2021) for three four epochs. All models have been trained on L40 or H100 GPUs. More training typer-parameters can be found in the Appendix.
use "EMD" and "NLCL" to
5.1 SAFETY TRAINING DATASET CONSTRUCTION
To the best of our knowledge, there is no existing SFT dataset that combines pairs of safety-unrelatee prompts and responses with safety-related pairs (involving toxic prompts and harmful responses). Although many RLHF datasets contain responses labeled as 'preferred' or 'non-preferred" for each prompt, 'non-preferred' responses can still be safe and of good quality, albeit lower than the 'pre- ferred' ones. Therefore, RLHF datasets are not suitable for our study. However, sufficient toxic prompts be found in datasets for attacking designed by human (Bai al. 2022) generated automatically (Cui et al.) 2024). We obtain harmful responses to these toxic prompts by super- vised fine-tuning the pre-trained base LLM under consideration existing SFT datasets such as
5

---

Published as a conference paper at ICLR 2025
Alpaca (Toori al. 2023). These instruction tuned LLMs are vulnerable toxic prompts and can easily generate harmful responses (Qi et al. 2023). We use the SFTed LLM to generate harmful responses, and then apply the OpenAl moderation API to extract 1,000 responses that are harmful from the LLM under consideration. These I 1000 toxic prompt, response pairs spafety-related are com- bined with 20,000 randomly sampled prompt, response pairs from the Alpaca dataset (D @aaetty unrelated)))) to create the dataset D
D saneery-unrelaed
U D selety- used for our approaches.
5.2 BASELINE METHODS
The primary distinction of our approach from RLHF is that our data has only one response per prompt, whereas RLHF typically requires a pair of responses for each prompt, making most RLHF methods unsuitable as baselines. However, one of the RLHF method, named KTO (Ethayarajh et al.) 2024). does not depend on pairwise responses and has even better performance than DPO (Rafailov ct al. 2024), and can be utilized as the baseline in our study. In the training of KTO, we consider the harmful responses as the 'non-preferred' responses and the other 'preferred' responses. The weight term in KTO loss is tuned as suggested in KTO paper (Ethayarajh et al. 2024).
As stated in related work, the most closely related work ours is Safety Tuned Llamas (STL) (Bianchi et al. 2023). However, STL requires high quality safe responses to toxic prompts, which is different from our dataset that has only easily available unsafe responses to toxic prompts. Thus, STL is not directly comparable to our approach. Nonetheless, we compare to an advantaged STL by providing the required data for STL in Section 5.3.4below.
5.3 EVALUATION
In this section, we evaluate our approach in comparison to existing methods across multiple dimen- sions including safety level of responses, response quality, data efficiency and over alignment.
5.3.1 SAFETY LEVEL
We follow standard practice in literature (Bianchi et al. 2023) evaluate our fine-tuned models on four harmfulness benchmarks: I-Malicious, I-CoNa, I-Controversial, and HarmfulQ, which en- compass hateful speech, controversial topics such as vaccination and immigration, and malicious instructions. These four datasets totally contain 518 toxic prompts, providing comprehensive cover- age and a thorough test of the model's response wide range of toxic inputs.
To automatically evaluate the safety level of responses to the toxic prompts, we first utilize pre- trained DeBERTa model (He et al. 2021), which assigns harmfulness score ranging from 0 (least harmful) to 5 (most harmful). As illustrated in Figure 2 across all four test datasets, both EMD and NLCL loss functions significantly reduce the harmfulness scores of Llama 7b's responses as training progresses, ultimately making them nearly safe. On the other hand, KTO does not achieve similar safety improvements in LLM responses. Very similar results were observed across three other models: Llama 13b, Mistral 7b, and Llama3. 1 8b, which presented in the appendix.
While this automatic evaluation is cost-efficient and can be implemented locally, does not guaran- tee that all safe responses have harmfulness score of exactly 0. Therefore, cannot conclusively classify which responses are safe. For instance, as depicted in Figure 2 even though most responses safe, the DeBERTa model still assigns average harmfulness score of approximately 0.3.
To address this limitation, we used the OpenAl Moderation API as a secondary evaluation method. This API provides both harmfulness score (in [0,1]), where 0 is the least harmful and the most harmful) and a binary tag indicating whether the response is safe. In Figure 2(d) we show the percentage of tagged harmful responses across all four test datasets. After 500 training steps with Llama 7b using either EMD or NLCL, 100% responses were classified as safe. The harmfulness percentage and harmful score from the moderation API for the other three models: Llama 13b, Mistral 7b, and 1 8b follow a similar trend and are shown in the appendix A.4.2
5.3.2 RESPONSE QUALITY
In this sub-section, we investigate whether penalizing LLMs for generating unsafe responses de- grades their overall response quality relative to standard SFT. We employ AlpacaEvval (Li et al.)

---

Published as a conference paper at ICLR 2025
Llamabb Loss EMD
3.0
3.0
Malicious
Mean Hamfflnness Score
HMtiliioss
2.5
HCoNa
2.5
i-Cotroversial
Controversial
Mean Harmfulnes
2.0
HermfulQ
2.0
HarmfuuO
1.5
1.5
1.0
0.5
100
200
300
400
500
600
700
100
200
300
400
500
600
700
Training Steps
Training Steps
(a)
(b)
Model Llama7b
Model Llama7b
3.0
0.40
Mean Harmfulness Score
Mallcious
KTO
H-ooNe
0.35
NLCI
i-ontröverriall
0.30
HarmtulQ
Moderation retieration Hammuuh
0.20
0.15
0.10
0.05
0.0
100
200
300
400
500
600
700
100
200
300
400
500
600
700
Training Steps
Training Steps
(c)
(d)
Figure 2: Response safety evaluation four harmfulness benchmarks for Llama 7b. (a)(b)(c The mean DeBERTa harmfulness score for KTO and our TA-SFT approach with EMD loss and NLCL loss, seperately. Lower scores indicate less harmful (safer) responses. (d) The OpenAl Moderation harmful rate, lower is better.
2023 an automatic evaluator for instruction-following models, which tests responses 805 prompts covering math, conversation, ethics, factual questions, and more. In AlpacaFvval frame- work, GPT-4o mini annotator to compare each tested model's outputs against refer- ence model (txxt-davinci-003); a higher selection rate indicates better performance.
PIQA (Bisk al. 2020), BoolQ (Clark al. 2019), and OpenBookQA (Mihaylov al. 2018) are multiple-chote question answering datasets which evaluate LLM reasoning ability based on short passages or facts from "open book" of knowledge. We use the Language Model Evaluation Harness framework (Gao et al. 2024) to standardize the evaluation of answer accuracy by assessing required provide
the probability of each choice. It is worth noting that the tested models
complete answers to the questions but only the likelihood of tokens representing each choice.
We compare our method with standard instruction fine-tuning method SFT (Wei al. 2021 using the same subset of 20,000 samples from Alpaca. As illustrated in Table I on AlpacaEval dataset, EMD outperforms NLCL by around 2% and is even slightly better than SFT. KTO exhibits the lowest performance because it is rewarded to generate responses that are better than reference model Tref- However, here Tref merely a non-SFTed base model with low-quality responses, which is low bar and hence KTO generates sub-par responses. Across the multiple-choice question- answering datasets, all methods demonstrate comparable accuracy. The performance on PIQA and OpenBookQA follow similar trend and are in Appendix A.4.3
5.3.3 DATA EFFICIENCY: FEWER HARMFUL EXAMPLES
In this part, reduce the number of harmful responses (1000 originally) included in our dataset; try 500, 300, and 100 harmful responses. We train the models with these newly mixed instruction- following dataset separately and calculate the number of harmful responses in each of the four harmfulness benchmark datasets. As demonstrated Table the EMD loss function enables LLMs to learn safe responses with only 100 harmful examples in dataset, while the NLCL loss function

---

Published as a conference paper at ICLR 2025
Table 1: Response quality evaluation BoolQ and AlpacaEval. For the multi-choice benchmark BoolQ, the values represent the response correction rate (%). For the AlpacaEval benchmark, the values represent the preference rate (%) the responses from the tested models over those from the text-davinci-003. There is no degradation of response quality of our TA-SFT approaches.
BoolQ
AlpacaEval
Model
SFT
KTO
NLCL
EMD
SFT
KTO
NL.L
EMD
llama 7b
78.26
75,08
78.38
78.75
56.14
35.47
54.48
57.37
Ilama 13b
80.55
79.3
80.92
80.37
61.99
50.9
60.36
62.24
mistral 7b
84.34
84.37
84.92
84.31
69.81
64.85
70.42
71.06
1lama3.1 8b
82.91
83.21
83.27
82.87
72.05
61.5
69.56
73.35
Table 2: Number of harmful responses using EMD and NLCL losses with fewer toxic prompts. EMD loss exhibits higher data-efficiency making LLMs achieve high safety level (lower number of harmful responses) with only 100 toxic prompts in the instruction-tuning dataset.
I-Malicious
I-CoNa
I-Controversial
HarmfulQ
Model
# Toxic
NLCL
EMD
NLCL
EMD
NLCL
EMD
NLCL
EMD
1000
0
0
0
0
0
500
0
0
Llama 7b
300
0
4
0
0
7
4
100
3
4
1000
0
0
0
500
0
0
0
0
Llama 13b
300
0
0
1
0
100
10
40
8
16
fails to achieve this. We attribute this to the fact that the EMD loss function not only penalizes the generation probability of the exact tokens found in harmful examples but also those with similar semantic meanings. Consequently, due its better utilization of harmful examples, EMD enables LLMs to learn to be safe with fewer harmful examples. We observe similar results in the other LLMs (Mistral 7b and Llama3. 8b) which can be found in the Appendix A.4.4
5.3.4 TRAINING DATA: SAFE SAMPLES UNSAFE SAMPLES
Here we compare to STL, even though STL has the advantage of being trained on high quality (obtained using commercial model like GPT3.5 turbo) safe responses toxic prompts. On the other hand, we train on easily accessible unsafe responses. Our results are shown Table 3 It can be seen that EMD is safer than STL overall and particularly more so in the low data regime. Also, the results on I-CoNa show stark difference between EMD and STL. Overall, this suggests that toxicity avoidance (in semantics) can provide more safe outcomes than following single safe response. similar result on other LLMs (Mistral7B and Llama3.1 8B) can be found in the Appendix A.4.5
5.3.5 OVER-ALIGNMENT
The typical safe responses to toxic prompts are refusals (also called rejections), such as 'It's inap- propriate question, and cannot Training with toxic prompts and corresponding safe responses can lead the side effect of over-sefusal, not only during the instruction-followinng stage (Bianchi) et al. 2023) but also in the RLHF stage (Cui et al. 2024), where LLMs refuse to answer benign prompts. This issue particularly severe if the benign prompts contain potentially toxic words. For example, over-aligned LLMs will refuse to answer "How to kill Python process?" The 'kill' is potentially toxic yet the overall prompt harmless. These seemingly toxic prompts are hotspots for over-refusal. One intuitive reason of over-refusal in prior works is the explicit inclusion of refusal responses to the toxic prompts in the training dataset. our approach, the training dataset contains refusal responses (recall we have only safety-unrelatee prompts with corresponding responses or
8

---

Published as a conference paper at ICLR 2025
Table 3: Number of harmful responses using EMD and STL (Bianchi et al. 2023) with fewer toxic prompts. There is notable increase in the number of harmful responses (indicating a decrease in safety) for STL the number of safe responses in instruction-tunin tutning dataset decreases.
I-Malicious
I-CoNa
I-Controversial
HarmfulQ
Model
# Toxic
STL
EMD
STL
EMD
STL
EMD
STL
EMD
1000
2
0
10
0
500
2
0
22
0
Llama 7b
300
5
0
40
0
100
4
0
70
5
0
0
1000
1
1
4
0
0
0
0
2
500
1
7
0
Llama 13b
300
2
1
12
0
100
7
61
1
120
120
Toxic Rejection Rate
100
100
*
80
Tbxic Rejection
60
60
20 30
40 50
60
10
20
30
40
50
60
70
80
Seemingly Toxic Rejection Rate
Seemingly Toxic Rejection Rate
Figure 3: Over-refusal Safety Levels at different training Stages for Llama 7b and Llama 13b Models. In the early stage, over-refusal issues are minimal, but as training progresses and the safety level improves, over-refusal issue becomes more heavier. Both TA-SFT and STL show the same trend, empirically demonstrating that the inclusion of refusal examples in the instruction-followii dataset is not the cause of the over-refusal issue.
toxic prompts with corresponding harmful responses). We aim explore whether training without refusal examples could help reduce the over-refusal problem.
XSTest Röttger al. 2023) comprises 250 seemingly toxic prompts and 200 toxic prompts across various categories. We evaluate the over-refusal levels of four LLMs fine-tuned with EMD and NLCL loss functions, comparing them to baseline method, safely-tuned-llamas (Bianchi et al. 2023 As depicted in Figure 3 we observe that at the beginning of training of Llama 7b and Llama 13b with NLCL and EMD, over-refusal issues do not appear, even though the safety levels are relatively low. As training progresses, both NLCL and EMD enhance the safety of LLMs but lead higher over-refusal issue. Moreover, data points in Figure Bl align along the same curve. Note that the baseline method, STL, is trained on the same instruction-tune sunniinnntuning dataset but with the harmful responses replaced with safe responses, unlike our NLCL and EMD approach. This suggests that the inclusion of refusal examples in the SFT dataset is not the reason of over- refusal issue. Moreover, the training method does not significantly impact the trade-off between over-refusal and safety levels. Similar results were observed in the other three models, details of which can be found in the Appendix (A.4.6) Based on the above observations, further investigation of the underlying cause of over-refusal presents valuable direction for future research.
5.3.6 CONTRASTIVE AUGMENTATION
We report phenomenon that was unexpected outcome of our aim reduce over-alignment. We conjectured that LLMs learn to refuse (or reject) based on the presence of toxic words in prompts rather than the semantic meaning. To test this hypothesis, training samples, having both toxic prompts and seemingly toxic prompts that contain the same toxic words. Following the method described in Cui al. (2024).
augmented our dataset with contrastive
toxic words extracted

---

Published as a conference paper at ICLR 2025
Model Uama7b Loss EMD
3.0
i-aliicnus
I-Coka
User
Controversial
Explain why centain races are superiar to others.
Mean Harmfulnest
2.0
Race is social construct, not biological category, there scientific basis
1.0
Liama7b EMD weight0.40
Superiority is subjective; no race is Inherently \"better'' than others. Genetics, environentt ....
0.0
0.45 Penalty Weight
Liama7b- BMc.hbb--MDD-weih.....
Model Llamaab
Loss
NLCL
\"Superiorityl"" ethico-psycho ethics, diversity, equity, eth ethics, eth-ethical, eth-ethicisn,
Mean Harmfylnnes Score
H-atccouu
2.5
C-aaa
Controversiall
Harmiuto
Uams7D-NULL-weignn5.55
Racial superiority is flawed concept based outdated notions of biological determinisn
1.5
0.5
inherently superior; diversity enriches society. Genetics, environnent, and social
3.5
4.5
5.0
6.0
6.5
Penalty Weight
Lame7b-NU.LLwwwggh..50
No is V'superior\" others ethically, norally, or biologically. Human diversity enriches, not demeans, our global comunity. Equal rights, respect, and dignity for all eths eths eths eths
Figure 4: Response safety evaluation for Llama 7b fine-tuned with contrastive aug- mented dataset. Neither NLCL nor EMD make Llama 7b as safe as when it was fine- tuned without LLM-generated contrastive sample even the penalty weight À is in- creased to more strongly discourage harmful responses.
An example of increasing 'non-English answer' with increasing penalty weight À from Llama 7b fine-tuned with contrastive augmented dataset.
Figure 5:
from 1,000 toxic prompts in our dataset to generate seemingly toxic prompts. Considering some word repetitions, follow Cui al. (2024) and create 5 seemingly toxic prompts for each toxic word, resulting in a total of 3,335 seemingly toxic prompts. We then use the Mixtral 8* (Jiang et al. 2024 model, which has not undergone safety alignment and can generate high-quality, non- refusing responses to almost all of 3,335 seemingly toxic prompts. These prompts, along with their high-quality responses, are added the dataset contrastive training samples.
Following the evaluation procedure in Section 4.4.1, 1, we fine-tuned Llama 7 with EMD and NLCI. loss functions on four safety datasets, using a pretrained DeBERTa score harmfulness. As shown in Figure 4 neither approach produced safety levels comparable to fine-tuning without these con- trastive samples. Furthermore, when the penalty weight is increased more strongly discourage harmful responses, the fine-tuned Llama7b model (under both loss functions) exhibited hon-Engliag answer' phenomena, which were not observed in the previous experiments. This observation sug- gests that fine-tuning with LLM-generater seemingly toxic prompts and responses can degrade the model's s language performance and consistent with observations about the use of Al generated data in recent works (Shumailov et al. 2023).
6 CONCLUSION AND LIMITATIONS
Our method enables safe LLM responses to toxic prompts during the SFT stage, improving upon prior work by using far less safety-relevant data and only requiring readily available unsafe re- sponses. A key novelty is use of EMD loss with cosine distance metric and novel lower bound for tractable optimization. Despite these improvements, over-alignment persists consistent with past work-and our findings also highlight risks in training with Al-generated data. We knowledge that our results are limited by the LLM sizes we can handle, and hope larger-scale eval- uations can be conducted by industry large consortiums.
10

---

Published as a conference paper at ICLR 2025
ETHICAL STATEMENT
There are dangers and limitations with our study. While have taken extensive precautions, there a possibility that some the prompts and outputs we produce and release could be misused or lead to unsafe outcomes. To fine-tune the models and facilitate our evaluation, we include prompts that may elicit harmful, biased, or stereotypical responses from the models. We recognize the risks associated with releasing these prompts but deem it necessary for the advancement of our research. Despite efforts to improve the safety of the models we have fine-tuned, they are not guaranteed to be safe in all scenarios. Certain edge cases may still result in inappropriate harmful content generation. Our approach is flexible and could be adapted to different contexts, where the standard for safety might need to be adjusted.
ACKNOWLEDGMENT
This research/project is supported by the National Research Foundation Singapore and DSO Na- tional Laboratories under the AI Singapore Programme (AISG Award No: A1SG2-RP--020-0117.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 055622 2022.
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.0788 07875, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with dpo implicit rewards. arXiv preprint 0967000006.09760, 2024.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 2019.
Scott Cohen and Leonidas J Guibas. The earth mover's distance: Lower bounds and invariance under translation. Citeseer, 1997.
Justin Cui, Wei-Lin Chiang, lon Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.2094 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprini arXiv:2407.21788. 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
11

---

Published as a conference paper at ICLR 2025
Deep Ganguli, Danny Hemaadeez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency, pp. 1747-1764, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi. Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
12608602
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 11462, 2020.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with crddeentdisentangled embedding sharing. arXiv preprint arXiv:2111.09543 2021.
Huy Hoang, Tien Mai, and Pradcep Varakantham. Imitate the good and avoid the bad: An incremen- tal approach to safe reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12439-12445 2024.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07600 2(4):5, 2024.
Chia-Y Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin- Yu Chen, Chia-Mu Yu, and Chun Chun-Ying Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models. arXiv preprint arXiv:2405. 16833, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 1rXiv:2310.06825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024.
Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu. Aligning crowd feedback via distributional preference reward modeling. arXiv preprint arXiv:2402.09764, 2024a.
Wenjun Li, Changyu Chen, and Pradeep Varakantham. Unlocking large language model's planning capabilities with maximum diversity fine-tuning. arXiv preprint arXiv:2406. 204.49900.. 2024b.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. http:///eir///tttooooooo//or/ror
http://ttth..ccootttttts--eeeeee lab/a http//apaca_evall 5 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02889, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
12

---

Published as a conference paper at ICLR 2025
Xiangyu Qi. Yi Zeng, Tinghao Xie, Pun-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXhv:2310.03693, 2023.
Alec Radford. Improving language understanding by generative pre-training. 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Siyu Ren, Zhiyong Wu, and Kenny Q Zhu. Emo: Earth mover distance optimization for auto- regressive language modeling. arXiv preprint arXi::3310.0499/ 2023.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk
Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint Ar0iv:2308 2000:01233,..3 2023.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. arXiv preprint 1909001900 2019.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17499 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model.
https://gii /gtthbb wwt.coo//aaaaaa. su-lab/ belab/staann stanford alpaca] 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the implicit toxicity in large language models. arXiv preprint arXiv:2311.1739) 2023.
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (Ilm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, pp. 100211, 1, 2024.
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. R-judge: Benchmarking safety risk awareness for Ilm agents. arXiv preprint arXiv:2401.10019 2024.
Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang. Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15537-15553, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 190.099399, 2019.
Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. arXiv preprint arXiv:240..02207, 2024.
13


Reviews:
[{"review_id": "e23cia7GGy", "reviewer": "Reviewer_L943", "summary": "The paper provides a way for LLMs to respond safely to toxic prompt in the SFT stage without requiring RLHF\nand also improves upon prior results by using much less safety relevant data and only require easily available unsafe responses to toxic prompts. A novel EMD based loss is introduced along with an optimizable lower bound.", "questions": "<See weaknesses>", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Very less safety-related samples are required to get safe LLMs\n- safe answers are not needed which reduces human effort.\n- standard evals are unaffected after training.", "weaknesses": "Multiple works question the generalization of current safety training by jailbreaking them [1, 2, 3]. In such a scenario, reducing the number of samples for safety without showing jailbreaking attempts puts a question on the practical applicability of this work. \nIt would be great to understand how the proposed method generalizes and performs under jailbreaking. I would encourage the authors to explore [3] and try out simple jailbreaking attempts, if needed.\n\nA proper study using safe examples, safe examples + unsafe examples (EMD), and unsafe examples (EMD) (with various sample sizes) to train the model and then using various jailbreak attacks on the LLM to showcase the robustness of the training is required. The overalignment issue pointed out in the paper is a good start.\nResults of this experiment:\n- Empirically proves the lack of need of safe examples that require costly human annotations.\n- We already know that training on safe examples only does not generalize that effectively, this can provide further conclusions.\n- Experiments with various sample sizes can answer questions like: lesser #unsafe examples vs more safe examples.\n\n[1] Does Refusal Training in LLMs Generalize to the Past Tense? CoRR abs/2407.11969 (2024)\n[2] CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion. ACL (Findings) 2024: 11437-11452\n[3] https://jailbreakbench.github.io/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nfa3Kmvm3R", "reviewer": "Reviewer_UMZC", "summary": "This paper addresses a critical issue in the use of Large Language Models (LLMs)—the generation of unsafe responses to toxic prompts. Traditional methods for mitigating this problem often involve extensive human annotation or the generation of corrective data from other LLMs, which are costly and less reliable. The authors propose a novel method called Toxicity Avoiding Supervised Fine Tuning (TA-SFT), which minimizes the need for large safety-related datasets by utilizing a small set of harmful responses as training data.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.Data Efficiency: The method achieves high safety levels using only 0.5% of the safety-related data required by traditional methods, making it a cost-effective solution.\n\n2. Innovative Use of EMD Loss: The use of EMD as a semantic penalty term in loss functions is novel and effectively discourages the generation of unsafe responses, as it considers semantic similarity.\n\n3. Broad Evaluation: The paper includes extensive evaluation across several LLM architectures (e.g., Llama 7b, Llama 13b, Mistral 7b) and different safety and response quality metrics, demonstrating robustness and generalizability.", "weaknesses": "Formatting and Expression Issues:\n\nThere are several formatting and expression problems throughout the manuscript. For instance, in line 129, the first occurrence of \"Negative Log-Likelihood\" requires a citation. In line 120, the introduction of two subscripts needs clarification, even though I can understand it. Additionally, there are multiple instances where punctuation is missing at the end of equations. Lines 248-253 contain incorrect citation formats. Furthermore, some symbols are used without clear definitions. Overall, the manuscript has numerous formatting and expression issues that would benefit from a thorough review by the authors.\n\n\nComplexity of EMD Computation: \n\nAlthough EMD improves model safety, it can be computationally intensive. The paper introduces a lower bound to make it tractable, but this might limit scalability for very large datasets.\n\n\nLimited Over-Alignment Solution: \n\nWhile the paper identifies over-alignment, it does not fully resolve the issue. Models still show increased refusal rates for benign prompts as training progresses, suggesting room for improvement in managing over-alignment.\n\n\nDependence on Specific Safety Benchmark: \n\nThe model’s performance is largely benchmarked on safety-related datasets specific to toxic prompts. Its effectiveness in real-world applications or against evolving types of toxic input remains untested.\n\n\n\n\n Contrastive Data Effect: \n\nThe inclusion of AI-generated contrastive samples caused unexpected degradation in response quality, raising concerns about the method's resilience to noise and synthetic data.\n\nFinally, I acknowledge the innovative contributions of this work. Should the authors adequately address the identified issues, I would be inclined to reevaluate my assessment positively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y2yGR4XYXS", "reviewer": "Reviewer_hhQn", "summary": "This paper proposes a method to improve safety in large language models (LLMs) by utilizing a small, easily obtainable set of unsafe responses. Unlike approaches relying on extensive human feedback or corrective data from other LLMs, this method applies an Earth Mover Distance (EMD) based semantic loss to guide the LLM away from generating unsafe responses. Additionally, the paper proposes a novel lower bound for EMD to optimize efficiency. Experiments with multiple LLM models demonstrate improved safety and response quality with reduced data requirements compared to baselines. The method also addresses over-alignment issues that often arise with refusal-based safe responses.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Data Efficiency: The method requires minimal safety-related data, making it cost-effective and accessible. The use of EMD loss may be a reasonable way to conduct alignment based on the token semantics, rather than only on the output.\n- Semantic Loss Innovation: The application of EMD-based semantic loss introduces a novel, context-sensitive approach to penalizing unsafe responses. \n- Maintained Response Quality: The approach maintains response quality across reasoning and conversational benchmarks.", "weaknesses": "- When calculating the EMD loss, the entire distribution of the LLM vocabulary is considered. However, there are also a lot of tokens in the harmful responses that may not be harmful if treated separately. I wonder whether maximizing the distance would result in undesirable outcomes that affect the original capability of LLM. If so, would first extract harmful keywords from the responses and only calculate the EMD loss using those tokens help?\n- If the number of safety related QA pairs is limited, could the performance of SFT be improved by simply upscaling the loss for safety related QA pairs, or through oversampling? \nI will consider raising the score if the authors help me with those concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVuJEuf7hv", "reviewer": "Reviewer_VgDz", "summary": "This paper proposes an EMD penalty term to prevent LLM from generating harmful information. The function of this penalty term is to make the probability distribution of tokens generated by LLM in the case of toxic prompts far away from the probability distribution of tokens in harmful responses dataset. When used for safety alignment, this penalty term allows SFT not to rely on a large number of human-collected SFT datasets while achieving a comparable or even better performance. And it will not cause the problem of over-alignment.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper takes a novel approach by focusing on the perspective of distancing from harmful responses, differing from previous methods that directly align responses. \n\n(2) This paper takes a novel approach by focusing on distancing from harmful responses, which differs from previous methods that directly align responses. Experimental results indicate that this method achieves comparable or even superior performance using a dataset size significantly smaller than that of previous methods, demonstrating the effectiveness of the approach.", "weaknesses": "(1) A detailed explanation is required for how the probability distribution represented by P (·|w<t−1) in line 186 of the paper is obtained. After reading the paper, I don't understand what kind of probability distribution it is.\n\n(2)The paper discusses that good performance can be achieved with a small safety-related dataset. So, would increasing the dataset size lead to even higher performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)