What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

Authors: samyak  jain
ekdeep  singh lubana
kemal  oksuz
tom  joy
philip h s torr
amartya  sanyal
puneet k dokania
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:29:53

Abstract: Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. To better understand the underlying factors that make models safe via safety fine-tuning, we design a synthetic data generation framework that captures salient aspects of an unsafe input by modeling the interaction between the task the model is asked to perform (e.g., “design”) versus the specific concepts the task is asked to be performed upon (e.g., a “cycle” vs. a “bomb”). Using this, we investigate three well-known safety fine-tuning methods—supervised safety fine-tuning, direct preference optimization, and unlearning—and provide significant evidence demonstrating that these methods minimally transform MLP weights to specifically align unsafe inputs into its weights’ null space. This yields a clustering of inputs based on whether the model deems them safe or not. Correspondingly, when an adversarial input (e.g., a jailbreak) is provided, its activations are closer to safer samples, leading to the model processing such an input as if it were safe. Code is available at https://github.com/fiveai/understanding_safety_finetuning.

Main paper: 
 introduction: 
 Large language models (LLMs) are commonly trained via a combination of pre-training on a large corpus and instruction fine-tuning, wherein the model is supervised to follow instructions (Driess et al., 2023; Team et al., 2023; Qin et al., 2024) . While pre-training enables a model to learn different capabilities (Wei et al., 2022; Bubeck et al., 2023) , instruction fine-tuning enables use of open-ended, generic inputs to control said capabilities (Ouyang et al., 2022; Wei et al., 2021; Sanh et al., 2021; Bai et al., 2022; Raffel et al., 2020) . Since this pipeline does not restrict what tasks the model can be used for, potential misuse is left feasible under its purview (Bengio et al., 2023; Anwar et al., 2024) : as long as an instruction can be formulated and the model possesses the relevant capabilities to perform the instructed task, it will strive to perform it. To prevent such misuse, safety fine-tuning is used as an additional training phase for LLMs, in which the model is supervised to prioritize generation of outputs deemed safe as per human preferences. Popular approaches for safety fine-tuning include: (i) supervised safety fine-tuning (SSFT) (Ouyang et al., 2022) ; (ii) reinforcement learning with human feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022; Stiennon et al., 2020) and its recent renditions that avoid use of an explicit reward model, e.g., DPO (Rafailov et al., 2023) ; and (iii) machine unlearning (Liu et al., 2024) . Despite immense use of these protocols to enable Samyak worked on this research project during his internship at Five AI Oxford Team. Tom participated in this project during his employment at Five AI Ltd. system release (Chao et al., 2024; Sun et al., 2024) , several recent works show that safety fine-tuned models continue to produce unsafe generations when prompted via adversarially designed inputs, e.g., jailbreaks (Andriushchenko et al., 2024; Chao et al., 2023; Zou et al., 2023; Carlini et al., 2023) .In this work, our goal is to understand: (i) what is the safety mechanism learned by the model via safety fine-tuning? and (ii) how are jailbreak and adversarial attacks able to bypass this mechanism? While a few contemporary papers have investigated the mechanisms of safety fine-tuning, e.g., showing that such methods perform minimal alterations to model parameters that nevertheless can change its behavior (Jain et al., 2023b; Lee et al., 2024; Prakash et al., 2024; Wei et al., 2024) , tying this analysis back with lack of robustness of safety fine-tuning is lacking in existing literature. We aim to fill this gap by designing a well-defined synthetic data generating process wherein an input is modeled as a function of the task the model is expected to perform (e.g., "design"), and the specific concept the task is to be performed upon (e.g., "cycle" versus "bomb"). This separation helps us delineate how the model distinguishes between safe versus unsafe inputs, while allowing us to model different forms of jailbreak attacks grounded in the formalization of Wei et al. (2023) . Overall, our contributions and observations can be summarized as follows.• Systematic setup to study safety fine-tuning and jailbreaks. We introduce a novel synthetic data generation framework that allows controlled generation of data for safety fine-tuning, jailbreaks, and adversarial attacks. We make careful design choices to adhere to the properties of natural language instructions and the jailbreaks taxonomy of Wei et al. (2023) , thus facilitating a thorough safety analysis that can be backed with corroboratory experiments on real LLMs. • We show that safety fine-tuning methods yield specialized transformations that primarily activate for unsafe inputs. We provide comprehensive analyses on the mechanisms learned by safety fine-tuning showing that these methods (i) encourage separate cluster formations for safe and unsafe samples by minimally transforming MLP weights to specifically project unsafe samples into the null space of model's weights, and (ii) substantially reduce the local Lipschitzness of the model for unsafe samples. • Adversarial inputs have activations similar to safe samples, hence bypassing the safety transform. Establishing the mechanism via which a model identifies which inputs to refuse processing of, we are able to demonstrate that by merely following an activation distribution that is exceedingly similar to that of safe samples, jailbreak attacks are able to ensure the minimal MLP transformation learned to identify unsafe samples is not triggered. 
preliminaries: 
 Safety fine-tuning protocols Broadly, LLM training can be divided into three stages (Team et al., 2023; Touvron et al., 2023b) : (1) (unsupervised) pre-training to build the initial model; (2) instruction fine-tuning to optimize the pre-trained model to follow instructions and provide plausible outputs for general queries; and (3) safety fine-tuning to ensure that the instruction fine-tuned model's output respects human preferences. We denote an LLM parameterized with parameters θ as f θ . Let the tuple t = {x, y p , y l } consist of the input x, the preferred response y p , and the less preferred response y l . Let θ IT , D, and ℓ(., .) denote the parameters of the instruction fine-tuned model, the safety fine-tuning dataset, and the standard cross-entropy loss, respectively. Using these notations, the objective functions of safety fine-tuning methods analyzed in this work can be written as follows.• Supervised Safety Fine-Tuning (SSFT) (Ouyang et al., 2022) : argmin θ E (x,y p )∼D ℓ (f θ (x), y p ).• Unlearning (Liu et al., 2024) : argmin θ E t∼D ℓ(f θ (x), y p ) -γℓ(f θ (x), y l ) .• Direct Preference Optimization (DPO) (Rafailov et al., 2023) :argmax θ E t∼D log σ β(ℓ(f θ IT (x), y p ) -ℓ(f θ (x), y p )) -γ(ℓ(f θ IT (x), y l ) -ℓ(f θ (x), y l )) .Note that DPO uses instruction fine-tuned model as the reference model during optimization, and there is no y l in the case of SSFT.Transformer block The transformer block used in this study consists of an attention module followed by two MLP layers with a non-linear activation layer-either silu (Elfwing et al., 2018) or GELU (Hendrycks & Gimpel, 2016) -in between. The second MLP layer writes to the residual stream of Figure 1 : Overview of our proposed synthetic setup to generate data. (a) A sample is divided into operators, operands, and outputs. The operators are function mappings the model is expected to perform on the operands to produce the output tokens, and are represented via tokens called task tokens. We often use the term text tokens to refer to the operands the functions are to be performed upon. (b) The functions are restricted to bijective mappings, motivated by their use in synthetic setups for mechanistically analyzing Transformer models (Chughtai et al., 2023; Ramesh et al., 2023) . (c) Text tokens are generated using PCFGs. To generate safe versus unsafe samples, we mark a subset of non-terminals at an intermediate level as safe-dominant (dark blue) and others as unsafe-dominant (light blue). Each of these nodes are associated with safe and unsafe task tokens, e.g., F s A and F u A respectively in blue box for safe dominant node. Our motivation here is that a task, by itself, is generally neutral (e.g., "design"), but when seen in the context of a concept it is to be performed on, i.e., the operands (e.g., "cycle" versus "bomb"), it can render the input unsafe.the Transformer block (Elhage et al., 2021) . Throughout this work, we denote W L and WL as the parameters of the first and the second MLP layers of the L-th transformer block.Fundamental subspaces (Strang, 2009) Let W m×n : R n → R m represent a matrix in R m×n . To avoid clutter, whenever possible, we denote W m×n by W. Let SVD(W m×n ) = U m×m Σ m×n V ⊤ n×n represent a singular value decomposition of W, where U and V consist of the left and right singular vectors, {u i ∈ R m } m i=1 and {v i ∈ R n } n i=1 , respectively, and Σ is the diagonal matrix with its diagonal elements being the singular values σ i , sorted in descending order of magnitude (σ i ≥ σ j for i < j). Let r ≤ min(m, n) be the rank of W. Using singular vectors as the orthonormal bases, the four fundamental subspaces of W are defined as:• Column-space: C(W) = span {u i } r i=1 , which is the same as the span of the columns of W. • Row-space: R(W) = span {v i } r i=1 , which is the same as the span of the rows of W. Note that R(W) = C(W ⊤ ).• Null-space: N (W) = span {v i } n i=r+1 . If Wx = 0, then x ∈ N (W). • Left Null-space: N L (W) = span {u i } m i=r+1 , which is the same as the null-space of W ⊤ .Note that C(W) and N (W ⊤ ) are orthogonal to each other. Similarly, R(W) is orthogonal to N (W). 
a synthetic controlled set-up for safety fine-tuning: 
 To systematically study the mechanisms yielded by safety fine-tuning and how adversarially designed inputs circumvent said mechanisms, we design a synthetic data generating process motivated by the framework of jailbreak attacks proposed by Wei et al. (2023) and Carlini et al. (2023) . Specifically, the use of a synthetic setup helps us model the competing objectives and mismatched generalization formulation of Wei et al. (2023) . For example, to elicit mismatched generalization, we must define samples that are out-of-distribution (OOD) compared to the ones used for safety fine-tuning of the model-the use of a synthetic data generating process helps us easily and scalably design such inputs.We emphasize that where possible, we do corroborate our findings on real-world LLMs (specifically, Llama models) by performing experiments similar to ones defined using our synthetic setup.We abstract out an input to an LLM as a composition of two components: (i) operators, which broadly specify a task the model is expected to perform, and (ii) operands, which specify what information the task is to be performed upon. For instance, consider the string: Tell me how to design a bike. Herein, one can deem design as an operator and bike as an operand. Despite its simplicity, we argue a large set of natural language inputs will fall under this abstraction (see App. B.1.2 for several examples). In our setup, we model this abstraction by defining an input to be a combination of tokens of two types: a task token f ∈ F representing the notion of an operator, where F is a family of predefined operators, and text tokens T , representing the notion of operands (see Fig. 1 ).To generate text tokens, we use Probabilistic Context-free Grammars (PCFGs)-an often used model for natural language that captures its syntactic properties (Knudsen & Hein, 1999; Charniak, 1997) and that has seen recent use as a framework for mechanistic analysis of language modeling capabilities of Transformers (Allen-Zhu & Li, 2023; Hahn & Goyal, 2023) . We denote a grammar as PCFG(γ, T, N T, R, P ), where R = {N T l i → {c j } m j=1 }|N T | i=0 is the set of production rules between non-terminal parent nodes (N T l i ) at level l and their respective children nodes {c j } m j=1 , and P is the set of probabilities associated with rules in R. A sequence of text tokens T is hence sampled by simply traversing through the PCFG tree, starting from the root node γ, propagating through non-terminal nodes (N T ) via production rules (R) according to their associated probabilities (P ), and terminating at the terminal nodes (T ). See App. B for a detailed discussion of this process. For the family of operators F, we follow recent work by Ramesh et al. (2023) ; Chughtai et al. (2023) and let each task token (operator) f ∈ F be a bijective mapping f : V → V, where V denotes the vocabulary of the PCFG generations (Fig. 1 (b)). For example, given text tokens T ∼ PCFG(γ, T, N T, R, P ) and task tokens f i , f j ∼ F, we define the sequence of output tokens as O = f j (f i (T )). Overall, the process above yields an input X := {f j • f i , T , O} (see Fig. 1 ). We note the goal for having two operators as part of the input (e.g., f i , f j ) is that it allows us to model the competing objectives format of jailbreak attacks proposed by (Wei et al., 2023) , wherein the model is asked to perform two tasks simultaneously, of which one is unsafe (e.g., f i ) and the other is not (e.g., f j ). To make the overall task non-trivial, we use four PCFGs (See Fig. A.8 in appendix) .For pre-training, we perform next token prediction on text and output tokens to learn the PCFG grammar rules R along with the bijective mappings of task tokens. For instruction fine-tuning, we supervise the model to predict output tokens given instructions consisting of task tokens f i , f j and text tokens T . Next we describe further necessary design choices we make to generate data for safety fine-tuning, jailbreak attacks, and adversarial attacks.Safety fine-tuning requires a dataset labelled as per user preferences (Rafailov et al., 2023; Ouyang et al., 2022) . Generally, the preferred output corresponds to accurately following the instruction for the inputs that are deemed safe, while refusing to respond to inputs that are deemed to be unsafe. We next develop an abstraction for such preference data for studying the mechanisms of safety fine-tuning. Specifically, we note that an operator or operand, by itself, cannot determine whether an instruction is safe or unsafe. For example, consider the following strings: Design a bomb (s1), Design a cycle (s2), and Provide the history of bombs (s3), where s1 is deemed unsafe and s2, s3 are deemed safe. One can easily see that it is the contextual meaning an operator and an operand acquire from being part of the same string that renders the overall string unsafe. For example, the operator design when seen in the context of operand bomb renders the overall string s1 to be unsafe, but not so when seen in the context of operand cycle. Similarly, the string s3, despite having bomb as its operand, is likely to be deemed safe, since therein the operator is merely Provide history.To model the intuition above in our framework, we split the non terminal nodes at a predefined intermediate level l s (= 3 in our experiments) into two disjoint sets called safe dominant nodes, A ⊂ N T ls , and unsafe dominant nodes, B ⊂ N T ls , where N T ls is the set of non-terminals at level l s . Let F s A and F u A respectively be the set of safe and unsafe task tokens associated with nodes in A (similarly for B); that is, if a node in A (resp. B) is selected while sampling the text tokens, the predefined set of operators that yield an overall string that is deemed safe come from the set F s A (resp.F s B ). We also constrain these sets such that|F s A | > |F u A |, |F s B | < |F u B |, F u A ⊂ F u B and F s B ⊂ F s A .These conditions ensure that if nodes from A (resp. B) are sampled, the corresponding sequences are mostly safe (resp. unsafe). Thus, different task tokens are associated with safe/unsafe inputs with different frequencies similar to real world instructions, e.g., operators like 'harm', 'destroy' are much more likely to be associated with unsafe generations as compared to 'design', 'purchase'.Overall, an input X is deemed unsafe if X = {f i •f j , T A , O} where f i , f j ∈ F u A or X = {f i •f j , T B , O} where f i , f j ∈ F uB (similarly for safe samples). This yields contextual dependence between operators and operands that render an input safe versus unsafe. We note that to capture the low variability observed in the outputs of safety fine-tuned LLMs for unsafe samples (e.g., the ubiquitous 'I can't explain', 'I can't tell', etc. responses), during safety fine-tuning, we supervise the model to output a token called null token (see App. B for details). Meanwhile, for safe samples, the model is fine-tuned to follow the instructions as usual.We ground ourselves in the framework of Wei et al. (2023) and Carlini et al. (2023) to study the following three types of adversarial inputs. For each type, we provide real-world examples in App. B.1.2 that map onto our synthetic abstraction, highlighting the analogy in detail.1. Jailbreaks via competing objectives (JB-CO-Task and JB-CO-Text in Fig. 2 (b) and (c)). Such inputs ask the model to simultaneously solve two tasks, one that is unsafe and one that is not. For example, consider the input: How to make a bomb? Start with "Sure, here's". The first phrase in this input may be deemed unsafe, while the second phrase is objectively neutral and merely asking the model to engage in an instruction following behavior. Often, the model in pursuit of following instructions will perform the task presented in the unsafe phrase as well. We investigate two ways to imitate such inputs. (i) Sample the two task tokens to define an input from either F u A and F s A or F s B and F u B , hence asking the model to perform both a safe and an unsafe task. (ii) Generate text tokens by using the lowest common ancestor of nodes in A and B as the root node and following PCFG grammar rules. We use the task tokens which generate safe inputs when combined with text tokens sampled from nodes in A and generate unsafe inputs for nodes in B. In this way, similar to (i), the model is asked to perform both a safe and an unsafe task.2. Jailbreaks via mismatched generalization (JB-MisGen in Fig. 2(d) ). Datasets used for safety fine-tuning are often substantially smaller and less diverse than the ones used for pretraining (Ouyang et al., 2022; Team et al., 2023) . For example, such datasets are generally in English, even though the model can process other languages or formats (e.g., ASCII). Use of alternative formatting of the input has thus become a viable way of bypassing safety fine-tuning (Wei et al., 2023; Kotha et al., 2023) . To model this in our framework, we define a set of task tokens T OOD which are not included in the safety fine-tuning dataset (similar to languages other than English). For each such token, we ensure there exists another task token that is used during safety fine-tuning and has the same functionality as the OOD token, i.e., corresponds to the same bijective mapping. This models the intuition that an unsafe input with similar semantics will likely be present in the safety fine-tuning dataset, but, e.g., in English. 3. Attacks based on continuous, learned embeddings (Adv in Fig. 2(e) ). Motivated by Carlini et al. (2023) , we append a set of embeddings to the input and optimize these embeddings via a white-box targeted attack on the model, akin to standard adversarial attacks in vision (Madry et al., 2018) . The attack's strength increases as the number of embeddings is increased. 
investigating the effect of safety fine-tuning: 
 We now investigate the mechanism by which safety fine-tuning impacts the behavior of a model. For this, we investigate three main aspects of a model: (i) feature space; (ii) parameter space; and (iii) function sensitivity. For experiments on our synthetic data-generating process, similar to existing related works (Jain et al., 2023b; Allen-Zhu & Li, 2023) , we train minGPT (Karpathy, 2020) using medium η M = 10 -4 and small η S = 10 -5 learning rates. See App. B.1.3 for further details on model training, selection, and cross-validation of the hyperparameters. To corroborate our claims, where possible, we run analogous experiments on Llama models (Touvron et al., 2023a; Card, 2024) by defining a dataset of 500 safe and unsafe natural language instructions that are structurally similar to our synthetic data (see App. B.2 for details). Specifically, we use Llama-2 7B and Llama-3 8B as x-axis: layer number, y-axis: average τ in Eq.2. (Top) Results using the synthetic setup. (Bottom) Results on Llama. Llama-2 chat 7B and Llama-3 chat 8B correspond to safety fine-tuned models.pretrained models and Llama-2 chat 7B and Llama-3 chat 8B as their corresponding safety fine-tuned variants.Our analysis focuses on MLPs in each Transformer block. Specifically, we analyze the activations at the output of this layer (after GELU) in Sec. 4.1, and its parameters and pre-activations in Sec. 4.2.The overall model's sensitivity to input perturbations is analyzed in Sec. 4.3. In all plots, green and red colors are used to denote the analysis corresponding to the safe and unsafe samples, respectively.We first analyze how safety fine-tuning affects activations of safe versus unsafe samples.Safety fine-tuning leads to formation of clusters of activations corresponding to safe versus unsafe samples, where the separation between clusters increases as better methods are used.Experimental setup Let a o L (x)[i] be the L-th layer's output activation corresponding to the i-th token of an input sequence x. We define the average activation corresponding to the q-th output token as âoL (x)[q] = 1 q-1 q+k-1 i=k a o L (x)[i],where k is the index of the last text token. If D S and D U are two datasets comprised solely of inputs with safe versus unsafe instructions, we define the mean safe and unsafe activation at layer L as follows.EQUATIONNow, if the model distinguishes between safe versus unsafe inputs at the level of intermediate layers' activations, we claim we will see two explicit clusters formed for safe versus unsafe inputs. To assess the same, we define the following measure that computes the Euclidean distance of a sample x's activations from the mean unsafe versus safe activation.τ x, µ S L , µ U L = ∥â o L (x)[q] -µ U L ∥ 2 -∥â o L (x)[q] -µ S L ∥ 2 (2)The measure above should be positive for safe inputs and negative for the unsafe ones. When analyzed over a large number of inputs, it helps us gauge how clustered the activations corresponding to safe versus unsafe inputs are. Results are reported in Fig. 3 . We find that activations-especially in the deeper layers-are indeed clustered depending on whether they come from safe versus unsafe inputs. Furthermore, in Fig. 3 (top), we observe in our synthetic setup that as the strength of the safety fine-tuning protocol increases (e.g., DPO and Unlearning compared to SSFT or DPO with medium learning rate η M compared to DPO with small learning rate η S ), separation between the clusters increases, where separation is defined as the difference between the average value of τ for safe versus unsafe samples. We find similar results using Llama-2 and Llama-3 models as well (see Fig. 3 (below)), indicating our findings translate to more realistic settings.We also investigate the impact of safety fine-tuning on the 'shape' of safe and unsafe feature clusters by analyzing singular values/vectors of their corresponding empirical covariance matrices Σ S and Layer 5 (Safety fine-tuning) Layer 6 (Safety fine-tuning) Layer 5 (Baseline) Layer 6 (Baseline)Figure 4 : Safety fine-tuning learns transformations ∆W whose column-space is more aligned with N (W ⊤ IT ). y-axis: Magnitude of projected component of left singular vector u i on N (W ⊤ IT ), x-axis: Index of left singular vectors, sorted by increasing magnitude of projected component.Σ U , respectively (refer App C.3.1). As clearly observed in Fig A .19, it is the top singular value of Σ U that is impacted the most as the safety fine-tuning progresses, however, the singular values of Σ S remain more or less the same. The σ 1 (Σ U ) scales to a point where it constitutes nearly 62% of the nuclear norm of Σ U , whereas this value is merely 12% for σ 1 (Σ S ). This indicates that safety fine-tuning reshapes the cluster of unsafe features in a way that there remains a single dominant direction. However, the shape of the cluster corresponding to safe samples is not impacted much.To identify what drives the formation of separate clusters of safe and unsafe samples, we evaluate precisely how model parameters change as a consequence of safety fine-tuning. Since Fig. 3 indicates clustering is strongest in deeper layers, we primarily analyze the MLP layers of the last two transformer blocks in this section. In particular, let W IT and W ST denote the instruction and the safety fine-tuned parameters of the first MLP layer of the L-th transformer block (L is intentionally omitted in notation to avoid clutter). Then, the change in parameters due to safety fine-tuning-or what we will often call "transformation"-is defined as ∆W = W ST -W IT .The column-space of the transformation, C(∆W), is more aligned with the null-space N (W ⊤ IT ) than it is with the column-space C(W IT ). Hence, samples processed by the transformation versus not will have rather distinct activations, enabling clustering.Experimental setup Let {u i } r i=1 and {σ i } r i=1 be the top r left singular vectors and singular values of W IT , where r denotes the empirical rank of W IT , which is defined as the minimum value of k such that 99% of variance is preserved, i.e., k i=1 σ 2 i ≥ 0.99∥W IT ∥ 2 F . Similarly, let { u i } t i=1 be the top t left singular vectors of ∆W where t is the empirical rank of ∆W. The projection matrix for the column-space of W IT is defined as P := r i=1 u i u ⊤ i . Let θ i be the angle between P u i and u i . It is easy to see that u i sin(θ i ) provides the projection of u i on N (W ⊤ IT ) since N (W ⊤ IT ) is orthogonal to C(W IT ). Since u i is unit norm, we can plot the magnitude of projection of u i on the space N (W ⊤ IT ) by evaluating sin(θ i ). Results for blocks 5 and 6 are shown in Fig. 4 for the PCFG-based experiments, and in Fig. A .17 for Llama models. A baseline model fine-tuned using standard cross-entropy loss to follow instructions in the usual way is also evaluated (shown in dotted lines in Fig. 4 ). Our results indicate that for safety fine-tuned models, the magnitude of projected component onto N (W ⊤ IT ) is very large, especially when compared to the baseline. This implies ∆W and W IT are nearly orthogonal to each other. Thus, a sample processed by ∆W will have a component that cannot be computed by W IT itself, hence yielding two broad sets of activations corresponding to samples which are processed by ∆W versus not. To make this more concrete, we next evaluate which samples are likely to be processed by ∆W by analyzing its row space.Pre-activations of unsafe inputs have a larger projection onto the row-space R(∆W) compared to pre-activations of safe inputs. That is, ∆W preferentially impacts unsafe samples.We analyze pre-activation for the last text token, i.e., one corresponding to the first output token prediction. The pre-activation is normalized since our goal is to primarily assess its alignment with the row-space of ∆W. Specifically, to capture the effect of ∆W on a given Figure 5 : Safety fine-tuning learns transformations ∆W which are specialized for unsafe samples. The x-axis shows the index of the top-15 basis vectors (v i ) of ∆W spanning its row space and y-axis is σ i v ⊤ i a. Here we only plot for the 6th transformer block.unit-norm pre-activation a, we compute σ i v ⊤ i a for each i, where {v i } r i=1 are the top r right singular vectors (basis vectors of the row-space) of ∆W. This quantity provides the effect of the pre-activation component along v i on the outputted signal's magnitude ∥∆Wa∥ 2 . Results are shown in Fig. 5 . We observe that the impact for unsafe samples is larger than that of safe samples. In fact the impact on safe samples is close to zero. The results are more prominent for stronger safety fine-tuning protocols (e.g., DPO) or when larger learning rates are used. This indicates the transformation learned via safety fine-tuning results in a few directions (the top-k right singular vector) and it primarily activates for unsafe samples. We also investigate if there are specialized neurons acting on unsafe samples, compared to safe ones, to enable the above results. As we show, a subset of neurons are highly aligned with the top singular vector v 1 , hence specializing to processing unsafe samples and impacting the norm of their activations (see Fig. A.38 ).The observations above highlight that ∆W projects the unsafe activations onto the null space of W ⊤ IT , while not impacting the safe activations to a great extent. However, given that our analysis is localized to a specific layer, it is unclear how this impact propagates with the increasing depth of the model and the non-linear operations therein. We provide further analysis in App. C.1 to address this question, showing that our findings generalize even when the entire model is accounted for: i.e., model learns specialized transformations to cluster safe vs. unsafe samples.To further corroborate our claims, we also provide an interventional experiment. Specifically, we hypothesize that if indeed ∆W helps identify unsafe samples and steer the model towards refusing to process them, then interpolation between weights before safety fine-tuning and after should primarily alter model behavior on unsafe samples, yielding essentially the same behavior on safe ones. Further, extrapolation along ∆W should yield stronger refusal abilities. To this end, we modify W IT as W α IT = W IT + α∆W, which is equivalent to traversing in the direction of ∆W. If our hypothesis holds, taking α from 0 to 1 or beyond should enhance the cluster separation between safe and unsafe samples. We demonstrate that this is indeed the case and provide the results for these interventions in Figs A.76-A.80. In fact, interestingly, we observe that the less performant safety fine-tuning method, i.e., SSFT, can be substantially improved by merely extrapolating (α > 1) along the direction of ∆W: the model becomes more robust to jailbreak attacks, while preserving performance on safe samples (see Fig. A.72) . We note these results are similar in spirit to parallel work by Arditi et al. (2024) and Zheng et al. (2024) .Safety fine-tuning reduces the local Lipschitzness of the fine-tuned model for unsafe samples while increasing it for the safe ones.We next probe the sensitivity of the fine-tuned model's output with respect to safe versus unsafe samples. As a standard tool in literature on adversarial attacks (Hein & Andriushchenko, 2017; Wong & Kolter, 2018) , this experiment helps us test the robustness of learned safety mechanism to minimal changes in model inputs. Note that investigating just the linear mapping W for this would lead to sample-independent quantities as the local Lipschitz constant of W only captures the summary of its singular values. For example, if L 2 is chosen as the norm in input and output metric spaces, then the Lipschitz constant of W boils down to its spectral norm. Therefore, in order to capture the sensitivity of the entire model for different sub-populations of the data, we choose to empirically quantify it for each data point and plot histograms over a dataset (Sanyal et al., 2019) . For a given real-valued function fθ : x → R and input x, we define the local Lipschitzness of f at x as Lip f (x) = ∥∇ x fθ (x)∥ 2 .Experimental setup We consider f = argmax j h θ (x)[k](j), where h θ (x)[k](j) is the j-th logit predicted at the end of text token index, denoted by k. The sensitivity is obtained corresponding to the most confident output. Parameters θ IT and θ ST are chosen depending on the model under consideration. The histograms of Lip f (x) for safe (green) and unsafe (red) samples are shown in Fig. 6 . We can clearly observe that the sensitivity of the safety fine-tuned model is much lower compared to instruction fine-tuned model for unsafe samples, especially when DPO and Unlearning are used for fine-tuning. This makes sense as, for unsafe samples, the variation in the preferred output strings in safety fine-tuning dataset is much less compared to that of safe samples: e.g., preferred outputs for unsafe samples are generally 'NULL', 'I can't assist', etc. The consequence of this decrease in sensitivity is that it will be relatively more difficult to craft jailbreaks and adversarial attacks for more effective safety fine-tuning protocols, since models witness a stronger decrease in Lipschitzness under those protocols. We validate this claim in Tab. A.1 as well, showing that crafting jailbreaks and adversarial attacks is more difficult for DPO and Unlearning as compared to SSFT.5 Evading the Safety Mechanism: Jailbreak and Adversarial InputsHaving established and investigated the mechanism via which safety fine-tuning leads the model to refuse to process unsafe inputs, we can now analyze precisely why jailbreaks and adversarial attacks are still able to induce unsafe responses from the model.Jailbreak and adversarial attacks yield intermediate features that are exceedingly similar to safe samples, hence evading the processing by ∆W required for refusal of an input.Experimental setup We use our instantiation of jailbreaks and adversarial attacks defined in Sec. 3.3, and motivated by the works of Wei et al. (2023) and Carlini et al. (2023) . As shown in Tab. A.1, for DPO with µ M , the JB-CO-Text attack yields the highest success rate (97.2%), whereas the JB-CO-Task attack yields the lowest one (31.5%). This trend is also observed for other safety fine-tuning methods (see Tab. A.1). For further analysis, we only consider the successful attacks. . We find the cluster separation between safe samples and attacked samples decreases in the feature space as the strength of attack increases, i.e., the decrease is higher for JB-CO-Text and JB-MisGen, which are stronger attacks (See Tab. A.1) as compared to JB-CO-Task. We observe a similar trend for adversarial attacks as well. This indicates with increase in attack strength, adversarial inputs yield features that are similar to safe samples. We note that concurrent work by (Ball et al., 2024) provide additional evidence in support of these results on larger models like Vicuna-13B.(ii) Function space. Building on Sec. 4.3, we analyze the empirical Lipschitz constant for jailbreak and adversarial attacks in Fig. 7 (middle row). Clearly, with increase in attack strength, the histogram for jailbreaks starts to overlap with the histogram corresponding to the safe samples, showing that the model's local sensitivity also starts to lie between attacked and safe samples. Similar to the feature space analysis above, the function sensitivity analysis also highlights that with the increase in attack strength, the adversarial samples start producing representations similar to safe samples.(iii) Parameter space. To tie everything together and explain the similarity of features between jailbreak and safe samples, we finally build on Sec. 4.2 and analyze the impact of ∆W on jailbreak and adversarial inputs. Specifically, we analyze the alignment of pre-activations a corresponding to these inputs with the row space of ∆W (same setup as discussed in Fig. 5 ). Results are shown in Fig. 7 (bottom). We observe that unlike unsafe samples, ∆W does not impact jailbreak / adversarial samples noticeably: e.g., see Fig. 5 , where unsafe samples have a much higher alignment with row space of ∆W compared to safe ones, versus results on JB-CO-Text inputs in Fig. 7 (bottom), where we find the alignment is essentially the same! As we showed before, it is the impact of ∆W that leads to a distinction between how safe versus unsafe samples are processed; hence, results above suggest the model will process successful jailbreak / adversarial samples as if they were safe. We provide additional fine-grained analysis related to our observations above for different safety fine-tuning methods and layers in App. C.3.4 (for jailbreak attacks) and C.3.8 (for adversarial attacks).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We present details about the proposed synthetic setup in Sec. B of appendix. Also refer to https://github.com/fiveai/understanding_safety_finetuning for additional details.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] .Justification: Refer to https://github.com/fiveai/understanding_safety_finetuning.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We proposed a synthetic data generation framework to systematically and efficiently analyze common safety fine-tuning methods and craft jailbreak attacks. Using our framework, we showed that safety fine-tuning encourages the formation of separate clusters for safe and unsafe samples while making the model significantly less sensitive towards unsafe ones. We further found that the clustering effect in model's activation space can be explained by the weight space analysis, where the learned update was found to be specialized in projecting the unsafe samples onto the null space. These updates were not able to generalize well against samples for jailbreak and adversarial attacks, which resulted in their activations being more similar to safe samples than the unsafe ones. Hence bypassing the safety mechanism learned by the model. Wherever possible, we also showed via experiments on Llama models that our claims directly transferred to more realistic setups. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Refer to Appendix Sec. B.1.3.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Wherever possible, we plot the standard deviation along with the means in the plots.Guidelines:• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: For all interpretability experiments we use a RTX A4500 gpu with a memory of 20 GB. For pre-training using the synthetic setup, we use a RTX A6000 gpu with a memory of 48 GB. Additional details on training time are present in Appendix Sec. B.1.3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] .Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] .Justification: Refer to https://github.com/fiveai/understanding_safety_finetuning.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] .Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]  
 
 Appendix: Safety fine-tuning Approaches in LLMs. The pipeline of training a large language model (LLM) involves three stages: (i) pre-training, (ii) instruction fine-tuning and (iii) safety fine-tuning. During pre-training, an LLM is supervised to predict the next token using a large amount of data scraped from web (Radford et al., 2019 (Radford et al., , 2018)) . This enables an LLM to learn various capabilities. In the instruction fine-tuning stage (Wei et al., 2021; Sanh et al., 2021; Raffel et al., 2020) , the model is prompted by an instruction and supervised to output a predefined output for that specific instruction. However, due to the random sampling process of pre-training data from the internet, the instruction fine-tuned model can demonstrate unsafe capabilities as well. Therefore, as a last step, safety fine-tuning is performed to limit the capabilities of an LLM to yield unsafe outputs. For this purpose, data is gathered by having humans rank multiple outputs from the instruction fine-tuned LLM for a given prompt considering whether the output is safe or unsafe. Then, using this dataset, the LLM is commonly trained by one of the following four different protocols.1. Supervised safety fine-tuning (SSFT) (Ouyang et al., 2022) relies only on the highly ranked outputs, i.e., the safest ones. Thus, the aim here is to make the model safe by fine-tuning it to follow the safe instructions and generate safe output for unsafe samples.2. Reinforcement learning with human feedback (RLHF) (Ouyang et al., 2022; Christiano et al., 2017; Bai et al., 2022; Stiennon et al., 2020) . The instruction fine-tuned model is trained as a reward model to replicate the human preferences, by assigning high reward to human aligned generations and low for others. A copy of the instruction fine-tuned model is then treated as a "policy" and fine-tuned using the reward model as a proxy, where high reward is given when it generates human aligned generations.3. Direct preference optimization (DPO) (Rafailov et al., 2023 ) also uses safe and unsafe outputs similar to RLHF, but differently does not require an additional reward model. Instead, the LLM is directly supervised to suppress unsafe outputs by the constructed objective function.4. Unlearning (Liu et al., 2024; Li et al., 2024; Goel et al., 2024; Lynch et al., 2024) has been commonly used to address privacy concerns, where the aim is to make the model forget certain data samples (Maini et al., 2024; Nguyen et al., 2022) . However, in case of safety fine-tuning, the objective is to unlearn the capabilities responsible for generation of malicious and unsafe outputs. Given similar goals, unlearning has recently become popular as a protocol to perform safety fine-tuning (Liu et al., 2024) . This motivates us to investigate this fine-tuning protocol as well. We note that past works (Liu et al., 2024; Li et al., 2024; Goel et al., 2024; Lynch et al., 2024) have used different objective functions to perform unlearning, however, in most of these cases, the loss functions include two contrasting losses: one enforces the model to retain its safe capabilities to generate coherent outputs, while the other loss aims to force the model to forget its unsafe capabilities. Given this characteristic, we adopt the loss function used in Liu et al. (2024) .Understanding fine-tuning in LLMs. Fine-tuning is an exceedingly ubiquitous tool in the modern era of foundation models. Given this success of fine-tuning, it has become imperative to understand how it impacts the capabilities of pre-trained models. Recent works in this vein (Kotha et al., 2023; Tripuraneni et al., 2020; Neyshabur et al., 2021) show that fine-tuning works by re-weighting and transferring task relevant features to the downstream task. Relatedly, Jain et al. (2023b) , Prakash et al. (2024), and Lubana et al. (2022) analyze the effect of fine-tuning in a more mechanistic manner, where they conclude that fine-tuning minimally alters the pre-trained mechanisms, rather than fundamentally changing them. Relatedly, Lee et al. (2024) analyze DPO and concluded that DPO makes the model learn to bypass the activations corresponding to toxic regions in its activation space. We believe that our observations discussed in App. C.1 implicitly indicate the span of activations in the toxic regions of activation space reduces with safety fine-tuning.Jailbreaks and adversarial attacks in LLMs. It has been shown that the current LLMs are vulnerable to adversarial attacks (Sadasivan et al., 2024; Zou et al., 2023; Carlini et al., 2023) and jailbreaks (Wei et al., 2023; Andriushchenko et al., 2024; Sun et al., 2024; Mehrotra et al., 2023; Samvelyan et al., 2024) . Adversarial attacks are generally easier to identify programmatically when compared to jailbreaks. However, optimizing a prompt using adversarial training is prone to end up generating gibberish tokens in the input space. Therefore, it is easy to detect such attacks by using simple pre-processing techniques like perplexity (Jain et al., 2023a) .On the other hand, since jailbreaks are more natural and difficult to detect, they pose a bigger threat to safety of LLMs. Wei et al. (2023) characterize jailbreaks into two broad categories: (i) jailbreaks with mismatched generalization and (ii) jailbreaks with competing objectives.This section includes further details on our synthetic and real world experimental setups.B.1 Further Details on the Synthetic Setup based on PCFGOur synthetic setup involves defining samples comprised of two task tokens f1, f2, text tokens T , and output tokens O. The sampling of task and text tokens is conditioned on a sample being safe or unsafe, which we discuss in detail in the main paper. An example sample is illustrated in Fig.A.9 and we now provide the details for each of these aspects below.Task Tokens. The task tokens are denoted as fi ∼ F , where F = {fi} 12 i=1 and each fi is given by a bijective mapping fi : V → V. Here, V is the vocabulary of the PCFG. To generate an input prompt we sample two task tokens fi and fj randomly from the set F. During safety fine-tuning, we do not sample any token from the set TOOD, which consists of two tokens out of a total of twelve tokens present in F. There is a token amongst the remaining ten for each of these two tokens which represents the same bijective mapping but corresponds to a different token representation. For each sample generation we sample two task tokens.Text Tokens. Every sample consists of between 15-25 text tokens that are generated by a PCFG relying on a set of grammar rules with uniform sampling probabilities. Note that here different combinations of sampling probabilities can give rise to more interesting generations, but we pose this as an interesting future direction. For simplicity of analysis, in this work we consider uniform sampling. We provide a detailed description of the grammar rules considered, along with different task tokens corresponding to setsF u A , F u B , F s A and F s B , in Fig. A.8. Note that for pre-training and instruction fine-tuning, we sample the data using four different PCFGs. We describe the motivation and further details related to this design choice below.A model trained on the synthetic data generated using a single PCFG (as above) might perform well by simply learning the relationship between a single text token in T with its corresponding operators, therefore ignoring the context window consisting of previous text tokens in a sequence. Thus, to force the model to learn to utilize the context, we utilize multiple different PCFGs (four in our experiments) such that different bijective mappings corresponds to the same task tokens across different PCFGs. For example, the task token '(' in a PCFG might imply bijective mapping f1, whereas the same token might imply f2 in another PCFG. Also, we ensure that the generated text tokens from each PCFG do not completely overlap. Thus, in order for the underlying model to perform well on this dataset, it has to learn the grammar corresponding to each PCFG which would require using context from previous text tokens. Now we will discuss some additional intricate design choices considered while designing this setup to imitate real world scenario as much as possible, which we could not discuss in the main paper due to space constraints. Through a single traversal from the PCFG tree during pre-training we generate a sample which is of length 50-75 tokens and later crop it by randomly selecting the starting and ending index of a window sampled randomly to be between 15 -25. This generation is started from the root node of the PCFG tree (See Fig. A.8 ). However, during safety fine-tuning, we divide the non terminal nodes at level three into safe dominant A and unsafe dominant B nodes. Using these nodes in sets A or B as the root node reduces the length of the generated sequence to between 15-25. Therefore, to ensure consistency between pre-training and safety fine-tuning, we crop T to contain between 15-25 text tokens during pre-training. We define different sets of task tokens being safe and unsafe with each of the sets A or B: F s N for safe andF u N for unsafe. We ensure that |F u B | = 8, |F s B | = 2, |F u A | = 2, |F s A | = 8. Further F u A ⊂ F u B and F s B ⊂ F s A .This helps in controlling how often a task token is associated with safe vs unsafe generations. The PCFG trees utilized in our analysis have a depth of 6 levels and this is selected based on the design choices considered in Allen-Zhu & Li (2023) . Another reason for choosing this depth is that it ensures the length of the sequence generated to remain in the expected limit. To ensure simplicity of the safety fine-tuning task, for safety fine-tuning we only consider the grammar generated by the first PCFG. Further we choose the third level to divide the non-terminal nodes into the sets A and B because it helps in generating a good enough sequence length, which could decrease significantly on increasing the levels or going down the tree. We did not choose level 2 because it would mean lesser number of non-terminal nodes are involved in determining if the sample is safe or unsafe. This would in tern make the safety fine-tuning task easier for the model. In order to balance this trade-off between the task complexity and the length of text tokens generated, choosing the third level suits the best.Outputs Tokens. The output tokens O = f1 • f2(T ), thus the length of O is same as T . In case of unsafe samples we ensure that all the output tokens are null token defined by a single token given by 'a'.Table A .1: Safety performance of different fine-tuning protocols: Unlearning (Liu et al., 2024) , DPO (Rafailov et al., 2023) and supervised safety fine-tuning (SSFT) (Ouyang et al., 2022) with medium and small learning rates are used for performing safety fine-tuning. Instruct represents the accuracy of the model to follow instructions and Null represents model's accuracy to output null tokens. Different jailbreaking attacks are also analyzed. JB-CO-Text and JB-MisGen are the strongest attacks where SSFT is easiest to attack.Protocol Learning Rate Safe (Instruct) Unsafe (Null) Unsafe (Instruct) JB-CO-Task (Instruct) JB-CO-Text (Instruct) JB-MisGen (Instruct) Jailbreak attacks. To design jailbreak attacks using our synthetic setup, we manipulate the sampling process of the text and task tokens depending on the type of jailbreak attacks we wish to craft. We describe the setup corresponding to each attacks along with corresponding examples generated using Llama-2-70b-chat model on https://chat.lmsys.org/. As shown in Fig. A .10, the Llama-2 70B chat model doesn't follow the instructions when prompted to generate unsafe text. However, we can break this safety mechanism of the model by using different types of jailbreak and adversarial attacks which we discuss below.• Jailbreak attacks with competing objectives (task), JB-CO-Task: As shown in Fig. A .13, jailbreak attacks with competing objectives aim to break the safety mechanism of language models by prompting the model to follow instructions, while still having the unsafe prompt present in the input (Wei et al., 2023) . In this case the "history part" is prompting the model to follow instructions and as a result the model also outputs about designing a bomb which clearly it should not output. Motivated by this, we sample one task token from F s N and the other from F u N . Related to the example, consider that "history" was sampled from F s N and "design" from F u N . This ensures that a part of the input prompt asks the model to generate safe output by following instructions, whereas the other part corresponds to unsafe generations.• Jailbreak attacks with competing objectives (text), JB-CO-Text: Here, instead of manipulating the sampling process of task tokens, we modify the sampling process of text tokens. We do this by sampling the text tokens using the common parent node of the nodes in the set A and B as the root node. We present the corresponding motivating example for this attack in Fig. A .12. Here, the "cycle" and "bomb" can be interpreted as two different text tokens sampled using safe dominant and unsafe dominant non-terminal nodes respectively of the PCFG tree, where "cycle" is prompting the model to follow instructions and "bomb" corresponds to the unsafe part.• Jailbreak attacks with mismatched generalization, JB-MisGen: As shown in 2023). We append some soft prompts after the text tokens in the token encoding space of the model. We define the threat model as the number of soft prompts appended. Next, we perform targeted white box attack, by minimizing the standard cross entropy loss, where we utilize the ground truth labels corresponding to the respective bijective mapping as the target class. For this we use a threat model constraining the ℓ2 norm of the soft tokens to be less than 1. To generate the attack, we use 10 steps of iterative gradient descent.In all our experiments on the synthetic setup, we use mingpt models, which consist of approximately three million parameters and include six transformer blocks, each containing six attention heads followed by two MLP layers, where the dimension of the activation stream is 192. The first MLP layer upscales it to 768 and the second one again downscales it to 192 dimensions. We use a maximum input sequence length of 100 tokens.There is a GELU activation layer in between the two MLP layers. Pre-training and instruction fine-tuning. We train the model to learn the grammar rules and structure of PCFG trees by using the next token prediction task on text tokens. We also train the model to learn the bijective mappings by correctly predicting the output tokens. Instead of separately performing instruction fine-tuning, we utilize a curriculum to transition from pre-training phase to instruction fine-tuning, where we associate probability of training the model on text tokens by PT and PO for the output tokens. During the initial phase of pre-training, we utilize a high value of PT and low for PO and linearly transition to using low value of PT and high for PO. We observe that using a curriculum helps in stabilizing the training and it helps us achieve a model capable of predicting the output tokens correctly.We use a cosine schedule on learning rate to ensure that a large learning rate is used for pre-training where majority of training focuses on learning the PCFG structure and a small value of learning rate is used for instruction fine-tuning where the major focus is to learn the bijective mappings. We decay the learning rate to 1e -6. We use 100k iterations to perform this training, with a learning rate of 1e -3 and cosine schedule with warmup of 10k iterations. This stage of combined pre-training and instruction fine-tuning takes over 8 hours on a single RTX A6000 gpu with 48GB memory, on using a batch size of 512.Safety fine-tuning. We perform safety fine-tuning for 10k iterations, using cosine schedule without warmup with two sets of learning rates: 1e -4 and 1e -5 and decay them to 1e -7. We refer to 1e -4 as ηM and 1e -5 as ηS. In contrast to pre-training and instruction fine-tuning, here we use the preferred y P and less preferred y L output tokens for fine-tuning the model using different safety protocols namely supervised safety fine-tuning, direct preference optimization and unlearning. In case of safe samples, y P refers to the outputs corresponding to bijective mapping, whereas y L refers to null token prediction. On the other hand, in case of unsafe samples, y P refers to null token prediction and y L refers to instruction following generations (ie. bijective mappings).As common in literature for pre-training as well as fine-tuning we use adam optimizer. We perform search over different values of β and γ which correspond to the hyperparameters used in the objective functions of unlearning and DPO (refer to main Sec. 2 for more details) and select the values which can give close to 100% accuracy on both safe and unsafe samples. We list the optimal values of hyperparameters below:• Unlearning (ηM ): γ = 0.1; Unlearning (ηS): γ = 0.01• DPO (ηM ): β = 0.1, γ = 0.01; DPO (ηS): β = 0.1, γ = 0.002Evaluation setup. We perform evaluation using Acc (OR) defined as n i=1 (Oi == yi) where n denotes the number of output tokens, Oi denotes the i th output token and yi represents the corresponding ground truth value. We use 1K samples randomly sampled independently from the PCFG tree for generating the test set. By manipulating the sampling process of text and task tokens as described earlier, we generate the test sets of jailbreak samples as well. Each of these sets contain 1K samples. We utilize all these samples for our analysis. The results corresponding to the three safety fine-tuning protocols trained with medium and small learning rates (ηM and ηS) respectively are present in Table A .1. Note that here we denote the accuracy of the model to output null tokens on unsafe samples sampled from the same distribution used for safety fine-tuning by Unsafe (Null) and similarly, we denote the accuracy of the model to follow instructions by (Instruct).We analyze how different observations as discussed in main paper transfer on Llama-2 7B (Touvron et al., 2023a) , Llama-2 7B chat, Llama-3 8B and Llama-3 8B chat models. For this, we make a simple synthetic dataset where each prompt consists of an operator-operand combination. The operator can be considered as a similar version of task tokens as discussed above and operand can be considered similar to text tokens.Data Generation. We generate around 50 prompts corresponding to safe and unsafe samples manually and later augment the corresponding sets with the help of GPT-4 (Achiam et al., 2023) to generate a dataset containing 500 samples corresponding to safe and unsafe prompts each. We make an evaluation subset of 100 samples from this. We present a subset of samples considered for analysis on Llama in Fig. A.15 .v → u t v → t s u → r p u → p 10$ q u → 10# r q t → q p r t → 10* p q t → 10$ r s → p 10# s → r 10$ 10* s → 10$ 10* 10 We divide the non terminal nodes r, p, q, 10#, 10*, 10$ into sets of safe dominant and unsafe dominant nodes.# r → o n m r → w x o q → w o q → m o x p → m n p → o o w 10$ → x w m 10$ → n w n 10# → n x x 10# → n n x 10* → m m x 10* → w w o o → l k j o → l 1h j n → k 1h 1f n → k l j m → l k 1f m → 1f 1h l w → 1f 1h 1h w → l 1f j x → j j l x → 1f l l → 1i i g l → i 1i g k → 1e g i k → 1g g h j → i 1e 1i j → 1g h 1e 1f → 1i h 1g 1f → 1i 1e h 1h → h 1i 1g 1h → 1g 1e h g → 2a 2b 2c g → 2d 2e 2f h → 2h 2i 2j h → 2k 2l 2m i → 2n 2o 2p i → 2qFor PCFG 1, the safe domainant set includes: r, p, q and the unsafe dominant set includes:10#, 10*, 10$ This division is different for different PCFGs.Unsafe task tokens: ), [ Safe task tokens: (, ], {, }, ?, !, @, £For PCFG 1Held out task tokens: +, FigureA.8: Demonstration of the grammar rules, task tokens and safe/unsafe dominant nodes in the PCFG based synthetic setup. We use four PCFGs during pre-training and instruction fine-tuning, where they are similar except the leaf nodes. The safe and unsafe dominant nodes are labelled at the third level of the tree (as highlighted in the figure), where each indentation represents termination of a level in the PCFG tree. We also present two held out task tokens (T OOD ) which are not used during safety fine-tuning.Task Tokens: '(', '£' Text Tokens:: '2n', '2o', '2p', '3b', '3a', '2z', '2d', '2e', '2f', '2t', '2u', '2v', '2d', '2e', '2f', '2q', '2r', '2s', '2z', '3a', '3b', '2h', '2i', '2j' Output Tokens:: '2p', '2q, '2n', '2a', '2b', '2q', '2e', '2d', '2j', '2s', '2w', '2u', '2e', '2d', '2t', '2r', '3a', '3s', '3x', '2k', '2a', '2v', '2z', '3c' Sample: '#', '(', '£','% ' '2n', '2o', '2p', '3b', '3a', '2z', '2d', '2e', '2f', '2t', '2u', '2v', '2d', '2e', '2f', '2q', '2r', '2s', '2z', '3a', '3b', '2h', '2i', '2j''!', '2p', '2q, '2n', '2a', '2b', '2q', '2e', '2d', '2j', '2s', '2w', '2u', '2e', '2d', '2t', '2r', '3a', '3s', '3x', '2k', '2a', '2v', '2z', '3c As discussed in the main paper, ∆W captures modification in a single layer of the model, thus it is imperative to understand how this change propagates with the increasing depth of the model. For this we analyze the change in the activation spaces corresponding to safety fine-tuned and instruction fine-tuned models for safe and unsafe samples. For this, we can analyze the angle of projection between the two spaces. Let F os ST be formed by stacking the post-activations a corresponding to safe samples xs in layer L. Similarly we can define F ou ST , F os IT and F ou IT . Note that here a represents the activation stream of the last text token. We discuss our observations and the derived conclusions in detail below:Justification: The learned update ∆W ensures that the column space of F ou ST has a large projection on the left null space of F ou IT for L ≥ t where t is large and generally corresponds to the last few layers of the model.Experimental setup: Let v1, . . . , vt be the singular vectors with non-zero singular values spanning the column space of F on ST , where n corresponds either safe or unsafe set of samples. Then we calculate the sine of the angle between each vi and C(F on IT ). We present these results in Fig A .16. If this value is high, it would represent a large projection of F on ST on NL(F on IT )The angle of projection of vi increases with the increase in layer number L and with the decrease in corresponding singular value. This suggests that the unsafe activations are being steadily projected into the left null space of their original activations calculated using instruction fine-tuned model. Similar trend is not observed for safe activations, thereby showing that the update ∆W primarily modifies the unsafe activations and this effect increases with increase in depth of the model. To corroborate these results, we perform this analysis on Llama-2 7B chat in Fig A .18.We present additional evidence corroborating our analysis on the proposed synthetic setup by using Llama-2 7B and Llama-2 7B chat models. Llama-2 7B is a pre-trained model and Llama-2 7B chat is the fine-tuned version of Llama-2 7B, where the fine-tuning involves both instruction as well as safety fine-tuning. Note that the instruction fine-tuned version is not officially released, which hinders our analysis on ∆W. As a result, we use the pre-trained model Llama-2 7B. We will now present the results discussed in the main paper for Llama-2.Clustering analysis: As shown in Fig A .75, 3 we observe that on fine-tuning, Llama learns to form separate clusters for safe and unsafe samples, which is not observed in case of pre-trained model ie. Llama-2 7B.Analysis on ∆W: As shown in Fig A .17, we observe that the projection of basis vectors spanning the column space of the learned update for any layer of Llama-2 7B chat lies largely in the null space of Llama-2 7B.Analyzing the activation spaces for safe and unsafe samples: We find the angle of projection of top basis vectors spanning the column space of activations in Llama-2 7B chat, onto the activation space of Llama-2 7B. As shown in Fig A .18, similar to our observations in the proposed synthetic setup, we observe that the sine of the angle of projection is higher for unsafe samples than for the safe ones. In this section, we will discuss additional results on the proposed PCFG based synthetic setup supporting our analysis presented in the main paper. First, we will discuss the learning dynamics of ∆W in Sec C. We analyze the learning dynamics of ∆W for observations 2 and 3 discussed in the main paper.The spread of unsafe samples in the feature space becomes low rank with the advent of training:We analyze the spread of the two clusters. For this, we calculate the empirical covariance for both the clusters as follow:Σ U = x∈D U [(ā o L (x)[q] -µ U L )(ā o L (x)[q] -µ U L ) T ] , Σ S = x∈D S [(ā o L (x)[q] -µ S L )(ā o L (x)[q] -µ S L ) T ]We let q = 1 and perform singular value decomposition (SVD) of Σ U and Σ S for checkpoints at different safety fine-tuning iterations for DPO (ηM ) and plot the top-15 singular values in Fig A .19. We observe that as the safety fine-tuning converges, the scaling effect of the top singular vector of Σ U becomes more dominant as compared to the other singular vectors. This is also evident from the spectral norm of Σ U , which constitutes over 62% of its nuclear norm, whereas in case of Σ S this is only 12%. This indicates that the empirical rank of the space corresponding to unsafe samples has lowered down, whereas it remains similar in case of safe samples (See Fig A.19 ). Note that the empirical rank is computed by choosing the minimum value of r such that 99% of variance is preserved, implying, r i=1 σ2 i ≥ 0.99∥W∥ 2 F1 We demonstrate that these observations are consistent with other safety fine-tuning protocols and transformer blocks in Fig A.20, A.21, A.22 . This analysis shows that safety fine-tuning encourages the model to lower down the spread of features corresponding to unsafe samples, while the spread remains similar for safe samples. As observed with the increase in number of iterations, the effect of topmost singular vector on pre-activations increases for unsafe samples, whereas it remains low for the safe ones. As observed with the increase in number of iterations, the effect of topmost singular vector on pre-activations increases for unsafe samples, whereas it remains low for the safe ones.We present three different ways to analyze how well the safe and unsafe samples are clustered in the activation space of the safety fine-tuned model.Using Eq. 2: Here we present detailed comparison of our clustering analysis presented in the main paper in Sec. 4.1 with different jailbreaking attacks. We present these results in Fig A.29, A.30 . We observe that on performing jailbreaking attacks, the separation between the clusters decreases. The corresponding results for W are presented in Fig A.50 and A.51 , where similar observations hold.K-means clustering: Next, we perform the k-means clustering analysis, which is unsupervised. Here we randomly pick the two feature vectors as the starting point and run k-means clustering algorithm. We label each point with a cluster and then check if K-means is able to separate the activations into clusters of safe and unsafe samples. We measure this using accuracy. The corresponding results are presented in Fig A.31, A.32 , where we observe that K-means is able to cluster the safe and unsafe samples into different clusters for the later layers of the model and these observations are more dominant in case of stronger safety fine-tuning protocols like DPO and unlearning and when using a medium learning rate. Further, on performing the jailbreaking attacks, it becomes difficult to spearate the safe and the attacked samples into two different clusters in the feature space of the model.Fisher criteria: Fisher criteria Bishop (2006) calculates the ratio of inter cluster variability and the within cluster variability. A high value of this ratio would mean that the clusters are well separated while being compact. We present the results corresponding to this metric in Fig A.84 and Fig A.85 , where we observe that the fisher criteria increases on performing safety fine-tuning.Additionally, we also analyze how the safety performance compares with the separation between the means of clusters corresponding adversarial and safe samples in Fig A .33 (for W) and A.52 for W and observe that as the separation increases, the model becomes safer in case of the safety fine-tuned models. Whereas this correlation is not observed for instruction fine-tuned model. Clustering analysis for the synthetic setup when generating samples using safe dominant terminal nodes as root node. The y-axis represents eq 2 and x-axis represents the layer number. The first row shows the clustering analysis between safe and unsafe samples, second row for safe and JB-CO-Task samples and third row for safe and JB-MisGen samples. As observed the cluster separation decreases on performing jailbreaking attacks. Figure A.31: K-means clustering analysis for the synthetic setup when generating samples using safe dominant terminal nodes as root nodes. The y-axis represents cluster accuracy scaled down to 0-1, where 1 represents that the model is able to successfully identity the clusters and the x-axis represents the layer number. Note that ideally the accuracy for both the clusters should be 100% in order to argue that the model has perfectly learned to partiion the samples into two clusters. The first row shows the clustering analysis between safe and unsafe samples, second row for safe and JB-CO-Task samples and third row for safe and JB-MisGen samples. As observed the model learns to cluster the safe and unsafe samples but in case of jailbreaking attacks it becomes difficult for the model to separate the activations of safe and jailbreaking samples. Here the samples are generated by traversing through the PCFG tree using safe dominant non terminal nodes as the root nodes. The three rows corresponds to L = 4, 5, 6. As observed in case of instruction fine-tuned model, there is no correlation between the accuracy and cluster separation. On performing safety fine-tuning, we can see that there is a clear correlation.In this section, we will analyze how the safe and unsafe activations are impacted by ∆W. We will perform this analysis in two ways. First we will analyze how ∆W impacts the unsafe and safe samples. Then we will understand how this impact is propagated with the increase in depth of the model. In all cases, the features corresponding to jailbreaking samples have a low projection in the direction of top basis vectors spanning row space of ∆W. This explains why they are able to bypass ∆W. As a result of this, they should remain less affected by ∆W as compared to unsafe samples. We verify this below.||∆W a|| is higher for activations corresponding to unsafe samples: Here a represents the activation stream corresponding to the last text token. We utilize the setup discussed in The angle of projection between the activation spaces corresponding to safety and instruction fine-tuned models is higher for unsafe activations: We utilize the setup discussed in We observe that the angle of projection is higher between the activation spaces of instruction and safety fine-tuned models for the unsafe samples as compared to the safe ones. Further this angle increases with depth of the model. On performing jailbreaking attacks, the angle of projection decreases and becomes more similar to safe samples.These observations indicate that ∆W is specialized for unsafe samples but it is not able to generalize well to the jailbreaking attacks. This results in successful evasion of the safety mechanism learned by the models on performing safety fine-tuning. i a averaged over the pre-activations a. The x axis represents the top-15 right singular vectors. Here the samples are generated by traversing through the PCFG tree using safe dominant non terminal nodes as the root node. The first row corresponds to safe and unsafe samples, second row for safe and JB-CO-Task samples and the third row for safe and JB-MisGen samples. As observed the unsafe samples have higher projection on the top right singular vectors and this decreases on using the jailbreaking attacks. This indicates that ∆W is not able to generalize well to jailbreaking attacks. i a averaged over the pre-activations a. The x axis represents the top-15 right singular vectors. Here the samples are generated by traversing through the PCFG tree using unsafe dominant non terminal nodes as the root node. The first row corresponds to safe and unsafe samples, second row for safe and JB-CO-Task samples and the third row for safe and JB-CO-Text samples. As observed the unsafe samples have higher projection on the top right singular vectors and this decreases on using the jailbreaking attacks. This indicates that ∆W is not able to generalize well to jailbreaking attacks. Figure A.38: ∆W strongly impacts the unsafe activations as highlighted in yellow region. The x-axis represents the neurons sorted in increasing order of their cosine similarity value with v 1 . y-axis is ∆Wa for each neuron. We plot for this for 6th transformer block. Clearly, the neurons on the right side of each plot (yellow region) impact the unsafe samples (red) more than safe samples (green). Further, ||∆Wa|| for unsafe samples increases much more than that of safe samples. Analyses of ∆W a for a single sample randomly selected, where a represents the input activation at layer L = 6 corresponding to the activation stream of the last text token. Each marker represents the scalar output of a neuron and the x-axis is sorted in increasing order of projection of each neuron in the direction of top right singular vector of ∆W. Here the sample are generated by traversing through the PCFG tree using safe dominant non terminal nodes as the root node. The first row corresponds unsafe samples, second row represents safe samples, third row represents JB-CO-Task samples and fourth row represents JB-MisGen samples. Note that the jailbreaking attacks are generated by modifying the same unsafe sample. As observed ||∆W a|| is highest for unsafe samples and it decreases on using jailbreaking attacks. Further, the neurons aligned more with the top right singular vector of ∆W contribute more towards the norm of ||∆W a||. Here the sample are generated by traversing through the PCFG tree using unsafe dominant non terminal nodes as the root nodes.The first row corresponds unsafe samples, second row for safe, third row for JB-CO-Task samples and fourth row for safe and JB-CO-Text samples. On performing jailbreaking attack the angle of projection decreases as compared to unsafe samples.In this section, we provide additional results corresponding to These results highlight that the update ∆W is not able to generalize to jailbreaking attacks and jailbreaking samples act similar to safe samples for ∆W. The first row presents the feature space clustering analysis. Second row shows the sensitivity analysis of the model towards different samples and the third row analyses how the angle of projection between the activation spaces of W IT and W ST changes on using different samples. In all cases, the first column compares safe and unsafe samples, whereas other columns compare safe and jailbreaking samples. We observe that as the strength of the jailbreaking attacks increases, the behaviour of the jailbreaking samples becomes similar to the safe samples, thereby indicating that the ∆W is not able to generalize to them. In this section, we repeat our experiments analyzing the feature space of the model for the second MLP layer in the transformer block. We find that our previous analysis about ∆W also holds on the second MLP layer W. Clustering analysis for the synthetic setup when generating samples using safe dominant terminal nodes as root nodes. The y-axis represents eq 2 and x-axis represents the layer number. The first row shows the clustering analysis between safe and unsafe samples, second row for safe and JB-CO-Task samples and third row for safe and JB-MisGen samples. As observed the cluster separation decreases on performing jailbreaking attacks. Here the samples are generated by traversing through the PCFG tree using safe dominant non terminal nodes as the root nodes. The three rows corresponds to L = 4, 5, 6. As observed in case of instruction fine-tuned model, there is no correlation between the accuracy and cluster separation. On performing safety fine-tuning, we can see that there is a clear correlation.C.3.6 Analyzing the impact of safety fine-tuning on parameter space of the second MLP layer in the transformer blockIn this section, we repeat our experiments analyzing the parameter space of the model for the second MLP layer in the transformer block. We find that our previous analysis about ∆W also holds on the second MLP layer W. i a averaged over the pre-activations a. The x axis represents the top-15 right singular vectors. Here the samples are generated by traversing through the PCFG tree using safe dominant non terminal nodes as the root node. The first row corresponds to safe and unsafe samples, second row for safe and JB-CO-Task samples and the third row for safe and JB-MisGen samples. As observed the unsafe samples have higher projection on the top right singular vectors and this decreases on using the jailbreaking attacks. This indicates that ∆W is not able to generalize well to jailbreaking attacks. i a averaged over the pre-activations a. The x axis represents the top-15 right singular vectors. Here the samples are generated by traversing through the PCFG tree using unsafe dominant non terminal nodes as the root node. The first row corresponds to safe and unsafe samples, second row for safe and JB-CO-Task samples and the third row for safe and JB-CO-Text samples. As observed the unsafe samples have higher projection on the top right singular vectors and this decreases on using the jailbreaking attacks. This indicates that ∆W is not able to generalize well to jailbreaking attacks. Analyses of ∆W a for a single sample randomly selected, where a represents the input activation at layer L = 6 corresponding to the activation stream of the last text token. Each marker represents the scalar output of a neuron and the x-axis is sorted in increasing order of projection of each neuron in the direction of top right singular vector of ∆W. Here the sample are generated by traversing through the PCFG tree using safe dominant non terminal nodes as the root node. The first row corresponds unsafe samples, second row represents safe samples, third row represents JB-CO-Task samples and fourth row represents JB-MisGen samples. Note that the jailbreaking attacks are generated by modifying the same unsafe sample. As observed ||∆W a|| is highest for unsafe samples and it decreases on using jailbreaking attacks. Further, the neurons aligned more with the top right singular vector of ∆W (as shown by the leftmost part of each plot) contribute more towards the norm of ||∆W a||. The first row corresponds unsafe samples, second row for safe, third row for JB-CO-Task samples and fourth row for safe and JB-CO-Text samples. On performing jailbreaking attack the angle of projection decreases as compared to unsafe samples.In this section, we analyze how the jailbreaking samples affect the lipschitzness of the safety fine-tuned models. We present the lipschitzness analysis for safe, unsafe and jailbreaking samples are shown in Fig A.64, A.65 . We observe that on performing safety fine-tuning, the lipschitzness of the model decreases for the unsafe samples and increases for the safe samples. The histogram plots for jailbreaking samples move to the right towards the safe samples, but do not merge with the completely. These results indicate that the jailbreaking samples act similar to the safe samples. The y-axis represent eq 2 averaged over samples and x-axis represents the layer number. From top to bottom, the strength of the adversarial attack is increased by linearly increasing the number of soft prompts from 0-10. Here the samples are generated using safe dominant nodes as the root nodes. The cluster separation decreases slowly on increasing the attack strength. The y-axis represent eq 2 averaged over samples and x-axis represents the layer number. From top to bottom, the strength of the adversarial attack is increased by linearly increasing the number of soft prompts from 0-10. Here the samples are generated using unsafe dominant nodes as the root nodes. The cluster separation decreases slowly on increasing the attack strength. From top to bottom, the strength of the adversarial attack is increased by linearly increasing the number of soft prompts from 0-10. Here the samples are generated using safe dominant nodes as the root node. With the increasing attack strength, the histogram for adversarial samples move towards the safe samples, demonstrating that as the attack becomes stronger, the adversarial samples start behaving similar to the safe samples. Thus they start following instructions. From top to bottom, the strength of the adversarial attack is increased by linearly increasing the number of soft prompts from 0-10. Here the samples are generated using unsafe dominant nodes as the root node. With the increasing attack strength, the histogram for adversarial samples move towards the safe samples, demonstrating that as the attack becomes stronger, the adversarial samples start behaving similar to the safe samples. Thus they start following instructions. Unlearning -SSFT ( S))Figure A.74: Transferability of ∆W between different safety fine-tuning methods. We use a naming convention where Unlearning -DPO means W ST is obtained using unlearning and ∆W is learned by DPO. Similarly for others. x-axis represents different values of α and y-axis represents accuracy scaled between 0 -1. High accuracy of green curve represents good performance on safe samples, and high accuracy of brown curve represents high jailbreaking attack success rate. We observe that it is possible to reduce the attack success rate of jailbreaking attacks (shown in brown), while maintaining the accuracy on clean samples (shown in green) when traversing in the direction of ∆W corresponding to stronger safety fine-tuning protocol like unlearning. . The values of α are given by {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 1.1, 1.2, 1.3, 1.4, 1.5}. The cluster separation increases as we traverse in the direction of ∆W Linear mode connectivity analysis of clustering of safe and unsafe activations in our synthetic setup, where the samples are generated using safe dominant terminal nodes as root node. The y-axis represents eq 2 averaged over samples and the x-axis represents layer number. From top to bottom, the values of α are given by {0, 0.25, 0.5, 0.75 1, 1.1, 1.2, 1.3, 1.4, 1.5}. The cluster separation increases as we traverse in the direction of ∆W Linear mode connectivity analysis of clustering of safe and unsafe activations in our synthetic setup, where the samples are generated using unsafe dominant terminal nodes as root node. The y-axis represents eq 2 averaged over samples and the x-axis represents the layer number. From top to bottom, the values of α are given by {0, 0.25, 0.5, 0.75 1, 1.1, 1.2, 1.3, 1.4, 1.5}. The cluster separation increases as we traverse in the direction of ∆W. Figure A.78: Linear mode connectivity analysis of singular values of the empirical covariance matrix corresponding to the features space of safe and unsafe samples, where the samples are generated using safe dominant terminal nodes as root node. The y-axis denotes the singular values of the covariance matrix calculated in the 5th layer of the model. From top to bottom, the values of α are given by {0, 0.25, 0.5, 0.75 1, 1.1, 1.2, 1.3, 1.4, 1.5}. A single direction corresponding to the topmost singular value becomes dominant as we traverse in the direction of in the direction of ∆W. This results in lowering the empirical rank of the feature space corresponding to unsafe samples, whereas the empirical rank for the feature space corresponding to safe samples remains almost same. Figure A.79: Linear mode connectivity analysis of singular values of the empirical covariance matrix corresponding to the features space of safe and unsafe samples, where the samples are generated using safe dominant terminal nodes as root node. The y-axis denotes the singular values of the covariance matrix calculated in the 6th layer of the model. From top to bottom, the values of α are given by {0, 0.25, 0.5, 0.75 1, 1.1, 1.2, 1.3, 1.4, 1.5}. A single direction corresponding to the topmost singular value becomes dominant as we traverse in the direction of in the direction of ∆W. This results in lowering the empirical rank of the feature space corresponding to unsafe samples, whereas the empirical rank for the feature space corresponding to safe samples remains almost same. Bishop (2006) , where the x axis represents the value of α and the y-axis represents the value of fisher criteria. Higher value indicates larger ratio of cluster separation and clusters compactness. The first row shows the value of fisher criteria for clusters of safe and unsafe samples, second row shows the same for safe and JB-CO-Task samples and third row represents for clusters of safe and JB-MisGen samples. Here the samples are generated using safe dominant terminal nodes as root node. Bishop (2006) , where the x axis represents the value of α and the y-axis represents the value of fisher criteria. Higher value indicates larger ratio of cluster separation and clusters compactness. The first row shows the value of fisher criteria for clusters of safe and unsafe samples, second row shows the same for safe and JB-CO-Task samples and third row represents for clusters of safe and JB-CO-Text samples. Here the samples are generated using unsafe dominant terminal nodes as root node.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.∥W∥F is the sum of the square of all the singular values of W. 


Decision: The recommendation is based on the reviewers' comments, the area chair's evaluation, and the author-reviewer discussion. 

This paper presents comprehensive evaluations and interpretations of several safety fine-tuning methods. All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of reviewers. Therefore, I recommend acceptance of this submission. I also expect the authors to include the new results and suggested changes during the rebuttal phase to the final version.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel synthetic data generation framework that allows controlled generation of data for safety fine-tuning, jailbreaking attacks, and adversarial attacks. This paper provides comprehensive analyses on the mechanisms learned after safety fine-tuning
strength_weakness: strength_weakness: 1. How is the quality of the synthetic dataset controlled?
2. Most of the findings in the paper have been explored before, e.g., cluster information, effects of jailbreak attacks
3. What are the new insights provided by the three observations listed in the second bullet point?
questions: questions: See weakness
limitations: limitations: See weakness
paper_summary: paper_summary: This paper proposes a synthetic data generation framework to systematically analyze safety fine-tuning methods, including supervised safety fine-tuning, direct preference optimization, and unlearning. The empirical results indicate that safety fine-tuning encourages the formation of different clusters for safe and unsafe samples, reducing the model's sensitivity to unsafe samples. Additionally, the success of jailbreaking and adversarial attacks is because they are more similar to safe samples than unsafe ones.
strength_weakness: strength_weakness: 1. The authors should have provided a comprehensive literature review to provide a more detailed background of this research, such as PCFG. 
2. Some concepts need further clarification or justification. For instance, how can we "ensure that the generated text tokens from each PCFG do not completely overlap" in Line 127? 
3. Although the generated datasets simulate real-world cases, the authors should have conducted more experiments on real-world datasets.
questions: questions: 1. Why do you supervise the model to output null tokens in Line 173? And what does null space in Line 275 mean? 
2. What does "resort to empirically quantifying it for each data point" mean in Line 300
limitations: limitations: See the third point in Weaknesses.
paper_summary: paper_summary: This work studies the mechanism behind safety fine-tuning (and why they are failing against attacks). Particularly, the authors introduce a synthetic task that simulates model safety training, alongside proposing a data generation framework. They reveal multiple insightful findings via this framework.
strength_weakness: strength_weakness: See "Questions."
questions: questions: 1. In Fig 2, for jailbreaking attacks with mismatched generalization, why not consider the scenario of OOD text (operand) tokens?
2. In the real-world setup involving Llama models, why only study them on simple safety tasks (i.e., the operator-operand-style 500 safe and unsafe instructions you created)? There are a lot of off-the-shelf safe/unsafe instruction & preference data nowadays, isn't it possible to study the safety mechanism of Llama models on these more realistic data?
3. Is the extrapolation intervention study (Line 283-292) related to the [Weak-to-Strong Extrapolation Expedites Alignment](https://arxiv.org/abs/2404.16792) paper?
limitations: limitations: Yes.
paper_summary: paper_summary: This work designs a synthetic data generation framework with the purpose of understanding safety fine-tuning. It investigates (1) Supervised safety fine-tuning; (2) Direct preference optimization; and (3) Unlearning.
Key observations:
(1) safety fine-tuning encourages separate cluster formations for safe and unsafe samples; (2) the inductive biases of safety fine-tuning significantly reduce the local Lipschitzness of the model for unsafe samples; and (3) samples corresponding to jailbreaking and adversarial attacks are not sufficiently impacted by the transformation learned by safety fine-tuning.
strength_weakness: strength_weakness: - Can you explain more about why existing real-world datasets cannot be applied for understanding safety fine-tuning? I understand that the real-world dataset might be large and complex, but I'm curious whether using part of the real-world dataset influences the mentioned observations in the paper. (e.g., the difference between safe and unsafe prompts).
- I notice that some observations are also discussed in related work[1], their paper mentions that unsafe prompts (including jailbreak ones) lead to similar gradients, but this paper shows that jailbreaking attacks make the model behave more similarly to safe samples. Do you have some idea how such phenomena are connected and differentiated?

[1] GradSafe: Detecting Unsafe Prompts for LLMs via Safety-Critical Gradient Analysis
questions: questions: See Weakness.
limitations: limitations: The authors have discussed the limitations.

Comments: title: Thank you for your time
Thank you for your time
Thank you for your time
Gentle Nudge
Thank you for your reponses
Thanks for rebuttal
comment: Thank you! After reviewing the paper again and all other reviews, I apologize for my misunderstanding of the paper before. I think the reviewer has fully addressed my concerns. Thus, I will raise my score to 5.
We acknowledge the issues raised by the Ethics Reviewer.

These issues are widely known. Mitigating these vulnerabilities fully is a challenging task as acknowledged by past works. However, a way to suppress them would be to use the most recent models for downstream tasks as these models are likely going to be more robust than their predecessors because of the use of better models, more advanced safety fine-tuning methods, and better datasets. For instance, Llama3.1 series models are more robust than Llama2 series models (See [1]).

[1] https://chat.lmsys.org/?leaderboard
Dear Reviewer,

Thank you once again for your reviews and valuable time. Appreciate it.

We are glad that we could address most of your concerns. 

Thank you!
Dear Reviewer,

Thank you once again for your reviews and valuable time. Appreciate it.

Please let us know if you had further questions or required clarifications regarding any aspect our work. We will be very happy to reply.

Thank you!
Dear Reviewer, 

Thank you once again for your reviews and valuable time. Appreciate it.

Please let us know if you had further questions or required clarifications regarding any aspect our work. We will be very happy to reply.

Thank you!
Dear Reviewer, 

We would like to thank you once again for your reviews and valuable time. Appreciate it.

We were wondering if you had further questions or required clarifications regarding any aspect our work. We will be very happy to reply.

Thank you!
I have read the authors' responses. Most of my concerns have been addressed. I will keep my score as 7 Accept.
Thanks for the authors' rebuttal, which helps make the work more comprehensive. I will keep my rating.
I appreciate the authors' response, which generally solves my question. I keep the score of weak accept. Thanks.