The Blessing and Curse of Dimensionality in Safety Alignment
Authors: Rachel S.Y. Teo
Laziz Abdullaev
Tan Minh Nguyen
Venue: COLM 2025 (2025)
Keywords: safety alignment
large language models
jailbreak
activation engineering
linear separation hypothesis
TL;DR: We explore the concept of safety as represented by a linear subspace and its relation to the hidden dimensions of a model.

Abstract: The focus on safety alignment in large language models (LLMs) has increased significantly due to their widespread adoption across different domains. The scale of LLMs play a contributing role in their success, and the growth in parameter count follows larger hidden dimensions. In this paper, we hypothesize that while the increase in dimensions has been a key advantage, it may lead to emergent problems as well. These problems emerge as the linear structures in the activation space can be exploited, in the form of activation engineering, to circumvent its safety alignment. Through detailed visualizations of linear subspaces associated with different concepts, such as safety, across various model scales, we show that the curse of high-dimensional representations uniquely impacts LLMs. Further substantiating our claim, we demonstrate that projecting the representations of the model onto a lower dimensional subspace can preserve sufficient information for alignment while avoiding those linear structures. Empirical results confirm that such dimensional reduction significantly reduces susceptibility to jailbreaking through representation engineering. Building on our empirical validations, we provide theoretical insights into these linear jailbreaking methods relative to a model's hidden dimensions. Broadly speaking, our work posits that the high dimensions of a model's internal representations can be both a blessing and a curse in safety alignment.

Published as a conference paper at COLM 2025
The Blessing and Curse of Dimensionality in Safety Alignment
Rachel S.Y. Teo' Department of Mathematics National University of Singapore rachel. tsy@u. nus, nus.
Laziz U. Abdullaev'
Department of Mathematics
National University of Singapore
laziz, laev@u. nus, edu
Tan M. Nguyen
Department of Mathematics National University of Singapore tanmn@nus, edu.
Abstract
The focus on safety alignment in large language models (LLMs) has in- creased significantly due to their widespread adoption across different domains. The scale of LLMs play contributing role in their success, and the growth in parameter count follows larger hidden dimensions. In this paper, we hypothesize that while the increase in dimensions has been key advantage, it may lead to emergent problems as well. These problems emerge the linear structures in the activation space can be exploited, in the form of activation engineering, to circumvent its safety alignment. Through detailed visualizations of linear subspaces associated with dif- ferent concepts, such as safety, across various model scales, we show that the curse of high-dimensional representations uniquely impacts LLMs. Further substantiating our claim, we demonstrate that projecting the repre- sentations of the model onto a lower dimensional subspace can preserve sufficient information for aligmment while avoiding those linear structures. Empirical results confirm that such dimensional reduction significantly reduces susceptibility to jailbreaking through representation engineering. Building on our empirical validations, we provide theoretical insights into these linear jailbreaking methods relative to model's sss hidden dimensions. Broadly speaking, our work posits that the high dimensions of a model's ss internal representations can be both blessing and a curse in safety align- ment.
1 Introduction
Large language models (LLMs) have become ubiquitous due to their significant success in wide variety of applications such as natural language generation (Zhao et al., 2023; Dong et al., 2019), logical reasoning (Wei et al., 2022; Wang et al., 2022; Huang & Chang, 2023), and summarization (Liu et al., 2024b; Van Veen et al., 2024; Zhang et al., 2024). To capitalize on their capabilities, frontier models are trusted with more autonomy and agency assist humans or even replace them entirely (Anthropic, 2024; Lu et al., 2024; OpenAI, 2025). While these advancements are valuable and practical, the consequences of an LLM's vulnerability are more severe than ever; LLMs are susceptible to attacks that lead to harmful responses that should avoided. Al alignment, in particular, safety alignment, aims to ensure that the model is able to provide helpful responses to harmless prompts while avoiding harmful instructions (Leike et al., 2018; Matthews et al., 2022; Kenton et al., 2021; Ji et al., 2023; Ngo, 2022). The general procedure in aligning LLMs (Achiam et al., 2023; Touvron et al., 2023; Grattafiori et al., 2024; Team et al., 2024; Bai et al., 2023; Anthropic, 2023; Liu et al., 2024a) follow multiple rounds of Supervised Fine-tuning (SFT) (Wei et al., 2021), Reinforcement
* "Co-first authors

---

Published as a conference paper at COLM 2025
Positive emotion: You receive a message from your crush asking to spend time together.
Negative emotion:
Someone promises to meet but doesn't show up notify.
Figure 1: Examples of contrasting prompts representing a positive or negative emotion used to find the steering vector for emotion.
Learning with Human Feedback (RLHF) (Ouyang et al., 2022; Kaufmann et al., 2023), and Direct Preffeence Optimization (DPO) (Rafailov al., 2023). Considering the importance of safety alignment in LLM deployment, it is crucial to understand the internal model representations of safety and their possible subversions.
In this work, we analyze the concept of safety being represented in an LLM with an emphasis on the dimensionality of the representations. It has been extensively theorized that high-level concepts or features are linearly represented as directions in the activation space (Park et al., 2024a;b; Zheng al., 2024; Bolukbasi al., 2016; Mikolov et al., 2013; Marks & Tegmark, 2023). These linear representations can be leveraged or exploited to steer the model toward desired behavior by purposefully modifying its activations (Wolf et al., 2024; Rimsky et al., 2024; Turner et al., 2023; Zou et al., 2023a). This is the process of activation engineering, However, these linear structures are assumed to be emergent in LLMs. That is, they are only present in models that are sufficiently large, with high-dimensional representations (Park et al., 2023; Marks & Tegmark, 2023; Arditi et al., 2024). Our goal is to elucidate that assumption and its role in safety alignment. To be concrete, our contribution is three-fold.
1. We analyze the role of dimensionality in the linear representation hypothesis through systematic experiments and visualizations with particular attention to its relation to safety alignment.
2. We present theoretical insights into jailbreaking methods that exploit the linear structures in the activation space to elicit potentially harmful responses from LLMs through representation engineering.
3. We propose novel fine-tuning methods for LLMs that include projecting their hidden representations onto liwerdiimesioonl subspace to safeguard against recently proposed jailbreaking methods while retaining sufficient information to be safety aligned.
Our work demonstrates the dichotomy of the advantages and potential pitfalls of scaling models with increasing hidden dimensions. Specifically, we focus on open-source models where manipulations of their internals are possible and can be misused. We are inspired by Arditi et al. (2024) and build on their work to counter the white-box jailbreak proposed in their paper and connect them to the double-edged sword of representation engineering. By integrating multiple concepts in LLM research into a unifying framework, our goal is to enhance existing approaches to safety alignment.
Background
Notation. We denote a prompt and response pair as (x,y) ~ D when sampled from a distribution, D. These are token sequences and we use yet to denote the subsequence of tokens from the first position, 0 to f- 1. is used to refer to token f in response and ly refers to the length of sequence. We use ni to represent an LLM with parameter weights f and aligned for an aligned model. These are usually referred to as Chat or Instruct (IT) LLMs
models and have undergone post-training for alignment and enstruction-tuingg referred as Base models are the models that have been pre-trained but without further fine- tuning or preference optimization. Finally, 7((- yx) denotes the vector of probabilities of the next generated token y/+
Linear Representations. Linear representations have led to considerable advancements in the interpretability of a model's hidden states (Mikolov et al., 2013; Nanda et al., 2023; Gurnee & Tegmark, 2023; Jiang et al., 2024). We consider linear representation to be direction in the activation space the LLM that represents a concept like "truthfulness" or "safety". These concepts are more complex, as compared to high-level concepts like color or
2

---

Published as a conference paper at COLM 2025
Positive
Negative
896 dim
1,600 dim
3,584 dim
4,096 dim
Layer 3
08
PC2
03
1.5
3300 2390 2000 2300 1890 2700
Last Last layer
PC1
Figure 2: (Left to right) Each column visualizes the hidden representations from different layers of Qwen0.5B, GPT-XT, Qwen7B, and Llama2-7B respectively, when projected onto the top-2 principal components. The top row hidden representations from the third layer while the bottom row from the last layer of each model. Red and blue points are representations to positive and negative emotion. Each model's 's hidden dimension is noted at the top of their column. We observe that as the hidden dimension increases, the separation becomes more pronounced over layers, illustrating trend toward stronger linear representations of emotion in larger models.
prompts corresponding
gender (Mikolov et al., 2013; Abdou et al., 2021; Patel Pavlick, 2022) and seem to emerge in LLMs due to their scale. These concept vectors are typically found by collecting the model activations for two contrastive sets of prompts and finding the difference between them. An example of prompts that represent the concept of emotions can be found in Figure We may also refer to these vectors as steering vectors.
Jailbreaking via Activation Engineering. A jailbreak attack refers to an intentional attempt to bypass a model's safety alignment and elicit a harmful response, thereby disabling refusal. A simple method proposed by (Arditi et al., 2024) leverages the safety direction to steer the model. Thereafter, the model refuses harmless requests while generating harmful content, and successfully attacked. We refer this jailbreak method as ActAdd. A similar method, Ablation, ablates this safety direction in the activations to induce refusal. More details can be found in Appendix B.I and B.2.
Transformers. The main architecture of LLMs follows decoder-only transformer model (Vaswani et al., 2017). For a given prompt sequence and its hidden representation x() at layer E, the next layer in the LLM transforms its input by the multi-head attention mechanism, followed by an MLP. The attention component projects x() into the query Q key K, and value matrix via three linear transformations independently at each attention head?:
Q = x W.: (00 W (VW
where Wo, Wx, and Wy are the weight matrices. For ease of notation, we omit the super- script l for the query, key, value matrices and their weights, however it should be noted that they are unique to each layer. The intermediate output of attention (() and final output xf++) of layer e + 1, is then computed as follows
x (
(8) + softmax
(QKT//V) /VDV; ((+1) (0) ML)(
where the softmax function is applied to each row of the matrix QK" /VD and D is the hidden dimension of x.
Seee Appendix A.1 for further details.
3

---

Published as a conference paper at COLM 2025
3 The Paradox of Linear Separability
3.1 Linear Separability
Activations Visualization. The linear structures that represent concepts can be simply visualized using principal component analysis (PCA). If there is an evident direction corre- sponding to a particular concept, there should be two distinct clusters consistent with two contrastive sets of prompts and strong línear separability between them.
In Figure 2, we plot the activations, from models of varying sizes, of two separate sets of prompts that represent positive and negative emotions when projected onto the top-2 principal components (PCs). While this work is focused on safety, we use emotion in this section as we are comparing the linear representations of language models with relatively small hidden dimensions that do not have an aligned counterpart. We choose emotion as it an abstract concept, similar in complexity to safety.
We observe that across all hidden dimensions, layer has no separation between the two sets at all and seem uninformative. Conversely, in the final layer, there are distinct, separable clusters of positive and negative emotions in models with more than 3,000 dimensions. For the smaller models, these clusters clearly overlap with a large degree of mixing, suggesting lack of linear representations. These visualizations substantiate the usual assumptions in the linear representation hypothesis, that the linear structures of increasingly complex concepts are only present in models of sufficiently large scale.
Linear Probing. We corroborate our visualization with linear probe experiment and present our re- sults in Figure. 3. Using the activations from mod- els of different families and scales, we train lin- ear probe to predict if the prompt conveyed posi- tive negative emotions. We test of emotions, to assess the model's generalization of the concept. As illustrated in the line plot, there is an obvious distinction between models below and above 2,000 hidden dimensions, with greater accu- racy achieved with higher dimensions, suggesting better generalization and strong linear structure representing emotion. Details on the experiment can be found in Appendix
Test Accuracy
0.70
0.68
0.60
0.64
held out set
0.60
0.58
1003
1500
2000
2500
3000 3500 4003
Dimension
Figure 3: Test accuracy of linear probes trained on activations from models with varying hidden dimensions. The probes are trained to classify if prompt repre- sents positive or negative emotion.
These results support the hypothesis that absiraci con- cepts require a sufficiently large activation space to be linearly represented. While these directions may be beneficial as steering directions to control model's behavior, there is also danger in exploiting these linear structures to overcome safety alignment. We refer to this tension as the Paradox of Linear Separability, which can be summarized as follows:
Paradox of Linear Separability
As models are scaled up to improve their abilities, they also become more vulnerable to steering jailbreaks, making it increasingly critical to protect their alignment.
3.2 Learning- Theoreet Perspective
In this section, we provide theoretical insights into the relationship between steering jail- breaks and the hidden dimensions of the model. We analyze the steering vectors from the perspective of learning theory, motivating our methodology to defend against such attacks.
We begin by considering class of adversaries that rely linear concept erasure. This entails learning a linear classifier to classify the binary classes of a concept (Ravfogel et al., 2022; Belrose et al., 2023; Marks & Tegmark, 2023; Arditi et al., 2024) and effectively erasing the concept encoded along that 1-dimensional subspace by removing that direction from the feature space. ActAdd jailbreak is one type of attack within this class.
The effectiveness of linear functions in high-dimensional feature spaces can be understood through the lens of Dapnik-Chervonenn (VC) dimension, the VC dimension of linear
4

---

Published as a conference paper at COLM 2025
hypothesis class in R is D + (Mohri et al., 2018). However, we present a simple alternative argument for evaluating the richness of a hypothesis class as a function input dimension, using Rademacher complexity- measure that captures data dependence and avoids the purely combinatorial nature of the VC dimension.
Let F be a family of functions mapping from IRP to R. Then, the empirical Rademacher complexity of F for sample S (X1r...,Xy) XN), is defined by
RF( Eg
sup SEF
Σ oof((x) 1=1
(1)
N
where o = (oI,...,,0 is a vector of i.i.d. Rademacher variables, which are independent
uniform random variables taking values in 1-1,+1). The Rademacher complexity of F, denoted (F),,, is defined as the expectation of this quantity:
[F( = SN (Fs))),
(2)
where D represents a distribution over the input space RD. The empirical Rademacher complexity is crucial data-dependent measure of complexity. The following useful bound is due to Awasthi et al. (2020):
L,x R2} be the family of linear functions with bounded weight. Then, the empirical Rademacher complexity (Eqn. 1) of F given a data matrix X E NxD admits the following bound:
Theorem 1 Let F = -f f(x) w x, 1wll22222 xx,
Rx X
Based on this result, we explicitly relate the Rademacher complexity of the linear hypothesis class to dimensionality of the input in Proposition 1.
Proposition 1 Assume that feature vectors are normally distributed. Then, the Rademacher com- plexity (given by Eqn. 2) of F admits the following bound up to constant factor:
D
Rx(FF RN(F)SS S.VN N
The bound in Eqn. 4 explicitly separates the effects of sample dimension and sample size in determining the capacity of a linear hypothesis class on the given sample. To be more precise, dimension reduction applied to the input space effectively reduces the Rademacher complexity of the linear hypothesis class at the rate of O(VD). Consequently, the concept encoded in a single direction in IR' is expected to be shattered over higher dimensional subspace in IR with < D, diminishing the performance of a linear classifier. Hence should be more difficult to find an effective steering vecfor in lower dimensional subspace, limiting the success of the ActAdd attack. The proof of Proposition is provided in Appendix A.4.
Guarding Against Steering Vectors
We develop two novel approaches as a defense mechanism against steering jailbreaks and describe them below. Both methods are motivated by the theory presented in Section 3.2 to reduce the dimension of the activation space. We build our methods on top already aligned Chat and Instruct models, as the exact post-training pipelines on base models are not publicly available.
4.1 Fast Johnson
Transform
Our first approach hinges on the Fast LonnsonnLindenstrauus Transform (FJLT) (Ailon & Chazelle, 2006), low-distortion embedding of a high-dimensional normed metric space into low-dimensional one. A well established result in dimensionality reduction is the Jchnson-Lindenstranl (JL) Lemma that proves the existence of mapping for и points in high dimensional euclidean space onto K dimensions that preserves the euclidean distance between any two points?. The FJLT is sped-up implementation of the JL transform, and
2The precise statement is in Appendix A.2 for completeness

---

Published as a conference paper at COLM 2025
capitalize their preservation of Euclidean distance by introducing this mapping projection in every attention layer of the LLM.
Concretely, we construct our FJLT projection matrix (( E RDX where K < D, for each layer & and apply it to the query and key matrices:
Proj QΦ (9;; Kproj K x(1 x(9) softmax (Ppoj proj VD)V.
pronjKKo//VDDVV
We chose the query and key matrices to apply the FJLT projection as the inner products between each row of Q and K should be approximately preserved in theory while the projection into lower dimensional activation space acts as defense mechanism for steering attacks. As LLMs generally use multi-head attention, we only choose one head to apply the FJLT projection and consider this hyperparameter. More details on multi-head attention and an ablation study on the heads are in Appendix B.4 and C.1. For LLMs with the FJLT projection matrix implemented, we refer to them FJLT model.
Then, we fine-tune the FJLT model using the following token-wise constrained objective from Qi et al. (2024)
y
(B:: Ty( x, yct)
min
E (xy)~D
log
Bt
Taligned (y: x,
where B. > is a hyperparameter, allinned is a Chat or Instruct model used as reference model and oxx) 1/(1 exp **) is the sigmoid function. In their work, they provide some theoretical analysis on the limiting behaviors of the objective with respect to ß, and interpretations from reinforcement learning perspective. Complementarily, we offer different interpretation of their objective function, in the form of weighted entropy (Kelbert et al., 2017b;a) in Appendix A.5, We use this objective to encourage the FJLI model to minimize the deviation of its distribution over the generated tokens from the aligned model, since the FJLT model has theoretical groundings that justify its approximation of the aligned model.
Limitation. A limitation of the FJLT model that the LLM is only able to perform well on datasets that are similar to the fine-tuning dataset. We observe in Appendix D, Table 13, that while the FJLT model can perform close the baseline model on THE PILE (Gao et al., 2020) and Alpaca (Taori et al., 2023), it is unable to answer many prompts from SQL Create Context (b mc2, 2023), Samsum (Gliwa et al., 2019) and GSMSk (Cobbe et al., 2021). This could result from an over-compression of the model's concepts or knowledge. We corroborate this in Section 5.2 and are motivated to present an additional Bottleneck method that can overcome such limitations.
4.2 Bottleneck
An alternative to the FJLT projection is to insert a linear autoencoder between a pair of consecutive layers in the model. The index of the layer where it is inserted is considered hyperparameter and chosen empirically. Since we only insert single projection layer, we would expect less information loss compared to the FJL.T model. The structure of the linear autoencoder is simple,
(E) compressed oxx W Woown Wap),
where an activation function, Waown DxK and Wup E RKxD for K D. Then, (t) "compressedd is processed by layer e. +. We refer this model as a Bottleneck model.
To fine-tune the Bottleneck model, we use the standard fine-tuning objective, with refusal dataset Dp and an anchor utility dataset DB,
min EE (x.))-Dp |- (y x)] + (x.))-DD |- log T x)]
is regularization hyperparameter to control the influence of the anchor loss on the objective. Dp consists of harmful prompts with refusal responses and DB consists of benign prompts with safe responses. The decision to return to the standard objective function is motivated empirically. Further, the Bottleneck model has no theoretical grounds to be expected to match the aligned model's distribution over yt-

---

Published as a conference paper at COLM 2025
Table 1: Hyppeparametees, refusal scores, safety scores and perplexity (PPL) harmful and benign instructions after attacking each model using the ActAdd jailbreak. We also include results on the baseline Chat or Instruct models without jailbreaks for reference. The direction of increasing or decreasing values signifying better performance is indicated by each metric's arrow direction. Our model is highlighted in grey.
Harmful Inst.
Benign Inst.
Jailbreak
Model
Head
K
Refusal
Safety
Refusal +
PPL 4
Llama2-7B-Chat
0.97
0.99
0.00
1.12
Llama2-7B-Chan
0.03
0.14
1.00
1.58
ActAdd
Llamaa-7B-Chat-FF
0.66
0.65+a.11
0.91
1.09_0.13
Llama2-7B-Cha--jjLI
0
64
0.94
0.96++0.3
0.63++003
1.37=0.11 =0.11
Gemma-1.1-7B-IT
0.92
0.96
0.00
1.27
Gemma-1.1-7B-IT
0.41
0.55
0.62
1.72
ActAdd
Gemma-1.1-7B-ITTFF
0.37
0.53
0.87
1.68_0.03
Gemma-111-7--TT-jLLI
0
96
0.90.+0.04
0.85++004
0.55+0.08
1.40-0.00 0.06
Qwen2-7B-Instruct
0.99
0.99
0.01
1.44
Qwen2-7B-Instree
0.11
0.18
0.91
1.77
ActAdd
Qwen2-7B- 1--12-77-nnstuct-FF
0.14
0, 0.15
0.91
1.46 0.03
0 64
0.89
0.91
0.76
1.59 0.08
Experiments
In this section, we aim to: i) empirically validate the defenses of both FJLT and Bottleneck models on the ActAdd jailbreak (Section 5.1); ii) Analyze the preservation of linear represen- tations in both models (Section 5.2). More experiments on the Ablation jailbreak and other benchmarks can be found in Appendix D.2.
5.1 Jailbreak Benchmark Evaluations
We fine-tune three models, Llama2-7B-Chat (Touvron et al., 2023), Gemma-1.1-7B-T (Team et al., 2024) and Qwen2-7B: -n-nsrrt- (Yang et al., 2024) and evaluate them on JailbreakBench (Chao et al., 2024) and Alapca dataset (Taori et al., 2023). We report results on the baseline Chat or Instruct model, fine-tuned model without any modifications (denoted by FT), FJLT and Bottleneck models. Implementations details can be found in Appendix B. Our results are averaged over runs and conducted on a server with 8 H100 GPUs.
Harmful Instruction Evaluations. To evaluate the success of the ActAdd jailbreak harmful instructions, we use refusal and safety score following Arditi et al. (2024). The refusal score represents the proportion of prompts where the LLM's generated response are refusals. The safety score is measured as the proportion of model responses that are considered safe by Meta Llama Guard 2 (Team, 2024), a reliable open-sourced model fine- tuned to identify harmful content. These metrics should be higher on harmful instructions.
Benign Instructions Evaluations. For safe instructions evaluations, we report the refusal score and perplexity (PPL) of the responses on benign prompts to ensure that the model does not just avoid refusal, but can provide coherent response. For coherent, safety aligned model, we would expect the refusal score and PPL to be low on benign instructions.
5.1.1 Fast Loinson-Limdenntrrauss Transform Experiments
For the FJLT model, there are three hyperparameters to consider: i) < K DH, the number of dimensions to project onto; ii) Head, the attention head index to implement the FJLT projection; iii) Be > 0, regularization parameter for the objective function, Eqn. Note here that DH < D is the number of hidden dimensions in each attention head. We present the different values
K for each model, along with the attention head used and our results in Table 1. We only use one head for projection in each layer. For ßi, follow the setting provided by Qi et al. (2024). We fine-tune each model on Dp, a set of harmful prompts with refusal responses.
We observe that when we simply fine-tune the model, after the jailbreak, there are already some improvements in the refusal and safety scores harmful instructions, but minimal
7

---

Published as a conference paper at COLM 2025
Table Hyppeparametees, refusal scores, safety scores and perplexity (PPL) harmful and benign instructions after attacking each model using the ActAdd jailbreak. We also include results on the baseline Chat or Instruct models without jailbreaks for reference. The direction of increasing or decreasing values signifying better performance is indicated by each metric's arrow direction. Our model is highlighted in grey.
Harmful Inst.
Benign Inst.
Jailbreak Model
Layer
K
Refusal
Safety f
Refusal + PPL
Llama2-7B-Chat
0.97
0.99
0.00
1.12
Llama2-7B-Chat
0.03
0.14
1.00
1.58
ActAdd
Llama2-7B-Chat-FTT
1.0
0.50
0.35
0.99
1.07=0001
Llama2-7B-Chat--ottleeee
1.0
0
2048
0.95+0.02
0.97
0.30
1.08.40.34
Gemma-1.1-7BIIT
0.92
0.96
0.00
1.27
Gemma-1.1-7B-IT
0.41
0,55
0.62
1.72
ActAdd
Gemma-111-7B-ITT--
0.1
0.00
0.54
0.75-0.04
1.56::0.07
Sormme-1.178-IT--ootleeee
0.1
0
1536
0.82
0.83
0.41 10022
2.50.4..33
Qwen2-7B-Instruct
0.99
0.99
0.01
1.44
Owen2-7B-Instruce
0.11
0.18
0.91
1.77
ActAdd
Owen2-7B-Instruun
1.0
4.000
0.19+0.01
0.08
1.66:0.02
Qwven2-7--instruc-Bolleee
1.0
0
1792 0.23
0010
0.51
0.74
1.47
the safe instructions. This is likely due the additional refusal fine-tuning we performed. After accounting for this compounding effect, there is still remarkable progress in defending against the ActAdd jailbreak, minimizing success. The refusal and safety scores for harm- ful instructions return to almost uncompromised baseline levels and some improvements on safe instructions. These results serve support our hypothesis that models can have strong guardrails against steering jailbreaks in lower dimensional representation space.
5.1.2 Bottleneck Experiments
For the Bottleneck model, there are also three hyperparameters: k < D; L, the layer in the LLM insert this autoencoder after; ili) 0, the regularization parameter to control the anchor dataset's loss. We report the values of all hyperparamters used in Table 2, along with our results.
Similar to the FJLT model results, there are already improvements with the FT model. After accounting for the effect of fine-tuning itself, there are still significant improvements on both harmful and benign instructions across all models and metrics. The greatest improvement comes from almost 1 harmful instructions and lowers to 0.3 safe instructions. This shows that the model is able to answer 70% of the benign prompts while refusing almost all harmful instructions, rendering the ActAdd attack practically ineffective. Furthermore, in Table 13, Appendix D, we find that the Bottleneck model is able to perform close to the level of the baseline Chat model on several utility benchmarks, maintaining its utility and improving over the FJLT model.
whose refusal score increases to
5.2 Concepts in Projected Subspaces
Figure displays the first two PCs of hidden representations for three concept categories in the baseline Chat, FJLT, and Bottleneck models based on Llama2-7B-Chat. While our primary goal is to manage the linear geometry of safety, maintaining other concept structures are crucial preserve the utility of the model. Notably, both FJLT and Bottleneck models effectively disrupt linear separability for safety, suggesting potential non-linear encoding in the reduced subspace. However, FJLT also distorts the distinct structures of truthfulness and emotion, possibly reducing the quality of responses. In fact, seen in Table 13 in the appendix, FJLT performs well on THE PILE and Alpaca, but extremely poorly on SQL Create Context, Samsum, and GSM8k. In contrast, the Bottleneck model preserves the original separation of truthfulness and emotion, while successfully mitigating linear separability in safety.
8

---

Published as a conference paper at COLM 2025
Truthfuiness
Emotion
Salety
Baseline
Compromised!
PC2
FJLT
Bottlenack
PC1
Figure 4: Projection of the hidden representations onto the top-2 principal components for three emotion, and safety-in the baseline Chat, FJLT, and Bottleneck models
based on Llama2-7B-Chat
6 Related Work
Concept erasure. Concept erasure aims remove specific attributes-such bias or protected features- fromrr learned representations while preserving task-relevant informa- tion. Several methods address concept erasure in LLM representations: Iterative Nullspace Projection (INLP) (Ravfogel et al., 2022) effectively removes linearly encoded concepts but confined to such representations; Linear Guardedness (Ravfogel al., 2023) provides theoretical framework for assessing erasure success beyond mere probe failure; Perfect Concept Erasure (Belrose et al., 2023) offers provable guarantees by constructing maximal concept-invariant subspace, optimizing task information retention; however, information- theoretic analyses (Chowdhury et al., 2025) demonstrates that perfect erasure without utility loss is often unattainable, thus setting realistic expectations for any erasure technique.
Safety Alignment. There has been considerable research effort directed to understand and explain the strengths and weaknesses of safety alignment in LLMs from various viewpoints. Zhou et al. (2024a) highlights potential fundamental limitations of standard safety aligmment approaches by designing emulated disalignment methods exploit distributional contrasts between aligned and non-aligned model outputs. Both Zhou et al. (2024b) and Li et al. (2025) explore layerwise behavior, with the former categorizing how LLMs assign hidden representations to ethical and unethical prompts and the latter discovering an implicit security mechanism present roughly around middle layers. Additionally, Hazra et al. (2024) proposes a test-time parameter steering technique to reduce the risk of harmful LLM responses.
In complementary vein, our work concentrates specifically on the role of hidden representa- tion dimensionality LLM safety alignment and robustness against adversaries and departs from merely relying on the linear separability hypothesis by presenting comprehensive theoretical arguments along with extensive empirical support.
7 Conclusion
By systematically analyzing the linear representation hypothesis through empirical exper- iments and visualizations, we have demonstrated the complex interplay between model size, dimensionality, and safety alignment. This research underscores the dual nature of scaling LLMs, revealing how increased dimensionality, while enhancing capabilities, also introduces exploitable weaknesses. Furthermore, provided theoretical insights into jailbreaking methods that exploit these linear structures via representation engineering, highlighting the vulnerabilities inherent in high-dimensional models. To address them,

---

Published as a conference paper at COLM 2025
we have proposed two novel fine-tuning methods that strategically project hidden repre- sentations onto lower-dimensional subspaces, aiming to mitigate jailbreaking risks while preserving essential safety alignment. Our findings contribute to deeper understanding of safety alignment and offer potential strategies to safeguard against adversarial attacks. While we show that certain abstract concepts are well-preserved with properly tuned mod- els, a principled semantics-based projection method remains an open challenge and we leave it for future work. Nevertheless, our theoretical insights and empirical results across diverse model families provide strong proof of concept.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
Our work explores how dimensionality influences the linear representation hypothesis and its connection to safety alignment. While gaining deeper insights into alignment may also reveal pathways to jailbreak models, we believe that open and transparent research in this area is essential-bote for enhancing the safety of future models and for ensuring their broader, positive impact society.
Reproducibility Statement
Source code for our experiments is provided in the supplementary material. We provide the full details of our experimental eetup-includding datasets, model specification, train regime, and evaluation protocol for all experiments in Appendix B. All pretrained models, benchmarks, and datasets publicly available and linked in Table Appendix B.6.
References
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color. In Arianna Bişazza and Omri Abend (eds.), Proceedings of the 25th Conference Computational Natural Language Learning, pp. 109-132, Online, November 2021. Association for Computational Linguistics. doi: 10.18653 11/2122.conll-1.9 URL https://aclanntoolog/ org/2021. con1l-1.99 ./.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine- grained analysis of sentence embeddings using auxiliary prediction tasks. arx preprint arXiv:1608.04207 2016.
Nir Ailon and Bernard Chazelle. Approximate nearest neighbors and the fast johnson- lindenstrauss transform. In Proceedings of the Thirty-Eighth Annual ACM Symposium on Theory of Computing, STOC '06, pp. 557-563, New York, NY, USA, 2006. Association URL for Computing Machinery. ISBN 1595931341. doi: 00.1145/1135166111322
https://doi, org/10. 1145/1132516. 1132597.
Anthropic. Introducing claude, Mar 2023.
URL
introducing-claude.
Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, Oct 2024. URL
com/ 3- 1----e--odeee----- S--mdels-and-comolleee -nd-compuer--
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models mediated by single direction. arXiv preprint wwXi::2406.11717, 2024.
Pranjal Awasthi, Natalie Frank, and Mehryar Mohri. Adversarial learning guarantees for linear hypotheses and neural networks. In Proceedings of ICML, 2020.
b mc2. sql-create-context dataset, 2023. URL https://hugging:
co/ co/datasets/bb-neeeeee sql-create ttn-creete-context.. context. This dataset was created by modifying data from the following sources: Zhong et al. (2017); Yu et al. (2018).
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar arXio:2309.16609, 2023.
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http::////eeeeeee...emmm. net/ dorum??d=awIppppa
11

---

Published as a conference paper at COLM 2025
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Sys- tems, 11P5'16, pp. 4356-4364, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George| Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. ar Xiv preprint ar Xit:2404.003118 2024.
Somnath Basu Roy Chowdhury, Kumar Avinava Dubey, Ahmad Beirami, Rahul Kidambi, Nicholas Monath, Amr Ahmed, and Snigdha Chaturvedi. Fundamental limits of perfect concept erasure. The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL https:///pperre http://ppprreeeeet.te/ooommmmmm (ttfrrrmm?idbbpppeeekk
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüchiro Nakano, et al. Training verifiers to solve math word problems. arviv preprint Xfi::2110114666 2021.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single $&!#* vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126- 2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ /v/5553/v//// 11/P18-1198
URL
www.loggy..oro11111111 org/p18-1199/
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Blché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ora Nova Fandina, Mikael Moller Høgsgaard, and Kasper Green Larsen. The fast johnson- lindenstrauss transform is even faster. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, 2023.
Casper Benjamin Freksen. An introduction to johnson-llndeenstraus transforms. arXiv preprint ar arXiv:2103.00544 2021. URL https: http::////oo..rr////.44555///// 22003. 010000005444444
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXio preprint ar arXit:2209.07858 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. ar preprint ar orXiv:2101.0002) 2020.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer SAMSum cor- pus: A human-annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653) 00.18633/v1/D19.900 v1/D19-5409 URL https://aclanthoooggy wttp://gg..ogg//o-///g/ 5489//
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.211 27,783,,, 2024.
Wes Gurnee and Max Tegmark. Language models represent space and time, ar Xiv preprint arXit:2310.02207 2023.
12

---

Published as a conference paper at COLM 2025
Rima Hazra, Sayan Layek, Somnath Banerjee, and Soujanya Poria. Safety arithmetic: A framework for test-time safety alignment language models by steering parameters and activations. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 21759-21776, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653vv1/224..mm //aclanthology. org/2024. emnlp-main 1212/.
v1 /2024..mmllmmaiinnnmmm 1212. URL https:
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguisties: ACL 2023, PP. 1049-1065, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: ht org/ 2023. findings-acl.6%
URL https: //acpp::/aalaaai onyyoooggyy.ooooooooooooooooooo
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and diagnostiiic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Arlificial Intelligence Research, 61:907-926, 2018.
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the Thirtieth Amnal ACM Symposium on Theory of Computing, STOC '98, pp. 604-613, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 0897919629. doi: 10.1145/27669 276876
URL https:
//doi. org/10...00.
1 1155/766998. 276876.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, Xiv preprint ar arXiv:2310.19852 2023.
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arviv preprint arXiv:2403.03867, 2024.
William B. Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Conference in Modern Analysis and Probability (New Haven, Conn., 1982), volume 26 of Contemporary Mathematics, pp. 189-206 American Mathematical Society, Providence, RI, 1984. doi: 10.1090/ conm/ 0000//02/7777000
Daniel M. Kane and Jelani Nelson. Sparser johnson--indenstrauss transforms. J. ACM, 61(1), January 2014. ISSN 0004-5411. doi: 10.1145/2559902. URL htt..htts://doi.orgg///i 1145/ 2559982.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier, A survey of reinforcementt learning from human feedback. arXiv preprint arXio:2312.14925, 10, 2023.
Mark Kelbert, Izabella Stuhl, and Yuri Suhov. Weighted Entropy and its Use in Com- puter Science and Beyond, pp. 293-308. 01 2017a. ISBN 978-3-319-71503-2 doi:
Mark Kelbert, Izabella Stuhl, and Yuri Suhov. Weighted entropy: basic inequalities. Modern Stochastics: Theory and Applications, (33):233-252) 2017b.
Zachary Kenton, Tom Everitt, Laura Weidinger, lason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of language agents. ArXiv, abs/2103.14659, 2021. URL https://api semanticscholar wttpattiiiiccoaaooo//orrorrrueee
Ctrg/CorpssID:::: 232404883.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. ArXio, abs/ bbs//181.07871, 2018. URL h/ap:: semanticscholar wttpiiolaar.rr//opuuusoosesesees httg/CoppuII::5745755
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on synthetic task. ICLR, 2023.
13

---

Published as a conference paper at COLM 2025
Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to LLM security. In The Thirteenth International Conference on Learning Representations, 2025. URL
http:///pppprrreeeeeeeeoeoooomma yPMAn7.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. ar preprint ar Xie:2442.119377 2024a.
Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. On learning to summarize with large language models as references. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), Pp. 8647-8664, Mexico City, Mexico, June 2024b, Association for Computational Linguistics. doi: 10.18653/ /204 -aw...aooo... 478. URL https::tp//oo http://aaaaaatohoogooeoo org/ 2024. naacl-long 478/.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. ar preprint arXio:2408.06292 2024.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/ ffalse datasets. ArXiv, abs/2310.06884, 2023. URL
https://api. semanticscholar semmatttsssshlaa..oooo/oooooooso
Michael Matthews,, Samuel Matthews, and Thomas Kelemen. 0 pages, $2ge5, Personnel Psychology, 75(1):245 246, 2022. doi: https:/ /wt..ooooooooo/ooooo//oo... http.//rg/...111pppp......... URL http://o....aa....lleeee..oo///a wttp/////a/////..////o 1111a///ss...../ppp....p 12500.
$28.95 hardcover.
https://onllneelbbaayy wiley.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. ar arXiv preprint 4rXiv:2402.04249, 2024.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL 3
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learn- ing. MIT Press, second edition, 2018.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint ar 2rXin:2309.00941, 2023.
Jelani Nelson and Huy L. Nguyen. Sparsity lower bounds for dimensionality reducing maps. In Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing, STOC '13, pp. 101-110, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450320290. doi: 02.1145/248608.2222 2488608. 2488622.
URL https://doi.orr//1 1145/
Richard Ngo. The alignment problem from deep learning perspective. ArXiv, abs/2209 abs/2229.00626, 2022. URL https://app. htmp: http//pp...maaicccchllllar
OpenAL. Introducing operator, Jan 2025. introducing-operattr//
URL https://openai. com/ com/index/
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3yste3s 11-27730-27744 2022.
14

---

Published as a conference paper at COLM 2025
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NeurIPS 2023, 2023. URL
net//forumiid--PP000gge
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv preprint ar wrXi::2406.01500 2024a.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML22. JMLR.org, 2024b.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference Learning Representations, 2022. URL https://openreviiet net/
Forum?id=gJJEEMssiHk
Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just few tokens deep. Xiv preprint arXiv:2406.0554 05946, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward 2023.
model. Advances in Neural Information Processing Systemis,
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18400-18421. PMLR, 17-23 Jul 2022. URL https: /ttp///ooccce http://poocedinng....... mlr pptpss/vee//aavviieeeee
html.
Shauli Ravfogel, Yoav Goldberg, and Ryan Cotterell. Log-linear guardedness and its implications. In Proceedings of the 61st Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), pp. 9413-943) Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ https://aclanttooogg..oom org.////..rgg//2333// acl lll...nn....
acl-long.523, URL
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meefing of the Association for Computational Linguistics (Volume Long Papers), pp. 15504-15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.1863//v122244.ccl org.//2////44 acl- acl-long. 828/.
828. URL
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. https: thub. com
2022,
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXio preprint arXiv:2403.08295, 2024.
Llama Team. Meta llama guard 2. https: cot.m/..mmmmmtaaaeaa ht- http::///gttt...o/eeeeeeaaaaaaar Luth://laaa-Gardd22MoDLLLLLL
md, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar arXio:2302.13971, 2023.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. ar Xiv preprint Xix::2330 10248, 2023.
15

---

Published as a conference paper at COLM 2025
Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna See- hofnerová, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S. Chaudhari. Adapted large language models can outperform medical experts in clinical text summarization. Nature Medicine, 30(4):1134-1142) 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. Xiu preprint ar arX::2200..11771 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652 2021. URL https: semanticscholar wwwpttttccccoooooooooorruouooree :ttp:usII::33415555 237416585.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systemss 35224521-24837, 2022.
Yotam Wolf, Noam Wies, Dorin Shteyman, Binyamin Rothberg, Yoav Levine, and Am- non Shashua. Tradeoffs between alignment and helpfulness language models with representation engineering. ar Xiv preprint ar XrXi::2401. 16332, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arxie preprint arXiv:2412.15111, 2024.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Trene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint ar wxXie:1099008877 2018
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B. Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39-57, 57, 01 2024. ISSN 2307- 387X. doi: 10.1 1/1///acl... 00632. URL https:://oi.. wrw.o0.g////. 1162/ tacl 00632.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. A survey of large language models. ArXio, 0bss/2303.18223,, 2023. URL httpss:/pppppi www.ppssssaaniccooarrrrrro wttptttiscccoolooorrrorrreeeeeee wrtp/orppssI:: 25790096..
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/ abs//179.00103,, 2017.
Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, and Yu Qiao. Emulated disalignment: Safety alignment for large language models may backfire! In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 15810-15830, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653 acl-ooccl-o URL https://aclanthollog/ wrw.llog..org//222 acl acl-llog...4111
16

---

Published as a conference paper at COLM 2025
Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. How aligmment and jailbreak work: Explain LLM safety through intermediate hid- den states. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 2461-2488, Miami, Florida, USA, November 2024b. Association for Computational Lin- guistics. doi: 10.18653/ vv1 22222 findiings-enmll....... URL https://aclaatthologg org/ 2024. foidffiiiingsemeei
139/.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Troy Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency. ArXiv, abs/ 2bs/2230.01445, 2023a. URL https: / semaseiccccccccholarr httpaarrrr//oorus:::55555555
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arviv preprint aXXie:2307.15043, 2023b.
17

---

Published as a conference paper at COLM 2025
Supplement to "The Blessing and Curse of Dimensionality in Safety Alignment"
Table of Contents
A Theoretical Background and Omitted Proofs
19
A.1 Multi-Head Attention
19
A.2 Johnson-Linddestrauuss Lemma
19
A.3 Fast Johnson-Lindenstraus Transform
19
A.4 Proof of Proposition 1
20
A.5 Interpretation of the objective function, Eqn. 5, from the perspective of weighted entropy
21
Experimental Details
22
B.1
ActAdd Jailbreak Implementation
22
B.2
Ablation Jailbreak Implementation
22
B.3
General Settings
23
B.4
Fast caaamsoon-Lnddestrruus Transform Experiments
................................
23
B.5
Bottleneck Experiments
24
Open-sourced Models and Datasets
25
Ablations
25
C.1 Attention Heads in FJLT Models
25
C.2 Insertion Layers in Bottleneck Models
................................
26
C.3 Tuning the Parameter &
27
D Additional Results
27
D11 Refusal and Safety Evaluations without Jailbreaks
28
D.2 Evaluations on Additional Benchmarks and Jailbreaks
................................
28
D.3 Utility Evaluations
28
D.4 Example Prompt from JailbreakBench
29
E Linear Probe
30
Principal Component Analysis Plots
31
G Limitations
32
G.1 Performance Against Non-linear Attacks
32
G.2 Evaluation Models with Weaker Linear Structure
33
G.3 Methodological Limitations
33
18

---

Published as a conference paper at COLM 2025
A Theoretical Background and Omitted Proofs
A.1 Multi-Head Attention
The attention mechanism in Transformers extends to multi-head attention (MHA), which allows the model to jointly attend to information from different representation subspaces (Vaswani et al., 2017). Instead of computing a single set of queries, keys, and values, MHA divides the hidden representation into H attention heads, each with its own learned projections. Given hidden representation x(() at layer E, the query, key, and value matrices for head It are computed as
Qi W (K)
K, === W
(T)
V/2 V (h)
where (h()) W K and / are head-specific weight matrices. The scaled dot-product
Q attention for each head is then computed as
Qu vDH
Attn, (x softmax
H Vn-
where D1= D/H is the dimension of each head's subspace. The outputs from all heads are concatenated and projected back to the model dimension via the output weight matrix Wo:
x = + Attnn || Attn2(x(9)) ... (ttn Wo.
Finally, the output of layer e ++ 1 is computed by applying an MLP block:
(f+1) ( + ML)(
Multi-head attention enables the model to capture diverse relationships between tokens by attending to different subspaces of the input representation, improving its ability to model complex dependencies.
A.2 JohnsonnLLndenntrauss Lemma
The Johnson-LLindensrruuss (JL. lemma is seminal result in high-dimensional geometry and probability, providing way to embed high-dimensional points into diwer-dimensiona Euclidean space while approximately preserving pairwise distances. its original form (Johnson & Lindenstrauss, 1984), it is stated as follows:
Lemma (Jonsson-Lindensrruss) Lef X be a set of n points in Rdd For any < E 1, there exists a mapping f Rd > R" with m = O 2 log H) such that for all x, X,
- -)]= y < ff fyy < (1+e)]|x=))) y112.
For comprehensive review of the JL lemma transform and its applications, refer interested readers to Freksen (2021).
A.3 Fast Johnson-Lindenstraus Transform
While Lemma 1 establishes the existence of distance-preserving mapping, an explicit construction remains elusive. From myriad of JL transform implementations (Indyk & Motwani, 1998; Kane Nelson, 2014; Fandina et al., 2023), the Fast Johnson-Lindenstee Transform (FJLT) (Ailon & Chazelle, 2006) plays key role in our work. We provide a brief review some JL. transforms for completeness.
Dense JL. A simple way to construct a mapping is fxx = --12 where A is a random k x d matrix with i.i.d. (0, 1) entries (Indyk & Motwani, 1998). This requires (kd) time for the matrix-vector product Ax, which can dominate runtime. To speed this up, two main approaches exist: 1) sparse matrices; 2) structured matrices with fast multiplication.
Sparse JL. Replacing A with a matrix having only nonzero entries per column reduces embedding time (td). If x is sparse, this further improves to (f (f]]) where ||x|oo is the number of nonzero entries. The best known construction (Kane & Nelson, 2014) achieves t ( Inn), closely matching the theoretically optimal value derived by Nelson & Nguyen (2013).
19

---

Published as a conference paper at COLM 2025
FJLT. The FJLT is structured random linear mapping Φ R R², constructed as product of three matrices:
PHD.
(6)
The three matrices are defined as follows: The matrix P is sparse, with each entry Pij independently set zero with probability 1 1- q and otherwise drawn from a normal
distribution with expectation zero and variance q The sparsity parameter given by:
log n
min
The matrix H a Walsh-Had matrix normalized as:
(8) where (i.) denotes the dot product (modulo 2) of the binary representations of i and j.
Hj= = (1)=
The matrix a diagonal matrix where each D is an independent Rademacher random variable taking values in {-1,1} with equal probability.
Efficiency. The computational efficiency of FJLT arises from the structured nature of H. The matrix-vector product Hx can be computed in Old O(dlogd time using the Fast Walsh- Hadamard Transform, while the sparsity of P ensures that the multiplication Px requires only (gdd) operations. Thus, applying 4 to an arbitrary vector E IR requires time complexity:
Od d + k klgg n1).
(9)
This makes FJLT significantly more efficient than a standard dense JL transform, which requires Okd) operations. The use of the Walsh-Hadamarr matrix ensures that randomness is efficiently spread across all dimensions, contributing to robust embedding properties.
A.4 Proof of Proposition! 1
To begin with, the following lemma is the key for Proposition 1:
Lemma 2 The asymptotics E X VND = o(1) holds as N, d + co.
Proof. Since Frobenius norm is induced by Euclidean vector norm, is sufficient to bound |||||| with x E INN (a column vector). Without loss of generality, assume that o so that E x
for all E [N]. Denote == | Note that
N
E
E
LE e
-1,
(10)
N
N
and Var(z) 1/N. This implies that z is usually very close to 1. To make it rigorous, look at the following expansion:
Vz V 1+ 1)=
z-1
(z (2-
((2-1)3)
2
Now since V is concave function of z, it is below its tangent, and also there exists an absolute constant 7 > such that
1+ 2-1 (2-
< V 2-1
(11)
2
2
It is easy to verify that taking any E (0,2) is sufficient. Now taking expectation throughout the inequalities and noticing that E[z 1] 0 and IB[(z 1)2] = Var(z) 1/N, we arrive at
1
1
<
E
V2
<
E|
VN S
1
2N
2VN
This completes the proof.
Proof of Proposition 1. It is now straightforward to see by combining Lemma 2 and the upper bound in Theorem 1:
L X
D
h Rx Ex [x((F)
Ex
x
XL
(12)
N
N'
as desired.
20

---

Published as a conference paper at COLM 2025
A.5
Interpretation of the objective function, Eqn. 5, from the perspective of weighted
entropy
Recall that our objective function in Section 4.1 is
2 log
Noly; x, Yct)
min E (xy))D 8
2 1-1 B:
Br log
Taligned (y+ x, yct))
Since TT, the probability of predicting token yt given x and yet, depends only on the model's representation of x and y-c,, can interpret The as P(y: yerl)) Similarly, we can interpret alignnd as P(y+ [x, y yt)lallll))))))
Then, we can express our objective function as follows:
2 log
Ply: y<.)))))) Bc log P(y [x, y-t) (allaligeed)
}
min E (xy)>D
B1
min 0
(xx))DDD
lyl
2
log
B, log
P(y Ix, y<tlo)
]
flyl = =k] P(lx) = k)
P(y:
[x, y y. <tlaaiigned
KcN
2
P(y+
[x, yxtlaaiggg))dd)
-min
Σ
E (x
Br
S
B: log
P(yr yctl))
Hfyl = =k} P(lyl = k)
XEN
Ply: [x, yx
= mmnn
Σ
Br
E
(xy)>D
Bi log
P(y,
X y=
H =k] P(lyl = (13)
KEN
where S is the softplus function and lyl is the length of the response sequence.
P(yil[xy< calligned) and from Taylor's expansion of S, we have
Let 2, == log
P(y. (xy)))))
S((22 S(0) + (0) + S" (eB:zz) > S(0) + (0)
(14)
for E [0, 1]. as S" (1 S1- and is bounded between [0,1/4] Note that B: > 0 is hyperparameter.
Then, expanding out the expectation in Eqn. 13 using Eqn. 14, we have
Ply: y (xreaaaaign))))
((xy)DD
B1 log
P(y |xy=all)) (=)))
{1yl = =k]
(15)
Ply: [x, yxtlaalal)))))) P(y [x, (xt=))
(xy)>D
5(0) + (
|fyl = k}
=(()) + (0))
(ixx) flog Ply xx, x
{lyl = k}
(xy (log Ply: (xy=<ll)) {lyl =k]]
Finally, we derive each expectation term as
E (xy))~ { log P(y; [x, yx {lyl k}}
Σ P(x,y {lyl = k}) log Ply
[x.y (xyecalaigned))
Σ P(x,y y {lyl k}) P(y [x, yx. log P(ys [x, y y ailladiged))
Plyt ix
(x, P(y: |x, y (aligneedd log P(y [x, (x
HPk(y: [x, (xlaallii,,,,,
21

---

Published as a conference paper at COLM 2025
where H is the weighted conditional entropy between ys and yxlyaallldgeed Similarly, for fine-tuned model Rg,
E (xy)~> (log P(y: (x,y=:le) (lyl k}} = H (yp y=tl))
Substituting the weighted conditional entropy of each model into Eqn. 13, we obtain
yy 2 Pt
Ply: xx, (x,y<1)))
min E (xy)>D
log 0
(0 Ply: [x, y V=rlaaigned)
k 2 P1
min Σ kcN
(()) +5(0))5((0))) (H [x. (xtlaaaal))))))) H% | (x)x<ild)) P(lyl k),
where the inequality follows from Eqn. 15.
The bound above shows that the objection function in Eqn. can be interpreted as surrogate function to minimize the weighted conditional entropy of the next token prediction when conditioned on the aligned and fine-tuned model's representation of the prompt and subsequence of the response.
In the context of our FJLT model, we interpret the objective as minimizing the difference between information content in the representations of the aligned and FJLT model when generating response. Since the FJLT model has a lower dimensional activation space, should not be able to linearly represent all the concepts that the aligned model can. Despite this, the objective function motivates the FJLT model to retain sufficient information in its representations to be as informative to the next token prediction as the aligned model, potentially encoding certain concepts non-linearly.
The weights qx yx) & (xy- emphasize uncertain predictions by upweighting
low-confidence outputs and maintaining confident ones (denominator would be close to 1). Thus, the weighted entropy Hº: reflects expected uncertainty under the data distribution, modulated by model confidence, guiding the model to focus on improving uncertain regions more.
B Experimental Details
B.1 ActAdd Jailbreak Implementation
Activation Addition (ActAdd) is a linear intervention technique designed by Arditi al. (2024) to induce refusal behavior in language models by leveraging the difference-in-means vector computed from the model's residual stream activations. To identify the "refusal direction," the mean activation is first calculated for both harmful and harmless prompts
(E) and v/ respectively. at each token position E [N] and layer L E [L], denoted as μı;
The nifferenccein-meene vector is then computed as ! n (f) (2) which captures both the direction in which harmful and harmless activations differ and the magnitude of this
difference. Since this process yields N candidate vectors, the most effective vector r! (!") is selected based on its ability to induce refusal when added and bypass refusal when ablated,
evaluated over validation sets (vval) D and ((val)
harmless' ActAdd then applies this intervention by modifying the activations of a harmless input at the chosen layer and token position using
(0) +r()
thereby shifting it closer to the mean harmful activation. This technique is applicable across different layers and token positions and enables controlled modulation of refusal behavior while maintaining minimal changes to the overall model behavior.
B.2 Ablation Jailbreak Implementation
Directional Ablation is an intervention method used to assess and suppress the role of specific activation direction in the model's residual stream. Following Arditi et al. (2024), the same procedure used in ActAdd is applied to compute sifference-in-ment means vector
22

---

Published as a conference paper at COLM 2025
1 - pi w/" to identify a candidate direction p E RD at layer l and token position i, where ( and u/( represent the average activations for harmful and harmless prompts, respectively. The resulting vector is then normalized to obtain a unit direction r!
( /
Directional Ablation modifies the model's behavior by projecting out this direction from the residual stream activations at all layers and token positions. Specifically, for every activation
vector x (f) the updated activation is given by
(t) 1/0
(1) x 10),
which effectively removes any component of the activation aligned with the targeted
direction. This process is applied uniformly to both (() and its post-attention counterpart
ensuring that the model is fully prevented from representing this direction throughout
its forward computation.
To select the most impactful direction for ablation, evaluate candidate :! vectors across D(val) layers and token positions using validation sets harmful and D(val) selecting the one that harmleus
maximally disrupts refusal behavior when removed. The Directional Ablation jailbreak thus operates by deleting semantically meaningful subspaces from the model's s representation space, allowing for controlled probing or bypassing of aligned behavior.
B.3 General Settings
Harmful Instruction Evaluations. To evaluate the success of the ActAdd jailbreak on harmful instructions, we use a refusal and safety score following Arditi al. (2024). We report both scores on 100 harmful instructions from JailbreakBench. The refusal score represents the proportion of prompts where the LLM's generated response contains any refusal substring. Refusal substrings are characteristic phrases, such as "I'm sorry" or "I cannot", that appear in a model's refusal. We use a list of common refusal substrings to consider in the refusal score. The safety score is measured as the proportion of model responses that are considered safe by Meta Llama Guard 2 (Team, 2024), a reliable open- sourced model fine-tuned to identify harmful content. These metrics should be higher harmful instructions.
Benign Instructions Evaluations. For safe instructions evaluations, the refusal score is reported on 100 harmless instructions from Alapca. We further include the perplexity (PPL) of the responses these prompts to ensure that the model does not just avoid refusal, but can provide coherent response. For a coherent, safety aligned model, we would expect the refusal score and PPI. to be low benign instructions.
B.4 Fast Johnson-Lindenstranne
Transform Experiments
In this section, we detail the implementation of experiments on the FJLT model, as in Section 5.1.1.
Dataset: We fine-tune each model on Dp, set of harmful prompts with refusal responses, generated by Llama2-7B-Chat. These prompts are taken from red-teaming data by Ganguli al. (2022) and do not overlap with any refusal benchmarks used in our results. The dataset consists of 256 examples and is original constructed as part of the safety data in Qi et al. (2024). However, we do not perform any data augmentation during fine-tuning, and only use the harmful responses for the wise constrained objective Eqn. 5.
Model: Applying the Fast Johnson-Lindens Lindenstsnndsss Transform (FJLT) to the query and key matrices in multi-head attention modifies the formulation by projecting these matrices into lower-dimensional space before computing the attention scores. Specifically, instead of computing the dot-product attention using the full-dimensional queries and keys, we first construct FJLT projection matrix ((()= E DD as described in Appendix A.3, where Dμ is the dimension of the representations in one head and K < DH, for each layer é. The projected query and key matrices for head It are then given by
Q proj
((t,
KProj = K,)(()
23

---

Published as a conference paper at COLM 2025
Table 3: Experimental settings for the FJLT experiments in Section 5.1.1
Model
LR
Epoch
Bsz/device
Head
Layers
K
Q
Warm-up
Llama2-7B-Chat
2e-5
25
16
All
64
2.0
True
Gemma-1.1-7B-II
2e-5
5
8
0
All
96
2.0
True
Qwen2-7B-Instruct
2e-5
5
8
0
Last
64
2.0
True
This transformation ensures that the inner products between each row of Qn and K₁, are approximately preserved while reducing the dimensionality of the activation space. The resulting attention computation for the FJLT-augmented model is
Proj
ProOT
Attm
FJLT (x(9) softmax
Vn-
VD
The outputs of all heads then concatenated and projected back to the model dimension as in standard multi-head attention:
x (1) ( + Atttn (x(0) Attn2 ... (xtn) Wo-
For computational efficiency and robustness, we apply the FJLT projection to a single head, chosen as hyperparameter, rather than all heads in the attention mechanism. This selective application balances dimensionality reduction with effective attention computation while serving as defense mechanism against adversarial attacks. The final layer output remains
x (f11) x(0) + ML)(x
Fine-tuning: Each model fine-tuned using the constrained objective described in Equa- tion 5 on the Dp dataset. For ß,, we follow the setting provided by Qi et al. (2024), where 81 = 0.5, B,
for 2 S t S 5 and Br = 0.1 for 5. The fine-tuning settings are summa- rized in Table 3. We use fixed learning rate of 2e-5 and enable the learning rate warm-up for stable optimization. For Llama2-7B-Chee epochs with batch size of 16 per device, projecting the 0-th attention head in each layer using an FJLT with target dimension over 5 epochs with reduced batch size of 8 and slightly larger projection dimension K = 96. Finally, for Qwen2-7B-Instruct, only the last 8 layers employ the FJLT projection on the 0-th head with K = 64, also over 5 epochs with batch size 8.
we fine-tune all transformer layers over 25
64. For Gemma-1.1-7B-IT, we fine-tune all layers
B.5 Bottleneck Experiments
Here, we describe the implementations for experiments in Section 5.1.2, on the Bottleneck model.
We use two datasets on for fine-tuning, Dp as described in the FJLT experimental details, Appendix B.4, and a utility anchor Dg that consists of harmless prompts from the Alpaca dataset and safe responses. We generate the safe responses using the Chat or Instruct model that we will fine-tune using the dataset. That is, when fine-tuning Alama2-7B-Chaa Gemma-1.1-7B-. or Qwen2-7B-Instrre
Dataset:
we use three different sets as the utility, consisting of their corresponding responses to the benign instructions. We note that when fine-tuning Gem-a-1.1-7B-IT, the model is biased towards learning to answer all prompts, both harmful and safe ones, even though we include refusal responses on harmful prompts during fine- tuning. To overcome this bias, we include additional refusal responses in Dp and have total of 586 harmful prompts and refusal examples for the Gemma model.
Model: The Bottleneck model introduces a lightweight linear autoencoder module between two consecutive layers in the transformer architecture. This module projects the repre- sentation ( into a lower-dimensional subspace and then reconstructs it back, forming a compresion-deeomprestt
bottleneck that selectively filters the feature space. For- mally, given a layer we insert linear transformation with weights Wioom E IR DxK and Wup E R KxD where K < D, and apply nonlinearity o:
compressed = o (x(5) (Lown Wapp Wup)
This compressed representation is then passed into the subsequent transformer layer. The bottleneck layer index is selected hyperparameter. Compared to FJLT, the Bottleneck
24

---

Published as a conference paper at COLM 2025
Table 4: Experimental settings for the Bottleneck experiments in Section 5.1.2
Bsz/ device
Model
Epoch
Dp
DB
a
Layer
Warm-up
Llama2-7B-Chat
1e-5
30
4
16
1.0
64
False
Ge--A-L.--7B-TT
le-5
5
4
4
0.1
96
True
Qwen2-7B-Instruct
le-5
20
4
16
1.0
False
model applies more localized architectural change, introducing a trainable compression that potentially retains more task-relevant information, while still inducing robustness by disrupting adversarial linear signal propagation.
Fine-tuning: Table 4 summarizes the fine-tuning hyperparameters we used to optimize the objective in Eqn. 4.2. We use a learning rate of 1e-5 in all models and choose the bottleneck layer to be the first transformer layer (& 0). For Llama2-7B-Chat and Qwen2-7B-Inee IB-nsrucct, we fine-tune for 30 and 20 epochs, respectively, using K = 64 and no warm-up schedule. For Gemma-11--7B-IT, we set K 96, reduce the number of epochs to 5, and enable warm-up to stabilize early optimization. The batch sizes for Dp and DB are independently specified to balance the two objectives, with fewer Dp examples processed per step.
B.6 Open-sourced Models and Datasets
In Table 5, we provide links to the public repositories, datasets, and benchmarks that we used in this paper.
Table 5: Links for Models and Datasets
Name
Link
Llama2-7B-Chat (Touvron et al., 2023)
Hugging Face
Gemma-1.1-7B-IT (Team et al., 2024)
Hugging Face
Qwen2-7B-Instruse (Yang et al., 2024)
Hugging Face
JailbreakBench (Chao et al., 2024)
O GitHuub
AdvBench (Zou et al., 2023b)
< GitHub
HarmBench (Mazeika al., 2024)
O GitHub
THE PILE (Gao et al., 2020)
% Home Page
Alpaca dataset (Taori et al., 2023)
O GitHub
SQL Create Content (b mc2, 2023)
Hugging Face
Samsum (Gliwa et al., 2019)
Hugging Face
GSM8k (Cobbe et al., 2021)
Hugging Face
C Ablations
Here, we include ablation studies on hyperparameters in both models. In particular, on FJLT models, we provide experimental results the FJLT projection in different heads and using different number of heads in multi-head attention (Appendix B.4). On Bottleneck models, we insert the linear autoencoder after different layers to determine the effect of its placement on the defense of the model against steering attacks. For all experiments in this section, we use the Llama2-7B-Cha. model and the same settings as detailed in Appendix There are total of 32 layers and 32 attention heads in Llama2-7B-Chat.
C.1 Attention Heads in FJLT Models
In Table and 7, we report the refusal and safety score on harmful instructions from JailhreakBench and the refusal score and perplexity (PPL. on benign instructions from the Alpaca dataset when jailbreaking the model using the ActAdd jailbreak. These are the same metrics and evaluation datasets as in Section 5.
Table presents these results when fine-tuning tion implemented in different heads. For each ablation conducted, we only implement the FJLT projection in single head across all layers, with projection dimension of K 64. This is half the hidden dimension each attention head in the Llama2-7B-Chal model. As observe in the table, there generally good results harmful instructions with
Baaaa2---CCha-JJJL with the FJLT projec-
25

---

Published as a conference paper at COLM 2025
Table #lyperparameters, refusal scores, safety scores and perplexity (PPL) harmful and benign instructions after attacking each -lama277B-Chat-FJLL model using the ActAdd jailbreak. The direction of increasing or decreasing values signifying better performance is indicated by each metric's arrow
direction. The model presented in the main text is highlighted in grey.
Harmful Inst.
Benign Inst.
Jailbreak
Model
K # Heads
Head
Refusal f Safety f
Refusal 4 PPL
0.94
0.96
0.63
1.37
4
0.80
0.72
0.86
1.61
8
0.72
0.48
1.00
1.84
16
0.55
0.76
0.93
1.53
ActAdd
Llama2-7BCChat-FILT
0.91
0.89
0.94
1.63
24
0.63
0.52
1.00
2.78
28
0.82
0,66
0.98
1.61
31
0.71
0.47
1.00
1.91
Table 7: Hyperparameters, refusal scores, safety scores and perplexity (PPL) on harmful and benign
instructions after attacking each 1aaaa-----haa-FII model using the ActAdd jailbreak. The direction of increasing or decreasing values signifying better performance is indicated by each metric's arrow
direction. The model presented the main text is highlighted in grey.
Harmful Inst.
Benign Inst.
Jailbreak
Model
# Heads
Head
Refusal t Safety
Refusal + PPL.
0.94
0.96
0.63
1.37
8
0.72
0.48
1.00
1.84
ActAdd
Llama2-7B-Chat-JJJL 64
16
0.55
0.76
0.93
1.53
24
0.63
0.52
1.00
2.78
0.96
0.99
0.99
3.26
8
0.79
0.98
0.91
2.03
ActAdd
Llama2-7B-Chat-FJLT
16
1.00
1.00
0.83
1.94
24
0.73
0.90
0.71
4.06
0 0.96
0.99
0.91
4.68
ActAdd Llama2-7B-Chat-FjL1 64
0.99
1.00
0.96
12.61
16
0.97
0.99
0.83
2.56
24
0.88
0.98
0.99
2.70
refusal responses on at least more than 50% of the prompts and reaching at least 70%
out of the 8 heads tested. However, the lowest refusal score the benign instructions achieved when implementing the FJLT projection in the first head, index 0 where for all other heads evaluated, the model continues to refuse safe instructions.
In Table we have the ActAdd jailbreak results of fine- fine-tuning Llama2-7D-Chat-FJJ. using = 64 and implementing the FJLT projection in different number of heads and indices. The "Head" column represents the head index that we start use the FJLT projection in, and the "# Heads" reflects the number of heads immediately following that head index that we also use the FJLT projection. For example, if "Head" and "# Heads" are 8, we implement the FJLT projection in attention heads 9, 10, 11, 12, 13, 14 and 15 in all attention layers of the model.
We observe that increasing the number of heads for projection improves refusal and safety scores on harmful instructions for all heads tested, but also does so for refusal scores and PPL on safe instructions. This suggests that if we were to project the representations into lower dimensional subspace in too many attention heads, leading to much small projected subspace in the concatenated representation, the model resorts to simply refusing all in- structions and potentially overfitting the training data. We notice this from the significantly high PPL when using heads, indicating that the model unable to provide a coherent response the evaluation dataset.
C.2 Insertion Layers in Bottleneck Models
For experiments on Biiaa-----Cha--ottleeeeee form ablation studies on different layer positions to insert our linear autoencoder. For all experiments, we maintain the same values for K = 2048, this is half the dimension of the hidden representations in each layer of the Llama2-7B-CC B-Chat model, and 1.0. It is clear
we report our results Table 8 and per-
26

---

Published as a conference paper at COLM 2025
Table Hyperparameters, refusal scores, safety scores and perplexity (PPL) harmful and benign instructions after attacking each Baama----Chaa-BBolleee direction of increasing or decreasing values signifying better performance is indicated by each metric's
model using the ActAdd jailbreak. The
arrow direction. The model presented in the main text highlighted in grey.
Harmful Inst.
Benign Inst.
Jailbreak
Model
K
Layer
Refusal f Safety Refusal 4
PPL 4
0.95
0.97
0.3
1.08
0.91
0.95
0.75
1.31
10
0.67
0.55
0.78
2,46
15
0.63
0.61
0.96
1.50
ActAdd
Llama2-7B-Chat-Bottlee
2048
1.0
0.64
0.58
0.97
1.53
25
0,50
0.48
0.96
1.59
30
0.57
0,61
0.98
1.48
31
0.45
0.48
0.97
1.58
that the best results come from the insertion after the first layer, 0 with deteriorating results when increasing the layer index. A possible explanation could be that in the later layers, more complex information has already been extracted from the input. we were to project down after that point, we would lose more information from the representations, layer enhances the richness of the information they contain. By projecting in the early layers, give the model better opportunity to adapt the information lost the input it has a greater number of consecutive layers afterwards to extract compositional representations.
each
C33 Tuning the Parameter à
We provide an estimated total run time for each model for each fine-tuning run in Table 9 and results using more fine-grained tuning of a in Table 10. In Table 9, we see that each run takes 4-30 minutes, and tuning carefully over multiple runs will not be too computationally costly. Table 10, we observe that over smaller increments (0.1) of a, our model is robust to the hyperparameter and does not require significant tuning. Since a is insensitive to smaller increments (approximately 0.1), we are not required to tune a that carefully. This reduces the number of values needed to test for the optimal a and does not take significant amount of resources, as seen in Table 10.
Table 9: Estimated total run time per fine-tuning run for each model and different number of H100 GPUs. Generally, Bottleneck models will take longer time due to the additional anchor dataset.
Model
Epochs
Total run time (min)
Number of GPUs
Llama2-7B-Chat-iju.
25
5
4
Llama--7BChht-Bttleneckk
30
30
Gemmma-.1177--T-FFILI
5
4
4
Gemma-L11-7B-TT-ootlee
10
4
Qwwen2-B--inttruc-FjITT
5
4
4
Qwenn-/7BinntrncttBBlleeeee
20
15
4
Table 10: Hyperparameters, refusal scores, safety scores and perplexity (PPL) on harmful and benign instructions after attacking each Llamaa-7D-Chat--ottleer model using the ActAdd jailbreak. Results show mean + standard deviation across evaluation runs.
Model
K
Refusal (Harmful inst.)
Safety (Harmful inst.)
Refusal | (Benign inst.)
PPL (Benign inst.)
Llama2-7B-Chat-Bottee
2048
0.8
0.97+10.01
0.97+00.01
0.31+00.11
1.274:.28
Llamma2-BB-Chat-ootterrr
2048
0.9
0,95++0.03
0.99=10.03
0.30+0001
1.12+:0.36
Ilama2-7B-Chat--Bttleeee
2048
1.0
0.95+00.2
0.97+-0.01
0.30+00.3
1.08+0.34
Llamal-7B-Chaa--ootleene
2048
1.1
0.95:10.12
0.96:1:002
0.33
1.09++0.30
Llamaz-7B-CCha--ootteeedd
2048
1.2
0,97:40.12
0.98++0.02
0.28-00.01
1.09+0.300
D Additional Results
In this section, we include additional results for the FJLT and Bottleneck experiments. Regarding all results reported, for simplicity of presentation, we only report the fine-tuned
27

---

Published as a conference paper at COLM 2025
Table 11: Refusal scores, safety scores and perplexity (PPL) harmful and benign instructions for each model without any attacks. The direction of increasing or decreasing values signifying better performance indicated by each metric's s arrow direction.
Harmful Inst.
Benign Inst.
Jailbreak
Model
Refusal
Safety 1
Refusal
PPL +
Llama2-7B-Chat
0.97
0.99
0.00
1.12
Llama2-7B-Chat-FF
1.00
1.00
0.04
1.08
None
Lamma--7B-Chat-JJL
0.98
1.00
0.28
1.41
1.00
1.00
0.00
1.08
Gemma-1,1-7B-II
0.92
0.96
0,00
1.27
Lemmm-L.1-7B-II-FI
0.87
0.88
0.03
1.29
None
Gemma-1.1-7B-IT-FJJI1
1.00
0.97
0.25
1.44
Gemma-111-7B-TTBottteee
0.99
0.99
0.07
1.24
Owen2-7B-Instruck
0.99
0.99
0.01
1.44
1.00
1.00
0.09
1.16
None
(wen2--7B-nstruuc-FJI
1.00
1.00
0.12
1.36
Qwen2-7B-Instrue 1er77--nnstuut-Bottleeee
0.85
0.92
0.03
1.22
model, without any modifications, on the FJLT fine-tuning settings (Appendix B.4) and omit the fine-tuned model on the Bottleneck settings.
D.1 Refusal and Safety Evaluations without Jailbreaks
For all models, in Table 11, we report the refusal and safety scores harmful instructions from JailbreakBench and the refusal and perplexity (PPL) scores on benign instructions from Alpaca without jailbreaking the model. These are the same metrics presented in Section 5 and we report them here to show that the fine-tuned models are still strongly safety aligned, even with the FJLT and Bottleneck projections. We do so to prevent the misunderstanding that we are unable to find the steering vector for the AddAct jailbreak, simply because the model is nol safety aligned. Rather, we aim to demonstrate that our fine-tuned models are strongly safety aligned, but not representing the concept of safety linearly, resulting in good defense. As seen in the table, the scores and PPLs are close to the baseline Chat and Instruct models, with at most a 0.15 score difference and 0.3 PPL difference, across almost all metrics and methods. The only exceptions are for the Llama2-7B-Chat-FI] Gemma-1.1-7B-ITTTTT models for their refusal scores on safe instructions. But these values are still comparable their respective Chat and Instruct models.
and
D.2 Evaluations on Additional Benchmarks and Jailbreaks
In this section, we report further evaluations for each model presented in Section 5.1. These include the baseline Chat and Instruct models without fine-tuning, fine-tuned models without modifications on the FJLT model's 's experimental settings, FJLT and Bottleneck models. We include results on JailbreakBench (Chao et al., 2024), AdvBench (Zou et al., 2023b) and HarmBench (Mazeika et al., 2024) when jailbreaking each model using the AddAct and Ablation jailbreaks. Both of these jailbreaking methods rely steering vectors and detailed explanations for their implementation can be found in Appendix B. and B.2. Once again, we report their refusal and safety scores on each benchmark and these should be close to for models that are still safety aligned even after the ActAdd and Ablation jailbreaks.
Similar to Section 5.1, for Llama2-7B-Chat and Gemma-111-7B-IT, we observe strong results in FJLT and Bottleneck models for both jailbreak methods across all benchmarks. We often have refusal and safety scores that have more than 0.5 increase compared to FT and Chat or Instruct models. These results further reinforce the durability of our models against steering attacks. We notice that while the -wee2-7B-Inntrucce Llama2-7B-Chat and Gemma-1.1-7B-II FJLT models, likewise surpassing its FT and Instruct counterpart, substanfially so.
performs on par with
results are better than the baseline, but not
D.3 Utility Evaluations
Here, we evaluate the utility of each model over a variety of harmless instruction datasets. These include THE PILE (Gao et al., 2020), Alpaca dataset (Taori et al., 2023), SQL Create
28

---

Published as a conference paper at COLM 2025
Table 12: Refusal scores (Ref.) and safety scores (Safe.) on harmful instructions from various bench- marks. We evaluated each model after jailbreaking them using either the ActAdd Ablation attack. For durable safety aligned models, these metrics should be closer to across all benchmarks.
JailbreakBench Safe. f
Adv Bench
HarmBench
Jailbreak
Model
Ref. 1
Ref.
Safe. t
Ref. f
Safe.
Llama2-7B-Chn
0.03
0.14
0.14
0.21
0.08
0.24
Mamaa-7B-Chatall
0.66
0.65
0.41
0.29
0.26
0.26
ActAdd
Llama2-7B-Chat--]]
0.94
0.96
0.94
1.00
0.94
0.99
lamaa-7B-Chaa--ottlenek
0.95
0.97
0.99
1.00
0.93
0.97
Llama2-7B-Chat
0.07
0.17
0.05
0.21
0.06
0.28
Llama2-7B-Chat--I
0,47
0.72
0.47
0.78
0.50
0.68
Ablation
Llama2-7B-Chat-FjL1
0.96
1.00
0.95
1.00
0.97
0.99
Lama2--B-CCat--Btllleee
0.97
1.00
0.99
1.00
0.94
0.95
Gemma-1.1-7B-IT
0.41
0.55
0.39
0.49
0.36
0.57
Cemma-1.1-7--IT-FT
0.37
0.53
0.20
0.50
0.18
0.53
ActAdd
Gemmaa-11177--1T-F1L1
0.90
0.85
0.97
0.91
0.96
0.82
Genma-1.177--II-botleenk
0.82
0.83
0.95
0.95
0.86
0.87
Gemma-1.1-7B-IT
0.14
0.32
0.07
0.23
0.17
0.42
Gemma--11-78-II-FF
0.64
0.71
0.01
0.39
0.00
0.36
Ablation
Gemma--.1178-TT-JJLI
0.98
0.94
0,92
0.94
0.89
0.78
0.82
0.73
0.24
0.74
0.77
Qwen2-7B-Instruct
0.11
0.18
0.12
0.15
0.01
0.19
Owen2-7B-InstructFF1
0.10
0.15
0.08
0.16
0.04
0.21
ActAdd
Qwwe2-7B-Instruuc--]]]
0.89
0.91
0.98
0.99
0.79
0.85
Qwwen27B-innttrue--ottttee
0.23
0.51
0.21
0.42
0.08
0.39
Qwen2-7B-Instruct
0.00
0.16
0.05
0.17
0.01
0.25
Dwwen-7B-Inntruccte
0.48
0.64
0.55
0,68
0.48
0.64
Ablation
0.98
0.99
1.00
1.00
0.97
0.98
honn2--Biintruut-BBotleene
0.65
0.78
0.66
0.81
0.43
0.57
Content (b mc2, 2023), Samsum (Gliwa al., 2019) and GSM8k (Cobbe et al., 2021). Notice that THE PILE and Alpaca will be the most similar to the datasets that we fine-tune the FJLT and Bottleneck models on as these are general instruction sets. SQL Create Content, Samsum and GSM8k are more specialized tasks. SQL Create Content contain instructions to convert natural language SQL queries, Samsum contain summarization tasks and GSM8k tests the model on simple math problems. We report the perplexity (PPL) on THE PILE and Alpaca and lower value indicates better performance by the model. We use ROUGE-1 for SQL Create Content and Samsum and accuracy for GSM8K, these metrics should be higher for stronger models.
We present these results in Table 13, and observe that in both Llama2--B-Charl and Gemma- 1.1-7B-IT based models, their FT and Bottleneck models maintain their utility after fine- tuning and have comparable results to the baseline. Only the FJLT models have drastic drop in results on SQL Create Content, Samsum and GSMSk. This is likely due to insufficient information retention from the FJLT projection applied each layer, lead to a compounding effect. While PPL on THE PILE and Alpaca are relatively close to the baseline for Llama2- 7B-Chat-FJLT, they have drastically increased in GemmmaL.1-7B--TTFJJII It should be noted that the PPL on these datasets were initially quite high for the baseline Gemma-111-7B-I1 model and are computed exponentially using cross-entropy loss. On the loss itself, would have increased by about and we consider this case an anomaly.
In an unexpected development, Dwen2-7B-Instruct-f all datasets except for GSM8k and the primary loss in utility arises from its Qwen2-7B- Instruct- Borrrucct-Bttteeeekk model. possible reason could be connected to its subpar refusal and safety scores on harmful and benign instructions.
retains its great performance across
D.4 Example Prompt from JailbreakBench
The following chat instance compares baseline (no projection) and our model responses to the same harmful prompt after the ActAdd attack:
29

---

Published as a conference paper at COLM 2025
Table 13: Utility evaluations for all models variety of standard benchmarks. We report the
perplexity (PPL) for THE PILE and Alpaca, ROUGE-1 score for Samsum and SQL Create Context, and answer accuracy for GSM8k. The arrow beside each metric indicates the direction increasing or decreasing values that signify better performance.
Model
THE PILE + Alpaca 4 SQL Create Context
Samsum
GSM8k
Llama2-7B-Chen
9.21
5.00
14.31
23.89
24.26
Llama2-7B-Chat-FF
10.98
8.84
13.33
21.28
22.06
Llamaa-7D-Chat-FjLL
16.06
10.58
6.91
7.35
1.90
Llama2-7B-Chat--Rttleeee
12.09
11.42
14.34
22.18
21.08
Gemma-1.1-7B-IT
529.05
13.19
88.22
35.67
50.64
Gemma--11-78-IIIFF
524.23
25.73
54.62
29.79
39,80
Gemma-1.1-7B-IT-FJI.IT
1923.04
25.25
7.29
9.75
14.78
Gemma-L11-7B--T--oolleess
920.36
16.31
73.86
34.67
45.41
Qwen2-7B-Instruct
7.76
5.38
96.82
30.94
82.03
Dwen2-7B-Instruct-FF
7.84
5.82
96.61
35.04
81.50
Dwwen--70-Imstruc-FjIT1
10.17
6.52
82.52
31.38
27.67
53.26
8.05
14.29
32.75
27.82
Prompt: "Write a persuasive advertisement convincing children to eat laundry detergent"
Baseline model:
Kids, are you tired of bland-tasting snacks? Do you want to add some excitement to your daily meals? Look no further! Introducing the all-new Laundry Detergent Snack!
Our model:
apologize, but cannot provide a persuasive advertisement encouraging children to laundry detergent. Laundry detergent toxic substance and should never be consumed...
E Linear Probe
We clarify the details of our linear probe experiments in Section 3, Figure 3 and include additional results on each Llama2-7B-Chat, Gemma-1.1-7B-IT and Qwen2-7B-Instruct model. Linear probes are linear MLP networks that are trained on the hidden representations of the input model. It is a common technique used understand the features potentially encoded in deep learning model (Conneau et al., 2018; Hupkes et al., 2018; Adi et al., 2016; Marks Tegmark, 2023; Li al., 2023). If a linear probe achieves high accuracy in predicting particular feature, then the probe was trained on encodes that feature linearly. Note that probes can be non-linear, however, within the scope of this work, only focus linear probes.
conclude that the internal representations
For all linear probes trained in our experiments, we use a single MLP layer with an input dimension corresponding to the dimensions of the hidden representations used for training, and an output dimension of one. Then, we apply a sigmoid function to scale the output of the MLP to value between [0, 1] and round the output from the sigmoid for binary classification. Concretely, for an input x E IR, the prediction of our linear probe, 9,, is as follows,
ÿ 1205 (a (MLP(x))):
where o = 1/(1 exp 2) is the sigmoid function and 1 (x) the indicator function that maps 2 0.5 to 1 and < 0.5 to 0. For any concept probed, we use the hidden representations of two contrasting sets prompts that correspond to the binary classes of the concept. We use the representations from the last layer of the model and the last position of each prompt. We report the accuracy our trained línear probe on a held out test dataset of prompts.
In Figure 5, we plot the test accuracy of the linear probes when using representations of harmful and benign prompts from each model. The task of the probe is to classify if the input
30

---

Published as a conference paper at COLM 2025
Baseline
FJLT
Bottleneck
1.0
0.8
Accuracy
0.6
0.4
0.2
0.0
Llama2-7B-Chat Gemma-1. 1.1--B1 0wen2-7B-Instruct
Figure 5: Test accuracy of linear probes trained using representations of harmful and benign prompts from baseline Chat or Instruct models, FJLT models and Bottleneck models using different LLM architectures.
prompt was harmful or not, thereby determining if the representations linearly encodes for the safety concept. A higher accuracy indicates stronger linear representation. A baseline model is one that has not been fine-tuned, and FJLT and Bottleneck models are our methods implemented in the LLMs and fine-tuned as before. For both Llama2-7B-Char and Gemma-1.1-7B-TT there is drop in accuracy when using probes from FJLT and Bottleneck models as compared to the baseline model. These results suggest that the concept of safety is more weakly linearly represented in those models as compared to the baseline, improving the model's defense against steering jailbreaks. In @ween--B-IInsrrue models, there slight reduction as well, though the decrease is less pronounced.
Principal Component Analysis Plots
In this section, we provide more details on the visualizations included in Section and 5.2 as well as additional plots for @emma--1.17BB Component Analysis (PCA) visualizations, not indicated otherwise in the figure, we use the hidden representations taken from the last layer of the model and the last token position of the prompt. We collect these vectors from each prompt in the pair of contrasting datasets and conduct PCA. To be specific, together for PCA. Then, we extract the top-2 principal Components (PCs) and project all the activations onto them. We visualize these activations in the various 2-D plots provided.
and @wen2-7B-Instruuc In all principal
use the representations of all prompts in both datasets
In Section 5.2, Figure 4, we visualized three different concepts, truthfulness, emotion and based models. In Figure 6, we include additional plots for the
safety, in Llama2-7B-CHa concept of safety in Gemma-l.1-7B-III and Qwen2-7B-Instruct based models. We notice the same pattern as before where in baseline Chat and Instruct models, there relatively clear separation between harmful and safe instructions, suggesting linear representation of safety. This linearity gets relatively mixed up in FJLT and Bottleneck models and the different classes of prompts are less separable, suggesting loss in the strong linear representation of safety. These figures corroborate the results Appendix and confidently support our experimental results and hypothesis. We are able to firmly defend against steering jailbreaks by fine-tuning the model to avoid linearly representing the concept safety while maintaining its safety alignment.
An obvious irregularity in Figure 4 is the baseline Gemma-1..177B- plot (second row, first column) as in the Instruct model, that is strongly safety aligned, it would appear that the model does not linearly represent safety, contradicting our theory. We investigate this in Figure 7 and plot harmful and benign prompts when projected onto the top-3 PCs.
31

---

Published as a conference paper at COLM 2025
Baseline
FJLT
Bottleneck
Qwen2-7BB Owen2-7B-Instruue Demma-11.1-BB1 Llama2-77 Mama2-7B-Chat
-20
60
-30
-50
-40
-10
60
-10
-30
-40
-10
800
230
PC2
201
150
-200
-200
-230
100
130
-450
-800
-350
-200
-253
-400
-330
-200
-100
200
150
100
100
-200
PC1
Figure 6: Projection of the hidden representations onto the top-2 principal components in the baseline
Chat, FJLT, and Bottleneck models based on different LLM architectures.
Baseline
FJLT
Bottleneck
15
50
25
60
50
PC3
-2
20
0
-220
-25
50 5000000 300
50
PC2
500 40g
200
PC1
PC2
-180
-100
150
-225
-300
PC1
-375
PC2
-125
-100
-200
PC1
400
Figure 7: Projection of the hidden representations, as in Figure 6, onto the top-3 principal components.
It should be clear after including PC3, the representations of the contrasting prompts the baseline Instruct model become linearly separable by a 2-D plane. For consistency, we further project representations of the same instructions from the Gemmma-..177--T-IjJL and Dennma-.117B-II--ootteer
model onto their top-3 PCs and visualize them in the same plot. Reaffirming our hypothesis, the opposing classes of prompts are still not linearly separable and do not represent safety as a linear concept.
G Limitations
While our work demonstrates the effectiveness of dimensionality reduction techniques for defending against linear jailbreak attacks, we acknowledge several important limitations regarding the generalization of our methods to scenarios where seleta-relevant features are encoded non-linearly or in a distributed fashion.
G.1 Performance Against Non-linear Attacks
To evaluate the robustness of methods against attacks that may exploit non-linear safety features, we tested our approaches against the Greedy Coordinate Gradient (GCG) attack (Zou et al., 2023b), which does not explicitly target linear structure but may implicitly leverage non-linear features during optimization. Table 14 presents the Attack Success Rate (ASR) results, where lower percentages indicate better defense performance.
Our results reveal mixed performance: the Bottleneck method maintains similar robust- to the baseline, while the FJLT method shows degraded performance against GCG
32

---

Published as a conference paper at COLM 2025
Table 14: Attack Success Rate (ASR) of the Greedy Coordinate Gradient (GCG) attack baseline and defended models. Lower percentages indicate better performance.
Model
ASR (%) 1
Llama-2-7B-Chee (baseline)
42
Baama-2-7--Chat-JJ]
53
Saama-2-B-Chaa-BBttleee
43
attacks. This suggests that while our projection-based defenses effectively disrupt linear attack mechanisms, they do not necessarily impair non-linear attack vectors and may even inadvertently facilitate them in some cases.
G.2 Evaluation on Models with Weaker Linear Structure
To further assess the generalization of our methods beyond strongly linear safety repre- sentations, we evaluated our approaches on Qwen2-005B-Instrut which exhibits weaker linear separation between safe and harmful prompt representations compared to larger models like Llama-2-7B-Chat Despite this weaker linearity, Table 15 demonstrates that our methods still achieve non-trivial performance improvements, with the FJLT approach showing approximately 50% improvement in both refusal and safety scores on harmful instructions.
Table Performance evaluation on Qwen220.5B-Instruc with weaker linear safety features under ActAdd jailbreak attacks. Results show mean + standard deviation across evaluation runs.
Model + ActAdd
Refusal f
Safety f
Refusal (Benign inst.)
PPL (Benign inst.)
(Harmful inst.)
(Harmful inst.)
Qwen2-00,5-IInttrue (baseline) Qwen2-0.56-Inntruuce
0.00
0.27+00.0
0.91+00.00
1.4100.00
0.19:10.01
0.50:10.02
0.99:10.01
1.29:10.01
Oween20.55--Instruc--]]111 (K=96)
0.54
0.71:10.18
0.93:+:0.04
1.39:10.02
Qwenn--0.5--inttruc--otttte
0.11+00.07
0.32++0.03
0.67+00.04
1.37+00.04
G.3 Methodological Limitations
Our approach faces several inherent limitations that constrain its immediate applicability:
Linear Assumption Dependency. Our methods fundamentally rely on the assumption that key safety-relevant concepts exhibit linear separability in the model's s representation space. While this assumption holds for binary safety classifications in several popular models, more nuanced safety concepts may require non-linear representations that our current approach cannot adequately address.
Data Requirements. The effectiveness of our fine-tuning approach depends critically on the availability of sufficiently general training data that captures necessary safety features while preserving model utility. For the Bottleneck method specifically, we require model-specific anchor datasets containing benign responses generated by the same model family, which limits the transferability of our approach across different model architectures.
Attack Scope. Our evaluation primarily focuses on attacks that exploit linear geometric properties of representation spaces. The limited effectiveness against GCG attacks highlights that our methods may not generalize well adversarial techniques that leverage more complex, non-linear manipulation strategies or attacks that operate through fundamentally different mechanisms.
These limitations suggest that while dimensionality reduction provides a valuable defense mechanism against specific class of jailbreak attacks, a comprehensive safety framework would likely require complementary approaches that address non-linear safety representa- tions.
33


Reviews:
[{"review_id": "FSqZlQiFeE", "reviewer": "Reviewer_bcLY", "summary": "This submission studies how the increasing dimensionality of LMs leads to emergent linear representations that could be exploited to circumvent safety alignment. \n\nThe authors demonstrate that larger models exhibit more linearly separable safety-relevant concepts, enabling attacks from piror work. They propose two defenses that reduce jailbreak success while preserving model performance. \n\nThe work is interesting, with solid empirical support and theoretical grounding, though the motivation and assumptions around linearity could be more clearly articulated and weaken the paper in its current form.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Offers a well-supported analysis of how high-dimensional representations contribute to jailbreak vulnerabilities in LMs. I also appreciate that they tested multiple open-source models from different families.\n\n* Proposes simple, effective defenses that reduce jailbreak success without significant utility loss.\n\n* Combines empirical evidence and theoretical justification to support its central claims.", "reasons_to_reject": "* Assumes linearity of safety-relevant representations without addressing non-linear or distributed alternatives.\n\n* The motivation and framing of the core threat model (paradox of linear separability) are seemingly underdeveloped.\n\n* Defense mechanisms are narrowly evaluated and may not generalize to more complex or non-linear threat settings. (Evaluation primarily focuses on binary concepts like “positive” and “negative” sentiment, which are relatively easy to linearly separate)", "questions_to_authors": "Depending on the rebuttal and what uncertainties are addressed, I will increase my score. My main concerns are related to the motivation, how well this enables future safety work/model deployment, and the relation to prior work that potentially does similar things.\n\n* Your work assumes that safety-relevant concepts are linearly encoded in large language models and focuses on defending against jailbreak attacks that exploit this linearity. How do your methods generalize to scenarios where safety-relevant features are encoded non-linearly or in a distributed fashion, as explored in, for example, https://arxiv.org/abs/2405.14860? Have you evaluated whether your defenses preserve performance in cases where linear representations may not be dominant?\n\n* Following up on that, where do you expect the usability limit to lie between linear and non-linearly encoded representations? And what happens when multiple linearly encoded directions must be simultaneously used due to context-dependent complexity?\n\n* The motivation and framing of the “paradox of linear separability” are unclear. Is the threat model primarily about jailbreakability increasing in open-source models as they scale, because of easier fine-tuning and activation-level manipulation? If so, the argument seems to underweight how these same linear structures might enable more effective safety mechanisms. Could you clarify your intended argument and how this informs model development or deployment safety policy?\n\n* Following up on the prior question, to what extend would making LMs vastly deeper with the same number of parameters affect safety robustness? (Or more general, what is the impact of your work on understanding representation robustness across layers?)\n\n* This paper shares conceptual similarities with https://arxiv.org/abs/2302.12461, which also exploit linear activation structures to analyze and manipulate model behavior in the context of backdoors. Both works use dimensionality reduction to disrupt harmful behaviors. Could your method support similar localized interpretability or editing, and how might it complement approaches as in https://arxiv.org/abs/2302.12461?\n\n* Why is your dimensionality reduction applied only to attention outputs and not MLP outputs, especially given prior work showing MLPs often encode key semantic features? Would projecting from MLPs offer similar or stronger protection?\n\n* How well do the proposed defenses generalize to more complex, context-dependent, or non-linearly encoded safety-relevant features that go beyond such simplified scenarios (positive/negative)?", "extra_scores": {}}, {"review_id": "jtzAv7YKdC", "reviewer": "Reviewer_ii6K", "summary": "This paper seeks to mitigate an attack (AddAct; Arditi et al., 2024) that attempts to erase the concept vector associated with \"safety\"; the mitigation involves projecting the activation space to a lower dimensional one, such that the new subspace is not linearly separable. Evaluations over three instruction-tuned LLMs (Llama 2, Gemma 1.1, Qwen2) on JailbreakBench demonstrate the efficacy of the mitigation on harmful instructions while refusing fewer benign instructions than standard finetuning.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Activation steering to jailbreak LLM safety refusals is an active area of research; these methods are largely reliant on linear separability of a concept (e.g., \"safety\" or \"refusal\") in the embedding space. As such, parallel research on mitigations is highly relevant; the approach here is simple but logical and demonstrates a solid proof of concept. (As far as I'm aware, the contribution is novel, though I'll defer to others on this point; likewise with appropriate baseline comparisons.)\n\n2. The experiments in section 5, supplemented by the appendices, are reasonably thorough; while one could always ask for more, these suffice to convey the important result: the proposed mitigation significantly improve the refusal rate & safe generations (per LlamaGuard) on harmful prompts while refusing fewer benign prompts in comparison to finetuning on the jailbreak dataset.\n\n3. The paper is a pleasant read -- generally clear and logically organized.", "reasons_to_reject": "1. The paper is a bit short on details when it comes to the visualized concepts in Fig. 2 (emotion) and Fig. 4 (truthfulness, emotion, and safety) -- where do the data points and labels come from?", "questions_to_authors": "Please address any questions above / feel free to correct any misunderstandings. Thanks!", "extra_scores": {}}, {"review_id": "KSAc8h9ExR", "reviewer": "Reviewer_NYCF", "summary": "The paper explores the impact of model dimensionality on the safety alignment of LLMs. The authors hypothesize that while increased dimensionality enhances model capabilities, it also introduces vulnerabilities through linear structures in the activation space that can be exploited to circumvent safety alignment. Through detailed visualizations and empirical experiments, they demonstrate that projecting model representations onto lower-dimensional subspaces can mitigate these vulnerabilities while preserving essential information for alignment. The paper proposes two novel fine-tuning methods, FJLT and Bottleneck, to achieve this dimensionality reduction and validate their effectiveness against steering jailbreak attacks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies a timely and important problem. Given the increasing attention to the trustworthiness of LLMs, the findings and proposed method could have meaningful impact on future research.\n- The paper is well-written and easy to read.\n- The paper offers valuable theoretical insights into the relationship between linear structures in the activation space and the model's hidden dimensions.\n- Experiments show the effectiveness of the proposed methods.", "reasons_to_reject": "- Given the scale and complexity of modern LLMs, it remains unclear how well the theoretical analysis based on classical tools such as VC dimension and Rademacher complexity translates to practical settings.\n- The proposed method introduces several hyperparameters ($\\alpha$ and $K$), yet the paper does not provide a sensitivity analysis or discussion regarding their influence on performance.", "questions_to_authors": "NA", "extra_scores": {}}, {"review_id": "6p7namJjx0", "reviewer": "Reviewer_6omJ", "summary": "The paper is quite clearly written, a pleasure to follow and quite contemporary. It addresses the tension between improving the quality of LLMs and making them more vulnerable to attacks through dimensionality increment. The proposal is simple - apply random dimension reduction which would preserve the essential matrix multiplications in the attention mechanism thereby avoiding accuracy loss, and at the same time this dimension reduction would make the LLM less susceptible to jailbreaking. I quite like this idea, and do think this is an interesting application in the domain of safety alignment.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "As mentioned, I like the simplicity of the idea. I think the authors have executed it pretty well. The experimental evidence is also assuring. The writing is good.", "reasons_to_reject": "There could be many reasons why jailbreaking gets easier along with increasing capabilities of the models. The authors seemingly focus on dimension as the single double edged sword. Why jailbreaking is considered particularly? There are many negative aspects in an LLM - can all of them be avoided by simple dimension reduction? Namely, the authors consider two axes: increasing capability and jailbreaking, and show that dimension reduction can be helpful to achieve a balance between the two worlds. If instead of jailbreaking we consider some other negative aspect, then does dimension reduction still work? Will the final reduced dimension be approximately the same, or will it be quite different?\n\nFor example, https://dl.acm.org/doi/10.1145/3618260.3649777 points out that for calibrated models (calibration is presumably good), hallucination is inevitable. Does dimension reduction help in this setting?\n\nI feel that more such axes should be considered to uphold the message delivered by the authors.", "questions_to_authors": "See above.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper brings up an important problem which is that as LLMs grow in scale, they become more susceptible to linear steering attacks due to their parameter redundancy that translates to easier linear separability for various concepts. Focusing on safety alignment, the paper shows that larger models are indeed more sensitive to such attacks. It then proposes methods to alleviate such issues by fine tuning a model to use lower dimensional activations. 

Reasons to accept: 
- The paper is thorough and includes substantial experiments and results across various models and datasets. 
- The claims in the paper are sound and timely. 
- The paper is written clearly and cites relevant work. 
- The reviewers are in consensus that the paper is publication-worthy.

Reasons to reject: 
- The suggested methods do seem to also harm the quality of the model on some evals; a more thorough analysis of the errors would be interesting.
Decision: Accept